2023-11-28T12:45:56.715 INFO:root:teuthology version: 0.0.1.dev209+g5ae09a5 2023-11-28T12:45:56.715 DEBUG:teuthology.run:Teuthology command: teuthology --name neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} --owner scheduled_neesingh@teuthology --archive /home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969 --verbose -- /home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969/orig.config.yaml 2023-11-28T12:45:56.759 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T12:45:56.842 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969 branch: wip-neesingh-testing-231123 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} email: neesingh@redhat.com first_in_suite: false job_id: '7469969' kernel: client: sha1: distro kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-neesingh-testing-231123 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - inode wrongly marked free - bad backtrace on inode - inode table repaired for inode - Scrub error on inode - Scrub error on dir - Metadata damage detected sha1: 2f5f6028b893ad41357cd06e827373f0fe2a1232 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 2f5f6028b893ad41357cd06e827373f0fe2a1232 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: issue_54557 sha1: 3ae99413970476692dfab1fe38dadb056700a61d owner: scheduled_neesingh@teuthology priority: 10 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 6754 sha1: 2f5f6028b893ad41357cd06e827373f0fe2a1232 sleep_before_teardown: 0 suite: fs suite_branch: issue_54557 suite_path: /home/teuthworker/src/github.com_neesingh-rh_ceph_3ae99413970476692dfab1fe38dadb056700a61d/qa suite_relpath: qa suite_repo: https://github.com/neesingh-rh/ceph suite_sha1: 3ae99413970476692dfab1fe38dadb056700a61d syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi149.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_forward_scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5ae09a5cc7064ca2bf994df19f3006b79794789a timestamp: 2023-11-28_12:38:30 tube: smithi user: neesingh verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-28T12:45:56.842 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_neesingh-rh_ceph_3ae99413970476692dfab1fe38dadb056700a61d/qa; will attempt to use it 2023-11-28T12:45:56.843 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_neesingh-rh_ceph_3ae99413970476692dfab1fe38dadb056700a61d/qa/tasks 2023-11-28T12:45:56.843 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-28T12:45:56.845 INFO:teuthology.task.internal:Checking packages... 2023-11-28T12:45:56.870 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '2f5f6028b893ad41357cd06e827373f0fe2a1232' 2023-11-28T12:45:56.870 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-28T12:45:56.870 INFO:teuthology.packaging:ref: None 2023-11-28T12:45:56.870 INFO:teuthology.packaging:tag: None 2023-11-28T12:45:56.870 INFO:teuthology.packaging:branch: wip-neesingh-testing-231123 2023-11-28T12:45:56.870 INFO:teuthology.packaging:sha1: 2f5f6028b893ad41357cd06e827373f0fe2a1232 2023-11-28T12:45:56.870 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-neesingh-testing-231123 2023-11-28T12:45:57.018 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-2697.g2f5f6028 2023-11-28T12:45:57.018 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-28T12:45:57.024 INFO:teuthology.task.internal:no buildpackages task found 2023-11-28T12:45:57.025 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-28T12:45:57.072 INFO:teuthology.task.internal:Saving configuration 2023-11-28T12:45:57.088 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-28T12:45:57.094 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T12:45:57.120 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 12:40:25.366247', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-28T12:45:57.143 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 12:40:25.367605', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-28T12:45:57.143 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-28T12:45:57.150 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-11-28T12:45:57.150 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-11-28T12:45:57.150 INFO:teuthology.run_tasks:Running task console_log... 2023-11-28T12:45:57.223 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6cc7928a60>, signals=[15]) 2023-11-28T12:45:57.224 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-28T12:45:57.230 INFO:teuthology.task.internal:Opening connections... 2023-11-28T12:45:57.231 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T12:45:57.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T12:45:57.298 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi149.front.sepia.ceph.com 2023-11-28T12:45:57.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T12:45:57.367 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-28T12:45:57.373 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2023-11-28T12:45:57.394 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2023-11-28T12:45:57.394 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2023-11-28T12:45:57.451 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="Red Hat Enterprise Linux" 2023-11-28T12:45:57.451 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="8.6 (Ootpa)" 2023-11-28T12:45:57.451 INFO:teuthology.orchestra.run.smithi111.stdout:ID="rhel" 2023-11-28T12:45:57.451 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="fedora" 2023-11-28T12:45:57.451 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="8.6" 2023-11-28T12:45:57.451 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el8" 2023-11-28T12:45:57.452 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-28T12:45:57.452 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2023-11-28T12:45:57.452 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-28T12:45:57.452 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://www.redhat.com/" 2023-11-28T12:45:57.452 INFO:teuthology.orchestra.run.smithi111.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-28T12:45:57.452 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-28T12:45:57.452 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:45:57.452 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-28T12:45:57.452 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-28T12:45:57.452 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-28T12:45:57.452 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-28T12:45:57.453 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2023-11-28T12:45:57.473 DEBUG:teuthology.orchestra.run.smithi149:> uname -m 2023-11-28T12:45:57.492 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2023-11-28T12:45:57.493 DEBUG:teuthology.orchestra.run.smithi149:> cat /etc/os-release 2023-11-28T12:45:57.550 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="Red Hat Enterprise Linux" 2023-11-28T12:45:57.551 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="8.6 (Ootpa)" 2023-11-28T12:45:57.551 INFO:teuthology.orchestra.run.smithi149.stdout:ID="rhel" 2023-11-28T12:45:57.551 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="fedora" 2023-11-28T12:45:57.551 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="8.6" 2023-11-28T12:45:57.551 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el8" 2023-11-28T12:45:57.551 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-28T12:45:57.551 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2023-11-28T12:45:57.551 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-28T12:45:57.551 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://www.redhat.com/" 2023-11-28T12:45:57.551 INFO:teuthology.orchestra.run.smithi149.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-28T12:45:57.551 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-28T12:45:57.551 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:45:57.552 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-28T12:45:57.552 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-28T12:45:57.552 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-28T12:45:57.552 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-28T12:45:57.552 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2023-11-28T12:45:57.573 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-28T12:45:57.581 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-28T12:45:57.587 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-28T12:45:57.587 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2023-11-28T12:45:57.590 DEBUG:teuthology.orchestra.run.smithi149:> test '!' -e /home/ubuntu/cephtest 2023-11-28T12:45:57.608 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-28T12:45:57.613 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-28T12:45:57.613 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2023-11-28T12:45:57.646 DEBUG:teuthology.orchestra.run.smithi149:> test -z $(ls -A /var/lib/ceph) 2023-11-28T12:45:57.679 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-28T12:45:57.733 INFO:teuthology.run_tasks:Running task kernel... 2023-11-28T12:45:57.748 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-11-28T12:45:57.748 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-11-28T12:45:57.748 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-11-28T12:45:57.748 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T12:45:57.749 DEBUG:teuthology.orchestra.run.smithi149:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T12:45:57.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T12:45:57.766 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2023-11-28T12:45:57.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T12:45:57.768 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2023-11-28T12:45:57.824 INFO:teuthology.orchestra.run.smithi111.stdout:4.18.0-372.9.1.el8.x86_64 2023-11-28T12:45:57.825 INFO:teuthology.task.kernel:Running kernel on smithi111: 4.18.0-372.9.1.el8.x86_64 2023-11-28T12:45:57.825 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum install -y kernel 2023-11-28T12:45:57.826 INFO:teuthology.orchestra.run.smithi149.stdout:4.18.0-372.9.1.el8.x86_64 2023-11-28T12:45:57.826 INFO:teuthology.task.kernel:Running kernel on smithi149: 4.18.0-372.9.1.el8.x86_64 2023-11-28T12:45:57.826 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum install -y kernel 2023-11-28T12:45:59.014 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-11-28T12:45:59.015 INFO:teuthology.orchestra.run.smithi149.stdout:Unable to read consumer identity 2023-11-28T12:45:59.027 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-11-28T12:45:59.027 INFO:teuthology.orchestra.run.smithi111.stdout:Unable to read consumer identity 2023-11-28T12:46:00.480 INFO:teuthology.orchestra.run.smithi149.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-11-28T12:46:00.496 INFO:teuthology.orchestra.run.smithi111.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2023-11-28T12:46:01.330 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2023-11-28T12:46:01.714 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-11-28T12:46:04.814 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 439 kB/s | 24 kB 00:00 2023-11-28T12:46:05.322 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 448 kB/s | 24 kB 00:00 2023-11-28T12:46:05.476 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-11-28T12:46:05.533 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-11-28T12:46:05.534 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2023-11-28T12:46:05.534 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-11-28T12:46:05.670 DEBUG:teuthology.orchestra.run.smithi149:> echo no | sudo yum reinstall kernel || true 2023-11-28T12:46:06.006 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-11-28T12:46:06.006 INFO:teuthology.orchestra.run.smithi149.stdout:Unable to read consumer identity 2023-11-28T12:46:06.007 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-11-28T12:46:06.062 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-11-28T12:46:06.063 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2023-11-28T12:46:06.063 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-11-28T12:46:06.174 DEBUG:teuthology.orchestra.run.smithi111:> echo no | sudo yum reinstall kernel || true 2023-11-28T12:46:06.278 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:00:02 ago on Tue 28 Nov 2023 12:46:04 PM UTC. 2023-11-28T12:46:06.320 INFO:teuthology.orchestra.run.smithi149.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-28T12:46:06.369 INFO:teuthology.orchestra.run.smithi149.stderr:Error: No packages marked for reinstall. 2023-11-28T12:46:06.416 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum reinstall -y kernel || true 2023-11-28T12:46:06.546 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-11-28T12:46:06.547 INFO:teuthology.orchestra.run.smithi111.stdout:Unable to read consumer identity 2023-11-28T12:46:06.748 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-11-28T12:46:06.748 INFO:teuthology.orchestra.run.smithi149.stdout:Unable to read consumer identity 2023-11-28T12:46:06.810 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:01 ago on Tue 28 Nov 2023 12:46:05 PM UTC. 2023-11-28T12:46:06.850 INFO:teuthology.orchestra.run.smithi111.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-28T12:46:06.889 INFO:teuthology.orchestra.run.smithi111.stderr:Error: No packages marked for reinstall. 2023-11-28T12:46:06.935 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum reinstall -y kernel || true 2023-11-28T12:46:07.015 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:00:03 ago on Tue 28 Nov 2023 12:46:04 PM UTC. 2023-11-28T12:46:07.055 INFO:teuthology.orchestra.run.smithi149.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-28T12:46:07.091 INFO:teuthology.orchestra.run.smithi149.stderr:Error: No packages marked for reinstall. 2023-11-28T12:46:07.143 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2023-11-28T12:46:07.224 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-11-28T12:46:07.225 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-11-28T12:46:07.225 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-11-28T12:46:07.225 DEBUG:teuthology.parallel:result is None 2023-11-28T12:46:07.264 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-11-28T12:46:07.265 INFO:teuthology.orchestra.run.smithi111.stdout:Unable to read consumer identity 2023-11-28T12:46:07.523 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:02 ago on Tue 28 Nov 2023 12:46:05 PM UTC. 2023-11-28T12:46:07.565 INFO:teuthology.orchestra.run.smithi111.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-28T12:46:07.603 INFO:teuthology.orchestra.run.smithi111.stderr:Error: No packages marked for reinstall. 2023-11-28T12:46:07.649 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2023-11-28T12:46:07.730 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-11-28T12:46:07.730 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-11-28T12:46:07.731 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-11-28T12:46:07.731 DEBUG:teuthology.parallel:result is None 2023-11-28T12:46:07.731 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-28T12:46:07.738 INFO:teuthology.task.internal:Creating test directory... 2023-11-28T12:46:07.738 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T12:46:07.741 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T12:46:07.759 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-28T12:46:07.765 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-28T12:46:07.776 INFO:teuthology.task.internal:Creating archive directory... 2023-11-28T12:46:07.776 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T12:46:07.798 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T12:46:07.823 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-28T12:46:07.829 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-28T12:46:07.830 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T12:46:07.855 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T12:46:07.898 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T12:46:07.901 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T12:46:07.908 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T12:46:07.942 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T12:46:07.944 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-28T12:46:07.955 INFO:teuthology.task.internal:Configuring sudo... 2023-11-28T12:46:07.955 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T12:46:07.958 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T12:46:08.011 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-28T12:46:08.022 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-28T12:46:08.022 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T12:46:08.025 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T12:46:08.068 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T12:46:08.123 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T12:46:08.169 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T12:46:08.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T12:46:08.295 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:46:08.295 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T12:46:08.362 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T12:46:08.405 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T12:46:08.453 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T12:46:08.536 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T12:46:08.578 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:46:08.578 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T12:46:08.645 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2023-11-28T12:46:08.648 DEBUG:teuthology.orchestra.run.smithi149:> sudo service rsyslog restart 2023-11-28T12:46:08.697 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T12:46:08.745 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T12:46:09.033 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-28T12:46:09.041 INFO:teuthology.task.internal:Starting timer... 2023-11-28T12:46:09.041 INFO:teuthology.run_tasks:Running task pcp... 2023-11-28T12:46:09.100 INFO:teuthology.run_tasks:Running task selinux... 2023-11-28T12:46:09.108 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-11-28T12:46:09.108 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2023-11-28T12:46:09.166 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2023-11-28T12:46:09.168 DEBUG:teuthology.orchestra.run.smithi149:> sudo service auditd rotate 2023-11-28T12:46:09.232 INFO:teuthology.orchestra.run.smithi149.stdout:Rotating logs: 2023-11-28T12:46:09.233 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-28T12:46:09.233 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2023-11-28T12:46:09.257 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2023-11-28T12:46:09.257 DEBUG:teuthology.orchestra.run.smithi149:> /usr/sbin/getenforce 2023-11-28T12:46:09.279 INFO:teuthology.orchestra.run.smithi149.stdout:Permissive 2023-11-28T12:46:09.279 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi111.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi149.front.sepia.ceph.com': 'permissive'} 2023-11-28T12:46:09.279 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T12:46:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T12:46:09.326 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T12:46:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T12:46:09.354 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-28T12:46:09.354 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2023-11-28T12:46:09.393 DEBUG:teuthology.orchestra.run.smithi149:> sudo /usr/sbin/setenforce permissive 2023-11-28T12:46:09.423 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-28T12:46:09.433 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-28T12:46:09.433 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-28T12:46:09.451 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-28T12:46:09.452 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi111.front.sepia.ceph.com,smithi149.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-28T12:55:33.325 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi149.front.sepia.ceph.com')] 2023-11-28T12:55:33.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-11-28T12:55:33.329 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T12:55:33.400 DEBUG:teuthology.orchestra.run.smithi111:> true 2023-11-28T12:55:33.471 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-11-28T12:55:33.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2023-11-28T12:55:33.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T12:55:33.540 DEBUG:teuthology.orchestra.run.smithi149:> true 2023-11-28T12:55:33.608 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2023-11-28T12:55:33.608 INFO:teuthology.run_tasks:Running task clock... 2023-11-28T12:55:33.620 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-28T12:55:33.620 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T12:55:33.620 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T12:55:33.623 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T12:55:33.623 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T12:55:33.653 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-28T12:55:33.669 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-28T12:55:33.680 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-28T12:55:33.694 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-28T12:55:33.758 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2023-11-28T12:55:33.762 INFO:teuthology.orchestra.run.smithi149.stderr:sudo: ntpd: command not found 2023-11-28T12:55:33.770 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2023-11-28T12:55:33.774 INFO:teuthology.orchestra.run.smithi149.stdout:506 Cannot talk to daemon 2023-11-28T12:55:33.784 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-28T12:55:33.789 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-28T12:55:33.797 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-28T12:55:33.802 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-28T12:55:33.874 INFO:teuthology.orchestra.run.smithi149.stderr:bash: ntpq: command not found 2023-11-28T12:55:33.877 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T12:55:33.877 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2023-11-28T12:55:33.877 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T12:55:33.877 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T12:55:33.878 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T12:55:33.878 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T12:55:33.880 INFO:teuthology.orchestra.run.smithi111.stderr:bash: ntpq: command not found 2023-11-28T12:55:33.884 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T12:55:33.884 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2023-11-28T12:55:33.884 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T12:55:33.884 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T12:55:33.885 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T12:55:33.885 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T12:55:33.885 INFO:teuthology.run_tasks:Running task install... 2023-11-28T12:55:33.893 DEBUG:teuthology.task.install:project ceph 2023-11-28T12:55:33.893 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2f5f6028b893ad41357cd06e827373f0fe2a1232'}} 2023-11-28T12:55:33.893 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '2f5f6028b893ad41357cd06e827373f0fe2a1232'} 2023-11-28T12:55:33.893 INFO:teuthology.task.install:Using flavor: default 2023-11-28T12:55:33.899 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-28T12:55:33.899 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-11-28T12:55:33.900 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2f5f6028b893ad41357cd06e827373f0fe2a1232', 'tag': None, 'wait_for_package': False} 2023-11-28T12:55:33.900 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2f5f6028b893ad41357cd06e827373f0fe2a1232 2023-11-28T12:55:33.902 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2f5f6028b893ad41357cd06e827373f0fe2a1232', 'tag': None, 'wait_for_package': False} 2023-11-28T12:55:33.902 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2f5f6028b893ad41357cd06e827373f0fe2a1232 2023-11-28T12:55:34.043 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-231123/2f5f6028b893ad41357cd06e827373f0fe2a1232/centos/8/flavors/default/ 2023-11-28T12:55:34.044 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2697.g2f5f6028 2023-11-28T12:55:34.049 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-231123/2f5f6028b893ad41357cd06e827373f0fe2a1232/centos/8/flavors/default/ 2023-11-28T12:55:34.050 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2697.g2f5f6028 2023-11-28T12:55:34.165 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-231123/2f5f6028b893ad41357cd06e827373f0fe2a1232/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-231123/2f5f6028b893ad41357cd06e827373f0fe2a1232/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-231123/2f5f6028b893ad41357cd06e827373f0fe2a1232/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-28T12:55:34.165 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:55:34.166 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-28T12:55:34.193 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-28T12:55:34.194 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2f5f6028b893ad41357cd06e827373f0fe2a1232/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-28T12:55:34.216 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-231123/2f5f6028b893ad41357cd06e827373f0fe2a1232/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-231123/2f5f6028b893ad41357cd06e827373f0fe2a1232/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-231123/2f5f6028b893ad41357cd06e827373f0fe2a1232/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-28T12:55:34.217 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:55:34.217 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-28T12:55:34.245 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-28T12:55:34.245 DEBUG:teuthology.orchestra.run.smithi149:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2f5f6028b893ad41357cd06e827373f0fe2a1232/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-28T12:55:34.261 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-28T12:55:34.313 DEBUG:teuthology.orchestra.run.smithi149:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-28T12:55:34.338 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-28T12:55:34.389 DEBUG:teuthology.orchestra.run.smithi149:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-28T12:55:34.405 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2023-11-28T12:55:34.406 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2023-11-28T12:55:34.455 INFO:teuthology.orchestra.run.smithi149.stdout:check_obsoletes = 1 2023-11-28T12:55:34.457 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum clean all 2023-11-28T12:55:34.776 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-11-28T12:55:34.832 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-11-28T12:55:35.765 INFO:teuthology.orchestra.run.smithi111.stdout:44 files removed 2023-11-28T12:55:35.818 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-28T12:55:36.071 INFO:teuthology.orchestra.run.smithi149.stdout:44 files removed 2023-11-28T12:55:36.126 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-28T12:55:36.150 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-11-28T12:55:36.460 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2023-11-28T12:55:37.688 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 145 kB/s | 80 kB 00:00 2023-11-28T12:55:37.963 INFO:teuthology.orchestra.run.smithi149.stdout:ceph packages for x86_64 191 kB/s | 80 kB 00:00 2023-11-28T12:55:38.072 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 27 kB/s | 9.4 kB 00:00 2023-11-28T12:55:38.364 INFO:teuthology.orchestra.run.smithi149.stdout:ceph noarch packages 25 kB/s | 9.4 kB 00:00 2023-11-28T12:55:38.398 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 4.9 kB/s | 1.5 kB 00:00 2023-11-28T12:55:38.548 INFO:teuthology.orchestra.run.smithi111.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-11-28T12:55:38.689 INFO:teuthology.orchestra.run.smithi149.stdout:ceph source packages 4.9 kB/s | 1.5 kB 00:00 2023-11-28T12:55:38.840 INFO:teuthology.orchestra.run.smithi149.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-11-28T12:55:39.166 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 27 MB/s | 16 MB 00:00 2023-11-28T12:55:39.659 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 20 MB/s | 16 MB 00:00 2023-11-28T12:55:42.691 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 352 kB/s | 24 kB 00:00 2023-11-28T12:55:43.191 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 459 kB/s | 24 kB 00:00 2023-11-28T12:55:43.851 INFO:teuthology.orchestra.run.smithi111.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 MB/s | 53 MB 00:01 2023-11-28T12:55:44.292 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 51 MB/s | 53 MB 00:01 2023-11-28T12:55:52.160 INFO:teuthology.orchestra.run.smithi111.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 MB/s | 47 MB 00:01 2023-11-28T12:55:52.810 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 MB/s | 47 MB 00:01 2023-11-28T12:56:03.033 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-28T12:56:03.033 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-28T12:56:03.047 INFO:teuthology.orchestra.run.smithi111.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-28T12:56:03.048 INFO:teuthology.orchestra.run.smithi111.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-28T12:56:03.049 INFO:teuthology.orchestra.run.smithi111.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-28T12:56:03.050 INFO:teuthology.orchestra.run.smithi111.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-28T12:56:03.053 INFO:teuthology.orchestra.run.smithi111.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-28T12:56:03.056 INFO:teuthology.orchestra.run.smithi111.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-28T12:56:03.057 INFO:teuthology.orchestra.run.smithi111.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-28T12:56:03.186 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-11-28T12:56:03.199 INFO:teuthology.orchestra.run.smithi111.stdout:=========================================================================================================================== 2023-11-28T12:56:03.199 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2023-11-28T12:56:03.200 INFO:teuthology.orchestra.run.smithi111.stdout:=========================================================================================================================== 2023-11-28T12:56:03.200 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-11-28T12:56:03.200 INFO:teuthology.orchestra.run.smithi111.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-28T12:56:03.200 INFO:teuthology.orchestra.run.smithi111.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-28T12:56:03.201 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 6.4 k 2023-11-28T12:56:03.201 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 5.4 M 2023-11-28T12:56:03.201 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 845 k 2023-11-28T12:56:03.202 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 165 k 2023-11-28T12:56:03.202 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 1.5 M 2023-11-28T12:56:03.202 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 123 k 2023-11-28T12:56:03.202 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 1.7 M 2023-11-28T12:56:03.202 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 7.4 M 2023-11-28T12:56:03.203 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 47 k 2023-11-28T12:56:03.203 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 13 M 2023-11-28T12:56:03.203 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 47 M 2023-11-28T12:56:03.203 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 263 k 2023-11-28T12:56:03.204 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 83 k 2023-11-28T12:56:03.204 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-mirror x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 257 k 2023-11-28T12:56:03.204 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-top noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 18 k 2023-11-28T12:56:03.204 INFO:teuthology.orchestra.run.smithi111.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-28T12:56:03.204 INFO:teuthology.orchestra.run.smithi111.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-28T12:56:03.205 INFO:teuthology.orchestra.run.smithi111.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-28T12:56:03.205 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 30 k 2023-11-28T12:56:03.205 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 767 k 2023-11-28T12:56:03.205 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 139 k 2023-11-28T12:56:03.206 INFO:teuthology.orchestra.run.smithi111.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-28T12:56:03.206 INFO:teuthology.orchestra.run.smithi111.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-28T12:56:03.206 INFO:teuthology.orchestra.run.smithi111.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-28T12:56:03.206 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-28T12:56:03.206 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-28T12:56:03.207 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-28T12:56:03.207 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 190 k 2023-11-28T12:56:03.207 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 371 k 2023-11-28T12:56:03.207 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 364 k 2023-11-28T12:56:03.208 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 107 k 2023-11-28T12:56:03.208 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 94 k 2023-11-28T12:56:03.208 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 3.4 M 2023-11-28T12:56:03.209 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 180 k 2023-11-28T12:56:03.209 INFO:teuthology.orchestra.run.smithi111.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-28T12:56:03.209 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2023-11-28T12:56:03.209 INFO:teuthology.orchestra.run.smithi111.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-28T12:56:03.209 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 3.7 M 2023-11-28T12:56:03.210 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 3.6 M 2023-11-28T12:56:03.210 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2023-11-28T12:56:03.210 INFO:teuthology.orchestra.run.smithi111.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-28T12:56:03.210 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 22 M 2023-11-28T12:56:03.211 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 23 k 2023-11-28T12:56:03.211 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 2.3 M 2023-11-28T12:56:03.211 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 236 k 2023-11-28T12:56:03.211 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 4.6 M 2023-11-28T12:56:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 18 M 2023-11-28T12:56:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 15 k 2023-11-28T12:56:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 24 k 2023-11-28T12:56:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-28T12:56:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-28T12:56:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-28T12:56:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-28T12:56:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-28T12:56:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 137 k 2023-11-28T12:56:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-28T12:56:03.213 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-28T12:56:03.213 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-28T12:56:03.213 INFO:teuthology.orchestra.run.smithi111.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-28T12:56:03.213 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-28T12:56:03.213 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-28T12:56:03.213 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 474 k 2023-11-28T12:56:03.213 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-28T12:56:03.213 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 6.5 M 2023-11-28T12:56:03.213 INFO:teuthology.orchestra.run.smithi111.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-28T12:56:03.213 INFO:teuthology.orchestra.run.smithi111.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-28T12:56:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-28T12:56:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-28T12:56:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-28T12:56:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-28T12:56:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-28T12:56:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-28T12:56:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-28T12:56:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-28T12:56:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-28T12:56:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-28T12:56:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-28T12:56:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-28T12:56:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-28T12:56:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-28T12:56:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 45 k 2023-11-28T12:56:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 117 k 2023-11-28T12:56:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-28T12:56:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-28T12:56:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-28T12:56:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-28T12:56:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-28T12:56:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-28T12:56:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-28T12:56:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-28T12:56:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-28T12:56:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-28T12:56:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-28T12:56:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-28T12:56:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-28T12:56:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-28T12:56:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-28T12:56:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-28T12:56:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-28T12:56:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-28T12:56:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-28T12:56:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-28T12:56:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-28T12:56:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-28T12:56:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-28T12:56:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-28T12:56:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-28T12:56:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-28T12:56:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-28T12:56:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-28T12:56:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-28T12:56:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-28T12:56:03.218 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-28T12:56:03.218 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-28T12:56:03.218 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-28T12:56:03.218 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-28T12:56:03.218 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-28T12:56:03.218 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-28T12:56:03.218 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-28T12:56:03.219 INFO:teuthology.orchestra.run.smithi111.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-28T12:56:03.219 INFO:teuthology.orchestra.run.smithi111.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-28T12:56:03.219 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-28T12:56:03.219 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-28T12:56:03.219 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-28T12:56:03.219 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-28T12:56:03.219 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-28T12:56:03.219 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-28T12:56:03.219 INFO:teuthology.orchestra.run.smithi111.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-28T12:56:03.219 INFO:teuthology.orchestra.run.smithi111.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-28T12:56:03.220 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-28T12:56:03.220 INFO:teuthology.orchestra.run.smithi111.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-28T12:56:03.220 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling module streams: 2023-11-28T12:56:03.220 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql 10 2023-11-28T12:56:03.220 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:56:03.220 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-11-28T12:56:03.220 INFO:teuthology.orchestra.run.smithi111.stdout:=========================================================================================================================== 2023-11-28T12:56:03.221 INFO:teuthology.orchestra.run.smithi111.stdout:Install 128 Packages 2023-11-28T12:56:03.221 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 3 Packages 2023-11-28T12:56:03.221 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:56:03.222 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 203 M 2023-11-28T12:56:03.222 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-11-28T12:56:03.527 INFO:teuthology.orchestra.run.smithi111.stdout:(1/131): ceph-18.0.0-2697.g2f5f6028.el8.x86_64. 50 kB/s | 6.4 kB 00:00 2023-11-28T12:56:03.761 INFO:teuthology.orchestra.run.smithi111.stdout:(2/131): ceph-fuse-18.0.0-2697.g2f5f6028.el8.x8 3.5 MB/s | 845 kB 00:00 2023-11-28T12:56:03.784 INFO:teuthology.orchestra.run.smithi149.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-28T12:56:03.785 INFO:teuthology.orchestra.run.smithi149.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-28T12:56:03.800 INFO:teuthology.orchestra.run.smithi149.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-28T12:56:03.801 INFO:teuthology.orchestra.run.smithi149.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-28T12:56:03.802 INFO:teuthology.orchestra.run.smithi149.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-28T12:56:03.803 INFO:teuthology.orchestra.run.smithi149.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-28T12:56:03.805 INFO:teuthology.orchestra.run.smithi149.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-28T12:56:03.808 INFO:teuthology.orchestra.run.smithi149.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-28T12:56:03.810 INFO:teuthology.orchestra.run.smithi149.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-28T12:56:03.836 INFO:teuthology.orchestra.run.smithi111.stdout:(3/131): ceph-immutable-object-cache-18.0.0-269 2.2 MB/s | 165 kB 00:00 2023-11-28T12:56:03.945 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2023-11-28T12:56:03.957 INFO:teuthology.orchestra.run.smithi149.stdout:=========================================================================================================================== 2023-11-28T12:56:03.958 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2023-11-28T12:56:03.958 INFO:teuthology.orchestra.run.smithi149.stdout:=========================================================================================================================== 2023-11-28T12:56:03.958 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2023-11-28T12:56:03.958 INFO:teuthology.orchestra.run.smithi149.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-28T12:56:03.959 INFO:teuthology.orchestra.run.smithi149.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-28T12:56:03.959 INFO:teuthology.orchestra.run.smithi149.stdout: ceph x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 6.4 k 2023-11-28T12:56:03.959 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-base x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 5.4 M 2023-11-28T12:56:03.959 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-fuse x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 845 k 2023-11-28T12:56:03.959 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 165 k 2023-11-28T12:56:03.960 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 1.5 M 2023-11-28T12:56:03.960 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 123 k 2023-11-28T12:56:03.960 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 1.7 M 2023-11-28T12:56:03.960 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 7.4 M 2023-11-28T12:56:03.960 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-rook noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 47 k 2023-11-28T12:56:03.960 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-radosgw x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 13 M 2023-11-28T12:56:03.961 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-test x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 47 M 2023-11-28T12:56:03.961 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-volume noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 263 k 2023-11-28T12:56:03.961 INFO:teuthology.orchestra.run.smithi149.stdout: cephadm noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 83 k 2023-11-28T12:56:03.961 INFO:teuthology.orchestra.run.smithi149.stdout: cephfs-mirror x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 257 k 2023-11-28T12:56:03.961 INFO:teuthology.orchestra.run.smithi149.stdout: cephfs-top noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 18 k 2023-11-28T12:56:03.962 INFO:teuthology.orchestra.run.smithi149.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-28T12:56:03.962 INFO:teuthology.orchestra.run.smithi149.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-28T12:56:03.962 INFO:teuthology.orchestra.run.smithi149.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-28T12:56:03.962 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs-devel x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 30 k 2023-11-28T12:56:03.962 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs2 x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 767 k 2023-11-28T12:56:03.963 INFO:teuthology.orchestra.run.smithi149.stdout: librados-devel x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 139 k 2023-11-28T12:56:03.963 INFO:teuthology.orchestra.run.smithi149.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-28T12:56:03.963 INFO:teuthology.orchestra.run.smithi149.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-28T12:56:03.963 INFO:teuthology.orchestra.run.smithi149.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-28T12:56:03.964 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-28T12:56:03.964 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-28T12:56:03.964 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-28T12:56:03.964 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cephfs x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 190 k 2023-11-28T12:56:03.964 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rados x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 371 k 2023-11-28T12:56:03.964 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rbd x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 364 k 2023-11-28T12:56:03.964 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rgw x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 107 k 2023-11-28T12:56:03.964 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-fuse x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 94 k 2023-11-28T12:56:03.964 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-mirror x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 3.4 M 2023-11-28T12:56:03.965 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-nbd x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 180 k 2023-11-28T12:56:03.965 INFO:teuthology.orchestra.run.smithi149.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-28T12:56:03.965 INFO:teuthology.orchestra.run.smithi149.stdout:Upgrading: 2023-11-28T12:56:03.965 INFO:teuthology.orchestra.run.smithi149.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-28T12:56:03.965 INFO:teuthology.orchestra.run.smithi149.stdout: librados2 x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 3.7 M 2023-11-28T12:56:03.965 INFO:teuthology.orchestra.run.smithi149.stdout: librbd1 x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 3.6 M 2023-11-28T12:56:03.965 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2023-11-28T12:56:03.965 INFO:teuthology.orchestra.run.smithi149.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-28T12:56:03.965 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-common x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 22 M 2023-11-28T12:56:03.965 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 23 k 2023-11-28T12:56:03.966 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mds x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 2.3 M 2023-11-28T12:56:03.966 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 236 k 2023-11-28T12:56:03.966 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mon x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 4.6 M 2023-11-28T12:56:03.966 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-osd x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 18 M 2023-11-28T12:56:03.966 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2697.g2f5f6028.el8 ceph-noarch 15 k 2023-11-28T12:56:03.966 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-selinux x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 24 k 2023-11-28T12:56:03.966 INFO:teuthology.orchestra.run.smithi149.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-28T12:56:03.966 INFO:teuthology.orchestra.run.smithi149.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-28T12:56:03.966 INFO:teuthology.orchestra.run.smithi149.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-28T12:56:03.966 INFO:teuthology.orchestra.run.smithi149.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-28T12:56:03.966 INFO:teuthology.orchestra.run.smithi149.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-28T12:56:03.966 INFO:teuthology.orchestra.run.smithi149.stdout: libcephsqlite x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 137 k 2023-11-28T12:56:03.967 INFO:teuthology.orchestra.run.smithi149.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-28T12:56:03.967 INFO:teuthology.orchestra.run.smithi149.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-28T12:56:03.967 INFO:teuthology.orchestra.run.smithi149.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-28T12:56:03.967 INFO:teuthology.orchestra.run.smithi149.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-28T12:56:03.967 INFO:teuthology.orchestra.run.smithi149.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-28T12:56:03.967 INFO:teuthology.orchestra.run.smithi149.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-28T12:56:03.967 INFO:teuthology.orchestra.run.smithi149.stdout: libradosstriper1 x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 474 k 2023-11-28T12:56:03.967 INFO:teuthology.orchestra.run.smithi149.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-28T12:56:03.967 INFO:teuthology.orchestra.run.smithi149.stdout: librgw2 x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 6.5 M 2023-11-28T12:56:03.967 INFO:teuthology.orchestra.run.smithi149.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-28T12:56:03.967 INFO:teuthology.orchestra.run.smithi149.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-28T12:56:03.967 INFO:teuthology.orchestra.run.smithi149.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-28T12:56:03.967 INFO:teuthology.orchestra.run.smithi149.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-28T12:56:03.968 INFO:teuthology.orchestra.run.smithi149.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-28T12:56:03.968 INFO:teuthology.orchestra.run.smithi149.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-28T12:56:03.968 INFO:teuthology.orchestra.run.smithi149.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-28T12:56:03.968 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-28T12:56:03.968 INFO:teuthology.orchestra.run.smithi149.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-28T12:56:03.968 INFO:teuthology.orchestra.run.smithi149.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-28T12:56:03.968 INFO:teuthology.orchestra.run.smithi149.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-28T12:56:03.968 INFO:teuthology.orchestra.run.smithi149.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-28T12:56:03.968 INFO:teuthology.orchestra.run.smithi149.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-28T12:56:03.968 INFO:teuthology.orchestra.run.smithi149.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-28T12:56:03.968 INFO:teuthology.orchestra.run.smithi149.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-28T12:56:03.968 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-28T12:56:03.968 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-argparse x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 45 k 2023-11-28T12:56:03.969 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-common x86_64 2:18.0.0-2697.g2f5f6028.el8 ceph 117 k 2023-11-28T12:56:03.969 INFO:teuthology.orchestra.run.smithi149.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-28T12:56:03.969 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-28T12:56:03.969 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-28T12:56:03.969 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-28T12:56:03.969 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-28T12:56:03.969 INFO:teuthology.orchestra.run.smithi149.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-28T12:56:03.969 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-28T12:56:03.969 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-28T12:56:03.969 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-28T12:56:03.969 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-28T12:56:03.969 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-28T12:56:03.969 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-28T12:56:03.970 INFO:teuthology.orchestra.run.smithi149.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-28T12:56:03.970 INFO:teuthology.orchestra.run.smithi149.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-28T12:56:03.970 INFO:teuthology.orchestra.run.smithi149.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-28T12:56:03.970 INFO:teuthology.orchestra.run.smithi149.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-28T12:56:03.970 INFO:teuthology.orchestra.run.smithi149.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-28T12:56:03.970 INFO:teuthology.orchestra.run.smithi149.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-28T12:56:03.970 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-28T12:56:03.970 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-28T12:56:03.970 INFO:teuthology.orchestra.run.smithi149.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-28T12:56:03.970 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-28T12:56:03.970 INFO:teuthology.orchestra.run.smithi149.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-28T12:56:03.970 INFO:teuthology.orchestra.run.smithi149.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-28T12:56:03.970 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-28T12:56:03.971 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-28T12:56:03.971 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-28T12:56:03.971 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-28T12:56:03.971 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-28T12:56:03.971 INFO:teuthology.orchestra.run.smithi149.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-28T12:56:03.971 INFO:teuthology.orchestra.run.smithi149.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-28T12:56:03.971 INFO:teuthology.orchestra.run.smithi149.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-28T12:56:03.971 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-28T12:56:03.971 INFO:teuthology.orchestra.run.smithi149.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-28T12:56:03.971 INFO:teuthology.orchestra.run.smithi149.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-28T12:56:03.971 INFO:teuthology.orchestra.run.smithi149.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-28T12:56:03.971 INFO:teuthology.orchestra.run.smithi149.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-28T12:56:03.971 INFO:teuthology.orchestra.run.smithi149.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-28T12:56:03.972 INFO:teuthology.orchestra.run.smithi149.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-28T12:56:03.972 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-28T12:56:03.972 INFO:teuthology.orchestra.run.smithi149.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-28T12:56:03.972 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-28T12:56:03.972 INFO:teuthology.orchestra.run.smithi149.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-28T12:56:03.972 INFO:teuthology.orchestra.run.smithi149.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-28T12:56:03.972 INFO:teuthology.orchestra.run.smithi149.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-28T12:56:03.972 INFO:teuthology.orchestra.run.smithi149.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-28T12:56:03.972 INFO:teuthology.orchestra.run.smithi149.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-28T12:56:03.972 INFO:teuthology.orchestra.run.smithi149.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-28T12:56:03.972 INFO:teuthology.orchestra.run.smithi149.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-28T12:56:03.972 INFO:teuthology.orchestra.run.smithi149.stdout:Enabling module streams: 2023-11-28T12:56:03.973 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql 10 2023-11-28T12:56:03.973 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:56:03.973 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2023-11-28T12:56:03.973 INFO:teuthology.orchestra.run.smithi149.stdout:=========================================================================================================================== 2023-11-28T12:56:03.973 INFO:teuthology.orchestra.run.smithi149.stdout:Install 128 Packages 2023-11-28T12:56:03.973 INFO:teuthology.orchestra.run.smithi149.stdout:Upgrade 3 Packages 2023-11-28T12:56:03.973 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:56:03.973 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 203 M 2023-11-28T12:56:03.974 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2023-11-28T12:56:04.194 INFO:teuthology.orchestra.run.smithi111.stdout:(4/131): ceph-mds-18.0.0-2697.g2f5f6028.el8.x86 6.3 MB/s | 2.3 MB 00:00 2023-11-28T12:56:04.503 INFO:teuthology.orchestra.run.smithi111.stdout:(5/131): ceph-mgr-18.0.0-2697.g2f5f6028.el8.x86 5.0 MB/s | 1.5 MB 00:00 2023-11-28T12:56:04.527 INFO:teuthology.orchestra.run.smithi149.stdout:(1/131): ceph-18.0.0-2697.g2f5f6028.el8.x86_64. 22 kB/s | 6.4 kB 00:00 2023-11-28T12:56:04.728 INFO:teuthology.orchestra.run.smithi111.stdout:(6/131): ceph-common-18.0.0-2697.g2f5f6028.el8. 16 MB/s | 22 MB 00:01 2023-11-28T12:56:05.211 INFO:teuthology.orchestra.run.smithi149.stdout:(2/131): ceph-fuse-18.0.0-2697.g2f5f6028.el8.x8 1.2 MB/s | 845 kB 00:00 2023-11-28T12:56:05.386 INFO:teuthology.orchestra.run.smithi149.stdout:(3/131): ceph-immutable-object-cache-18.0.0-269 946 kB/s | 165 kB 00:00 2023-11-28T12:56:05.404 INFO:teuthology.orchestra.run.smithi111.stdout:(7/131): ceph-mon-18.0.0-2697.g2f5f6028.el8.x86 5.1 MB/s | 4.6 MB 00:00 2023-11-28T12:56:05.453 INFO:teuthology.orchestra.run.smithi149.stdout:(4/131): ceph-base-18.0.0-2697.g2f5f6028.el8.x8 4.4 MB/s | 5.4 MB 00:01 2023-11-28T12:56:05.795 INFO:teuthology.orchestra.run.smithi149.stdout:(5/131): ceph-mgr-18.0.0-2697.g2f5f6028.el8.x86 4.5 MB/s | 1.5 MB 00:00 2023-11-28T12:56:06.387 INFO:teuthology.orchestra.run.smithi149.stdout:(6/131): ceph-mds-18.0.0-2697.g2f5f6028.el8.x86 2.3 MB/s | 2.3 MB 00:01 2023-11-28T12:56:06.587 INFO:teuthology.orchestra.run.smithi149.stdout:(7/131): ceph-mon-18.0.0-2697.g2f5f6028.el8.x86 5.8 MB/s | 4.6 MB 00:00 2023-11-28T12:56:06.845 INFO:teuthology.orchestra.run.smithi111.stdout:(8/131): ceph-osd-18.0.0-2697.g2f5f6028.el8.x86 8.4 MB/s | 18 MB 00:02 2023-11-28T12:56:06.929 INFO:teuthology.orchestra.run.smithi111.stdout:(9/131): ceph-selinux-18.0.0-2697.g2f5f6028.el8 292 kB/s | 24 kB 00:00 2023-11-28T12:56:07.980 INFO:teuthology.orchestra.run.smithi111.stdout:(10/131): ceph-radosgw-18.0.0-2697.g2f5f6028.el 5.1 MB/s | 13 MB 00:02 2023-11-28T12:56:08.157 INFO:teuthology.orchestra.run.smithi111.stdout:(11/131): cephfs-mirror-18.0.0-2697.g2f5f6028.e 1.4 MB/s | 257 kB 00:00 2023-11-28T12:56:08.179 INFO:teuthology.orchestra.run.smithi149.stdout:(8/131): ceph-common-18.0.0-2697.g2f5f6028.el8. 5.5 MB/s | 22 MB 00:03 2023-11-28T12:56:08.254 INFO:teuthology.orchestra.run.smithi149.stdout:(9/131): ceph-selinux-18.0.0-2697.g2f5f6028.el8 325 kB/s | 24 kB 00:00 2023-11-28T12:56:08.313 INFO:teuthology.orchestra.run.smithi111.stdout:(12/131): libcephfs-devel-18.0.0-2697.g2f5f6028 195 kB/s | 30 kB 00:00 2023-11-28T12:56:08.564 INFO:teuthology.orchestra.run.smithi111.stdout:(13/131): libcephfs2-18.0.0-2697.g2f5f6028.el8. 3.0 MB/s | 767 kB 00:00 2023-11-28T12:56:09.011 INFO:teuthology.orchestra.run.smithi111.stdout:(14/131): libcephsqlite-18.0.0-2697.g2f5f6028.e 307 kB/s | 137 kB 00:00 2023-11-28T12:56:09.139 INFO:teuthology.orchestra.run.smithi111.stdout:(15/131): librados-devel-18.0.0-2697.g2f5f6028. 1.1 MB/s | 139 kB 00:00 2023-11-28T12:56:09.446 INFO:teuthology.orchestra.run.smithi149.stdout:(10/131): ceph-radosgw-18.0.0-2697.g2f5f6028.el 4.6 MB/s | 13 MB 00:02 2023-11-28T12:56:09.473 INFO:teuthology.orchestra.run.smithi111.stdout:(16/131): libradosstriper1-18.0.0-2697.g2f5f602 1.4 MB/s | 474 kB 00:00 2023-11-28T12:56:09.521 INFO:teuthology.orchestra.run.smithi149.stdout:(11/131): cephfs-mirror-18.0.0-2697.g2f5f6028.e 3.4 MB/s | 257 kB 00:00 2023-11-28T12:56:09.596 INFO:teuthology.orchestra.run.smithi149.stdout:(12/131): libcephfs-devel-18.0.0-2697.g2f5f6028 407 kB/s | 30 kB 00:00 2023-11-28T12:56:09.763 INFO:teuthology.orchestra.run.smithi149.stdout:(13/131): libcephfs2-18.0.0-2697.g2f5f6028.el8. 4.5 MB/s | 767 kB 00:00 2023-11-28T12:56:09.855 INFO:teuthology.orchestra.run.smithi149.stdout:(14/131): libcephsqlite-18.0.0-2697.g2f5f6028.e 1.5 MB/s | 137 kB 00:00 2023-11-28T12:56:09.939 INFO:teuthology.orchestra.run.smithi149.stdout:(15/131): librados-devel-18.0.0-2697.g2f5f6028. 1.6 MB/s | 139 kB 00:00 2023-11-28T12:56:10.097 INFO:teuthology.orchestra.run.smithi149.stdout:(16/131): libradosstriper1-18.0.0-2697.g2f5f602 2.9 MB/s | 474 kB 00:00 2023-11-28T12:56:10.856 INFO:teuthology.orchestra.run.smithi149.stdout:(17/131): ceph-osd-18.0.0-2697.g2f5f6028.el8.x8 4.0 MB/s | 18 MB 00:04 2023-11-28T12:56:10.957 INFO:teuthology.orchestra.run.smithi149.stdout:(18/131): python3-ceph-argparse-18.0.0-2697.g2f 448 kB/s | 45 kB 00:00 2023-11-28T12:56:10.998 INFO:teuthology.orchestra.run.smithi111.stdout:(17/131): librgw2-18.0.0-2697.g2f5f6028.el8.x86 4.3 MB/s | 6.5 MB 00:01 2023-11-28T12:56:11.520 INFO:teuthology.orchestra.run.smithi111.stdout:(18/131): python3-ceph-argparse-18.0.0-2697.g2f 599 kB/s | 45 kB 00:00 2023-11-28T12:56:11.520 INFO:teuthology.orchestra.run.smithi111.stdout:(19/131): python3-ceph-common-18.0.0-2697.g2f5f 1.4 MB/s | 117 kB 00:00 2023-11-28T12:56:11.520 INFO:teuthology.orchestra.run.smithi111.stdout:(20/131): python3-cephfs-18.0.0-2697.g2f5f6028. 2.5 MB/s | 190 kB 00:00 2023-11-28T12:56:11.521 INFO:teuthology.orchestra.run.smithi149.stdout:(19/131): python3-ceph-common-18.0.0-2697.g2f5f 1.4 MB/s | 117 kB 00:00 2023-11-28T12:56:11.521 INFO:teuthology.orchestra.run.smithi149.stdout:(20/131): python3-cephfs-18.0.0-2697.g2f5f6028. 1.6 MB/s | 190 kB 00:00 2023-11-28T12:56:11.521 INFO:teuthology.orchestra.run.smithi149.stdout:(21/131): librgw2-18.0.0-2697.g2f5f6028.el8.x86 5.4 MB/s | 6.5 MB 00:01 2023-11-28T12:56:11.521 INFO:teuthology.orchestra.run.smithi149.stdout:(22/131): python3-rados-18.0.0-2697.g2f5f6028.e 2.2 MB/s | 371 kB 00:00 2023-11-28T12:56:11.521 INFO:teuthology.orchestra.run.smithi149.stdout:(23/131): python3-rbd-18.0.0-2697.g2f5f6028.el8 5.3 MB/s | 364 kB 00:00 2023-11-28T12:56:11.521 INFO:teuthology.orchestra.run.smithi149.stdout:(24/131): python3-rgw-18.0.0-2697.g2f5f6028.el8 1.6 MB/s | 107 kB 00:00 2023-11-28T12:56:11.521 INFO:teuthology.orchestra.run.smithi149.stdout:(25/131): rbd-fuse-18.0.0-2697.g2f5f6028.el8.x8 1.6 MB/s | 94 kB 00:00 2023-11-28T12:56:11.525 INFO:teuthology.orchestra.run.smithi149.stdout:(26/131): rbd-nbd-18.0.0-2697.g2f5f6028.el8.x86 1.8 MB/s | 180 kB 00:00 2023-11-28T12:56:11.584 INFO:teuthology.orchestra.run.smithi149.stdout:(27/131): ceph-grafana-dashboards-18.0.0-2697.g 397 kB/s | 23 kB 00:00 2023-11-28T12:56:11.642 INFO:teuthology.orchestra.run.smithi149.stdout:(28/131): ceph-mgr-cephadm-18.0.0-2697.g2f5f602 2.1 MB/s | 123 kB 00:00 2023-11-28T12:56:11.691 INFO:teuthology.orchestra.run.smithi111.stdout:(21/131): ceph-test-18.0.0-2697.g2f5f6028.el8.x 9.8 MB/s | 47 MB 00:04 2023-11-28T12:56:11.716 INFO:teuthology.orchestra.run.smithi111.stdout:(22/131): python3-rados-18.0.0-2697.g2f5f6028.e 768 kB/s | 371 kB 00:00 2023-11-28T12:56:11.734 INFO:teuthology.orchestra.run.smithi149.stdout:(29/131): rbd-mirror-18.0.0-2697.g2f5f6028.el8. 10 MB/s | 3.4 MB 00:00 2023-11-28T12:56:11.758 INFO:teuthology.orchestra.run.smithi111.stdout:(23/131): python3-rbd-18.0.0-2697.g2f5f6028.el8 5.3 MB/s | 364 kB 00:00 2023-11-28T12:56:11.784 INFO:teuthology.orchestra.run.smithi111.stdout:(24/131): python3-rgw-18.0.0-2697.g2f5f6028.el8 1.6 MB/s | 107 kB 00:00 2023-11-28T12:56:11.818 INFO:teuthology.orchestra.run.smithi111.stdout:(25/131): rbd-fuse-18.0.0-2697.g2f5f6028.el8.x8 1.6 MB/s | 94 kB 00:00 2023-11-28T12:56:11.910 INFO:teuthology.orchestra.run.smithi149.stdout:(30/131): ceph-mgr-dashboard-18.0.0-2697.g2f5f6 6.5 MB/s | 1.7 MB 00:00 2023-11-28T12:56:12.709 INFO:teuthology.orchestra.run.smithi111.stdout:(26/131): rbd-nbd-18.0.0-2697.g2f5f6028.el8.x86 202 kB/s | 180 kB 00:00 2023-11-28T12:56:12.754 INFO:teuthology.orchestra.run.smithi111.stdout:(27/131): ceph-grafana-dashboards-18.0.0-2697.g 556 kB/s | 23 kB 00:00 2023-11-28T12:56:12.818 INFO:teuthology.orchestra.run.smithi111.stdout:(28/131): ceph-mgr-cephadm-18.0.0-2697.g2f5f602 1.8 MB/s | 123 kB 00:00 2023-11-28T12:56:12.836 INFO:teuthology.orchestra.run.smithi149.stdout:(31/131): ceph-mgr-modules-core-18.0.0-2697.g2f 255 kB/s | 236 kB 00:00 2023-11-28T12:56:12.902 INFO:teuthology.orchestra.run.smithi149.stdout:(32/131): ceph-mgr-rook-18.0.0-2697.g2f5f6028.e 714 kB/s | 47 kB 00:00 2023-11-28T12:56:12.911 INFO:teuthology.orchestra.run.smithi111.stdout:(29/131): rbd-mirror-18.0.0-2697.g2f5f6028.el8. 3.1 MB/s | 3.4 MB 00:01 2023-11-28T12:56:12.994 INFO:teuthology.orchestra.run.smithi149.stdout:(33/131): ceph-prometheus-alerts-18.0.0-2697.g2 159 kB/s | 15 kB 00:00 2023-11-28T12:56:13.052 INFO:teuthology.orchestra.run.smithi111.stdout:(30/131): ceph-mgr-dashboard-18.0.0-2697.g2f5f6 7.4 MB/s | 1.7 MB 00:00 2023-11-28T12:56:13.086 INFO:teuthology.orchestra.run.smithi149.stdout:(34/131): ceph-volume-18.0.0-2697.g2f5f6028.el8 2.8 MB/s | 263 kB 00:00 2023-11-28T12:56:13.119 INFO:teuthology.orchestra.run.smithi111.stdout:(31/131): ceph-mgr-modules-core-18.0.0-2697.g2f 3.5 MB/s | 236 kB 00:00 2023-11-28T12:56:13.161 INFO:teuthology.orchestra.run.smithi149.stdout:(35/131): cephadm-18.0.0-2697.g2f5f6028.el8.noa 1.1 MB/s | 83 kB 00:00 2023-11-28T12:56:13.186 INFO:teuthology.orchestra.run.smithi111.stdout:(32/131): ceph-mgr-rook-18.0.0-2697.g2f5f6028.e 711 kB/s | 47 kB 00:00 2023-11-28T12:56:13.212 INFO:teuthology.orchestra.run.smithi149.stdout:(36/131): cephfs-top-18.0.0-2697.g2f5f6028.el8. 362 kB/s | 18 kB 00:00 2023-11-28T12:56:13.330 INFO:teuthology.orchestra.run.smithi149.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 4.7 MB/s | 7.4 MB 00:01 2023-11-28T12:56:13.404 INFO:teuthology.orchestra.run.smithi111.stdout:(33/131): ceph-prometheus-alerts-18.0.0-2697.g2 67 kB/s | 15 kB 00:00 2023-11-28T12:56:13.478 INFO:teuthology.orchestra.run.smithi111.stdout:(34/131): ceph-volume-18.0.0-2697.g2f5f6028.el8 3.5 MB/s | 263 kB 00:00 2023-11-28T12:56:13.528 INFO:teuthology.orchestra.run.smithi111.stdout:(35/131): cephadm-18.0.0-2697.g2f5f6028.el8.noa 1.6 MB/s | 83 kB 00:00 2023-11-28T12:56:13.583 INFO:teuthology.orchestra.run.smithi111.stdout:(36/131): cephfs-top-18.0.0-2697.g2f5f6028.el8. 360 kB/s | 18 kB 00:00 2023-11-28T12:56:13.654 INFO:teuthology.orchestra.run.smithi149.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 181 kB/s | 59 kB 00:00 2023-11-28T12:56:13.796 INFO:teuthology.orchestra.run.smithi149.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 524 kB/s | 306 kB 00:00 2023-11-28T12:56:13.838 INFO:teuthology.orchestra.run.smithi111.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 8.0 MB/s | 7.4 MB 00:00 2023-11-28T12:56:13.850 INFO:teuthology.orchestra.run.smithi149.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 383 kB/s | 75 kB 00:00 2023-11-28T12:56:14.021 INFO:teuthology.orchestra.run.smithi149.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.1 MB/s | 476 kB 00:00 2023-11-28T12:56:14.038 INFO:teuthology.orchestra.run.smithi149.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 232 kB/s | 44 kB 00:00 2023-11-28T12:56:14.113 INFO:teuthology.orchestra.run.smithi149.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.0 MB/s | 185 kB 00:00 2023-11-28T12:56:14.130 INFO:teuthology.orchestra.run.smithi149.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 359 kB/s | 33 kB 00:00 2023-11-28T12:56:14.206 INFO:teuthology.orchestra.run.smithi149.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 178 kB/s | 16 kB 00:00 2023-11-28T12:56:14.290 INFO:teuthology.orchestra.run.smithi149.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.1 MB/s | 173 kB 00:00 2023-11-28T12:56:14.314 INFO:teuthology.orchestra.run.smithi149.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.5 MB/s | 384 kB 00:00 2023-11-28T12:56:14.381 INFO:teuthology.orchestra.run.smithi149.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 886 kB/s | 81 kB 00:00 2023-11-28T12:56:14.407 INFO:teuthology.orchestra.run.smithi149.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2023-11-28T12:56:14.474 INFO:teuthology.orchestra.run.smithi149.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 190 kB/s | 18 kB 00:00 2023-11-28T12:56:14.479 INFO:teuthology.orchestra.run.smithi111.stdout:(38/131): gperftools-libs-2.7-9.el8.x86_64.rpm 340 kB/s | 306 kB 00:00 2023-11-28T12:56:14.521 INFO:teuthology.orchestra.run.smithi111.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 86 kB/s | 59 kB 00:00 2023-11-28T12:56:14.566 INFO:teuthology.orchestra.run.smithi149.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 532 kB/s | 49 kB 00:00 2023-11-28T12:56:14.630 INFO:teuthology.orchestra.run.smithi111.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 499 kB/s | 75 kB 00:00 2023-11-28T12:56:14.658 INFO:teuthology.orchestra.run.smithi149.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 642 kB/s | 59 kB 00:00 2023-11-28T12:56:14.692 INFO:teuthology.orchestra.run.smithi149.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 4.1 MB/s | 1.2 MB 00:00 2023-11-28T12:56:14.755 INFO:teuthology.orchestra.run.smithi111.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 349 kB/s | 44 kB 00:00 2023-11-28T12:56:14.758 INFO:teuthology.orchestra.run.smithi149.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 601 kB/s | 60 kB 00:00 2023-11-28T12:56:14.792 INFO:teuthology.orchestra.run.smithi149.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.8 MB/s | 283 kB 00:00 2023-11-28T12:56:14.851 INFO:teuthology.orchestra.run.smithi149.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2023-11-28T12:56:14.884 INFO:teuthology.orchestra.run.smithi149.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 366 kB/s | 34 kB 00:00 2023-11-28T12:56:14.889 INFO:teuthology.orchestra.run.smithi111.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 MB/s | 185 kB 00:00 2023-11-28T12:56:14.943 INFO:teuthology.orchestra.run.smithi149.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 2.1 MB/s | 196 kB 00:00 2023-11-28T12:56:14.973 INFO:teuthology.orchestra.run.smithi111.stdout:(43/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.0 MB/s | 476 kB 00:00 2023-11-28T12:56:14.976 INFO:teuthology.orchestra.run.smithi149.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 699 kB/s | 64 kB 00:00 2023-11-28T12:56:15.015 INFO:teuthology.orchestra.run.smithi111.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 263 kB/s | 33 kB 00:00 2023-11-28T12:56:15.035 INFO:teuthology.orchestra.run.smithi149.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 204 kB/s | 19 kB 00:00 2023-11-28T12:56:15.069 INFO:teuthology.orchestra.run.smithi149.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 260 kB/s | 24 kB 00:00 2023-11-28T12:56:15.106 INFO:teuthology.orchestra.run.smithi111.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 122 kB/s | 16 kB 00:00 2023-11-28T12:56:15.136 INFO:teuthology.orchestra.run.smithi149.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 291 kB/s | 29 kB 00:00 2023-11-28T12:56:15.149 INFO:teuthology.orchestra.run.smithi111.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.3 MB/s | 173 kB 00:00 2023-11-28T12:56:15.161 INFO:teuthology.orchestra.run.smithi149.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 299 kB/s | 27 kB 00:00 2023-11-28T12:56:15.244 INFO:teuthology.orchestra.run.smithi149.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 2.3 MB/s | 255 kB 00:00 2023-11-28T12:56:15.270 INFO:teuthology.orchestra.run.smithi149.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.3 MB/s | 251 kB 00:00 2023-11-28T12:56:15.299 INFO:teuthology.orchestra.run.smithi111.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 542 kB/s | 81 kB 00:00 2023-11-28T12:56:15.345 INFO:teuthology.orchestra.run.smithi149.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 610 kB/s | 61 kB 00:00 2023-11-28T12:56:15.366 INFO:teuthology.orchestra.run.smithi111.stdout:(48/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.5 MB/s | 384 kB 00:00 2023-11-28T12:56:15.426 INFO:teuthology.orchestra.run.smithi111.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 88 kB/s | 11 kB 00:00 2023-11-28T12:56:15.437 INFO:teuthology.orchestra.run.smithi149.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 251 kB/s | 23 kB 00:00 2023-11-28T12:56:15.508 INFO:teuthology.orchestra.run.smithi111.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 124 kB/s | 18 kB 00:00 2023-11-28T12:56:15.650 INFO:teuthology.orchestra.run.smithi111.stdout:(51/131): ceph-base-18.0.0-2697.g2f5f6028.el8.x 449 kB/s | 5.4 MB 00:12 2023-11-28T12:56:15.675 INFO:teuthology.orchestra.run.smithi111.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 292 kB/s | 49 kB 00:00 2023-11-28T12:56:15.729 INFO:teuthology.orchestra.run.smithi149.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 186 kB/s | 85 kB 00:00 2023-11-28T12:56:15.763 INFO:teuthology.orchestra.run.smithi149.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.0 MB/s | 70 kB 00:00 2023-11-28T12:56:15.767 INFO:teuthology.orchestra.run.smithi111.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 3.4 MB/s | 1.2 MB 00:00 2023-11-28T12:56:15.793 INFO:teuthology.orchestra.run.smithi111.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 515 kB/s | 60 kB 00:00 2023-11-28T12:56:15.846 INFO:teuthology.orchestra.run.smithi149.stdout:(70/131): thrift-0.13.0-2.el8.x86_64.rpm 4.2 MB/s | 1.7 MB 00:00 2023-11-28T12:56:15.876 INFO:teuthology.orchestra.run.smithi111.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.7 MB/s | 283 kB 00:00 2023-11-28T12:56:15.896 INFO:teuthology.orchestra.run.smithi149.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 351 kB/s | 47 kB 00:00 2023-11-28T12:56:15.901 INFO:teuthology.orchestra.run.smithi111.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2023-11-28T12:56:15.985 INFO:teuthology.orchestra.run.smithi111.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 311 kB/s | 34 kB 00:00 2023-11-28T12:56:16.027 INFO:teuthology.orchestra.run.smithi111.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 1.5 MB/s | 196 kB 00:00 2023-11-28T12:56:16.064 INFO:teuthology.orchestra.run.smithi149.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 922 kB/s | 200 kB 00:00 2023-11-28T12:56:16.086 INFO:teuthology.orchestra.run.smithi111.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 642 kB/s | 64 kB 00:00 2023-11-28T12:56:16.130 INFO:teuthology.orchestra.run.smithi149.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 182 kB/s | 43 kB 00:00 2023-11-28T12:56:16.144 INFO:teuthology.orchestra.run.smithi111.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 160 kB/s | 19 kB 00:00 2023-11-28T12:56:16.195 INFO:teuthology.orchestra.run.smithi111.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 220 kB/s | 24 kB 00:00 2023-11-28T12:56:16.231 INFO:teuthology.orchestra.run.smithi149.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.3 MB/s | 220 kB 00:00 2023-11-28T12:56:16.258 INFO:teuthology.orchestra.run.smithi111.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 269 kB/s | 29 kB 00:00 2023-11-28T12:56:16.278 INFO:teuthology.orchestra.run.smithi111.stdout:(63/131): python3-more-itertools-7.2.0-3.el8.no 93 kB/s | 59 kB 00:00 2023-11-28T12:56:16.304 INFO:teuthology.orchestra.run.smithi111.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 253 kB/s | 27 kB 00:00 2023-11-28T12:56:16.371 INFO:teuthology.orchestra.run.smithi111.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 2.1 MB/s | 255 kB 00:00 2023-11-28T12:56:16.415 INFO:teuthology.orchestra.run.smithi149.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 170 kB/s | 48 kB 00:00 2023-11-28T12:56:16.430 INFO:teuthology.orchestra.run.smithi111.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 488 kB/s | 61 kB 00:00 2023-11-28T12:56:16.448 INFO:teuthology.orchestra.run.smithi149.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 401 kB/s | 87 kB 00:00 2023-11-28T12:56:16.488 INFO:teuthology.orchestra.run.smithi111.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 731 kB/s | 85 kB 00:00 2023-11-28T12:56:16.547 INFO:teuthology.orchestra.run.smithi111.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 196 kB/s | 23 kB 00:00 2023-11-28T12:56:16.564 INFO:teuthology.orchestra.run.smithi111.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 4.1 MB/s | 70 kB 00:00 2023-11-28T12:56:16.614 INFO:teuthology.orchestra.run.smithi111.stdout:(70/131): python3-webob-1.8.5-1.el8.1.noarch.rp 749 kB/s | 251 kB 00:00 2023-11-28T12:56:16.674 INFO:teuthology.orchestra.run.smithi149.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 485 kB/s | 109 kB 00:00 2023-11-28T12:56:16.700 INFO:teuthology.orchestra.run.smithi149.stdout:(78/131): python3-requests-oauthlib-1.0.0-1.el8 150 kB/s | 43 kB 00:00 2023-11-28T12:56:16.723 INFO:teuthology.orchestra.run.smithi111.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 295 kB/s | 47 kB 00:00 2023-11-28T12:56:16.781 INFO:teuthology.orchestra.run.smithi111.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.2 MB/s | 200 kB 00:00 2023-11-28T12:56:16.848 INFO:teuthology.orchestra.run.smithi111.stdout:(73/131): thrift-0.13.0-2.el8.x86_64.rpm 4.8 MB/s | 1.7 MB 00:00 2023-11-28T12:56:16.865 INFO:teuthology.orchestra.run.smithi111.stdout:(74/131): python3-jwt-1.6.1-2.el8.noarch.rpm 300 kB/s | 43 kB 00:00 2023-11-28T12:56:16.899 INFO:teuthology.orchestra.run.smithi111.stdout:(75/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.8 MB/s | 220 kB 00:00 2023-11-28T12:56:16.957 INFO:teuthology.orchestra.run.smithi111.stdout:(76/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 442 kB/s | 48 kB 00:00 2023-11-28T12:56:16.983 INFO:teuthology.orchestra.run.smithi111.stdout:(77/131): libsepol-devel-2.9-3.el8.x86_64.rpm 743 kB/s | 87 kB 00:00 2023-11-28T12:56:17.066 INFO:teuthology.orchestra.run.smithi111.stdout:(78/131): python3-requests-oauthlib-1.0.0-1.el8 255 kB/s | 43 kB 00:00 2023-11-28T12:56:17.108 INFO:teuthology.orchestra.run.smithi111.stdout:(79/131): python3-pycparser-2.14-14.el8.noarch. 725 kB/s | 109 kB 00:00 2023-11-28T12:56:17.208 INFO:teuthology.orchestra.run.smithi149.stdout:(79/131): ceph-test-18.0.0-2697.g2f5f6028.el8.x 5.2 MB/s | 47 MB 00:08 2023-11-28T12:56:17.208 INFO:teuthology.orchestra.run.smithi111.stdout:(80/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.0 MB/s | 229 kB 00:00 2023-11-28T12:56:17.241 INFO:teuthology.orchestra.run.smithi149.stdout:(80/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 403 kB/s | 229 kB 00:00 2023-11-28T12:56:17.283 INFO:teuthology.orchestra.run.smithi149.stdout:(81/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 959 kB/s | 560 kB 00:00 2023-11-28T12:56:17.300 INFO:teuthology.orchestra.run.smithi111.stdout:(81/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 2.3 MB/s | 560 kB 00:00 2023-11-28T12:56:17.407 INFO:teuthology.orchestra.run.smithi111.stdout:(82/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 63 kB/s | 19 kB 00:00 2023-11-28T12:56:17.464 INFO:teuthology.orchestra.run.smithi111.stdout:(83/131): python3-oauthlib-2.1.0-1.el8.noarch.r 618 kB/s | 155 kB 00:00 2023-11-28T12:56:17.511 INFO:teuthology.orchestra.run.smithi149.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 2.4 MB/s | 559 kB 00:00 2023-11-28T12:56:17.533 INFO:teuthology.orchestra.run.smithi149.stdout:(83/131): python3-oauthlib-2.1.0-1.el8.noarch.r 540 kB/s | 155 kB 00:00 2023-11-28T12:56:17.567 INFO:teuthology.orchestra.run.smithi149.stdout:(84/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 52 kB/s | 19 kB 00:00 2023-11-28T12:56:17.626 INFO:teuthology.orchestra.run.smithi111.stdout:(84/131): python3-cryptography-3.2.1-5.el8.x86_ 1.7 MB/s | 559 kB 00:00 2023-11-28T12:56:17.727 INFO:teuthology.orchestra.run.smithi111.stdout:(85/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 56 kB/s | 18 kB 00:00 2023-11-28T12:56:17.742 INFO:teuthology.orchestra.run.smithi149.stdout:(85/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 77 kB/s | 18 kB 00:00 2023-11-28T12:56:17.760 INFO:teuthology.orchestra.run.smithi111.stdout:(86/131): pcre2-devel-10.32-2.el8.x86_64.rpm 2.0 MB/s | 605 kB 00:00 2023-11-28T12:56:17.776 INFO:teuthology.orchestra.run.smithi149.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.1 MB/s | 238 kB 00:00 2023-11-28T12:56:17.802 INFO:teuthology.orchestra.run.smithi111.stdout:(87/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.3 MB/s | 238 kB 00:00 2023-11-28T12:56:17.926 INFO:teuthology.orchestra.run.smithi149.stdout:(87/131): pcre2-devel-10.32-2.el8.x86_64.rpm 1.5 MB/s | 605 kB 00:00 2023-11-28T12:56:18.010 INFO:teuthology.orchestra.run.smithi111.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 289 kB/s | 72 kB 00:00 2023-11-28T12:56:18.026 INFO:teuthology.orchestra.run.smithi149.stdout:(88/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 8.2 MB/s | 2.3 MB 00:00 2023-11-28T12:56:18.052 INFO:teuthology.orchestra.run.smithi149.stdout:(89/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 262 kB/s | 72 kB 00:00 2023-11-28T12:56:18.108 INFO:teuthology.orchestra.run.smithi111.stdout:(89/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 6.1 MB/s | 2.3 MB 00:00 2023-11-28T12:56:18.136 INFO:teuthology.orchestra.run.smithi111.stdout:(90/131): fuse-2.9.7-15.el8.x86_64.rpm 248 kB/s | 83 kB 00:00 2023-11-28T12:56:18.161 INFO:teuthology.orchestra.run.smithi111.stdout:(91/131): elfutils-libelf-devel-0.186-1.el8.x86 398 kB/s | 60 kB 00:00 2023-11-28T12:56:18.210 INFO:teuthology.orchestra.run.smithi149.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 326 kB/s | 60 kB 00:00 2023-11-28T12:56:18.235 INFO:teuthology.orchestra.run.smithi149.stdout:(91/131): fuse-2.9.7-15.el8.x86_64.rpm 268 kB/s | 83 kB 00:00 2023-11-28T12:56:18.270 INFO:teuthology.orchestra.run.smithi149.stdout:(92/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 452 kB/s | 98 kB 00:00 2023-11-28T12:56:18.305 INFO:teuthology.orchestra.run.smithi111.stdout:(92/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 504 kB/s | 98 kB 00:00 2023-11-28T12:56:18.404 INFO:teuthology.orchestra.run.smithi111.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 159 kB/s | 39 kB 00:00 2023-11-28T12:56:18.419 INFO:teuthology.orchestra.run.smithi149.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 210 kB/s | 39 kB 00:00 2023-11-28T12:56:18.429 INFO:teuthology.orchestra.run.smithi111.stdout:(94/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 197 kB/s | 58 kB 00:00 2023-11-28T12:56:18.445 INFO:teuthology.orchestra.run.smithi149.stdout:(94/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 246 kB/s | 58 kB 00:00 2023-11-28T12:56:18.478 INFO:teuthology.orchestra.run.smithi149.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 3.0 MB/s | 644 kB 00:00 2023-11-28T12:56:18.546 INFO:teuthology.orchestra.run.smithi111.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 2.6 MB/s | 644 kB 00:00 2023-11-28T12:56:18.687 INFO:teuthology.orchestra.run.smithi149.stdout:(96/131): NetworkManager-initscripts-updown-1.3 569 kB/s | 138 kB 00:00 2023-11-28T12:56:18.737 INFO:teuthology.orchestra.run.smithi149.stdout:(97/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 539 kB/s | 171 kB 00:00 2023-11-28T12:56:18.763 INFO:teuthology.orchestra.run.smithi111.stdout:(96/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 477 kB/s | 171 kB 00:00 2023-11-28T12:56:18.854 INFO:teuthology.orchestra.run.smithi149.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 464 kB/s | 54 kB 00:00 2023-11-28T12:56:18.922 INFO:teuthology.orchestra.run.smithi111.stdout:(97/131): NetworkManager-initscripts-updown-1.3 279 kB/s | 138 kB 00:00 2023-11-28T12:56:18.938 INFO:teuthology.orchestra.run.smithi111.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 628 kB/s | 110 kB 00:00 2023-11-28T12:56:19.013 INFO:teuthology.orchestra.run.smithi149.stdout:(99/131): python3-scipy-1.0.0-21.module+el8.5.0 26 MB/s | 14 MB 00:00 2023-11-28T12:56:19.038 INFO:teuthology.orchestra.run.smithi149.stdout:(100/131): python3-pyasn1-modules-0.3.7-6.el8.n 314 kB/s | 110 kB 00:00 2023-11-28T12:56:19.039 INFO:teuthology.orchestra.run.smithi111.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 463 kB/s | 54 kB 00:00 2023-11-28T12:56:19.113 INFO:teuthology.orchestra.run.smithi149.stdout:(101/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2023-11-28T12:56:19.114 INFO:teuthology.orchestra.run.smithi111.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2023-11-28T12:56:19.256 INFO:teuthology.orchestra.run.smithi111.stdout:(101/131): python3-scipy-1.0.0-21.module+el8.5. 19 MB/s | 14 MB 00:00 2023-11-28T12:56:19.272 INFO:teuthology.orchestra.run.smithi149.stdout:(102/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.2 MB/s | 323 kB 00:00 2023-11-28T12:56:19.281 INFO:teuthology.orchestra.run.smithi111.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 232 kB/s | 39 kB 00:00 2023-11-28T12:56:19.307 INFO:teuthology.orchestra.run.smithi111.stdout:(103/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.2 MB/s | 323 kB 00:00 2023-11-28T12:56:19.313 INFO:teuthology.orchestra.run.smithi149.stdout:(103/131): python3-markupsafe-0.23-19.el8.x86_6 141 kB/s | 39 kB 00:00 2023-11-28T12:56:19.372 INFO:teuthology.orchestra.run.smithi149.stdout:(104/131): bison-3.0.4-10.el8.x86_64.rpm 2.6 MB/s | 688 kB 00:00 2023-11-28T12:56:19.455 INFO:teuthology.orchestra.run.smithi111.stdout:(104/131): bison-3.0.4-10.el8.x86_64.rpm 3.4 MB/s | 688 kB 00:00 2023-11-28T12:56:19.564 INFO:teuthology.orchestra.run.smithi149.stdout:(105/131): python3-jinja2-2.10.1-3.el8.noarch.r 1.8 MB/s | 538 kB 00:00 2023-11-28T12:56:19.574 INFO:teuthology.orchestra.run.smithi111.stdout:(105/131): python3-jinja2-2.10.1-3.el8.noarch.r 1.8 MB/s | 538 kB 00:00 2023-11-28T12:56:19.591 INFO:teuthology.orchestra.run.smithi111.stdout:(106/131): python3-mako-1.0.6-13.el8.noarch.rpm 553 kB/s | 157 kB 00:00 2023-11-28T12:56:19.606 INFO:teuthology.orchestra.run.smithi149.stdout:(106/131): python3-mako-1.0.6-13.el8.noarch.rpm 537 kB/s | 157 kB 00:00 2023-11-28T12:56:19.631 INFO:teuthology.orchestra.run.smithi149.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 724 kB/s | 187 kB 00:00 2023-11-28T12:56:19.725 INFO:teuthology.orchestra.run.smithi111.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 697 kB/s | 187 kB 00:00 2023-11-28T12:56:19.908 INFO:teuthology.orchestra.run.smithi111.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 145 kB/s | 27 kB 00:00 2023-11-28T12:56:19.915 INFO:teuthology.orchestra.run.smithi149.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 94 kB/s | 27 kB 00:00 2023-11-28T12:56:20.016 INFO:teuthology.orchestra.run.smithi111.stdout:(109/131): automake-1.16.1-7.el8.noarch.rpm 1.6 MB/s | 713 kB 00:00 2023-11-28T12:56:20.024 INFO:teuthology.orchestra.run.smithi149.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 11 MB/s | 4.8 MB 00:00 2023-11-28T12:56:20.057 INFO:teuthology.orchestra.run.smithi149.stdout:(110/131): automake-1.16.1-7.el8.noarch.rpm 1.4 MB/s | 713 kB 00:00 2023-11-28T12:56:20.149 INFO:teuthology.orchestra.run.smithi111.stdout:(110/131): python3-babel-2.5.1-7.el8.noarch.rpm 8.6 MB/s | 4.8 MB 00:00 2023-11-28T12:56:20.157 INFO:teuthology.orchestra.run.smithi149.stdout:(111/131): flex-2.6.1-9.el8.x86_64.rpm 1.3 MB/s | 320 kB 00:00 2023-11-28T12:56:20.201 INFO:teuthology.orchestra.run.smithi111.stdout:(111/131): flex-2.6.1-9.el8.x86_64.rpm 1.1 MB/s | 320 kB 00:00 2023-11-28T12:56:20.234 INFO:teuthology.orchestra.run.smithi111.stdout:(112/131): python3-prettytable-0.7.2-14.el8.noa 202 kB/s | 44 kB 00:00 2023-11-28T12:56:20.300 INFO:teuthology.orchestra.run.smithi149.stdout:(112/131): python3-numpy-1.14.3-10.el8.x86_64.r 15 MB/s | 3.7 MB 00:00 2023-11-28T12:56:20.325 INFO:teuthology.orchestra.run.smithi149.stdout:(113/131): python3-prettytable-0.7.2-14.el8.noa 146 kB/s | 44 kB 00:00 2023-11-28T12:56:20.351 INFO:teuthology.orchestra.run.smithi111.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 19 MB/s | 3.7 MB 00:00 2023-11-28T12:56:20.358 INFO:teuthology.orchestra.run.smithi149.stdout:(114/131): python3-werkzeug-0.12.2-4.el8.noarch 2.2 MB/s | 457 kB 00:00 2023-11-28T12:56:20.451 INFO:teuthology.orchestra.run.smithi149.stdout:(115/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.5 MB/s | 225 kB 00:00 2023-11-28T12:56:20.501 INFO:teuthology.orchestra.run.smithi149.stdout:(116/131): uuid-1.6.2-43.el8.x86_64.rpm 365 kB/s | 64 kB 00:00 2023-11-28T12:56:20.501 INFO:teuthology.orchestra.run.smithi111.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 843 kB/s | 225 kB 00:00 2023-11-28T12:56:20.527 INFO:teuthology.orchestra.run.smithi111.stdout:(115/131): python3-werkzeug-0.12.2-4.el8.noarch 1.4 MB/s | 457 kB 00:00 2023-11-28T12:56:20.668 INFO:teuthology.orchestra.run.smithi149.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 92 kB/s | 20 kB 00:00 2023-11-28T12:56:20.668 INFO:teuthology.orchestra.run.smithi111.stdout:(116/131): uuid-1.6.2-43.el8.x86_64.rpm 202 kB/s | 64 kB 00:00 2023-11-28T12:56:20.694 INFO:teuthology.orchestra.run.smithi111.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 120 kB/s | 20 kB 00:00 2023-11-28T12:56:20.736 INFO:teuthology.orchestra.run.smithi111.stdout:(118/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 540 kB/s | 126 kB 00:00 2023-11-28T12:56:20.760 INFO:teuthology.orchestra.run.smithi149.stdout:(118/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 315 kB/s | 126 kB 00:00 2023-11-28T12:56:20.826 INFO:teuthology.orchestra.run.smithi149.stdout:(119/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 318 kB/s | 103 kB 00:00 2023-11-28T12:56:20.911 INFO:teuthology.orchestra.run.smithi111.stdout:(119/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 427 kB/s | 103 kB 00:00 2023-11-28T12:56:21.035 INFO:teuthology.orchestra.run.smithi149.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 949 kB/s | 198 kB 00:00 2023-11-28T12:56:21.044 INFO:teuthology.orchestra.run.smithi111.stdout:(120/131): openblas-threads-0.3.15-3.el8.x86_64 16 MB/s | 4.9 MB 00:00 2023-11-28T12:56:21.127 INFO:teuthology.orchestra.run.smithi149.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 13 MB/s | 4.9 MB 00:00 2023-11-28T12:56:21.153 INFO:teuthology.orchestra.run.smithi111.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 818 kB/s | 198 kB 00:00 2023-11-28T12:56:21.186 INFO:teuthology.orchestra.run.smithi149.stdout:(122/131): libtool-2.4.6-25.el8.x86_64.rpm 1.3 MB/s | 709 kB 00:00 2023-11-28T12:56:21.203 INFO:teuthology.orchestra.run.smithi111.stdout:(122/131): libtool-2.4.6-25.el8.x86_64.rpm 1.4 MB/s | 709 kB 00:00 2023-11-28T12:56:21.337 INFO:teuthology.orchestra.run.smithi111.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-11-28T12:56:21.395 INFO:teuthology.orchestra.run.smithi111.stdout:(124/131): jq-1.6-3.el8.x86_64.rpm 834 kB/s | 202 kB 00:00 2023-11-28T12:56:21.513 INFO:teuthology.orchestra.run.smithi111.stdout:(125/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.1 MB/s | 354 kB 00:00 2023-11-28T12:56:21.536 INFO:teuthology.orchestra.run.smithi149.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.0 MB/s | 354 kB 00:00 2023-11-28T12:56:21.561 INFO:teuthology.orchestra.run.smithi149.stdout:(124/131): jq-1.6-3.el8.x86_64.rpm 466 kB/s | 202 kB 00:00 2023-11-28T12:56:21.622 INFO:teuthology.orchestra.run.smithi149.stdout:(125/131): openblas-0.3.15-3.el8.x86_64.rpm 8.1 MB/s | 4.7 MB 00:00 2023-11-28T12:56:21.696 INFO:teuthology.orchestra.run.smithi111.stdout:(126/131): postgresql-server-10.21-2.module+el8 14 MB/s | 5.1 MB 00:00 2023-11-28T12:56:21.728 INFO:teuthology.orchestra.run.smithi149.stdout:(126/131): postgresql-contrib-10.21-2.module+el 4.7 MB/s | 808 kB 00:00 2023-11-28T12:56:21.730 INFO:teuthology.orchestra.run.smithi111.stdout:(127/131): postgresql-contrib-10.21-2.module+el 2.4 MB/s | 808 kB 00:00 2023-11-28T12:56:21.845 INFO:teuthology.orchestra.run.smithi149.stdout:(127/131): postgresql-server-10.21-2.module+el8 16 MB/s | 5.1 MB 00:00 2023-11-28T12:56:21.864 INFO:teuthology.orchestra.run.smithi111.stdout:(128/131): postgresql-10.21-2.module+el8.6.0+15 4.3 MB/s | 1.5 MB 00:00 2023-11-28T12:56:21.912 INFO:teuthology.orchestra.run.smithi149.stdout:(128/131): postgresql-10.21-2.module+el8.6.0+15 5.1 MB/s | 1.5 MB 00:00 2023-11-28T12:56:22.114 INFO:teuthology.orchestra.run.smithi111.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 3.1 MB/s | 798 kB 00:00 2023-11-28T12:56:22.137 INFO:teuthology.orchestra.run.smithi149.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 3.5 MB/s | 798 kB 00:00 2023-11-28T12:56:22.173 INFO:teuthology.orchestra.run.smithi111.stdout:(130/131): librados2-18.0.0-2697.g2f5f6028.el8. 7.7 MB/s | 3.7 MB 00:00 2023-11-28T12:56:22.262 INFO:teuthology.orchestra.run.smithi149.stdout:(130/131): librados2-18.0.0-2697.g2f5f6028.el8. 6.8 MB/s | 3.7 MB 00:00 2023-11-28T12:56:22.346 INFO:teuthology.orchestra.run.smithi149.stdout:(131/131): librbd1-18.0.0-2697.g2f5f6028.el8.x8 7.1 MB/s | 3.6 MB 00:00 2023-11-28T12:56:22.349 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2023-11-28T12:56:22.349 INFO:teuthology.orchestra.run.smithi149.stdout:Total 11 MB/s | 203 MB 00:18 2023-11-28T12:56:23.196 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2023-11-28T12:56:23.317 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2023-11-28T12:56:23.317 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2023-11-28T12:56:23.556 INFO:teuthology.orchestra.run.smithi111.stdout:(131/131): librbd1-18.0.0-2697.g2f5f6028.el8.x8 2.0 MB/s | 3.6 MB 00:01 2023-11-28T12:56:23.561 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-11-28T12:56:23.561 INFO:teuthology.orchestra.run.smithi111.stdout:Total 10 MB/s | 203 MB 00:20 2023-11-28T12:56:24.419 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-11-28T12:56:24.429 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2023-11-28T12:56:24.433 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2023-11-28T12:56:24.541 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-11-28T12:56:24.541 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-11-28T12:56:25.650 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-11-28T12:56:25.654 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-11-28T12:56:29.279 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2023-11-28T12:56:29.461 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-28T12:56:29.645 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-28T12:56:29.814 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-28T12:56:29.940 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-28T12:56:30.163 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-28T12:56:30.538 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-11-28T12:56:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-28T12:56:30.748 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-28T12:56:30.798 INFO:teuthology.orchestra.run.smithi149.stdout: Upgrading : librados2-2:18.0.0-2697.g2f5f6028.el8.x86_64 5/134 2023-11-28T12:56:30.939 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-28T12:56:31.117 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-28T12:56:31.199 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librados2-2:18.0.0-2697.g2f5f6028.el8.x86_64 5/134 2023-11-28T12:56:31.281 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-28T12:56:31.388 INFO:teuthology.orchestra.run.smithi149.stdout: Upgrading : librbd1-2:18.0.0-2697.g2f5f6028.el8.x86_64 6/134 2023-11-28T12:56:31.492 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-28T12:56:31.498 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librbd1-2:18.0.0-2697.g2f5f6028.el8.x86_64 6/134 2023-11-28T12:56:31.663 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephfs2-2:18.0.0-2697.g2f5f6028.el8.x86_64 7/134 2023-11-28T12:56:31.775 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libcephfs2-2:18.0.0-2697.g2f5f6028.el8.x86_64 7/134 2023-11-28T12:56:31.893 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-28T12:56:31.935 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rados-2:18.0.0-2697.g2f5f6028.el8.x86_64 8/134 2023-11-28T12:56:32.091 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:18.0.0-2697.g2f5f6028.el8.x86_64 5/134 2023-11-28T12:56:32.138 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-11-28T12:56:32.384 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-28T12:56:32.471 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:18.0.0-2697.g2f5f6028.el8.x86_64 5/134 2023-11-28T12:56:32.591 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-28T12:56:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:18.0.0-2697.g2f5f6028.el8.x86_64 6/134 2023-11-28T12:56:32.813 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:18.0.0-2697.g2f5f6028.el8.x86_64 6/134 2023-11-28T12:56:32.992 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:18.0.0-2697.g2f5f6028.el8.x86_64 7/134 2023-11-28T12:56:33.040 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-28T12:56:33.085 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:18.0.0-2697.g2f5f6028.el8.x86_64 7/134 2023-11-28T12:56:33.276 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-28T12:56:33.305 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:18.0.0-2697.g2f5f6028.el8.x86_64 8/134 2023-11-28T12:56:33.455 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephsqlite-2:18.0.0-2697.g2f5f6028.el8.x86_64 14/134 2023-11-28T12:56:33.499 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-11-28T12:56:33.558 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2697.g2f5f6028.el8.x86_64 14/134 2023-11-28T12:56:33.754 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libradosstriper1-2:18.0.0-2697.g2f5f6028.el8.x86 15/134 2023-11-28T12:56:33.755 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-28T12:56:33.884 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2697.g2f5f6028.el8.x86 15/134 2023-11-28T12:56:33.962 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-28T12:56:34.091 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-11-28T12:56:34.352 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-28T12:56:34.649 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-28T12:56:34.654 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-11-28T12:56:34.824 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-11-28T12:56:34.841 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:18.0.0-2697.g2f5f6028.el8.x86_64 14/134 2023-11-28T12:56:34.958 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2697.g2f5f6028.el8.x86_64 14/134 2023-11-28T12:56:34.962 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-28T12:56:35.098 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-28T12:56:35.149 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:18.0.0-2697.g2f5f6028.el8.x86 15/134 2023-11-28T12:56:35.237 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-28T12:56:35.286 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2697.g2f5f6028.el8.x86 15/134 2023-11-28T12:56:35.469 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-11-28T12:56:35.613 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-11-28T12:56:35.764 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-11-28T12:56:35.858 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-28T12:56:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-28T12:56:36.029 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-28T12:56:36.092 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-28T12:56:36.216 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-28T12:56:36.632 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-28T12:56:36.818 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-28T12:56:36.820 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-11-28T12:56:36.982 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-28T12:56:37.514 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-11-28T12:56:37.583 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-28T12:56:37.770 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-11-28T12:56:37.782 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-11-28T12:56:37.969 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-11-28T12:56:38.214 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-28T12:56:38.369 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-ceph-common-2:18.0.0-2697.g2f5f6028.el8. 26/134 2023-11-28T12:56:38.498 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-11-28T12:56:38.505 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-ceph-argparse-2:18.0.0-2697.g2f5f6028.el 27/134 2023-11-28T12:56:38.735 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cephfs-2:18.0.0-2697.g2f5f6028.el8.x86_6 28/134 2023-11-28T12:56:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-11-28T12:56:38.921 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-11-28T12:56:38.937 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-28T12:56:39.149 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-28T12:56:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:18.0.0-2697.g2f5f6028.el8. 26/134 2023-11-28T12:56:39.434 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:18.0.0-2697.g2f5f6028.el 27/134 2023-11-28T12:56:39.695 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:18.0.0-2697.g2f5f6028.el8.x86_6 28/134 2023-11-28T12:56:39.924 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-28T12:56:40.695 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-28T12:56:41.677 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-28T12:56:41.711 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-11-28T12:56:42.018 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-11-28T12:56:42.273 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-11-28T12:56:42.475 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-11-28T12:56:42.690 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-28T12:56:42.709 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-11-28T12:56:42.872 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-28T12:56:43.003 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-11-28T12:56:43.042 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rbd-2:18.0.0-2697.g2f5f6028.el8.x86_64 36/134 2023-11-28T12:56:43.192 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librados-devel-2:18.0.0-2697.g2f5f6028.el8.x86_6 37/134 2023-11-28T12:56:43.319 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-11-28T12:56:43.343 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-11-28T12:56:43.530 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-11-28T12:56:43.541 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-11-28T12:56:43.703 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-11-28T12:56:43.752 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-28T12:56:43.774 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-11-28T12:56:43.950 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-28T12:56:43.992 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-11-28T12:56:44.137 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:18.0.0-2697.g2f5f6028.el8.x86_64 36/134 2023-11-28T12:56:44.190 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-11-28T12:56:44.288 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:18.0.0-2697.g2f5f6028.el8.x86_6 37/134 2023-11-28T12:56:44.353 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-28T12:56:44.436 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-28T12:56:44.464 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-11-28T12:56:44.609 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-11-28T12:56:44.640 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-11-28T12:56:44.756 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-11-28T12:56:44.798 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-11-28T12:56:44.829 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-11-28T12:56:44.957 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-11-28T12:56:45.033 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-11-28T12:56:45.129 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-11-28T12:56:45.243 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-11-28T12:56:45.347 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-11-28T12:56:45.390 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-28T12:56:45.465 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-28T12:56:45.506 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-11-28T12:56:45.678 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-11-28T12:56:45.701 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-11-28T12:56:45.852 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-11-28T12:56:45.899 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-11-28T12:56:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-11-28T12:56:46.190 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-11-28T12:56:46.290 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-11-28T12:56:46.367 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-11-28T12:56:46.518 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-11-28T12:56:46.546 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-11-28T12:56:46.689 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-11-28T12:56:46.730 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-11-28T12:56:46.871 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-11-28T12:56:46.964 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-11-28T12:56:47.182 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-11-28T12:56:47.249 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-11-28T12:56:47.477 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-11-28T12:56:47.516 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-11-28T12:56:47.702 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-11-28T12:56:47.750 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-11-28T12:56:47.925 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-11-28T12:56:48.004 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-11-28T12:56:48.141 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-11-28T12:56:48.219 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-11-28T12:56:48.316 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-11-28T12:56:48.472 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-11-28T12:56:48.558 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-11-28T12:56:48.750 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-11-28T12:56:48.763 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-11-28T12:56:48.923 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-11-28T12:56:49.040 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-11-28T12:56:49.143 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-11-28T12:56:49.265 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-11-28T12:56:49.305 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-11-28T12:56:49.467 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-11-28T12:56:49.486 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-11-28T12:56:49.626 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-28T12:56:49.774 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-11-28T12:56:49.821 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-28T12:56:50.036 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-11-28T12:56:50.093 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-28T12:56:50.246 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-11-28T12:56:50.292 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-28T12:56:50.431 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-28T12:56:50.490 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-11-28T12:56:50.612 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-28T12:56:50.655 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-28T12:56:50.792 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-28T12:56:50.816 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-28T12:56:50.995 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-28T12:56:51.098 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-28T12:56:51.161 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-28T12:56:51.306 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-28T12:56:51.321 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-28T12:56:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-28T12:56:51.490 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-28T12:56:51.591 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-28T12:56:51.693 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-28T12:56:51.797 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-28T12:56:51.879 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-28T12:56:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-28T12:56:52.048 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-28T12:56:52.133 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-28T12:56:52.266 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-28T12:56:52.293 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-28T12:56:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-28T12:56:52.534 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-28T12:56:52.635 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-28T12:56:52.699 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-28T12:56:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-28T12:56:52.910 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-28T12:56:53.078 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-28T12:56:53.335 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-28T12:56:53.580 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-28T12:56:53.769 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-28T12:56:53.982 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-28T12:56:54.001 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-28T12:56:54.280 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-28T12:56:54.509 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-28T12:56:54.785 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-28T12:56:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-28T12:56:55.068 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-28T12:56:55.283 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-28T12:56:55.308 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-28T12:56:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-28T12:56:55.847 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-28T12:56:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-28T12:56:56.107 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-28T12:56:56.276 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librgw2-2:18.0.0-2697.g2f5f6028.el8.x86_64 92/134 2023-11-28T12:56:56.323 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-28T12:56:56.351 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librgw2-2:18.0.0-2697.g2f5f6028.el8.x86_64 92/134 2023-11-28T12:56:56.527 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 93/134 2023-11-28T12:56:57.097 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-28T12:56:57.257 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:18.0.0-2697.g2f5f6028.el8.x86_64 92/134 2023-11-28T12:56:57.330 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:18.0.0-2697.g2f5f6028.el8.x86_64 92/134 2023-11-28T12:56:57.496 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 93/134 2023-11-28T12:57:00.135 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-common-2:18.0.0-2697.g2f5f6028.el8.x86_64 94/134 2023-11-28T12:57:00.634 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-common-2:18.0.0-2697.g2f5f6028.el8.x86_64 94/134 2023-11-28T12:57:01.108 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:18.0.0-2697.g2f5f6028.el8.x86_64 94/134 2023-11-28T12:57:01.225 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-common-2:18.0.0-2697.g2f5f6028.el8.x86_64 94/134 2023-11-28T12:57:01.296 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:18.0.0-2697.g2f5f6028.el8.x86_64 94/134 2023-11-28T12:57:01.546 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-base-2:18.0.0-2697.g2f5f6028.el8.x86_64 95/134 2023-11-28T12:57:01.629 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-base-2:18.0.0-2697.g2f5f6028.el8.x86_64 95/134 2023-11-28T12:57:01.767 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-selinux-2:18.0.0-2697.g2f5f6028.el8.x86_64 96/134 2023-11-28T12:57:01.893 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:18.0.0-2697.g2f5f6028.el8.x86_64 94/134 2023-11-28T12:57:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:18.0.0-2697.g2f5f6028.el8.x86_64 95/134 2023-11-28T12:57:02.275 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:18.0.0-2697.g2f5f6028.el8.x86_64 95/134 2023-11-28T12:57:02.445 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:18.0.0-2697.g2f5f6028.el8.x86_64 96/134 2023-11-28T12:57:20.676 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2697.g2f5f6028.el8.x86_64 96/134 2023-11-28T12:57:20.676 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /sys 2023-11-28T12:57:20.676 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /proc 2023-11-28T12:57:20.676 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /dev 2023-11-28T12:57:20.676 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /run 2023-11-28T12:57:20.676 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /mnt 2023-11-28T12:57:20.676 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /var/tmp 2023-11-28T12:57:20.677 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /home 2023-11-28T12:57:20.677 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /tmp 2023-11-28T12:57:20.677 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /dev 2023-11-28T12:57:20.677 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:57:21.014 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-volume-2:18.0.0-2697.g2f5f6028.el8.noarch 97/134 2023-11-28T12:57:21.233 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2697.g2f5f6028.el8.x86_64 96/134 2023-11-28T12:57:21.233 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2023-11-28T12:57:21.233 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2023-11-28T12:57:21.233 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /dev 2023-11-28T12:57:21.233 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /run 2023-11-28T12:57:21.233 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2023-11-28T12:57:21.233 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2023-11-28T12:57:21.233 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2023-11-28T12:57:21.233 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2023-11-28T12:57:21.234 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /dev 2023-11-28T12:57:21.234 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:57:21.567 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-volume-2:18.0.0-2697.g2f5f6028.el8.noarch 97/134 2023-11-28T12:57:22.587 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-volume-2:18.0.0-2697.g2f5f6028.el8.noarch 97/134 2023-11-28T12:57:22.743 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-osd-2:18.0.0-2697.g2f5f6028.el8.x86_64 98/134 2023-11-28T12:57:22.961 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-osd-2:18.0.0-2697.g2f5f6028.el8.x86_64 98/134 2023-11-28T12:57:23.106 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mds-2:18.0.0-2697.g2f5f6028.el8.x86_64 99/134 2023-11-28T12:57:23.140 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-volume-2:18.0.0-2697.g2f5f6028.el8.noarch 97/134 2023-11-28T12:57:23.309 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:18.0.0-2697.g2f5f6028.el8.x86_64 98/134 2023-11-28T12:57:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:18.0.0-2697.g2f5f6028.el8.x86_64 98/134 2023-11-28T12:57:23.567 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mds-2:18.0.0-2697.g2f5f6028.el8.x86_64 99/134 2023-11-28T12:57:23.676 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:18.0.0-2697.g2f5f6028.el8.x86_64 99/134 2023-11-28T12:57:23.748 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mon-2:18.0.0-2697.g2f5f6028.el8.x86_64 100/134 2023-11-28T12:57:23.764 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mon-2:18.0.0-2697.g2f5f6028.el8.x86_64 100/134 2023-11-28T12:57:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:18.0.0-2697.g2f5f6028.el8.x86_64 99/134 2023-11-28T12:57:24.334 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:18.0.0-2697.g2f5f6028.el8.x86_64 100/134 2023-11-28T12:57:24.351 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:18.0.0-2697.g2f5f6028.el8.x86_64 100/134 2023-11-28T12:57:25.454 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: cephadm-2:18.0.0-2697.g2f5f6028.el8.noarch 101/134 2023-11-28T12:57:25.603 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : cephadm-2:18.0.0-2697.g2f5f6028.el8.noarch 101/134 2023-11-28T12:57:25.728 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2697.g2f5f6028.e 102/134 2023-11-28T12:57:25.934 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2697.g2f5f6028. 103/134 2023-11-28T12:57:25.991 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:18.0.0-2697.g2f5f6028.el8.noarch 101/134 2023-11-28T12:57:26.121 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2697.g2f5f6028.el8.noa 104/134 2023-11-28T12:57:26.144 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:18.0.0-2697.g2f5f6028.el8.noarch 101/134 2023-11-28T12:57:26.239 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2697.g2f5f6028.e 102/134 2023-11-28T12:57:26.335 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2697.g2f5f6028.el8.noa 104/134 2023-11-28T12:57:26.455 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2697.g2f5f6028. 103/134 2023-11-28T12:57:26.625 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2697.g2f5f6028.el8.n 105/134 2023-11-28T12:57:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2697.g2f5f6028.el8.noa 104/134 2023-11-28T12:57:26.868 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2697.g2f5f6028.el8.noa 104/134 2023-11-28T12:57:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2697.g2f5f6028.el8.n 105/134 2023-11-28T12:57:27.592 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2697.g2f5f6028.el8.n 105/134 2023-11-28T12:57:27.754 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2697.g2f5 106/134 2023-11-28T12:57:27.844 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2697.g2f5 106/134 2023-11-28T12:57:28.146 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2697.g2f5f6028.el8.n 105/134 2023-11-28T12:57:28.220 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2697.g2f5f6028.el 107/134 2023-11-28T12:57:28.299 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2697.g2f5 106/134 2023-11-28T12:57:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2697.g2f5 106/134 2023-11-28T12:57:28.400 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-2:18.0.0-2697.g2f5f6028.el8.x86_64 108/134 2023-11-28T12:57:28.437 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2697.g2f5f6028.el8.x86_64 108/134 2023-11-28T12:57:28.596 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-rook-2:18.0.0-2697.g2f5f6028.el8.noarch 109/134 2023-11-28T12:57:28.607 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2697.g2f5f6028.el8.noarch 109/134 2023-11-28T12:57:28.745 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2697.g2f5f6028.el 107/134 2023-11-28T12:57:28.908 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:18.0.0-2697.g2f5f6028.el8.x86_64 108/134 2023-11-28T12:57:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2697.g2f5f6028.el8.x86_64 108/134 2023-11-28T12:57:29.083 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:18.0.0-2697.g2f5f6028.el8.noarch 109/134 2023-11-28T12:57:29.094 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2697.g2f5f6028.el8.noarch 109/134 2023-11-28T12:57:29.846 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-2:18.0.0-2697.g2f5f6028.el8.x86_64 110/134 2023-11-28T12:57:30.055 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-radosgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 111/134 2023-11-28T12:57:30.089 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 111/134 2023-11-28T12:57:30.229 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2697.g2f5f6 112/134 2023-11-28T12:57:30.271 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2697.g2f5f6 112/134 2023-11-28T12:57:30.356 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:18.0.0-2697.g2f5f6028.el8.x86_64 110/134 2023-11-28T12:57:30.405 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : cephfs-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 113/134 2023-11-28T12:57:30.612 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 111/134 2023-11-28T12:57:30.646 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 111/134 2023-11-28T12:57:30.751 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 113/134 2023-11-28T12:57:30.799 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2697.g2f5f6 112/134 2023-11-28T12:57:30.841 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2697.g2f5f6 112/134 2023-11-28T12:57:30.904 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 114/134 2023-11-28T12:57:30.975 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 113/134 2023-11-28T12:57:31.321 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 113/134 2023-11-28T12:57:31.474 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 114/134 2023-11-28T12:57:35.326 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 114/134 2023-11-28T12:57:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 114/134 2023-11-28T12:57:36.212 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-test-2:18.0.0-2697.g2f5f6028.el8.x86_64 115/134 2023-11-28T12:57:36.507 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-11-28T12:57:36.674 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-fuse-2:18.0.0-2697.g2f5f6028.el8.x86_64 117/134 2023-11-28T12:57:36.806 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:18.0.0-2697.g2f5f6028.el8.x86_64 115/134 2023-11-28T12:57:36.912 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-11-28T12:57:37.061 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-11-28T12:57:37.131 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-11-28T12:57:37.254 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-11-28T12:57:37.294 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:18.0.0-2697.g2f5f6028.el8.x86_64 117/134 2023-11-28T12:57:37.491 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-11-28T12:57:37.527 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-11-28T12:57:37.665 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephfs-devel-2:18.0.0-2697.g2f5f6028.el8.x86_ 121/134 2023-11-28T12:57:37.723 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-11-28T12:57:37.837 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-28T12:57:37.914 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-11-28T12:57:38.144 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-11-28T12:57:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:18.0.0-2697.g2f5f6028.el8.x86_ 121/134 2023-11-28T12:57:38.491 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-28T12:57:40.116 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-28T12:57:40.344 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-28T12:57:40.363 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-28T12:57:40.545 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : cephfs-top-2:18.0.0-2697.g2f5f6028.el8.noarch 124/134 2023-11-28T12:57:40.719 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-28T12:57:40.819 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-fuse-2:18.0.0-2697.g2f5f6028.el8.x86_64 125/134 2023-11-28T12:57:40.939 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-28T12:57:40.957 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-28T12:57:41.099 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-nbd-2:18.0.0-2697.g2f5f6028.el8.x86_64 126/134 2023-11-28T12:57:41.136 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-top-2:18.0.0-2697.g2f5f6028.el8.noarch 124/134 2023-11-28T12:57:41.406 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:18.0.0-2697.g2f5f6028.el8.x86_64 125/134 2023-11-28T12:57:41.446 INFO:teuthology.orchestra.run.smithi149.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-28T12:57:41.644 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-11-28T12:57:41.686 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:18.0.0-2697.g2f5f6028.el8.x86_64 126/134 2023-11-28T12:57:41.728 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-11-28T12:57:41.921 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-11-28T12:57:41.932 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-11-28T12:57:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-28T12:57:42.077 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-28T12:57:42.085 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-28T12:57:42.205 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-11-28T12:57:42.221 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-11-28T12:57:42.221 INFO:teuthology.orchestra.run.smithi149.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-28T12:57:42.284 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-11-28T12:57:42.430 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-28T12:57:42.430 INFO:teuthology.orchestra.run.smithi149.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-28T12:57:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-11-28T12:57:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-11-28T12:57:42.656 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-28T12:57:42.664 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-28T12:57:42.666 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-28T12:57:42.791 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-11-28T12:57:42.791 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-28T12:57:42.843 INFO:teuthology.orchestra.run.smithi149.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-28T12:57:42.967 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-28T12:57:42.967 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-28T12:57:43.240 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-28T12:57:43.369 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-28T12:57:43.370 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-2:18.0.0-2697.g2f5f6028.el8.x86_64 1/134 2023-11-28T12:57:43.370 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-base-2:18.0.0-2697.g2f5f6028.el8.x86_64 2/134 2023-11-28T12:57:43.370 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-common-2:18.0.0-2697.g2f5f6028.el8.x86_64 3/134 2023-11-28T12:57:43.370 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-fuse-2:18.0.0-2697.g2f5f6028.el8.x86_64 4/134 2023-11-28T12:57:43.370 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2697.g2f5f6 5/134 2023-11-28T12:57:43.370 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mds-2:18.0.0-2697.g2f5f6028.el8.x86_64 6/134 2023-11-28T12:57:43.371 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-2:18.0.0-2697.g2f5f6028.el8.x86_64 7/134 2023-11-28T12:57:43.371 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mon-2:18.0.0-2697.g2f5f6028.el8.x86_64 8/134 2023-11-28T12:57:43.371 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-osd-2:18.0.0-2697.g2f5f6028.el8.x86_64 9/134 2023-11-28T12:57:43.371 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-radosgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 10/134 2023-11-28T12:57:43.371 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-selinux-2:18.0.0-2697.g2f5f6028.el8.x86_64 11/134 2023-11-28T12:57:43.371 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-test-2:18.0.0-2697.g2f5f6028.el8.x86_64 12/134 2023-11-28T12:57:43.371 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : cephfs-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 13/134 2023-11-28T12:57:43.372 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephfs-devel-2:18.0.0-2697.g2f5f6028.el8.x86_ 14/134 2023-11-28T12:57:43.372 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephfs2-2:18.0.0-2697.g2f5f6028.el8.x86_64 15/134 2023-11-28T12:57:43.372 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephsqlite-2:18.0.0-2697.g2f5f6028.el8.x86_64 16/134 2023-11-28T12:57:43.372 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados-devel-2:18.0.0-2697.g2f5f6028.el8.x86_6 17/134 2023-11-28T12:57:43.372 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libradosstriper1-2:18.0.0-2697.g2f5f6028.el8.x86 18/134 2023-11-28T12:57:43.372 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librgw2-2:18.0.0-2697.g2f5f6028.el8.x86_64 19/134 2023-11-28T12:57:43.372 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2697.g2f5f6028.el 20/134 2023-11-28T12:57:43.372 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-ceph-common-2:18.0.0-2697.g2f5f6028.el8. 21/134 2023-11-28T12:57:43.372 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cephfs-2:18.0.0-2697.g2f5f6028.el8.x86_6 22/134 2023-11-28T12:57:43.372 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rados-2:18.0.0-2697.g2f5f6028.el8.x86_64 23/134 2023-11-28T12:57:43.372 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rbd-2:18.0.0-2697.g2f5f6028.el8.x86_64 24/134 2023-11-28T12:57:43.372 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 25/134 2023-11-28T12:57:43.373 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-fuse-2:18.0.0-2697.g2f5f6028.el8.x86_64 26/134 2023-11-28T12:57:43.373 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 27/134 2023-11-28T12:57:43.373 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-nbd-2:18.0.0-2697.g2f5f6028.el8.x86_64 28/134 2023-11-28T12:57:43.373 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2697.g2f5f6028. 29/134 2023-11-28T12:57:43.373 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2697.g2f5f6028.el8.noa 30/134 2023-11-28T12:57:43.373 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2697.g2f5f6028.el8.n 31/134 2023-11-28T12:57:43.373 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2697.g2f5 32/134 2023-11-28T12:57:43.373 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2697.g2f5f6028.el 33/134 2023-11-28T12:57:43.373 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2697.g2f5f6028.el8.noarch 34/134 2023-11-28T12:57:43.373 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2697.g2f5f6028.e 35/134 2023-11-28T12:57:43.373 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-volume-2:18.0.0-2697.g2f5f6028.el8.noarch 36/134 2023-11-28T12:57:43.373 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : cephadm-2:18.0.0-2697.g2f5f6028.el8.noarch 37/134 2023-11-28T12:57:43.373 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : cephfs-top-2:18.0.0-2697.g2f5f6028.el8.noarch 38/134 2023-11-28T12:57:43.374 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-28T12:57:43.374 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-28T12:57:43.374 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-28T12:57:43.374 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-28T12:57:43.374 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-28T12:57:43.374 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-28T12:57:43.374 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-28T12:57:43.374 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-28T12:57:43.374 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-28T12:57:43.374 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-28T12:57:43.374 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-28T12:57:43.375 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-28T12:57:43.375 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-28T12:57:43.375 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-28T12:57:43.375 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-28T12:57:43.375 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-28T12:57:43.376 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-28T12:57:43.376 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-28T12:57:43.376 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-28T12:57:43.376 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-28T12:57:43.376 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-28T12:57:43.376 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-28T12:57:43.377 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-28T12:57:43.377 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-28T12:57:43.377 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-28T12:57:43.377 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-28T12:57:43.377 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-28T12:57:43.377 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-28T12:57:43.377 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-28T12:57:43.378 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-28T12:57:43.378 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-28T12:57:43.378 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-28T12:57:43.378 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-28T12:57:43.378 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-11-28T12:57:43.378 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-11-28T12:57:43.378 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-11-28T12:57:43.379 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-11-28T12:57:43.379 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-11-28T12:57:43.379 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-11-28T12:57:43.379 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-11-28T12:57:43.379 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-11-28T12:57:43.379 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-11-28T12:57:43.379 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-11-28T12:57:43.380 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-11-28T12:57:43.380 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-11-28T12:57:43.380 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-11-28T12:57:43.380 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-11-28T12:57:43.380 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-11-28T12:57:43.381 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-11-28T12:57:43.381 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-11-28T12:57:43.381 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-11-28T12:57:43.381 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-11-28T12:57:43.381 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-11-28T12:57:43.381 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-11-28T12:57:43.382 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-11-28T12:57:43.382 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-11-28T12:57:43.382 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-11-28T12:57:43.382 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-11-28T12:57:43.382 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-11-28T12:57:43.382 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-11-28T12:57:43.382 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-11-28T12:57:43.382 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-11-28T12:57:43.382 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-11-28T12:57:43.383 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-11-28T12:57:43.383 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-11-28T12:57:43.383 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-11-28T12:57:43.383 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-11-28T12:57:43.383 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-11-28T12:57:43.383 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-11-28T12:57:43.383 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-11-28T12:57:43.383 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-11-28T12:57:43.383 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-11-28T12:57:43.383 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-11-28T12:57:43.383 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-11-28T12:57:43.383 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-11-28T12:57:43.383 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-11-28T12:57:43.384 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-11-28T12:57:43.384 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-11-28T12:57:43.384 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-11-28T12:57:43.384 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-11-28T12:57:43.384 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-11-28T12:57:43.384 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-11-28T12:57:43.384 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-11-28T12:57:43.384 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-11-28T12:57:43.384 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-11-28T12:57:43.384 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-11-28T12:57:43.384 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-11-28T12:57:43.384 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-11-28T12:57:43.384 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-11-28T12:57:43.385 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-11-28T12:57:43.385 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados2-2:18.0.0-2697.g2f5f6028.el8.x86_64 129/134 2023-11-28T12:57:43.385 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-28T12:57:43.385 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librbd1-2:18.0.0-2697.g2f5f6028.el8.x86_64 131/134 2023-11-28T12:57:43.385 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-28T12:57:43.385 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-28T12:57:43.430 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-28T12:57:43.927 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-28T12:57:43.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:18.0.0-2697.g2f5f6028.el8.x86_64 1/134 2023-11-28T12:57:43.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:18.0.0-2697.g2f5f6028.el8.x86_64 2/134 2023-11-28T12:57:43.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:18.0.0-2697.g2f5f6028.el8.x86_64 3/134 2023-11-28T12:57:43.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:18.0.0-2697.g2f5f6028.el8.x86_64 4/134 2023-11-28T12:57:43.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2697.g2f5f6 5/134 2023-11-28T12:57:43.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:18.0.0-2697.g2f5f6028.el8.x86_64 6/134 2023-11-28T12:57:43.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:18.0.0-2697.g2f5f6028.el8.x86_64 7/134 2023-11-28T12:57:43.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:18.0.0-2697.g2f5f6028.el8.x86_64 8/134 2023-11-28T12:57:43.927 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:18.0.0-2697.g2f5f6028.el8.x86_64 9/134 2023-11-28T12:57:43.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 10/134 2023-11-28T12:57:43.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:18.0.0-2697.g2f5f6028.el8.x86_64 11/134 2023-11-28T12:57:43.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:18.0.0-2697.g2f5f6028.el8.x86_64 12/134 2023-11-28T12:57:43.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 13/134 2023-11-28T12:57:43.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:18.0.0-2697.g2f5f6028.el8.x86_ 14/134 2023-11-28T12:57:43.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:18.0.0-2697.g2f5f6028.el8.x86_64 15/134 2023-11-28T12:57:43.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:18.0.0-2697.g2f5f6028.el8.x86_64 16/134 2023-11-28T12:57:43.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:18.0.0-2697.g2f5f6028.el8.x86_6 17/134 2023-11-28T12:57:43.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:18.0.0-2697.g2f5f6028.el8.x86 18/134 2023-11-28T12:57:43.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:18.0.0-2697.g2f5f6028.el8.x86_64 19/134 2023-11-28T12:57:43.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2697.g2f5f6028.el 20/134 2023-11-28T12:57:43.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:18.0.0-2697.g2f5f6028.el8. 21/134 2023-11-28T12:57:43.928 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:18.0.0-2697.g2f5f6028.el8.x86_6 22/134 2023-11-28T12:57:43.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:18.0.0-2697.g2f5f6028.el8.x86_64 23/134 2023-11-28T12:57:43.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:18.0.0-2697.g2f5f6028.el8.x86_64 24/134 2023-11-28T12:57:43.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 25/134 2023-11-28T12:57:43.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:18.0.0-2697.g2f5f6028.el8.x86_64 26/134 2023-11-28T12:57:43.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 27/134 2023-11-28T12:57:43.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:18.0.0-2697.g2f5f6028.el8.x86_64 28/134 2023-11-28T12:57:43.929 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2697.g2f5f6028. 29/134 2023-11-28T12:57:43.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2697.g2f5f6028.el8.noa 30/134 2023-11-28T12:57:43.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2697.g2f5f6028.el8.n 31/134 2023-11-28T12:57:43.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2697.g2f5 32/134 2023-11-28T12:57:43.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2697.g2f5f6028.el 33/134 2023-11-28T12:57:43.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2697.g2f5f6028.el8.noarch 34/134 2023-11-28T12:57:43.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2697.g2f5f6028.e 35/134 2023-11-28T12:57:43.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-volume-2:18.0.0-2697.g2f5f6028.el8.noarch 36/134 2023-11-28T12:57:43.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:18.0.0-2697.g2f5f6028.el8.noarch 37/134 2023-11-28T12:57:43.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-top-2:18.0.0-2697.g2f5f6028.el8.noarch 38/134 2023-11-28T12:57:43.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-28T12:57:43.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-28T12:57:43.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-28T12:57:43.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-28T12:57:43.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-28T12:57:43.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-28T12:57:43.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-28T12:57:43.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-28T12:57:43.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-28T12:57:43.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-28T12:57:43.939 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-28T12:57:43.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-28T12:57:43.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-28T12:57:43.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-28T12:57:43.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-28T12:57:43.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-28T12:57:43.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-28T12:57:43.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-28T12:57:43.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-28T12:57:43.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-28T12:57:43.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-28T12:57:43.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-28T12:57:43.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-28T12:57:43.940 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-28T12:57:43.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-28T12:57:43.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-28T12:57:43.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-28T12:57:43.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-28T12:57:43.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-28T12:57:43.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-28T12:57:43.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-28T12:57:43.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-28T12:57:43.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-28T12:57:43.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-11-28T12:57:43.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-11-28T12:57:43.941 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-11-28T12:57:43.942 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-11-28T12:57:43.942 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-11-28T12:57:43.942 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-11-28T12:57:43.942 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-11-28T12:57:43.942 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-11-28T12:57:43.942 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-11-28T12:57:43.942 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-11-28T12:57:43.942 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-11-28T12:57:43.942 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-11-28T12:57:43.942 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-11-28T12:57:43.942 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-11-28T12:57:43.942 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-11-28T12:57:43.943 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-11-28T12:57:43.943 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-11-28T12:57:43.943 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-11-28T12:57:43.943 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-11-28T12:57:43.943 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-11-28T12:57:43.943 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-11-28T12:57:43.943 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-11-28T12:57:43.943 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-11-28T12:57:43.943 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-11-28T12:57:43.943 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-11-28T12:57:43.943 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-11-28T12:57:43.943 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-11-28T12:57:43.943 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-11-28T12:57:43.944 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-11-28T12:57:43.944 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-11-28T12:57:43.944 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-11-28T12:57:43.944 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-11-28T12:57:43.944 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-11-28T12:57:43.944 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-11-28T12:57:43.944 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-11-28T12:57:43.944 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-11-28T12:57:43.944 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-11-28T12:57:43.944 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-11-28T12:57:43.944 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-11-28T12:57:43.944 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-11-28T12:57:43.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-11-28T12:57:43.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-11-28T12:57:43.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-11-28T12:57:43.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-11-28T12:57:43.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-11-28T12:57:43.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-11-28T12:57:43.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-11-28T12:57:43.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-11-28T12:57:43.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-11-28T12:57:43.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-11-28T12:57:43.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-11-28T12:57:43.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-11-28T12:57:43.945 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-11-28T12:57:43.946 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-11-28T12:57:43.946 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-11-28T12:57:43.946 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-11-28T12:57:43.946 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-11-28T12:57:43.946 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:18.0.0-2697.g2f5f6028.el8.x86_64 129/134 2023-11-28T12:57:43.946 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-28T12:57:43.946 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:18.0.0-2697.g2f5f6028.el8.x86_64 131/134 2023-11-28T12:57:43.946 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-28T12:57:43.946 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-28T12:57:46.029 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-28T12:57:46.029 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:01:54 ago on Tue 28 Nov 2023 12:55:52 PM UTC. 2023-11-28T12:57:46.635 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-28T12:57:46.635 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:55 ago on Tue 28 Nov 2023 12:55:51 PM UTC. 2023-11-28T12:57:55.078 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-11-28T12:57:55.302 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:57:55.302 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2023-11-28T12:57:55.302 INFO:teuthology.orchestra.run.smithi111.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-28T12:57:55.302 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.302 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.303 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-11-28T12:57:55.303 INFO:teuthology.orchestra.run.smithi111.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-28T12:57:55.303 INFO:teuthology.orchestra.run.smithi111.stdout: automake-1.16.1-7.el8.noarch 2023-11-28T12:57:55.303 INFO:teuthology.orchestra.run.smithi111.stdout: bison-3.0.4-10.el8.x86_64 2023-11-28T12:57:55.303 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.303 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.303 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.303 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.303 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:55.303 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.303 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.303 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.303 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:55.304 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:55.304 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:55.304 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:55.304 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:55.304 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.304 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.304 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:55.304 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.304 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.304 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.304 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:55.304 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:55.304 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.305 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-top-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:55.305 INFO:teuthology.orchestra.run.smithi111.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-28T12:57:55.305 INFO:teuthology.orchestra.run.smithi111.stdout: flex-2.6.1-9.el8.x86_64 2023-11-28T12:57:55.305 INFO:teuthology.orchestra.run.smithi111.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-28T12:57:55.305 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-28T12:57:55.305 INFO:teuthology.orchestra.run.smithi111.stdout: jq-1.6-3.el8.x86_64 2023-11-28T12:57:55.305 INFO:teuthology.orchestra.run.smithi111.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-28T12:57:55.305 INFO:teuthology.orchestra.run.smithi111.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-28T12:57:55.305 INFO:teuthology.orchestra.run.smithi111.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-28T12:57:55.305 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.305 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.305 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.306 INFO:teuthology.orchestra.run.smithi111.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-28T12:57:55.306 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-28T12:57:55.306 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-28T12:57:55.306 INFO:teuthology.orchestra.run.smithi111.stdout: libpq-13.5-1.el8.x86_64 2023-11-28T12:57:55.306 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-28T12:57:55.306 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-28T12:57:55.306 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.306 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.306 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-28T12:57:55.306 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.306 INFO:teuthology.orchestra.run.smithi111.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-28T12:57:55.306 INFO:teuthology.orchestra.run.smithi111.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-28T12:57:55.306 INFO:teuthology.orchestra.run.smithi111.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-28T12:57:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-28T12:57:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-28T12:57:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-28T12:57:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-28T12:57:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-28T12:57:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-28T12:57:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-28T12:57:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-28T12:57:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-28T12:57:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-28T12:57:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-28T12:57:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-28T12:57:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-28T12:57:55.308 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-28T12:57:55.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-28T12:57:55.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-28T12:57:55.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-28T12:57:55.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-28T12:57:55.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-28T12:57:55.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-28T12:57:55.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-28T12:57:55.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-28T12:57:55.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-28T12:57:55.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-28T12:57:55.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-28T12:57:55.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-28T12:57:55.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-28T12:57:55.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-28T12:57:55.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-28T12:57:55.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-28T12:57:55.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-28T12:57:55.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-28T12:57:55.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-28T12:57:55.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-28T12:57:55.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-28T12:57:55.310 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-28T12:57:55.310 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-28T12:57:55.310 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-28T12:57:55.310 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-28T12:57:55.310 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-28T12:57:55.310 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-28T12:57:55.310 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-2.6-1.el8.noarch 2023-11-28T12:57:55.310 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-28T12:57:55.310 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-28T12:57:55.310 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-28T12:57:55.310 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-28T12:57:55.310 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-28T12:57:55.310 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-28T12:57:55.311 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.311 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.311 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-28T12:57:55.311 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-28T12:57:55.311 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.311 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-28T12:57:55.311 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-28T12:57:55.311 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-28T12:57:55.311 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-28T12:57:55.311 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-28T12:57:55.311 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-28T12:57:55.311 INFO:teuthology.orchestra.run.smithi111.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-28T12:57:55.311 INFO:teuthology.orchestra.run.smithi111.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-28T12:57:55.312 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-28T12:57:55.312 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-28T12:57:55.312 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-28T12:57:55.312 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-28T12:57:55.312 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-28T12:57:55.312 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.312 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.312 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:55.312 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-28T12:57:55.312 INFO:teuthology.orchestra.run.smithi111.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-28T12:57:55.312 INFO:teuthology.orchestra.run.smithi111.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-28T12:57:55.312 INFO:teuthology.orchestra.run.smithi111.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-28T12:57:55.312 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-28T12:57:55.313 INFO:teuthology.orchestra.run.smithi111.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-28T12:57:55.313 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:57:55.313 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-11-28T12:57:55.498 DEBUG:teuthology.parallel:result is None 2023-11-28T12:57:56.238 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2023-11-28T12:57:56.457 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:57:56.457 INFO:teuthology.orchestra.run.smithi149.stdout:Upgraded: 2023-11-28T12:57:56.458 INFO:teuthology.orchestra.run.smithi149.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-28T12:57:56.458 INFO:teuthology.orchestra.run.smithi149.stdout: librados2-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.458 INFO:teuthology.orchestra.run.smithi149.stdout: librbd1-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.458 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2023-11-28T12:57:56.458 INFO:teuthology.orchestra.run.smithi149.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-28T12:57:56.458 INFO:teuthology.orchestra.run.smithi149.stdout: automake-1.16.1-7.el8.noarch 2023-11-28T12:57:56.458 INFO:teuthology.orchestra.run.smithi149.stdout: bison-3.0.4-10.el8.x86_64 2023-11-28T12:57:56.458 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.458 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-base-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.458 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-common-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.458 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-fuse-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.458 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-grafana-dashboards-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:56.459 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-immutable-object-cache-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.459 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mds-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.459 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.459 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-cephadm-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:56.459 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-dashboard-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:56.459 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:56.459 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-modules-core-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:56.459 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-rook-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:56.459 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mon-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.459 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-osd-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.459 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-prometheus-alerts-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:56.459 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-radosgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.459 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-selinux-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.460 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-test-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.460 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-volume-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:56.460 INFO:teuthology.orchestra.run.smithi149.stdout: cephadm-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:56.460 INFO:teuthology.orchestra.run.smithi149.stdout: cephfs-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.460 INFO:teuthology.orchestra.run.smithi149.stdout: cephfs-top-2:18.0.0-2697.g2f5f6028.el8.noarch 2023-11-28T12:57:56.460 INFO:teuthology.orchestra.run.smithi149.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-28T12:57:56.460 INFO:teuthology.orchestra.run.smithi149.stdout: flex-2.6.1-9.el8.x86_64 2023-11-28T12:57:56.460 INFO:teuthology.orchestra.run.smithi149.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-28T12:57:56.460 INFO:teuthology.orchestra.run.smithi149.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-28T12:57:56.460 INFO:teuthology.orchestra.run.smithi149.stdout: jq-1.6-3.el8.x86_64 2023-11-28T12:57:56.460 INFO:teuthology.orchestra.run.smithi149.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-28T12:57:56.460 INFO:teuthology.orchestra.run.smithi149.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-28T12:57:56.461 INFO:teuthology.orchestra.run.smithi149.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-28T12:57:56.461 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs-devel-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.461 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs2-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.461 INFO:teuthology.orchestra.run.smithi149.stdout: libcephsqlite-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.461 INFO:teuthology.orchestra.run.smithi149.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-28T12:57:56.461 INFO:teuthology.orchestra.run.smithi149.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-28T12:57:56.461 INFO:teuthology.orchestra.run.smithi149.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-28T12:57:56.461 INFO:teuthology.orchestra.run.smithi149.stdout: libpq-13.5-1.el8.x86_64 2023-11-28T12:57:56.461 INFO:teuthology.orchestra.run.smithi149.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-28T12:57:56.461 INFO:teuthology.orchestra.run.smithi149.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-28T12:57:56.461 INFO:teuthology.orchestra.run.smithi149.stdout: librados-devel-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.461 INFO:teuthology.orchestra.run.smithi149.stdout: libradosstriper1-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.461 INFO:teuthology.orchestra.run.smithi149.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-28T12:57:56.462 INFO:teuthology.orchestra.run.smithi149.stdout: librgw2-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.462 INFO:teuthology.orchestra.run.smithi149.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-28T12:57:56.462 INFO:teuthology.orchestra.run.smithi149.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-28T12:57:56.462 INFO:teuthology.orchestra.run.smithi149.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-28T12:57:56.462 INFO:teuthology.orchestra.run.smithi149.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-28T12:57:56.462 INFO:teuthology.orchestra.run.smithi149.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-28T12:57:56.462 INFO:teuthology.orchestra.run.smithi149.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-28T12:57:56.462 INFO:teuthology.orchestra.run.smithi149.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-28T12:57:56.462 INFO:teuthology.orchestra.run.smithi149.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-28T12:57:56.462 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-28T12:57:56.462 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-28T12:57:56.462 INFO:teuthology.orchestra.run.smithi149.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-28T12:57:56.463 INFO:teuthology.orchestra.run.smithi149.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-28T12:57:56.463 INFO:teuthology.orchestra.run.smithi149.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-28T12:57:56.463 INFO:teuthology.orchestra.run.smithi149.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-28T12:57:56.463 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-28T12:57:56.463 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-28T12:57:56.464 INFO:teuthology.orchestra.run.smithi149.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-28T12:57:56.464 INFO:teuthology.orchestra.run.smithi149.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-28T12:57:56.464 INFO:teuthology.orchestra.run.smithi149.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-28T12:57:56.464 INFO:teuthology.orchestra.run.smithi149.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-28T12:57:56.464 INFO:teuthology.orchestra.run.smithi149.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-28T12:57:56.464 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-28T12:57:56.464 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-argparse-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.464 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-common-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.464 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cephfs-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.464 INFO:teuthology.orchestra.run.smithi149.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-28T12:57:56.464 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-28T12:57:56.464 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-28T12:57:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-28T12:57:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-28T12:57:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-28T12:57:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-28T12:57:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-28T12:57:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-28T12:57:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-28T12:57:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-28T12:57:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-28T12:57:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-28T12:57:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-28T12:57:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-28T12:57:56.465 INFO:teuthology.orchestra.run.smithi149.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-28T12:57:56.466 INFO:teuthology.orchestra.run.smithi149.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-28T12:57:56.466 INFO:teuthology.orchestra.run.smithi149.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-28T12:57:56.466 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-28T12:57:56.466 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-28T12:57:56.466 INFO:teuthology.orchestra.run.smithi149.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-28T12:57:56.466 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-28T12:57:56.466 INFO:teuthology.orchestra.run.smithi149.stdout: python3-portend-2.6-1.el8.noarch 2023-11-28T12:57:56.466 INFO:teuthology.orchestra.run.smithi149.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-28T12:57:56.466 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-28T12:57:56.466 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-28T12:57:56.466 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-28T12:57:56.466 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-28T12:57:56.467 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-28T12:57:56.467 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rados-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.467 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rbd-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.467 INFO:teuthology.orchestra.run.smithi149.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-28T12:57:56.467 INFO:teuthology.orchestra.run.smithi149.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-28T12:57:56.467 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rgw-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.467 INFO:teuthology.orchestra.run.smithi149.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-28T12:57:56.467 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-28T12:57:56.467 INFO:teuthology.orchestra.run.smithi149.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-28T12:57:56.467 INFO:teuthology.orchestra.run.smithi149.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-28T12:57:56.467 INFO:teuthology.orchestra.run.smithi149.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-28T12:57:56.467 INFO:teuthology.orchestra.run.smithi149.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-28T12:57:56.467 INFO:teuthology.orchestra.run.smithi149.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-28T12:57:56.468 INFO:teuthology.orchestra.run.smithi149.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-28T12:57:56.468 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-28T12:57:56.468 INFO:teuthology.orchestra.run.smithi149.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-28T12:57:56.468 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-28T12:57:56.468 INFO:teuthology.orchestra.run.smithi149.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-28T12:57:56.468 INFO:teuthology.orchestra.run.smithi149.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-28T12:57:56.468 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-fuse-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.468 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-mirror-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.468 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-nbd-2:18.0.0-2697.g2f5f6028.el8.x86_64 2023-11-28T12:57:56.468 INFO:teuthology.orchestra.run.smithi149.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-28T12:57:56.468 INFO:teuthology.orchestra.run.smithi149.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-28T12:57:56.468 INFO:teuthology.orchestra.run.smithi149.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-28T12:57:56.468 INFO:teuthology.orchestra.run.smithi149.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-28T12:57:56.469 INFO:teuthology.orchestra.run.smithi149.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-28T12:57:56.469 INFO:teuthology.orchestra.run.smithi149.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-28T12:57:56.469 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:57:56.469 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2023-11-28T12:57:56.642 DEBUG:teuthology.parallel:result is None 2023-11-28T12:57:56.643 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2f5f6028b893ad41357cd06e827373f0fe2a1232 2023-11-28T12:57:56.785 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-28T12:57:56.813 INFO:teuthology.orchestra.run.smithi111.stdout:18.0.0-2697.g2f5f6028.el8 2023-11-28T12:57:56.814 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2697.g2f5f6028.el8 2023-11-28T12:57:56.814 INFO:teuthology.task.install:The correct ceph version 18.0.0-2697.g2f5f6028 is installed. 2023-11-28T12:57:56.814 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2f5f6028b893ad41357cd06e827373f0fe2a1232 2023-11-28T12:57:57.031 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-28T12:57:57.056 INFO:teuthology.orchestra.run.smithi149.stdout:18.0.0-2697.g2f5f6028.el8 2023-11-28T12:57:57.056 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2697.g2f5f6028.el8 2023-11-28T12:57:57.056 INFO:teuthology.task.install:The correct ceph version 18.0.0-2697.g2f5f6028 is installed. 2023-11-28T12:57:57.056 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-28T12:57:57.057 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:57:57.057 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T12:57:57.105 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:57:57.105 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T12:57:57.149 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-28T12:57:57.149 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:57:57.149 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T12:57:57.179 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T12:57:57.249 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:57:57.249 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T12:57:57.277 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T12:57:57.345 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-28T12:57:57.346 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:57:57.346 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T12:57:57.374 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T12:57:57.441 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:57:57.441 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T12:57:57.469 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T12:57:57.536 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-28T12:57:57.537 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:57:57.537 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T12:57:57.565 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T12:57:57.632 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:57:57.632 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T12:57:57.659 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T12:57:57.726 INFO:teuthology.run_tasks:Running task ceph... 2023-11-28T12:57:57.791 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-28T12:57:57.791 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /var/log/ceph 2023-11-28T12:57:57.793 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 777 /var/log/ceph 2023-11-28T12:57:57.819 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-28T12:57:57.819 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-28T12:57:57.860 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-28T12:57:57.886 INFO:tasks.ceph:Creating extra log directories... 2023-11-28T12:57:57.887 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-28T12:57:57.927 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-28T12:57:57.996 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-11-28T12:57:57.996 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-11-28T12:57:57.996 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-11-28T12:57:57.997 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:57:57.997 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-28T12:57:58.013 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-28T12:57:58.144 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-28T12:57:58.170 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:57:58.170 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-28T12:57:58.187 DEBUG:teuthology.orchestra.run.smithi149:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-28T12:57:58.319 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-28T12:57:58.351 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-28T12:57:58.352 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-28T12:57:58.352 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969', 'branch': 'wip-neesingh-testing-231123', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}', 'email': 'neesingh@redhat.com', 'first_in_suite': False, 'job_id': '7469969', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-neesingh-testing-231123'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': '2f5f6028b893ad41357cd06e827373f0fe2a1232'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '2f5f6028b893ad41357cd06e827373f0fe2a1232'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'issue_54557', 'sha1': '3ae99413970476692dfab1fe38dadb056700a61d'}}, 'owner': 'scheduled_neesingh@teuthology', 'priority': 10, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 6754, 'sha1': '2f5f6028b893ad41357cd06e827373f0fe2a1232', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'issue_54557', 'suite_path': '/home/teuthworker/src/github.com_neesingh-rh_ceph_3ae99413970476692dfab1fe38dadb056700a61d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/neesingh-rh/ceph', 'suite_sha1': '3ae99413970476692dfab1fe38dadb056700a61d', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi111.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=', 'smithi149.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '2f5f6028b893ad41357cd06e827373f0fe2a1232'}}, {'ceph': None}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_forward_scrub']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '5ae09a5cc7064ca2bf994df19f3006b79794789a', 'timestamp': '2023-11-28_12:38:30', 'tube': 'smithi', 'user': 'neesingh', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-28T12:57:58.352 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-28T12:57:58.355 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-28T12:57:58.403 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/run/ceph 2023-11-28T12:57:58.412 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m0777 -- /var/run/ceph 2023-11-28T12:57:58.470 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:57:58.470 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T12:57:58.494 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T12:57:58.494 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2023-11-28T12:57:58.554 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T12:57:58.554 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T12:57:58.554 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 368631 Links: 1 2023-11-28T12:57:58.554 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T12:57:58.554 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T12:57:58.554 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-11-28 12:57:43.627011227 +0000 2023-11-28T12:57:58.555 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-11-28 12:54:48.781455191 +0000 2023-11-28T12:57:58.555 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-11-28 12:54:48.781455191 +0000 2023-11-28T12:57:58.555 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-11-28T12:57:58.555 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T12:57:58.619 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-11-28T12:57:58.619 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-11-28T12:57:58.619 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000249886 s, 2.0 MB/s 2023-11-28T12:57:58.620 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T12:57:58.678 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2023-11-28T12:57:58.735 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T12:57:58.735 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T12:57:58.735 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 390463 Links: 1 2023-11-28T12:57:58.736 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T12:57:58.736 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T12:57:58.736 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-11-28 12:57:43.627011227 +0000 2023-11-28T12:57:58.736 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-11-28 12:54:49.300446046 +0000 2023-11-28T12:57:58.736 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-11-28 12:54:49.300446046 +0000 2023-11-28T12:57:58.737 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-11-28T12:57:58.737 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T12:57:58.800 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-11-28T12:57:58.800 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-11-28T12:57:58.800 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000212569 s, 2.4 MB/s 2023-11-28T12:57:58.802 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T12:57:58.859 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2023-11-28T12:57:58.916 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T12:57:58.917 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T12:57:58.917 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 390543 Links: 1 2023-11-28T12:57:58.917 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T12:57:58.917 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T12:57:58.917 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-11-28 12:57:43.627011227 +0000 2023-11-28T12:57:58.917 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-11-28 12:54:49.829436724 +0000 2023-11-28T12:57:58.917 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-11-28 12:54:49.829436724 +0000 2023-11-28T12:57:58.917 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-11-28T12:57:58.918 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T12:57:58.982 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-11-28T12:57:58.983 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-11-28T12:57:58.983 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000215636 s, 2.4 MB/s 2023-11-28T12:57:58.984 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T12:57:59.043 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2023-11-28T12:57:59.100 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T12:57:59.100 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T12:57:59.100 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 394468 Links: 1 2023-11-28T12:57:59.100 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T12:57:59.100 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T12:57:59.100 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-11-28 12:57:43.627011227 +0000 2023-11-28T12:57:59.100 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-11-28 12:54:50.338427755 +0000 2023-11-28T12:57:59.101 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-11-28 12:54:50.338427755 +0000 2023-11-28T12:57:59.101 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2023-11-28T12:57:59.101 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T12:57:59.165 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-11-28T12:57:59.165 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-11-28T12:57:59.165 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000179986 s, 2.8 MB/s 2023-11-28T12:57:59.166 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T12:57:59.223 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T12:57:59.224 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:57:59.224 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T12:57:59.244 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T12:57:59.244 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_1 2023-11-28T12:57:59.305 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T12:57:59.305 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T12:57:59.306 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 386865 Links: 1 2023-11-28T12:57:59.306 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T12:57:59.306 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T12:57:59.306 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2023-11-28 12:57:43.041900426 +0000 2023-11-28T12:57:59.306 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2023-11-28 12:55:03.152519802 +0000 2023-11-28T12:57:59.306 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2023-11-28 12:55:03.152519802 +0000 2023-11-28T12:57:59.306 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2023-11-28T12:57:59.307 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T12:57:59.375 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2023-11-28T12:57:59.375 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2023-11-28T12:57:59.375 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000269501 s, 1.9 MB/s 2023-11-28T12:57:59.376 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T12:57:59.440 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_2 2023-11-28T12:57:59.500 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T12:57:59.548 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T12:57:59.548 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 391478 Links: 1 2023-11-28T12:57:59.549 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T12:57:59.549 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T12:57:59.549 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2023-11-28 12:57:43.041900426 +0000 2023-11-28T12:57:59.549 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2023-11-28 12:55:03.659509743 +0000 2023-11-28T12:57:59.549 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2023-11-28 12:55:03.659509743 +0000 2023-11-28T12:57:59.549 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2023-11-28T12:57:59.550 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T12:57:59.623 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2023-11-28T12:57:59.623 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2023-11-28T12:57:59.623 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000267668 s, 1.9 MB/s 2023-11-28T12:57:59.624 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T12:57:59.684 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_3 2023-11-28T12:57:59.743 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T12:57:59.743 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T12:57:59.743 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 392501 Links: 1 2023-11-28T12:57:59.743 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T12:57:59.744 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T12:57:59.744 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2023-11-28 12:57:43.041900426 +0000 2023-11-28T12:57:59.744 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2023-11-28 12:55:04.173499545 +0000 2023-11-28T12:57:59.744 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2023-11-28 12:55:04.173499545 +0000 2023-11-28T12:57:59.744 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2023-11-28T12:57:59.744 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T12:57:59.812 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2023-11-28T12:57:59.812 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2023-11-28T12:57:59.812 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000230575 s, 2.2 MB/s 2023-11-28T12:57:59.813 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T12:57:59.873 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_4 2023-11-28T12:57:59.930 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T12:57:59.931 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T12:57:59.931 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 391721 Links: 1 2023-11-28T12:57:59.931 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T12:57:59.931 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T12:57:59.931 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2023-11-28 12:57:43.041900426 +0000 2023-11-28T12:57:59.931 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2023-11-28 12:55:04.682489447 +0000 2023-11-28T12:57:59.932 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2023-11-28 12:55:04.682489447 +0000 2023-11-28T12:57:59.932 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2023-11-28T12:57:59.932 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T12:57:59.997 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2023-11-28T12:57:59.997 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2023-11-28T12:57:59.998 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000205896 s, 2.5 MB/s 2023-11-28T12:57:59.999 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T12:58:00.057 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T12:58:00.057 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi149.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-28T12:58:00.057 INFO:tasks.ceph:Generating config... 2023-11-28T12:58:00.058 INFO:tasks.ceph:[client] client mount timeout = 600 2023-11-28T12:58:00.059 INFO:tasks.ceph:[client] debug client = 20 2023-11-28T12:58:00.059 INFO:tasks.ceph:[client] debug ms = 1 2023-11-28T12:58:00.059 INFO:tasks.ceph:[client] pid file = 2023-11-28T12:58:00.059 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-11-28T12:58:00.059 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-11-28T12:58:00.059 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-11-28T12:58:00.059 INFO:tasks.ceph:[mds] debug mds = 20 2023-11-28T12:58:00.059 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-11-28T12:58:00.059 INFO:tasks.ceph:[mds] debug ms = 1 2023-11-28T12:58:00.059 INFO:tasks.ceph:[mds] mds debug frag = True 2023-11-28T12:58:00.059 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-11-28T12:58:00.060 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-11-28T12:58:00.060 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-11-28T12:58:00.060 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-11-28T12:58:00.060 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-11-28T12:58:00.060 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-11-28T12:58:00.060 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-28T12:58:00.060 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-28T12:58:00.060 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-28T12:58:00.060 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-28T12:58:00.061 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-28T12:58:00.061 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-11-28T12:58:00.061 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-28T12:58:00.061 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-28T12:58:00.061 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-28T12:58:00.061 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-28T12:58:00.061 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-28T12:58:00.061 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-28T12:58:00.061 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-28T12:58:00.061 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-28T12:58:00.061 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-28T12:58:00.062 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-28T12:58:00.062 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-28T12:58:00.062 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-28T12:58:00.062 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-11-28T12:58:00.063 INFO:tasks.ceph:Setting up mon.a... 2023-11-28T12:58:00.063 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-28T12:58:00.127 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/ceph.keyring 2023-11-28T12:58:00.131 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-28T12:58:00.179 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T12:58:00.206 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.111'), ('mon.b', '172.21.15.149'), ('mon.c', '[v2:172.21.15.149:3301,v1:172.21.15.149:6790]')] 2023-11-28T12:58:00.206 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.111,172.21.15.149,[v2:172.21.15.149:3301,v1:172.21.15.149:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-11-28T12:58:00.207 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:00.207 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T12:58:00.263 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.111 --add b 172.21.15.149 --addv c '[v2:172.21.15.149:3301,v1:172.21.15.149:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:00.350 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:00.350 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: generated fsid a1b2d00c-aeec-4b5d-93e3-41820c4189ba 2023-11-28T12:58:00.351 INFO:teuthology.orchestra.run.smithi111.stdout:setting min_mon_release = pacific 2023-11-28T12:58:00.351 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 0 2023-11-28T12:58:00.351 INFO:teuthology.orchestra.run.smithi111.stdout:fsid a1b2d00c-aeec-4b5d-93e3-41820c4189ba 2023-11-28T12:58:00.351 INFO:teuthology.orchestra.run.smithi111.stdout:last_changed 2023-11-28T12:58:00.350556+0000 2023-11-28T12:58:00.351 INFO:teuthology.orchestra.run.smithi111.stdout:created 2023-11-28T12:58:00.350556+0000 2023-11-28T12:58:00.351 INFO:teuthology.orchestra.run.smithi111.stdout:min_mon_release 16 (pacific) 2023-11-28T12:58:00.351 INFO:teuthology.orchestra.run.smithi111.stdout:election_strategy: 1 2023-11-28T12:58:00.351 INFO:teuthology.orchestra.run.smithi111.stdout:0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.a 2023-11-28T12:58:00.351 INFO:teuthology.orchestra.run.smithi111.stdout:1: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2023-11-28T12:58:00.351 INFO:teuthology.orchestra.run.smithi111.stdout:2: [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] mon.c 2023-11-28T12:58:00.351 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-28T12:58:00.353 DEBUG:teuthology.orchestra.run.smithi111:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T12:58:00.369 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a1b2d00c-aeec-4b5d-93e3-41820c4189ba... 2023-11-28T12:58:00.370 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-28T12:58:00.412 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-28T12:58:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2023-11-28T12:58:00.455 INFO:teuthology.orchestra.run.smithi111.stdout: chdir = "" 2023-11-28T12:58:00.455 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-28T12:58:00.455 INFO:teuthology.orchestra.run.smithi111.stdout: auth supported = cephx 2023-11-28T12:58:00.455 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.455 INFO:teuthology.orchestra.run.smithi111.stdout: filestore xattr use omap = true 2023-11-28T12:58:00.455 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.455 INFO:teuthology.orchestra.run.smithi111.stdout: mon clock drift allowed = 1.000 2023-11-28T12:58:00.455 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.455 INFO:teuthology.orchestra.run.smithi111.stdout: osd crush chooseleaf type = 0 2023-11-28T12:58:00.456 INFO:teuthology.orchestra.run.smithi111.stdout: auth debug = true 2023-11-28T12:58:00.456 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.456 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on old message = true 2023-11-28T12:58:00.456 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on bug = true 2023-11-28T12:58:00.456 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.456 INFO:teuthology.orchestra.run.smithi111.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-28T12:58:00.456 INFO:teuthology.orchestra.run.smithi111.stdout: mon pg warn max object skew = 0 2023-11-28T12:58:00.456 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.456 INFO:teuthology.orchestra.run.smithi111.stdout: # disable pg_autoscaler by default for new pools 2023-11-28T12:58:00.456 INFO:teuthology.orchestra.run.smithi111.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-28T12:58:00.456 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.456 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default size = 2 2023-11-28T12:58:00.457 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.457 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow primary affinity = true 2023-11-28T12:58:00.457 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow pg remap = true 2023-11-28T12:58:00.457 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on legacy crush tunables = false 2023-11-28T12:58:00.457 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on crush straw calc version zero = false 2023-11-28T12:58:00.457 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on no sortbitwise = false 2023-11-28T12:58:00.457 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on osd down out interval zero = false 2023-11-28T12:58:00.457 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on too few osds = false 2023-11-28T12:58:00.457 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-28T12:58:00.457 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-28T12:58:00.457 INFO:teuthology.orchestra.run.smithi111.stdout: mon_allow_pool_size_one = true 2023-11-28T12:58:00.457 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.458 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-28T12:58:00.458 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.458 INFO:teuthology.orchestra.run.smithi111.stdout: osd default data pool replay window = 5 2023-11-28T12:58:00.458 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.458 INFO:teuthology.orchestra.run.smithi111.stdout: mon allow pool delete = true 2023-11-28T12:58:00.458 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.458 INFO:teuthology.orchestra.run.smithi111.stdout: mon cluster log file level = debug 2023-11-28T12:58:00.458 INFO:teuthology.orchestra.run.smithi111.stdout: debug asserts on shutdown = true 2023-11-28T12:58:00.458 INFO:teuthology.orchestra.run.smithi111.stdout: mon health detail to clog = false 2023-11-28T12:58:00.458 INFO:teuthology.orchestra.run.smithi111.stdout: mon host = "172.21.15.111,172.21.15.149,[v2:172.21.15.149:3301,v1:172.21.15.149:6790]" 2023-11-28T12:58:00.458 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on skipped message = False 2023-11-28T12:58:00.458 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = a1b2d00c-aeec-4b5d-93e3-41820c4189ba 2023-11-28T12:58:00.458 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.459 INFO:teuthology.orchestra.run.smithi111.stdout:[osd] 2023-11-28T12:58:00.459 INFO:teuthology.orchestra.run.smithi111.stdout: osd journal size = 100 2023-11-28T12:58:00.459 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.459 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub load threshold = 5.0 2023-11-28T12:58:00.459 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub max interval = 600 2023-11-28T12:58:00.459 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.459 INFO:teuthology.orchestra.run.smithi111.stdout: osd recover clone overlap = true 2023-11-28T12:58:00.459 INFO:teuthology.orchestra.run.smithi111.stdout: osd recovery max chunk = 1048576 2023-11-28T12:58:00.459 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.459 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug shutdown = true 2023-11-28T12:58:00.459 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug op order = true 2023-11-28T12:58:00.459 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify stray on activate = true 2023-11-28T12:58:00.459 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.460 INFO:teuthology.orchestra.run.smithi111.stdout: osd open classes on start = true 2023-11-28T12:58:00.460 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug pg log writeout = true 2023-11-28T12:58:00.460 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.460 INFO:teuthology.orchestra.run.smithi111.stdout: osd deep scrub update digest min age = 30 2023-11-28T12:58:00.460 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.460 INFO:teuthology.orchestra.run.smithi111.stdout: osd map max advance = 10 2023-11-28T12:58:00.460 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.460 INFO:teuthology.orchestra.run.smithi111.stdout: journal zero on create = true 2023-11-28T12:58:00.460 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.461 INFO:teuthology.orchestra.run.smithi111.stdout: filestore ondisk finisher threads = 3 2023-11-28T12:58:00.461 INFO:teuthology.orchestra.run.smithi111.stdout: filestore apply finisher threads = 3 2023-11-28T12:58:00.461 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.461 INFO:teuthology.orchestra.run.smithi111.stdout: bdev debug aio = true 2023-11-28T12:58:00.461 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug misdirected ops = true 2023-11-28T12:58:00.461 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore block size = 96636764160 2023-11-28T12:58:00.461 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore fsck on mount = True 2023-11-28T12:58:00.461 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluefs = 20 2023-11-28T12:58:00.461 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluestore = 20 2023-11-28T12:58:00.461 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-11-28T12:58:00.461 INFO:teuthology.orchestra.run.smithi111.stdout: debug osd = 20 2023-11-28T12:58:00.461 INFO:teuthology.orchestra.run.smithi111.stdout: debug rocksdb = 10 2023-11-28T12:58:00.462 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd backfillfull_ratio = 0.85 2023-11-28T12:58:00.462 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd full ratio = 0.9 2023-11-28T12:58:00.462 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd nearfull ratio = 0.8 2023-11-28T12:58:00.462 INFO:teuthology.orchestra.run.smithi111.stdout: osd failsafe full ratio = 0.95 2023-11-28T12:58:00.462 INFO:teuthology.orchestra.run.smithi111.stdout: osd objectstore = bluestore 2023-11-28T12:58:00.462 INFO:teuthology.orchestra.run.smithi111.stdout: osd op complaint time = 180 2023-11-28T12:58:00.462 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.462 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr] 2023-11-28T12:58:00.462 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-11-28T12:58:00.462 INFO:teuthology.orchestra.run.smithi111.stdout: debug mgr = 20 2023-11-28T12:58:00.462 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2023-11-28T12:58:00.462 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2023-11-28T12:58:00.463 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2023-11-28T12:58:00.463 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2023-11-28T12:58:00.463 INFO:teuthology.orchestra.run.smithi111.stdout: mgr/telemetry/nag = false 2023-11-28T12:58:00.463 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.463 INFO:teuthology.orchestra.run.smithi111.stdout:[mon] 2023-11-28T12:58:00.463 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-11-28T12:58:00.463 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2023-11-28T12:58:00.463 INFO:teuthology.orchestra.run.smithi111.stdout: debug paxos = 20 2023-11-28T12:58:00.463 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2023-11-28T12:58:00.463 INFO:teuthology.orchestra.run.smithi111.stdout: mon data avail warn = 5 2023-11-28T12:58:00.463 INFO:teuthology.orchestra.run.smithi111.stdout: mon mgr mkfs grace = 240 2023-11-28T12:58:00.463 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2023-11-28T12:58:00.464 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd reporter subtree level = osd 2023-11-28T12:58:00.464 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd prime pg temp = true 2023-11-28T12:58:00.464 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2023-11-28T12:58:00.464 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.464 INFO:teuthology.orchestra.run.smithi111.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-28T12:58:00.464 INFO:teuthology.orchestra.run.smithi111.stdout: auth mon ticket ttl = 660 # 11m 2023-11-28T12:58:00.464 INFO:teuthology.orchestra.run.smithi111.stdout: auth service ticket ttl = 240 # 4m 2023-11-28T12:58:00.464 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.464 INFO:teuthology.orchestra.run.smithi111.stdout: # don't complain about insecure global_id in the test suite 2023-11-28T12:58:00.464 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-28T12:58:00.464 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-28T12:58:00.465 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.465 INFO:teuthology.orchestra.run.smithi111.stdout: # 1m isn't quite enough 2023-11-28T12:58:00.465 INFO:teuthology.orchestra.run.smithi111.stdout: mon_down_mkfs_grace = 2m 2023-11-28T12:58:00.465 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.465 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_filestore_osds = false 2023-11-28T12:58:00.465 INFO:teuthology.orchestra.run.smithi111.stdout: mon op complaint time = 120 2023-11-28T12:58:00.465 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:00.465 INFO:teuthology.orchestra.run.smithi111.stdout:[client] 2023-11-28T12:58:00.465 INFO:teuthology.orchestra.run.smithi111.stdout: rgw cache enabled = true 2023-11-28T12:58:00.465 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable ops log = true 2023-11-28T12:58:00.465 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable usage log = true 2023-11-28T12:58:00.465 INFO:teuthology.orchestra.run.smithi111.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-28T12:58:00.465 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-28T12:58:00.466 INFO:teuthology.orchestra.run.smithi111.stdout: client mount timeout = 600 2023-11-28T12:58:00.466 INFO:teuthology.orchestra.run.smithi111.stdout: debug client = 20 2023-11-28T12:58:00.466 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-11-28T12:58:00.466 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = "" 2023-11-28T12:58:00.466 INFO:teuthology.orchestra.run.smithi111.stdout: rados mon op timeout = 900 2023-11-28T12:58:00.466 INFO:teuthology.orchestra.run.smithi111.stdout: rados osd op timeout = 900 2023-11-28T12:58:00.466 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.a] 2023-11-28T12:58:00.466 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.b] 2023-11-28T12:58:00.466 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.c] 2023-11-28T12:58:00.466 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.a] 2023-11-28T12:58:00.466 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.c] 2023-11-28T12:58:00.466 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.b] 2023-11-28T12:58:00.466 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.d] 2023-11-28T12:58:00.467 INFO:teuthology.orchestra.run.smithi111.stdout:[mds] 2023-11-28T12:58:00.467 INFO:teuthology.orchestra.run.smithi111.stdout: debug mds = 20 2023-11-28T12:58:00.467 INFO:teuthology.orchestra.run.smithi111.stdout: debug mds balancer = 20 2023-11-28T12:58:00.467 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2023-11-28T12:58:00.467 INFO:teuthology.orchestra.run.smithi111.stdout: mds debug frag = True 2023-11-28T12:58:00.467 INFO:teuthology.orchestra.run.smithi111.stdout: mds debug scatterstat = True 2023-11-28T12:58:00.467 INFO:teuthology.orchestra.run.smithi111.stdout: mds op complaint time = 180 2023-11-28T12:58:00.467 INFO:teuthology.orchestra.run.smithi111.stdout: mds verify scatter = True 2023-11-28T12:58:00.468 INFO:teuthology.orchestra.run.smithi111.stdout: osd op complaint time = 180 2023-11-28T12:58:00.468 INFO:teuthology.orchestra.run.smithi111.stdout: rados mon op timeout = 900 2023-11-28T12:58:00.468 INFO:teuthology.orchestra.run.smithi111.stdout: rados osd op timeout = 900 2023-11-28T12:58:00.468 INFO:teuthology.orchestra.run.smithi149.stdout:[global] 2023-11-28T12:58:00.469 INFO:teuthology.orchestra.run.smithi149.stdout: chdir = "" 2023-11-28T12:58:00.469 INFO:teuthology.orchestra.run.smithi149.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-28T12:58:00.469 INFO:teuthology.orchestra.run.smithi149.stdout: auth supported = cephx 2023-11-28T12:58:00.469 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.469 INFO:teuthology.orchestra.run.smithi149.stdout: filestore xattr use omap = true 2023-11-28T12:58:00.469 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.469 INFO:teuthology.orchestra.run.smithi149.stdout: mon clock drift allowed = 1.000 2023-11-28T12:58:00.469 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.469 INFO:teuthology.orchestra.run.smithi149.stdout: osd crush chooseleaf type = 0 2023-11-28T12:58:00.469 INFO:teuthology.orchestra.run.smithi149.stdout: auth debug = true 2023-11-28T12:58:00.469 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.470 INFO:teuthology.orchestra.run.smithi149.stdout: ms die on old message = true 2023-11-28T12:58:00.470 INFO:teuthology.orchestra.run.smithi149.stdout: ms die on bug = true 2023-11-28T12:58:00.470 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.470 INFO:teuthology.orchestra.run.smithi149.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-28T12:58:00.470 INFO:teuthology.orchestra.run.smithi149.stdout: mon pg warn max object skew = 0 2023-11-28T12:58:00.470 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.470 INFO:teuthology.orchestra.run.smithi149.stdout: # disable pg_autoscaler by default for new pools 2023-11-28T12:58:00.470 INFO:teuthology.orchestra.run.smithi149.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-28T12:58:00.470 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.471 INFO:teuthology.orchestra.run.smithi149.stdout: osd pool default size = 2 2023-11-28T12:58:00.471 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.471 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd allow primary affinity = true 2023-11-28T12:58:00.471 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd allow pg remap = true 2023-11-28T12:58:00.471 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on legacy crush tunables = false 2023-11-28T12:58:00.471 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on crush straw calc version zero = false 2023-11-28T12:58:00.471 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on no sortbitwise = false 2023-11-28T12:58:00.471 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on osd down out interval zero = false 2023-11-28T12:58:00.471 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on too few osds = false 2023-11-28T12:58:00.471 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-28T12:58:00.472 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-28T12:58:00.472 INFO:teuthology.orchestra.run.smithi149.stdout: mon_allow_pool_size_one = true 2023-11-28T12:58:00.472 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.472 INFO:teuthology.orchestra.run.smithi149.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-28T12:58:00.472 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.472 INFO:teuthology.orchestra.run.smithi149.stdout: osd default data pool replay window = 5 2023-11-28T12:58:00.472 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.472 INFO:teuthology.orchestra.run.smithi149.stdout: mon allow pool delete = true 2023-11-28T12:58:00.472 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.473 INFO:teuthology.orchestra.run.smithi149.stdout: mon cluster log file level = debug 2023-11-28T12:58:00.473 INFO:teuthology.orchestra.run.smithi149.stdout: debug asserts on shutdown = true 2023-11-28T12:58:00.473 INFO:teuthology.orchestra.run.smithi149.stdout: mon health detail to clog = false 2023-11-28T12:58:00.473 INFO:teuthology.orchestra.run.smithi149.stdout: mon host = "172.21.15.111,172.21.15.149,[v2:172.21.15.149:3301,v1:172.21.15.149:6790]" 2023-11-28T12:58:00.473 INFO:teuthology.orchestra.run.smithi149.stdout: ms die on skipped message = False 2023-11-28T12:58:00.473 INFO:teuthology.orchestra.run.smithi149.stdout: fsid = a1b2d00c-aeec-4b5d-93e3-41820c4189ba 2023-11-28T12:58:00.473 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.473 INFO:teuthology.orchestra.run.smithi149.stdout:[osd] 2023-11-28T12:58:00.473 INFO:teuthology.orchestra.run.smithi149.stdout: osd journal size = 100 2023-11-28T12:58:00.473 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.473 INFO:teuthology.orchestra.run.smithi149.stdout: osd scrub load threshold = 5.0 2023-11-28T12:58:00.474 INFO:teuthology.orchestra.run.smithi149.stdout: osd scrub max interval = 600 2023-11-28T12:58:00.474 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.474 INFO:teuthology.orchestra.run.smithi149.stdout: osd recover clone overlap = true 2023-11-28T12:58:00.474 INFO:teuthology.orchestra.run.smithi149.stdout: osd recovery max chunk = 1048576 2023-11-28T12:58:00.474 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.474 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug shutdown = true 2023-11-28T12:58:00.474 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug op order = true 2023-11-28T12:58:00.474 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug verify stray on activate = true 2023-11-28T12:58:00.474 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.474 INFO:teuthology.orchestra.run.smithi149.stdout: osd open classes on start = true 2023-11-28T12:58:00.474 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug pg log writeout = true 2023-11-28T12:58:00.475 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.475 INFO:teuthology.orchestra.run.smithi149.stdout: osd deep scrub update digest min age = 30 2023-11-28T12:58:00.475 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.475 INFO:teuthology.orchestra.run.smithi149.stdout: osd map max advance = 10 2023-11-28T12:58:00.475 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.475 INFO:teuthology.orchestra.run.smithi149.stdout: journal zero on create = true 2023-11-28T12:58:00.475 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.475 INFO:teuthology.orchestra.run.smithi149.stdout: filestore ondisk finisher threads = 3 2023-11-28T12:58:00.475 INFO:teuthology.orchestra.run.smithi149.stdout: filestore apply finisher threads = 3 2023-11-28T12:58:00.475 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.475 INFO:teuthology.orchestra.run.smithi149.stdout: bdev debug aio = true 2023-11-28T12:58:00.475 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug misdirected ops = true 2023-11-28T12:58:00.476 INFO:teuthology.orchestra.run.smithi149.stdout: bluestore block size = 96636764160 2023-11-28T12:58:00.476 INFO:teuthology.orchestra.run.smithi149.stdout: bluestore fsck on mount = True 2023-11-28T12:58:00.476 INFO:teuthology.orchestra.run.smithi149.stdout: debug bluefs = 20 2023-11-28T12:58:00.476 INFO:teuthology.orchestra.run.smithi149.stdout: debug bluestore = 20 2023-11-28T12:58:00.476 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2023-11-28T12:58:00.476 INFO:teuthology.orchestra.run.smithi149.stdout: debug osd = 20 2023-11-28T12:58:00.476 INFO:teuthology.orchestra.run.smithi149.stdout: debug rocksdb = 10 2023-11-28T12:58:00.476 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd backfillfull_ratio = 0.85 2023-11-28T12:58:00.476 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd full ratio = 0.9 2023-11-28T12:58:00.476 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd nearfull ratio = 0.8 2023-11-28T12:58:00.476 INFO:teuthology.orchestra.run.smithi149.stdout: osd failsafe full ratio = 0.95 2023-11-28T12:58:00.476 INFO:teuthology.orchestra.run.smithi149.stdout: osd objectstore = bluestore 2023-11-28T12:58:00.477 INFO:teuthology.orchestra.run.smithi149.stdout: osd op complaint time = 180 2023-11-28T12:58:00.477 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.477 INFO:teuthology.orchestra.run.smithi149.stdout:[mgr] 2023-11-28T12:58:00.477 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2023-11-28T12:58:00.477 INFO:teuthology.orchestra.run.smithi149.stdout: debug mgr = 20 2023-11-28T12:58:00.477 INFO:teuthology.orchestra.run.smithi149.stdout: debug mon = 20 2023-11-28T12:58:00.477 INFO:teuthology.orchestra.run.smithi149.stdout: debug auth = 20 2023-11-28T12:58:00.477 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min pgs per osd = 4 2023-11-28T12:58:00.477 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min bytes per osd = 10 2023-11-28T12:58:00.478 INFO:teuthology.orchestra.run.smithi149.stdout: mgr/telemetry/nag = false 2023-11-28T12:58:00.478 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.478 INFO:teuthology.orchestra.run.smithi149.stdout:[mon] 2023-11-28T12:58:00.478 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2023-11-28T12:58:00.478 INFO:teuthology.orchestra.run.smithi149.stdout: debug mon = 20 2023-11-28T12:58:00.478 INFO:teuthology.orchestra.run.smithi149.stdout: debug paxos = 20 2023-11-28T12:58:00.478 INFO:teuthology.orchestra.run.smithi149.stdout: debug auth = 20 2023-11-28T12:58:00.478 INFO:teuthology.orchestra.run.smithi149.stdout: mon data avail warn = 5 2023-11-28T12:58:00.478 INFO:teuthology.orchestra.run.smithi149.stdout: mon mgr mkfs grace = 240 2023-11-28T12:58:00.478 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min pgs per osd = 4 2023-11-28T12:58:00.479 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd reporter subtree level = osd 2023-11-28T12:58:00.479 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd prime pg temp = true 2023-11-28T12:58:00.479 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min bytes per osd = 10 2023-11-28T12:58:00.479 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.479 INFO:teuthology.orchestra.run.smithi149.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-28T12:58:00.479 INFO:teuthology.orchestra.run.smithi149.stdout: auth mon ticket ttl = 660 # 11m 2023-11-28T12:58:00.479 INFO:teuthology.orchestra.run.smithi149.stdout: auth service ticket ttl = 240 # 4m 2023-11-28T12:58:00.479 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.479 INFO:teuthology.orchestra.run.smithi149.stdout: # don't complain about insecure global_id in the test suite 2023-11-28T12:58:00.480 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-28T12:58:00.480 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-28T12:58:00.480 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.480 INFO:teuthology.orchestra.run.smithi149.stdout: # 1m isn't quite enough 2023-11-28T12:58:00.480 INFO:teuthology.orchestra.run.smithi149.stdout: mon_down_mkfs_grace = 2m 2023-11-28T12:58:00.480 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.480 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_filestore_osds = false 2023-11-28T12:58:00.480 INFO:teuthology.orchestra.run.smithi149.stdout: mon op complaint time = 120 2023-11-28T12:58:00.480 INFO:teuthology.orchestra.run.smithi149.stdout: 2023-11-28T12:58:00.480 INFO:teuthology.orchestra.run.smithi149.stdout:[client] 2023-11-28T12:58:00.480 INFO:teuthology.orchestra.run.smithi149.stdout: rgw cache enabled = true 2023-11-28T12:58:00.480 INFO:teuthology.orchestra.run.smithi149.stdout: rgw enable ops log = true 2023-11-28T12:58:00.481 INFO:teuthology.orchestra.run.smithi149.stdout: rgw enable usage log = true 2023-11-28T12:58:00.481 INFO:teuthology.orchestra.run.smithi149.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-28T12:58:00.481 INFO:teuthology.orchestra.run.smithi149.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-28T12:58:00.481 INFO:teuthology.orchestra.run.smithi149.stdout: client mount timeout = 600 2023-11-28T12:58:00.481 INFO:teuthology.orchestra.run.smithi149.stdout: debug client = 20 2023-11-28T12:58:00.481 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2023-11-28T12:58:00.481 INFO:teuthology.orchestra.run.smithi149.stdout: pid file = "" 2023-11-28T12:58:00.481 INFO:teuthology.orchestra.run.smithi149.stdout: rados mon op timeout = 900 2023-11-28T12:58:00.481 INFO:teuthology.orchestra.run.smithi149.stdout: rados osd op timeout = 900 2023-11-28T12:58:00.481 INFO:teuthology.orchestra.run.smithi149.stdout:[mon.a] 2023-11-28T12:58:00.482 INFO:teuthology.orchestra.run.smithi149.stdout:[mon.b] 2023-11-28T12:58:00.482 INFO:teuthology.orchestra.run.smithi149.stdout:[mon.c] 2023-11-28T12:58:00.482 INFO:teuthology.orchestra.run.smithi149.stdout:[mds.a] 2023-11-28T12:58:00.482 INFO:teuthology.orchestra.run.smithi149.stdout:[mds.c] 2023-11-28T12:58:00.482 INFO:teuthology.orchestra.run.smithi149.stdout:[mds.b] 2023-11-28T12:58:00.482 INFO:teuthology.orchestra.run.smithi149.stdout:[mds.d] 2023-11-28T12:58:00.482 INFO:teuthology.orchestra.run.smithi149.stdout:[mds] 2023-11-28T12:58:00.482 INFO:teuthology.orchestra.run.smithi149.stdout: debug mds = 20 2023-11-28T12:58:00.482 INFO:teuthology.orchestra.run.smithi149.stdout: debug mds balancer = 20 2023-11-28T12:58:00.482 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2023-11-28T12:58:00.482 INFO:teuthology.orchestra.run.smithi149.stdout: mds debug frag = True 2023-11-28T12:58:00.483 INFO:teuthology.orchestra.run.smithi149.stdout: mds debug scatterstat = True 2023-11-28T12:58:00.483 INFO:teuthology.orchestra.run.smithi149.stdout: mds op complaint time = 180 2023-11-28T12:58:00.483 INFO:teuthology.orchestra.run.smithi149.stdout: mds verify scatter = True 2023-11-28T12:58:00.483 INFO:teuthology.orchestra.run.smithi149.stdout: osd op complaint time = 180 2023-11-28T12:58:00.483 INFO:teuthology.orchestra.run.smithi149.stdout: rados mon op timeout = 900 2023-11-28T12:58:00.483 INFO:teuthology.orchestra.run.smithi149.stdout: rados osd op timeout = 900 2023-11-28T12:58:00.485 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-28T12:58:00.485 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-28T12:58:00.582 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-28T12:58:00.582 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:00.582 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-28T12:58:00.599 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:00.599 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-28T12:58:00.655 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T12:58:00.656 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:00.656 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-28T12:58:00.656 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T12:58:00.731 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:00.732 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:00.788 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi149.front.sepia.ceph.com 2023-11-28T12:58:00.788 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:00.788 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-28T12:58:00.790 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T12:58:00.836 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:00.836 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:00.897 INFO:tasks.ceph:Setting up mon nodes... 2023-11-28T12:58:00.897 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-28T12:58:00.897 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-11-28T12:58:00.950 INFO:teuthology.orchestra.run.smithi111.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-11-28T12:58:00.952 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T12:58:01.039 INFO:teuthology.orchestra.run.smithi149.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T12:58:01.042 INFO:tasks.ceph:Setting up mds nodes... 2023-11-28T12:58:01.042 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-11-28T12:58:01.095 INFO:teuthology.orchestra.run.smithi111.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-11-28T12:58:01.098 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-11-28T12:58:01.124 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-11-28T12:58:01.217 INFO:teuthology.orchestra.run.smithi111.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-11-28T12:58:01.219 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-11-28T12:58:01.245 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-11-28T12:58:01.308 INFO:teuthology.orchestra.run.smithi149.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-11-28T12:58:01.310 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-11-28T12:58:01.338 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-11-28T12:58:01.434 INFO:teuthology.orchestra.run.smithi149.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-11-28T12:58:01.436 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-11-28T12:58:01.463 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-28T12:58:01.463 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-28T12:58:01.506 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-28T12:58:01.519 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-11-28T12:58:01.603 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-11-28T12:58:01.615 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-11-28T12:58:01.659 INFO:teuthology.orchestra.run.smithi149.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-11-28T12:58:01.672 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-11-28T12:58:01.715 INFO:teuthology.orchestra.run.smithi149.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-11-28T12:58:01.727 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-28T12:58:01.727 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi149.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-28T12:58:01.727 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-28T12:58:01.754 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T12:58:01.754 INFO:tasks.ceph:role: osd.0 2023-11-28T12:58:01.754 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T12:58:01.754 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-28T12:58:01.829 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T12:58:01.830 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T12:58:01.830 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T12:58:01.830 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T12:58:01.830 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T12:58:01.830 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-11-28T12:58:01.831 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T12:58:01.831 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T12:58:01.831 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T12:58:01.832 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T12:58:02.259 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2023-11-28T12:58:02.260 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-11-28T12:58:02.261 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-28T12:58:02.430 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-28T12:58:02.464 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-28T12:58:02.531 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T12:58:02.531 INFO:tasks.ceph:role: osd.1 2023-11-28T12:58:02.531 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T12:58:02.531 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-28T12:58:02.605 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T12:58:02.605 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T12:58:02.606 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T12:58:02.606 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T12:58:02.606 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T12:58:02.606 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-11-28T12:58:02.606 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T12:58:02.606 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T12:58:02.606 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T12:58:02.606 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T12:58:03.025 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2023-11-28T12:58:03.028 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-11-28T12:58:03.028 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-28T12:58:03.152 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-28T12:58:03.185 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-28T12:58:03.255 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T12:58:03.256 INFO:tasks.ceph:role: osd.2 2023-11-28T12:58:03.256 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T12:58:03.256 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-28T12:58:03.332 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T12:58:03.332 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T12:58:03.332 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T12:58:03.332 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T12:58:03.332 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T12:58:03.332 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-11-28T12:58:03.332 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T12:58:03.332 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T12:58:03.332 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T12:58:03.332 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T12:58:03.772 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2023-11-28T12:58:03.774 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-11-28T12:58:03.774 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-28T12:58:03.924 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-28T12:58:03.962 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-28T12:58:04.031 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T12:58:04.031 INFO:tasks.ceph:role: osd.3 2023-11-28T12:58:04.032 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T12:58:04.032 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-28T12:58:04.108 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T12:58:04.108 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T12:58:04.108 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T12:58:04.109 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T12:58:04.109 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T12:58:04.109 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2023-11-28T12:58:04.109 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T12:58:04.109 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T12:58:04.109 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T12:58:04.109 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T12:58:04.565 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2023-11-28T12:58:04.567 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2023-11-28T12:58:04.567 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-11-28T12:58:04.747 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-28T12:58:04.782 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:04.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:04.889+0000 7fb88666f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-28T12:58:04.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:04.890+0000 7fb88666f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-28T12:58:04.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:04.890+0000 7fb88666f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T12:58:04.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:04.890+0000 7fb88666f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T12:58:04.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:04.890+0000 7fb88666f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T12:58:04.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:04.890+0000 7fb88666f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-28T12:58:07.763 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-28T12:58:07.794 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:07.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:07.884+0000 7f5f2b8f1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-28T12:58:07.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:07.884+0000 7f5f2b8f1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-28T12:58:07.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:07.884+0000 7f5f2b8f1640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T12:58:07.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:07.884+0000 7f5f2b8f1640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T12:58:07.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:07.884+0000 7f5f2b8f1640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T12:58:07.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:07.885+0000 7f5f2b8f1640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-28T12:58:10.768 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-28T12:58:10.802 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:10.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:10.890+0000 7f37d8e44640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-28T12:58:10.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:10.891+0000 7f37d8e44640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-28T12:58:10.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:10.891+0000 7f37d8e44640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T12:58:10.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:10.891+0000 7f37d8e44640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T12:58:10.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:10.891+0000 7f37d8e44640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T12:58:10.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:10.891+0000 7f37d8e44640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-28T12:58:13.760 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-28T12:58:13.790 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:13.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:13.878+0000 7fd300bfd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-28T12:58:13.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:13.879+0000 7fd300bfd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-28T12:58:13.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:13.879+0000 7fd300bfd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T12:58:13.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:13.879+0000 7fd300bfd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T12:58:13.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:13.879+0000 7fd300bfd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T12:58:13.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:13.879+0000 7fd300bfd640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-28T12:58:16.748 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-28T12:58:16.780 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-28T12:58:16.815 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T12:58:16.815 INFO:tasks.ceph:role: osd.4 2023-11-28T12:58:16.815 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi149.front.sepia.ceph.com 2023-11-28T12:58:16.815 DEBUG:teuthology.orchestra.run.smithi149:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-28T12:58:16.893 INFO:teuthology.orchestra.run.smithi149.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T12:58:16.893 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T12:58:16.893 INFO:teuthology.orchestra.run.smithi149.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T12:58:16.894 INFO:teuthology.orchestra.run.smithi149.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T12:58:16.894 INFO:teuthology.orchestra.run.smithi149.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T12:58:16.894 INFO:teuthology.orchestra.run.smithi149.stdout: = sunit=0 swidth=0 blks 2023-11-28T12:58:16.894 INFO:teuthology.orchestra.run.smithi149.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T12:58:16.894 INFO:teuthology.orchestra.run.smithi149.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T12:58:16.894 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T12:58:16.894 INFO:teuthology.orchestra.run.smithi149.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T12:58:17.330 INFO:teuthology.orchestra.run.smithi149.stdout:Discarding blocks...Done. 2023-11-28T12:58:17.333 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi149.front.sepia.ceph.com -o noatime 2023-11-28T12:58:17.333 DEBUG:teuthology.orchestra.run.smithi149:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-11-28T12:58:17.516 DEBUG:teuthology.orchestra.run.smithi149:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-28T12:58:17.552 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-28T12:58:17.620 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T12:58:17.620 INFO:tasks.ceph:role: osd.5 2023-11-28T12:58:17.620 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi149.front.sepia.ceph.com 2023-11-28T12:58:17.620 DEBUG:teuthology.orchestra.run.smithi149:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-28T12:58:17.695 INFO:teuthology.orchestra.run.smithi149.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T12:58:17.695 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T12:58:17.696 INFO:teuthology.orchestra.run.smithi149.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T12:58:17.696 INFO:teuthology.orchestra.run.smithi149.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T12:58:17.696 INFO:teuthology.orchestra.run.smithi149.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T12:58:17.696 INFO:teuthology.orchestra.run.smithi149.stdout: = sunit=0 swidth=0 blks 2023-11-28T12:58:17.696 INFO:teuthology.orchestra.run.smithi149.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T12:58:17.696 INFO:teuthology.orchestra.run.smithi149.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T12:58:17.696 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T12:58:17.696 INFO:teuthology.orchestra.run.smithi149.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T12:58:18.115 INFO:teuthology.orchestra.run.smithi149.stdout:Discarding blocks...Done. 2023-11-28T12:58:18.117 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi149.front.sepia.ceph.com -o noatime 2023-11-28T12:58:18.117 DEBUG:teuthology.orchestra.run.smithi149:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-11-28T12:58:18.224 DEBUG:teuthology.orchestra.run.smithi149:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-28T12:58:18.257 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-11-28T12:58:18.326 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T12:58:18.326 INFO:tasks.ceph:role: osd.6 2023-11-28T12:58:18.326 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi149.front.sepia.ceph.com 2023-11-28T12:58:18.326 DEBUG:teuthology.orchestra.run.smithi149:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-28T12:58:18.401 INFO:teuthology.orchestra.run.smithi149.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T12:58:18.401 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T12:58:18.401 INFO:teuthology.orchestra.run.smithi149.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T12:58:18.401 INFO:teuthology.orchestra.run.smithi149.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T12:58:18.402 INFO:teuthology.orchestra.run.smithi149.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T12:58:18.402 INFO:teuthology.orchestra.run.smithi149.stdout: = sunit=0 swidth=0 blks 2023-11-28T12:58:18.402 INFO:teuthology.orchestra.run.smithi149.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T12:58:18.402 INFO:teuthology.orchestra.run.smithi149.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T12:58:18.402 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T12:58:18.402 INFO:teuthology.orchestra.run.smithi149.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T12:58:18.845 INFO:teuthology.orchestra.run.smithi149.stdout:Discarding blocks...Done. 2023-11-28T12:58:18.846 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi149.front.sepia.ceph.com -o noatime 2023-11-28T12:58:18.847 DEBUG:teuthology.orchestra.run.smithi149:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-11-28T12:58:19.010 DEBUG:teuthology.orchestra.run.smithi149:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-11-28T12:58:19.043 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-11-28T12:58:19.109 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T12:58:19.109 INFO:tasks.ceph:role: osd.7 2023-11-28T12:58:19.109 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi149.front.sepia.ceph.com 2023-11-28T12:58:19.110 DEBUG:teuthology.orchestra.run.smithi149:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-28T12:58:19.184 INFO:teuthology.orchestra.run.smithi149.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T12:58:19.184 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T12:58:19.184 INFO:teuthology.orchestra.run.smithi149.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T12:58:19.185 INFO:teuthology.orchestra.run.smithi149.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-28T12:58:19.185 INFO:teuthology.orchestra.run.smithi149.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T12:58:19.185 INFO:teuthology.orchestra.run.smithi149.stdout: = sunit=0 swidth=0 blks 2023-11-28T12:58:19.185 INFO:teuthology.orchestra.run.smithi149.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T12:58:19.185 INFO:teuthology.orchestra.run.smithi149.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-28T12:58:19.185 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T12:58:19.186 INFO:teuthology.orchestra.run.smithi149.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T12:58:19.618 INFO:teuthology.orchestra.run.smithi149.stdout:Discarding blocks...Done. 2023-11-28T12:58:19.620 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi149.front.sepia.ceph.com -o noatime 2023-11-28T12:58:19.620 DEBUG:teuthology.orchestra.run.smithi149:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-11-28T12:58:19.797 DEBUG:teuthology.orchestra.run.smithi149:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-11-28T12:58:19.829 DEBUG:teuthology.orchestra.run.smithi149:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:19.933 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:19.932+0000 7f2511b32640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-28T12:58:19.934 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:19.932+0000 7f2511b32640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-28T12:58:19.934 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:19.932+0000 7f2511b32640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T12:58:19.934 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:19.932+0000 7f2511b32640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T12:58:19.934 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:19.932+0000 7f2511b32640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T12:58:19.934 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:19.933+0000 7f2511b32640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-28T12:58:22.808 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-28T12:58:22.837 DEBUG:teuthology.orchestra.run.smithi149:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:22.927 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:22.926+0000 7fa85792d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-28T12:58:22.927 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:22.926+0000 7fa85792d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-28T12:58:22.928 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:22.926+0000 7fa85792d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T12:58:22.928 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:22.926+0000 7fa85792d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T12:58:22.928 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:22.926+0000 7fa85792d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T12:58:22.928 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:22.927+0000 7fa85792d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-28T12:58:25.790 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-28T12:58:25.819 DEBUG:teuthology.orchestra.run.smithi149:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:25.909 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:25.908+0000 7f2ecdf7d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-11-28T12:58:25.909 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:25.908+0000 7f2ecdf7d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-11-28T12:58:25.909 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:25.908+0000 7f2ecdf7d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T12:58:25.909 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:25.908+0000 7f2ecdf7d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T12:58:25.909 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:25.908+0000 7f2ecdf7d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T12:58:25.910 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:25.909+0000 7f2ecdf7d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-11-28T12:58:28.353 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T12:58:28.356 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T12:58:28.768 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-11-28T12:58:28.795 DEBUG:teuthology.orchestra.run.smithi149:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:28.882 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:28.880+0000 7f9a704a5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-11-28T12:58:28.882 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:28.880+0000 7f9a704a5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-11-28T12:58:28.882 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:28.880+0000 7f9a704a5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T12:58:28.883 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:28.880+0000 7f9a704a5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T12:58:28.883 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:28.880+0000 7f9a704a5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T12:58:28.883 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:28.881+0000 7f9a704a5640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-11-28T12:58:31.747 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-11-28T12:58:31.776 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-28T12:58:31.777 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:31.777 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-11-28T12:58:31.809 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:31.809 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-11-28T12:58:31.878 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:31.878 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-11-28T12:58:31.950 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:31.950 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-28T12:58:32.017 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:32.018 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-28T12:58:32.085 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:32.085 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-28T12:58:32.151 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:32.151 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-28T12:58:32.217 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:32.217 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-28T12:58:32.249 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:32.249 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-11-28T12:58:32.320 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:32.320 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-11-28T12:58:32.389 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:32.389 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-28T12:58:32.457 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:32.457 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-28T12:58:32.529 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:32.529 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-11-28T12:58:32.598 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:32.598 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-11-28T12:58:32.665 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:32.665 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-28T12:58:32.682 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:32.682 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-11-28T12:58:32.739 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:32.740 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-11-28T12:58:32.757 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:32.757 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-11-28T12:58:32.814 INFO:tasks.ceph:Adding keys to all mons... 2023-11-28T12:58:32.815 DEBUG:teuthology.orchestra.run.smithi111:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-28T12:58:32.817 DEBUG:teuthology.orchestra.run.smithi149:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-28T12:58:32.857 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.y] 2023-11-28T12:58:32.858 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDY42Vl6WCiOBAALRDd8ZjOOlxxLKQsgXP9Jg== 2023-11-28T12:58:32.858 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.a] 2023-11-28T12:58:32.858 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDZ42VlH823BRAArK8kE5qUgPoqFZPxIudN+w== 2023-11-28T12:58:32.858 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.c] 2023-11-28T12:58:32.858 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDZ42VlQqHxDBAA8RDJ2PETpTHNjnn1i1T3Ow== 2023-11-28T12:58:32.858 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.0] 2023-11-28T12:58:32.858 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDc42VlfB0eNRAAJEa0AL3iDLArHc4n6CWK9g== 2023-11-28T12:58:32.858 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.1] 2023-11-28T12:58:32.858 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDf42VlnrbLNBAAU+sVJ/6EsadNo3BXpyrbSw== 2023-11-28T12:58:32.858 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.2] 2023-11-28T12:58:32.859 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDi42VlwCUsNRAAwZKJ/o1qTBtWwR35B1r2zQ== 2023-11-28T12:58:32.859 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.3] 2023-11-28T12:58:32.859 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDl42Vl3/t2NBAAMPuhlpUi37Fpd/6NclsErw== 2023-11-28T12:58:32.859 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.x] 2023-11-28T12:58:32.859 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDZ42Vl+AxXAhAADTJ8Jt/WNo8qr5zAyQJ4rw== 2023-11-28T12:58:32.859 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.b] 2023-11-28T12:58:32.859 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDZ42VlIo9ZEhAA5NmyiPE68OmMcsG1hTiTgw== 2023-11-28T12:58:32.859 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.d] 2023-11-28T12:58:32.859 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDZ42Vl8RDmGRAAkiGKEQMr9rHahKJ/3lw9BA== 2023-11-28T12:58:32.859 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.4] 2023-11-28T12:58:32.859 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDr42VlkrWgNxAAuu2dXZZ3U/XRrq3cWCEa8w== 2023-11-28T12:58:32.860 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.5] 2023-11-28T12:58:32.860 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDu42VldDFENxAAYKD2Dqgka5G+XxGjOR3M6w== 2023-11-28T12:58:32.860 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.6] 2023-11-28T12:58:32.860 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDx42VlvjotNhAAWD2E1sgAZLdxb5CG1Rg26g== 2023-11-28T12:58:32.860 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.7] 2023-11-28T12:58:32.860 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQD042VlZheVNBAAlMIKaDmgUMmt/hA+vx2/PA== 2023-11-28T12:58:32.860 INFO:teuthology.orchestra.run.smithi111.stdout:[client.0] 2023-11-28T12:58:32.860 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDZ42Vl3MswHhAA/oDIgj9lzhcnU7f0ksZ9cw== 2023-11-28T12:58:32.860 INFO:teuthology.orchestra.run.smithi111.stdout:[client.2] 2023-11-28T12:58:32.860 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDZ42VlJiXlIxAAYuFMFLUqIDN5f0E008GKZw== 2023-11-28T12:58:32.860 INFO:teuthology.orchestra.run.smithi111.stdout:[client.1] 2023-11-28T12:58:32.860 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDZ42VlV3dQJxAAdzQk1q+DoxFDEr/TYJi/6A== 2023-11-28T12:58:32.861 INFO:teuthology.orchestra.run.smithi111.stdout:[client.3] 2023-11-28T12:58:32.861 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQDZ42VlagSZKhAAkLS8RwTZZTDyO2slb0MzIQ== 2023-11-28T12:58:32.879 INFO:teuthology.orchestra.run.smithi149.stdout:[mgr.y] 2023-11-28T12:58:32.879 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDY42Vl6WCiOBAALRDd8ZjOOlxxLKQsgXP9Jg== 2023-11-28T12:58:32.879 INFO:teuthology.orchestra.run.smithi149.stdout:[mds.a] 2023-11-28T12:58:32.879 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDZ42VlH823BRAArK8kE5qUgPoqFZPxIudN+w== 2023-11-28T12:58:32.880 INFO:teuthology.orchestra.run.smithi149.stdout:[mds.c] 2023-11-28T12:58:32.880 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDZ42VlQqHxDBAA8RDJ2PETpTHNjnn1i1T3Ow== 2023-11-28T12:58:32.880 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.0] 2023-11-28T12:58:32.880 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDc42VlfB0eNRAAJEa0AL3iDLArHc4n6CWK9g== 2023-11-28T12:58:32.880 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.1] 2023-11-28T12:58:32.880 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDf42VlnrbLNBAAU+sVJ/6EsadNo3BXpyrbSw== 2023-11-28T12:58:32.880 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.2] 2023-11-28T12:58:32.880 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDi42VlwCUsNRAAwZKJ/o1qTBtWwR35B1r2zQ== 2023-11-28T12:58:32.880 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.3] 2023-11-28T12:58:32.880 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDl42Vl3/t2NBAAMPuhlpUi37Fpd/6NclsErw== 2023-11-28T12:58:32.880 INFO:teuthology.orchestra.run.smithi149.stdout:[mgr.x] 2023-11-28T12:58:32.880 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDZ42Vl+AxXAhAADTJ8Jt/WNo8qr5zAyQJ4rw== 2023-11-28T12:58:32.881 INFO:teuthology.orchestra.run.smithi149.stdout:[mds.b] 2023-11-28T12:58:32.881 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDZ42VlIo9ZEhAA5NmyiPE68OmMcsG1hTiTgw== 2023-11-28T12:58:32.881 INFO:teuthology.orchestra.run.smithi149.stdout:[mds.d] 2023-11-28T12:58:32.881 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDZ42Vl8RDmGRAAkiGKEQMr9rHahKJ/3lw9BA== 2023-11-28T12:58:32.881 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.4] 2023-11-28T12:58:32.881 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDr42VlkrWgNxAAuu2dXZZ3U/XRrq3cWCEa8w== 2023-11-28T12:58:32.881 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.5] 2023-11-28T12:58:32.881 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDu42VldDFENxAAYKD2Dqgka5G+XxGjOR3M6w== 2023-11-28T12:58:32.881 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.6] 2023-11-28T12:58:32.881 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDx42VlvjotNhAAWD2E1sgAZLdxb5CG1Rg26g== 2023-11-28T12:58:32.881 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.7] 2023-11-28T12:58:32.881 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQD042VlZheVNBAAlMIKaDmgUMmt/hA+vx2/PA== 2023-11-28T12:58:32.881 INFO:teuthology.orchestra.run.smithi149.stdout:[client.0] 2023-11-28T12:58:32.882 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDZ42Vl3MswHhAA/oDIgj9lzhcnU7f0ksZ9cw== 2023-11-28T12:58:32.882 INFO:teuthology.orchestra.run.smithi149.stdout:[client.2] 2023-11-28T12:58:32.882 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDZ42VlJiXlIxAAYuFMFLUqIDN5f0E008GKZw== 2023-11-28T12:58:32.882 INFO:teuthology.orchestra.run.smithi149.stdout:[client.1] 2023-11-28T12:58:32.882 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDZ42VlV3dQJxAAdzQk1q+DoxFDEr/TYJi/6A== 2023-11-28T12:58:32.882 INFO:teuthology.orchestra.run.smithi149.stdout:[client.3] 2023-11-28T12:58:32.882 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQDZ42VlagSZKhAAkLS8RwTZZTDyO2slb0MzIQ== 2023-11-28T12:58:32.883 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T12:58:32.904 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T12:58:32.976 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T12:58:32.978 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T12:58:33.035 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T12:58:33.037 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T12:58:33.093 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.096 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.148 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.151 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.204 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.206 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.259 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.261 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.315 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T12:58:33.317 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T12:58:33.369 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T12:58:33.371 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T12:58:33.427 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T12:58:33.429 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T12:58:33.485 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.488 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.541 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.543 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.596 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.598 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.651 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.653 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T12:58:33.707 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T12:58:33.709 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T12:58:33.765 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T12:58:33.767 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T12:58:33.823 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T12:58:33.825 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T12:58:33.882 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T12:58:33.884 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T12:58:33.940 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-28T12:58:33.940 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-28T12:58:33.967 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T12:58:34.065 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-28T12:58:34.091 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-28T12:58:34.117 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T12:58:34.215 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-28T12:58:34.241 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-28T12:58:34.308 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T12:58:34.407 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-28T12:58:34.433 DEBUG:teuthology.orchestra.run.smithi111:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:34.435 DEBUG:teuthology.orchestra.run.smithi149:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-28T12:58:34.490 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-28T12:58:34.490 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-28T12:58:34.491 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-28T12:58:34.493 INFO:tasks.ceph.mon.a:Started 2023-11-28T12:58:34.493 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-28T12:58:34.493 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-28T12:58:34.533 INFO:tasks.ceph.mon.b:Started 2023-11-28T12:58:34.533 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-28T12:58:34.533 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-28T12:58:34.535 INFO:tasks.ceph.mon.c:Started 2023-11-28T12:58:34.535 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-28T12:58:34.535 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-11-28T12:58:34.535 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-11-28T12:58:34.537 INFO:tasks.ceph.mgr.y:Started 2023-11-28T12:58:34.538 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-28T12:58:34.538 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-28T12:58:34.540 INFO:tasks.ceph.mgr.x:Started 2023-11-28T12:58:34.540 INFO:tasks.ceph:Setting crush tunables to default 2023-11-28T12:58:34.540 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd crush tunables default 2023-11-28T12:58:34.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.661+0000 7f6703a06700 1 Processor -- start 2023-11-28T12:58:34.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.662+0000 7f6703a06700 1 -- start start 2023-11-28T12:58:34.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.662+0000 7f6703a06700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66fc0b0d70 0x7f66fc1604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:34.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.662+0000 7f6703a06700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f66fc0b15b0 0x7f66fc0cc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:34.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.662+0000 7f6703a06700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66fc15cd90 con 0x7f66fc0b1140 2023-11-28T12:58:34.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.662+0000 7f6703a06700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66fc0cedd0 con 0x7f66fc0b15b0 2023-11-28T12:58:34.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.662+0000 7f6703a06700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc1613e0 con 0x7f66fc0b0d70 2023-11-28T12:58:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.662+0000 7f6701fa3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f66fc0b15b0 0x7f66fc0cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.662+0000 7f6700fa1700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f66fc0b1140 0x7f66fc0ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:60952/0 (socket says 172.21.15.111:60952) 2023-11-28T12:58:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.663+0000 7f6701fa3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f66fc0b15b0 0x7f66fc0cc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60178/0 (socket says 172.21.15.111:60178) 2023-11-28T12:58:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.663+0000 7f6700fa1700 1 -- 172.21.15.111:0/3784647585 learned_addr learned my addr 172.21.15.111:0/3784647585 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:58:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.663+0000 7f67017a2700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66fc0b0d70 0x7f66fc1604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.663+0000 7f6701fa3700 1 -- 172.21.15.111:0/3784647585 >> v2:172.21.15.111:3300/0 conn(0x7f66fc0b15b0 msgr2=0x7f66fc0cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T12:58:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.663+0000 7f6701fa3700 1 -- 172.21.15.111:0/3784647585 >> v2:172.21.15.111:3300/0 conn(0x7f66fc0b15b0 msgr2=0x7f66fc0cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T12:58:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.663+0000 7f6701fa3700 1 --2- 172.21.15.111:0/3784647585 >> v2:172.21.15.111:3300/0 conn(0x7f66fc0b15b0 0x7f66fc0cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T12:58:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.663+0000 7f6701fa3700 1 --2- 172.21.15.111:0/3784647585 >> v2:172.21.15.111:3300/0 conn(0x7f66fc0b15b0 0x7f66fc0cc550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T12:58:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.663+0000 7f67017a2700 1 -- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66fc0b0d70 msgr2=0x7f66fc1604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T12:58:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.663+0000 7f67017a2700 1 -- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66fc0b0d70 msgr2=0x7f66fc1604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T12:58:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.663+0000 7f67017a2700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66fc0b0d70 0x7f66fc1604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T12:58:34.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.663+0000 7f67017a2700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66fc0b0d70 0x7f66fc1604b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T12:58:34.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.855+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832359317 0 0) 0x7f66f8001110 con 0x7f66fc0b1140 2023-11-28T12:58:34.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.855+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66e80035b0 con 0x7f66fc0b1140 2023-11-28T12:58:34.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 173925190 0 0) 0x7f66e80035b0 con 0x7f66fc0b1140 2023-11-28T12:58:34.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66fc0b0d70 msgr2=0x7f66fc1604b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T12:58:34.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66fc0b0d70 0x7f66fc1604b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:34.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 >> v2:172.21.15.111:3300/0 conn(0x7f66fc0b15b0 msgr2=0x7f66fc0cc550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T12:58:34.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 --2- 172.21.15.111:0/3784647585 >> v2:172.21.15.111:3300/0 conn(0x7f66fc0b15b0 0x7f66fc0cc550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:34.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66fc160b40 con 0x7f66fc0b1140 2023-11-28T12:58:34.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f66f8001800 con 0x7f66fc0b1140 2023-11-28T12:58:34.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 >> v1:172.21.15.111:6789/0 conn(0x7f66fc0b1140 legacy=0x7f66fc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:34.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66e8003a00 0x7f66e8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:34.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66fc0b0d70 0x7f66e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:34.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66e8004d90 0x7f66e8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:34.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc0cedd0 con 0x7f66e8003a00 2023-11-28T12:58:34.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc1613e0 con 0x7f66e8004d90 2023-11-28T12:58:34.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e800bac0 con 0x7f66fc0b0d70 2023-11-28T12:58:34.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f67017a2700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66e8003a00 0x7f66e8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:34.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f6700fa1700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66fc0b0d70 0x7f66e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:34.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f6701fa3700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66e8004d90 0x7f66e8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:34.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f67017a2700 1 -- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66fc0b0d70 msgr2=0x7f66e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:34.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f67017a2700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66fc0b0d70 0x7f66e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:34.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f67017a2700 1 -- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66e8004d90 msgr2=0x7f66e8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:34.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f67017a2700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66e8004d90 0x7f66e8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:34.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f67017a2700 1 -- 172.21.15.111:0/3784647585 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e80035b0 con 0x7f66e8003a00 2023-11-28T12:58:34.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f6701fa3700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66e8004d90 0x7f66e8005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:58:34.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f67017a2700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66e8003a00 0x7f66e8003e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00c840 tx=0x7f66e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=132a7e2f48db1d8b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:34.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.856+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec005c20 con 0x7f66e8003a00 2023-11-28T12:58:34.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.857+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66ec005d80 con 0x7f66e8003a00 2023-11-28T12:58:34.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.857+0000 7f66f3fff700 1 -- 172.21.15.111:0/3784647585 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec004270 con 0x7f66e8003a00 2023-11-28T12:58:34.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.857+0000 7f6703a06700 1 -- 172.21.15.111:0/3784647585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66e8003a00 msgr2=0x7f66e8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:34.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.857+0000 7f6703a06700 1 --2- 172.21.15.111:0/3784647585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66e8003a00 0x7f66e8003e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00c840 tx=0x7f66e8005880 comp rx=0 tx=0).stop 2023-11-28T12:58:34.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.857+0000 7f6701fa3700 1 -- 172.21.15.111:0/3784647585 reap_dead start 2023-11-28T12:58:34.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.857+0000 7f6703a06700 1 -- 172.21.15.111:0/3784647585 shutdown_connections 2023-11-28T12:58:34.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.857+0000 7f6703a06700 1 -- 172.21.15.111:0/3784647585 >> 172.21.15.111:0/3784647585 conn(0x7f66fc098ee0 msgr2=0x7f66fc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:34.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.857+0000 7f6703a06700 1 -- 172.21.15.111:0/3784647585 shutdown_connections 2023-11-28T12:58:34.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.857+0000 7f6703a06700 1 -- 172.21.15.111:0/3784647585 wait complete. 2023-11-28T12:58:34.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.859+0000 7f6703a06700 1 Processor -- start 2023-11-28T12:58:34.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.859+0000 7f6703a06700 1 -- start start 2023-11-28T12:58:34.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6703a06700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66e8003a00 0x7f66fc0c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:34.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6703a06700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66fc0c7580 0x7f66fc0c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:34.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6703a06700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66fc0c5840 0x7f66fc0c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:34.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6703a06700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc0c4480 con 0x7f66fc0c7580 2023-11-28T12:58:34.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6703a06700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc0c8c50 con 0x7f66fc0c5840 2023-11-28T12:58:34.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6703a06700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc148ab0 con 0x7f66e8003a00 2023-11-28T12:58:34.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6700fa1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66fc0c7580 0x7f66fc0c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:34.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6700fa1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66fc0c7580 0x7f66fc0c79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60196/0 (socket says 172.21.15.111:60196) 2023-11-28T12:58:34.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6700fa1700 1 -- 172.21.15.111:0/1699337176 learned_addr learned my addr 172.21.15.111:0/1699337176 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:34.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f67017a2700 1 --2- 172.21.15.111:0/1699337176 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66e8003a00 0x7f66fc0c3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:34.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6701fa3700 1 --2- 172.21.15.111:0/1699337176 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66fc0c5840 0x7f66fc0c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:34.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6700fa1700 1 -- 172.21.15.111:0/1699337176 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66e8003a00 msgr2=0x7f66fc0c3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:34.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6700fa1700 1 --2- 172.21.15.111:0/1699337176 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66e8003a00 0x7f66fc0c3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:34.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6700fa1700 1 -- 172.21.15.111:0/1699337176 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66fc0c5840 msgr2=0x7f66fc0c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:34.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6700fa1700 1 --2- 172.21.15.111:0/1699337176 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66fc0c5840 0x7f66fc0c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:34.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6700fa1700 1 -- 172.21.15.111:0/1699337176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66fc161100 con 0x7f66fc0c7580 2023-11-28T12:58:34.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6700fa1700 1 --2- 172.21.15.111:0/1699337176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66fc0c7580 0x7f66fc0c79f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f66f80072c0 tx=0x7f66f80075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:34.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f66f1ffb700 1 -- 172.21.15.111:0/1699337176 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f800f600 con 0x7f66fc0c7580 2023-11-28T12:58:34.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f66f1ffb700 1 -- 172.21.15.111:0/1699337176 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f800f7a0 con 0x7f66fc0c7580 2023-11-28T12:58:34.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f6703a06700 1 -- 172.21.15.111:0/1699337176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66fc160e20 con 0x7f66fc0c7580 2023-11-28T12:58:34.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.860+0000 7f66f1ffb700 1 -- 172.21.15.111:0/1699337176 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f800fb50 con 0x7f66fc0c7580 2023-11-28T12:58:34.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.861+0000 7f66f1ffb700 1 -- 172.21.15.111:0/1699337176 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f66f801d460 con 0x7f66fc0c7580 2023-11-28T12:58:34.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.861+0000 7f6703a06700 1 -- 172.21.15.111:0/1699337176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66fc0af340 con 0x7f66fc0c7580 2023-11-28T12:58:34.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.861+0000 7f66f1ffb700 1 -- 172.21.15.111:0/1699337176 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f66f801d940 con 0x7f66fc0c7580 2023-11-28T12:58:34.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.862+0000 7f6703a06700 1 -- 172.21.15.111:0/1699337176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66fc067170 con 0x7f66fc0c7580 2023-11-28T12:58:34.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.864+0000 7f66f1ffb700 1 -- 172.21.15.111:0/1699337176 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f66fc15cd90 con 0x7f66fc0c7580 2023-11-28T12:58:34.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.952+0000 7f6703a06700 1 -- 172.21.15.111:0/1699337176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f66fc1602c0 con 0x7f66fc0c7580 2023-11-28T12:58:34.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.954+0000 7f66f1ffb700 1 -- 172.21.15.111:0/1699337176 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f66f801dbb0 con 0x7f66fc0c7580 2023-11-28T12:58:34.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.955+0000 7f6703a06700 1 -- 172.21.15.111:0/1699337176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66fc0c7580 msgr2=0x7f66fc0c79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:34.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.955+0000 7f6703a06700 1 --2- 172.21.15.111:0/1699337176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66fc0c7580 0x7f66fc0c79f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f66f80072c0 tx=0x7f66f80075d0 comp rx=0 tx=0).stop 2023-11-28T12:58:34.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.955+0000 7f6703a06700 1 -- 172.21.15.111:0/1699337176 shutdown_connections 2023-11-28T12:58:34.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.955+0000 7f6703a06700 1 --2- 172.21.15.111:0/1699337176 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f66fc0c5840 0x7f66fc0c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:34.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.955+0000 7f6703a06700 1 --2- 172.21.15.111:0/1699337176 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f66e8003a00 0x7f66fc0c3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:34.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.956+0000 7f6703a06700 1 --2- 172.21.15.111:0/1699337176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66fc0c7580 0x7f66fc0c79f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:34.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.956+0000 7f6703a06700 1 -- 172.21.15.111:0/1699337176 >> 172.21.15.111:0/1699337176 conn(0x7f66fc098ee0 msgr2=0x7f66fc06f010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:34.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.956+0000 7f6703a06700 1 -- 172.21.15.111:0/1699337176 shutdown_connections 2023-11-28T12:58:34.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:34.956+0000 7f6703a06700 1 -- 172.21.15.111:0/1699337176 wait complete. 2023-11-28T12:58:34.957 INFO:teuthology.orchestra.run.smithi111.stderr:adjusted tunables profile to default 2023-11-28T12:58:34.968 INFO:tasks.ceph:check_enable_crimson: False 2023-11-28T12:58:34.968 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-28T12:58:34.969 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:34.969 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-28T12:58:34.974 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:34.972+0000 7fbbe43e2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T12:58:34.990 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:34.990+0000 7f2d977bd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T12:58:34.996 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:34.996 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-28T12:58:35.039 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:35.037+0000 7fbbe43e2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T12:58:35.058 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:35.058+0000 7f2d977bd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T12:58:35.062 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:35.062 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-28T12:58:35.114 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:35.112+0000 7fbbe43e2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T12:58:35.130 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:58:35.130 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-28T12:58:35.138 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:35.137+0000 7f2d977bd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T12:58:35.163 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:35.163 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-28T12:58:35.191 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:35.191 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-28T12:58:35.196 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:35.194+0000 7fbbe43e2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T12:58:35.221 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:35.221+0000 7f2d977bd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T12:58:35.222 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:35.222 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-11-28T12:58:35.291 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:58:35.292 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-11-28T12:58:35.348 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:35.346+0000 7fbbe43e2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T12:58:35.363 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new e275bd31-fa21-4fe2-8a50-63c225873180 0 2023-11-28T12:58:35.364 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:35.363+0000 7f2d977bd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T12:58:35.528 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.526+0000 7fa421374700 1 Processor -- start 2023-11-28T12:58:35.528 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.526+0000 7fa421374700 1 -- start start 2023-11-28T12:58:35.530 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.527+0000 7fa421374700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa41c06a980 0x7fa41c0c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:35.530 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.527+0000 7fa421374700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41c159b10 con 0x7fa41c142e80 2023-11-28T12:58:35.531 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.527+0000 7fa421374700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41c143250 con 0x7fa41c0ca430 2023-11-28T12:58:35.531 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.527+0000 7fa421374700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c0c9380 con 0x7fa41c06a980 2023-11-28T12:58:35.531 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.527+0000 7fa41b7fe700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7fa41c0ca430 0x7fa41c0cd0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.149:40026/0 (socket says 172.21.15.149:40026) 2023-11-28T12:58:35.531 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.527+0000 7fa41b7fe700 1 -- 172.21.15.149:0/3415656925 learned_addr learned my addr 172.21.15.149:0/3415656925 (peer_addr_for_me v1:172.21.15.149:0/0) 2023-11-28T12:58:35.531 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.527+0000 7fa41a7fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa41c06a980 0x7fa41c0c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:35.532 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.528+0000 7fa41a7fc700 1 -- 172.21.15.149:0/3415656925 >> v1:172.21.15.149:6789/0 conn(0x7fa41c0ca430 legacy=0x7fa41c0cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:35.532 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.528+0000 7fa419ffb700 1 -- 172.21.15.149:0/3415656925 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795900525 0 0) 0x7fa41c143250 con 0x7fa41c0ca430 2023-11-28T12:58:35.532 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.528+0000 7fa41a7fc700 1 -- 172.21.15.149:0/3415656925 >> v1:172.21.15.111:6789/0 conn(0x7fa41c142e80 legacy=0x7fa41c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:35.532 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.528+0000 7fa41a7fc700 1 -- 172.21.15.149:0/3415656925 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c0ca870 con 0x7fa41c06a980 2023-11-28T12:58:35.532 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.528+0000 7fa41a7fc700 1 --2- 172.21.15.149:0/3415656925 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa41c06a980 0x7fa41c0c9ef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa410009870 tx=0x7fa410009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c921d94c75fe546a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:35.532 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.528+0000 7fa419ffb700 1 -- 172.21.15.149:0/3415656925 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa410011830 con 0x7fa41c06a980 2023-11-28T12:58:35.533 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.528+0000 7fa419ffb700 1 -- 172.21.15.149:0/3415656925 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa410011990 con 0x7fa41c06a980 2023-11-28T12:58:35.533 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.528+0000 7fa419ffb700 1 -- 172.21.15.149:0/3415656925 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa410011d20 con 0x7fa41c06a980 2023-11-28T12:58:35.533 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.529+0000 7fa421374700 1 -- 172.21.15.149:0/3415656925 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa41c06a980 msgr2=0x7fa41c0c9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:35.533 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.529+0000 7fa421374700 1 --2- 172.21.15.149:0/3415656925 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa41c06a980 0x7fa41c0c9ef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa410009870 tx=0x7fa410009b80 comp rx=0 tx=0).stop 2023-11-28T12:58:35.533 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.529+0000 7fa421374700 1 -- 172.21.15.149:0/3415656925 shutdown_connections 2023-11-28T12:58:35.534 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.529+0000 7fa421374700 1 --2- 172.21.15.149:0/3415656925 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa41c06a980 0x7fa41c0c9ef0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.534 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.529+0000 7fa421374700 1 -- 172.21.15.149:0/3415656925 >> 172.21.15.149:0/3415656925 conn(0x7fa41c0cb490 msgr2=0x7fa41c0cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:35.534 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.529+0000 7fa421374700 1 -- 172.21.15.149:0/3415656925 shutdown_connections 2023-11-28T12:58:35.534 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.529+0000 7fa421374700 1 -- 172.21.15.149:0/3415656925 wait complete. 2023-11-28T12:58:35.537 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.530+0000 7fa421374700 1 Processor -- start 2023-11-28T12:58:35.537 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.530+0000 7fa421374700 1 -- start start 2023-11-28T12:58:35.537 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa421374700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa41c06a980 0x7fa41c1e2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:35.537 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa421374700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa41c0ca430 0x7fa41c14a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:35.537 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa421374700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa41c142e80 0x7fa41c0c4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:35.537 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa421374700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c147570 con 0x7fa41c0ca430 2023-11-28T12:58:35.537 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa421374700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c14a580 con 0x7fa41c142e80 2023-11-28T12:58:35.537 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa421374700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c1e2f00 con 0x7fa41c06a980 2023-11-28T12:58:35.537 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa41affd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa41c06a980 0x7fa41c1e2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:35.538 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa41b7fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa41c142e80 0x7fa41c0c4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:35.538 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa41affd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa41c06a980 0x7fa41c1e2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.149:60208/0 (socket says 172.21.15.149:60208) 2023-11-28T12:58:35.538 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa41affd700 1 -- 172.21.15.149:0/2266680761 learned_addr learned my addr 172.21.15.149:0/2266680761 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-11-28T12:58:35.538 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa41a7fc700 1 --2- 172.21.15.149:0/2266680761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa41c0ca430 0x7fa41c14a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:35.538 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa41affd700 1 -- 172.21.15.149:0/2266680761 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa41c142e80 msgr2=0x7fa41c0c4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:35.538 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa41affd700 1 --2- 172.21.15.149:0/2266680761 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa41c142e80 0x7fa41c0c4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.538 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa41affd700 1 -- 172.21.15.149:0/2266680761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa41c0ca430 msgr2=0x7fa41c14a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:35.538 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa41affd700 1 --2- 172.21.15.149:0/2266680761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa41c0ca430 0x7fa41c14a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.538 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa41affd700 1 -- 172.21.15.149:0/2266680761 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4100094a0 con 0x7fa41c06a980 2023-11-28T12:58:35.538 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa41affd700 1 --2- 172.21.15.149:0/2266680761 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa41c06a980 0x7fa41c1e2870 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00c450 tx=0x7fa40c00c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:35.538 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa4037fe700 1 -- 172.21.15.149:0/2266680761 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c0052d0 con 0x7fa41c06a980 2023-11-28T12:58:35.539 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa4037fe700 1 -- 172.21.15.149:0/2266680761 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40c005430 con 0x7fa41c06a980 2023-11-28T12:58:35.539 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa421374700 1 -- 172.21.15.149:0/2266680761 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41c0cc5f0 con 0x7fa41c06a980 2023-11-28T12:58:35.539 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa4037fe700 1 -- 172.21.15.149:0/2266680761 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c0038c0 con 0x7fa41c06a980 2023-11-28T12:58:35.539 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.531+0000 7fa421374700 1 -- 172.21.15.149:0/2266680761 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41c0cab00 con 0x7fa41c06a980 2023-11-28T12:58:35.539 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.532+0000 7fa4037fe700 1 -- 172.21.15.149:0/2266680761 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fa40c019030 con 0x7fa41c06a980 2023-11-28T12:58:35.539 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.532+0000 7fa4037fe700 1 -- 172.21.15.149:0/2266680761 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa40c011af0 con 0x7fa41c06a980 2023-11-28T12:58:35.539 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.533+0000 7fa421374700 1 -- 172.21.15.149:0/2266680761 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41c0cc8d0 con 0x7fa41c06a980 2023-11-28T12:58:35.539 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.535+0000 7fa4037fe700 1 -- 172.21.15.149:0/2266680761 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fa40c017070 con 0x7fa41c06a980 2023-11-28T12:58:35.621 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.618+0000 7fa421374700 1 -- 172.21.15.149:0/2266680761 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e275bd31-fa21-4fe2-8a50-63c225873180", "id": 0} v 0) v1 -- 0x7fa41c06cbd0 con 0x7fa41c06a980 2023-11-28T12:58:35.625 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.624+0000 7fa4037fe700 1 -- 172.21.15.149:0/2266680761 <== mon.2 v2:172.21.15.149:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e275bd31-fa21-4fe2-8a50-63c225873180", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa40c011cb0 con 0x7fa41c06a980 2023-11-28T12:58:35.626 INFO:teuthology.orchestra.run.smithi149.stdout:0 2023-11-28T12:58:35.627 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.625+0000 7fa421374700 1 -- 172.21.15.149:0/2266680761 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa41c06a980 msgr2=0x7fa41c1e2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:35.627 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.625+0000 7fa421374700 1 --2- 172.21.15.149:0/2266680761 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa41c06a980 0x7fa41c1e2870 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00c450 tx=0x7fa40c00c760 comp rx=0 tx=0).stop 2023-11-28T12:58:35.627 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.625+0000 7fa421374700 1 -- 172.21.15.149:0/2266680761 shutdown_connections 2023-11-28T12:58:35.627 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.625+0000 7fa421374700 1 --2- 172.21.15.149:0/2266680761 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa41c142e80 0x7fa41c0c4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.627 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.625+0000 7fa421374700 1 --2- 172.21.15.149:0/2266680761 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa41c06a980 0x7fa41c1e2870 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.627 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.625+0000 7fa421374700 1 --2- 172.21.15.149:0/2266680761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa41c0ca430 0x7fa41c14a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.627 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.625+0000 7fa421374700 1 -- 172.21.15.149:0/2266680761 >> 172.21.15.149:0/2266680761 conn(0x7fa41c0cb490 msgr2=0x7fa41c0710d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:35.627 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.625+0000 7fa421374700 1 -- 172.21.15.149:0/2266680761 shutdown_connections 2023-11-28T12:58:35.627 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.625+0000 7fa421374700 1 -- 172.21.15.149:0/2266680761 wait complete. 2023-11-28T12:58:35.639 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new 457b5866-4c8b-4f07-a3c7-95914260e791 1 2023-11-28T12:58:35.763 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.758+0000 7f2531820700 1 Processor -- start 2023-11-28T12:58:35.763 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.758+0000 7f2531820700 1 -- start start 2023-11-28T12:58:35.763 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.759+0000 7f2531820700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f252c15a5d0 0x7f252c0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:35.763 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.759+0000 7f2531820700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f252c0b0160 0x7f252c0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:35.763 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.759+0000 7f2531820700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252c0ae8b0 con 0x7f252c0cb2d0 2023-11-28T12:58:35.763 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.759+0000 7f2531820700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f252c0cc1b0 con 0x7f252c15a5d0 2023-11-28T12:58:35.764 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.759+0000 7f2531820700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252c0cc390 con 0x7f252c0b0160 2023-11-28T12:58:35.764 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.759+0000 7f252a7fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f252c0b0160 0x7f252c0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:35.764 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.759+0000 7f252b7fe700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f252c0cb2d0 0x7f252c15e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.149:40038/0 (socket says 172.21.15.149:40038) 2023-11-28T12:58:35.764 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.759+0000 7f252b7fe700 1 -- 172.21.15.149:0/2128846478 learned_addr learned my addr 172.21.15.149:0/2128846478 (peer_addr_for_me v1:172.21.15.149:0/0) 2023-11-28T12:58:35.764 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.760+0000 7f252affd700 1 --2- 172.21.15.149:0/2128846478 >> v2:172.21.15.111:3300/0 conn(0x7f252c15a5d0 0x7f252c0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:35.764 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.760+0000 7f2529ffb700 1 -- 172.21.15.149:0/2128846478 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28430999 0 0) 0x7f252c0ae8b0 con 0x7f252c0cb2d0 2023-11-28T12:58:35.764 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.760+0000 7f2529ffb700 1 -- 172.21.15.149:0/2128846478 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25140035b0 con 0x7f252c0cb2d0 2023-11-28T12:58:35.764 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.760+0000 7f252a7fc700 1 -- 172.21.15.149:0/2128846478 >> v2:172.21.15.111:3300/0 conn(0x7f252c15a5d0 msgr2=0x7f252c0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:35.765 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.760+0000 7f252a7fc700 1 --2- 172.21.15.149:0/2128846478 >> v2:172.21.15.111:3300/0 conn(0x7f252c15a5d0 0x7f252c0ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.765 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.760+0000 7f252a7fc700 1 -- 172.21.15.149:0/2128846478 >> v1:172.21.15.149:6789/0 conn(0x7f252c0cb2d0 legacy=0x7f252c15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:35.765 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.760+0000 7f252a7fc700 1 -- 172.21.15.149:0/2128846478 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252c0cc670 con 0x7f252c0b0160 2023-11-28T12:58:35.765 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.760+0000 7f252affd700 1 --2- 172.21.15.149:0/2128846478 >> v2:172.21.15.111:3300/0 conn(0x7f252c15a5d0 0x7f252c0ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:58:35.765 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.760+0000 7f252a7fc700 1 --2- 172.21.15.149:0/2128846478 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f252c0b0160 0x7f252c0ced90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2520009840 tx=0x7f2520009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec5c7504229ea3bd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:35.765 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.760+0000 7f2529ffb700 1 -- 172.21.15.149:0/2128846478 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520011830 con 0x7f252c0b0160 2023-11-28T12:58:35.765 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.760+0000 7f2529ffb700 1 -- 172.21.15.149:0/2128846478 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2520011990 con 0x7f252c0b0160 2023-11-28T12:58:35.765 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.760+0000 7f2529ffb700 1 -- 172.21.15.149:0/2128846478 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520011d20 con 0x7f252c0b0160 2023-11-28T12:58:35.766 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.761+0000 7f2531820700 1 -- 172.21.15.149:0/2128846478 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f252c0b0160 msgr2=0x7f252c0ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:35.766 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.761+0000 7f2531820700 1 --2- 172.21.15.149:0/2128846478 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f252c0b0160 0x7f252c0ced90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2520009840 tx=0x7f2520009b50 comp rx=0 tx=0).stop 2023-11-28T12:58:35.766 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.761+0000 7f2531820700 1 -- 172.21.15.149:0/2128846478 shutdown_connections 2023-11-28T12:58:35.766 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.761+0000 7f2531820700 1 --2- 172.21.15.149:0/2128846478 >> v2:172.21.15.111:3300/0 conn(0x7f252c15a5d0 0x7f252c0ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.766 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.761+0000 7f2531820700 1 --2- 172.21.15.149:0/2128846478 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f252c0b0160 0x7f252c0ced90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.766 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.761+0000 7f2531820700 1 -- 172.21.15.149:0/2128846478 >> 172.21.15.149:0/2128846478 conn(0x7f252c09a040 msgr2=0x7f252c0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:35.766 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.761+0000 7f2531820700 1 -- 172.21.15.149:0/2128846478 shutdown_connections 2023-11-28T12:58:35.766 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.761+0000 7f2531820700 1 -- 172.21.15.149:0/2128846478 wait complete. 2023-11-28T12:58:35.769 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.762+0000 7f2531820700 1 Processor -- start 2023-11-28T12:58:35.769 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.762+0000 7f2531820700 1 -- start start 2023-11-28T12:58:35.769 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f2531820700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f252c0b0160 0x7f252c1f6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:35.770 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f2531820700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f252c0cb2d0 0x7f252c0c8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:35.770 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f2531820700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f252c15a5d0 0x7f252c14c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:35.770 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f2531820700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c0b2500 con 0x7f252c15a5d0 2023-11-28T12:58:35.770 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f2531820700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c0cac20 con 0x7f252c0b0160 2023-11-28T12:58:35.770 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f2531820700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252c14c710 con 0x7f252c0cb2d0 2023-11-28T12:58:35.770 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f252a7fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f252c0cb2d0 0x7f252c0c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:35.770 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f252a7fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f252c0cb2d0 0x7f252c0c8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.149:60222/0 (socket says 172.21.15.149:60222) 2023-11-28T12:58:35.771 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f252a7fc700 1 -- 172.21.15.149:0/862732384 learned_addr learned my addr 172.21.15.149:0/862732384 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-11-28T12:58:35.771 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f252affd700 1 --2- 172.21.15.149:0/862732384 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f252c0b0160 0x7f252c1f6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:35.771 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f252b7fe700 1 --2- 172.21.15.149:0/862732384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f252c15a5d0 0x7f252c14c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:35.771 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f252a7fc700 1 -- 172.21.15.149:0/862732384 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f252c0b0160 msgr2=0x7f252c1f6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:35.771 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f252a7fc700 1 --2- 172.21.15.149:0/862732384 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f252c0b0160 0x7f252c1f6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.771 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f252a7fc700 1 -- 172.21.15.149:0/862732384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f252c15a5d0 msgr2=0x7f252c14c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:35.771 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f252a7fc700 1 --2- 172.21.15.149:0/862732384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f252c15a5d0 0x7f252c14c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.771 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f252a7fc700 1 -- 172.21.15.149:0/862732384 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25200094a0 con 0x7f252c0cb2d0 2023-11-28T12:58:35.771 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f252b7fe700 1 --2- 172.21.15.149:0/862732384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f252c15a5d0 0x7f252c14c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:58:35.771 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f252a7fc700 1 --2- 172.21.15.149:0/862732384 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f252c0cb2d0 0x7f252c0c8520 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2520000c00 tx=0x7f2520004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:35.771 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.763+0000 7f2513fff700 1 -- 172.21.15.149:0/862732384 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25200036a0 con 0x7f252c0cb2d0 2023-11-28T12:58:35.772 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.764+0000 7f2513fff700 1 -- 172.21.15.149:0/862732384 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252001f070 con 0x7f252c0cb2d0 2023-11-28T12:58:35.772 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.764+0000 7f2513fff700 1 -- 172.21.15.149:0/862732384 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520003a40 con 0x7f252c0cb2d0 2023-11-28T12:58:35.772 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.764+0000 7f2531820700 1 -- 172.21.15.149:0/862732384 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252c0ccc30 con 0x7f252c0cb2d0 2023-11-28T12:58:35.772 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.764+0000 7f2531820700 1 -- 172.21.15.149:0/862732384 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252c0cc950 con 0x7f252c0cb2d0 2023-11-28T12:58:35.772 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.764+0000 7f2513fff700 1 -- 172.21.15.149:0/862732384 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f252000f050 con 0x7f252c0cb2d0 2023-11-28T12:58:35.772 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.764+0000 7f2513fff700 1 -- 172.21.15.149:0/862732384 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f2520021070 con 0x7f252c0cb2d0 2023-11-28T12:58:35.772 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.764+0000 7f2531820700 1 -- 172.21.15.149:0/862732384 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f252c0ccf10 con 0x7f252c0cb2d0 2023-11-28T12:58:35.772 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.767+0000 7f2513fff700 1 -- 172.21.15.149:0/862732384 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f252002b020 con 0x7f252c0cb2d0 2023-11-28T12:58:35.853 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.851+0000 7f2531820700 1 -- 172.21.15.149:0/862732384 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "osd new", "uuid": "457b5866-4c8b-4f07-a3c7-95914260e791", "id": 1} v 0) v1 -- 0x7f252c0b2640 con 0x7f252c0cb2d0 2023-11-28T12:58:35.860 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.858+0000 7f2513fff700 1 -- 172.21.15.149:0/862732384 <== mon.2 v2:172.21.15.149:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "457b5866-4c8b-4f07-a3c7-95914260e791", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f252c0b2640 con 0x7f252c0cb2d0 2023-11-28T12:58:35.860 INFO:teuthology.orchestra.run.smithi149.stdout:1 2023-11-28T12:58:35.861 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.859+0000 7f2531820700 1 -- 172.21.15.149:0/862732384 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f252c0cb2d0 msgr2=0x7f252c0c8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:35.861 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.859+0000 7f2531820700 1 --2- 172.21.15.149:0/862732384 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f252c0cb2d0 0x7f252c0c8520 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2520000c00 tx=0x7f2520004750 comp rx=0 tx=0).stop 2023-11-28T12:58:35.861 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.859+0000 7f2531820700 1 -- 172.21.15.149:0/862732384 shutdown_connections 2023-11-28T12:58:35.862 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.859+0000 7f2531820700 1 --2- 172.21.15.149:0/862732384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f252c15a5d0 0x7f252c14c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.862 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.859+0000 7f2531820700 1 --2- 172.21.15.149:0/862732384 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f252c0b0160 0x7f252c1f6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.862 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.859+0000 7f2531820700 1 --2- 172.21.15.149:0/862732384 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f252c0cb2d0 0x7f252c0c8520 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:35.862 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.859+0000 7f2531820700 1 -- 172.21.15.149:0/862732384 >> 172.21.15.149:0/862732384 conn(0x7f252c09a040 msgr2=0x7f252c0d0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:35.862 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.859+0000 7f2531820700 1 -- 172.21.15.149:0/862732384 shutdown_connections 2023-11-28T12:58:35.862 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.859+0000 7f2531820700 1 -- 172.21.15.149:0/862732384 wait complete. 2023-11-28T12:58:35.873 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new 4bb372c7-9bd8-4865-b4ce-b2e09840b087 2 2023-11-28T12:58:35.997 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.995+0000 7fe410955700 1 Processor -- start 2023-11-28T12:58:35.998 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.996+0000 7fe410955700 1 -- start start 2023-11-28T12:58:36.000 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.996+0000 7fe410955700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe40c15a5d0 0x7fe40c0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.996+0000 7fe410955700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 0x7fe40c0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.996+0000 7fe410955700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40c0ae8b0 con 0x7fe40c0b0160 2023-11-28T12:58:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.996+0000 7fe410955700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40c0cc1b0 con 0x7fe40c15a5d0 2023-11-28T12:58:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.996+0000 7fe410955700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0cc390 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.997+0000 7fe40ad9d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 0x7fe40c0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.997+0000 7fe40ad9d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 0x7fe40c0ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.149:60230/0 (socket says 172.21.15.149:60230) 2023-11-28T12:58:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.997+0000 7fe40ad9d700 1 -- 172.21.15.149:0/1201814190 learned_addr learned my addr 172.21.15.149:0/1201814190 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-11-28T12:58:36.001 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.997+0000 7fe40a59c700 1 --2- 172.21.15.149:0/1201814190 >> v2:172.21.15.111:3300/0 conn(0x7fe40c15a5d0 0x7fe40c0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.002 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.997+0000 7fe40ad9d700 1 -- 172.21.15.149:0/1201814190 >> v2:172.21.15.111:3300/0 conn(0x7fe40c15a5d0 msgr2=0x7fe40c0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:36.002 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.997+0000 7fe40ad9d700 1 --2- 172.21.15.149:0/1201814190 >> v2:172.21.15.111:3300/0 conn(0x7fe40c15a5d0 0x7fe40c0ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.002 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.997+0000 7fe40ad9d700 1 -- 172.21.15.149:0/1201814190 >> v1:172.21.15.111:6789/0 conn(0x7fe40c0b0160 legacy=0x7fe40c15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.002 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.997+0000 7fe40ad9d700 1 -- 172.21.15.149:0/1201814190 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c0cc670 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.002 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.997+0000 7fe40ad9d700 1 --2- 172.21.15.149:0/1201814190 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 0x7fe40c0ced90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe404009e50 tx=0x7fe40400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b1bc30f8ad7ae70 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:36.002 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.997+0000 7fe40959a700 1 -- 172.21.15.149:0/1201814190 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40400e070 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.002 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.997+0000 7fe40959a700 1 -- 172.21.15.149:0/1201814190 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40400b910 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.003 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.997+0000 7fe40959a700 1 -- 172.21.15.149:0/1201814190 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40400bcc0 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.003 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.998+0000 7fe410955700 1 -- 172.21.15.149:0/1201814190 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 msgr2=0x7fe40c0ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.003 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.998+0000 7fe410955700 1 --2- 172.21.15.149:0/1201814190 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 0x7fe40c0ced90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe404009e50 tx=0x7fe40400b330 comp rx=0 tx=0).stop 2023-11-28T12:58:36.003 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.998+0000 7fe410955700 1 -- 172.21.15.149:0/1201814190 shutdown_connections 2023-11-28T12:58:36.003 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.998+0000 7fe410955700 1 --2- 172.21.15.149:0/1201814190 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 0x7fe40c0ced90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.003 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.998+0000 7fe410955700 1 --2- 172.21.15.149:0/1201814190 >> v2:172.21.15.111:3300/0 conn(0x7fe40c15a5d0 0x7fe40c0ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.003 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.998+0000 7fe410955700 1 -- 172.21.15.149:0/1201814190 >> 172.21.15.149:0/1201814190 conn(0x7fe40c09a040 msgr2=0x7fe40c0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:36.003 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.998+0000 7fe410955700 1 -- 172.21.15.149:0/1201814190 shutdown_connections 2023-11-28T12:58:36.003 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.998+0000 7fe410955700 1 -- 172.21.15.149:0/1201814190 wait complete. 2023-11-28T12:58:36.007 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.999+0000 7fe410955700 1 Processor -- start 2023-11-28T12:58:36.008 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:35.999+0000 7fe410955700 1 -- start start 2023-11-28T12:58:36.008 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.000+0000 7fe410955700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe40c0b0160 0x7fe40c06cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.008 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.000+0000 7fe410955700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 0x7fe40c06f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.008 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.000+0000 7fe410955700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe40c15a5d0 0x7fe40c06fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.008 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.000+0000 7fe410955700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c15adb0 con 0x7fe40c15a5d0 2023-11-28T12:58:36.009 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.000+0000 7fe410955700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c154b20 con 0x7fe40c0b0160 2023-11-28T12:58:36.009 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.000+0000 7fe410955700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0b2180 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.009 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.000+0000 7fe409d9b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 0x7fe40c06f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.009 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.000+0000 7fe40a59c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe40c0b0160 0x7fe40c06cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.009 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.000+0000 7fe40ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe40c15a5d0 0x7fe40c06fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.009 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.000+0000 7fe409d9b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 0x7fe40c06f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.149:60234/0 (socket says 172.21.15.149:60234) 2023-11-28T12:58:36.009 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.000+0000 7fe40ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe40c15a5d0 0x7fe40c06fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.149:37188/0 (socket says 172.21.15.149:37188) 2023-11-28T12:58:36.009 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.000+0000 7fe409d9b700 1 -- 172.21.15.149:0/3353019715 learned_addr learned my addr 172.21.15.149:0/3353019715 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-11-28T12:58:36.009 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe409d9b700 1 -- 172.21.15.149:0/3353019715 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe40c0b0160 msgr2=0x7fe40c06cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.010 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe409d9b700 1 --2- 172.21.15.149:0/3353019715 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe40c0b0160 0x7fe40c06cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.010 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe409d9b700 1 -- 172.21.15.149:0/3353019715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe40c15a5d0 msgr2=0x7fe40c06fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.010 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe409d9b700 1 --2- 172.21.15.149:0/3353019715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe40c15a5d0 0x7fe40c06fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.010 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe409d9b700 1 -- 172.21.15.149:0/3353019715 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe404009b30 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.010 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe40ad9d700 1 --2- 172.21.15.149:0/3353019715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe40c15a5d0 0x7fe40c06fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:36.010 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe409d9b700 1 --2- 172.21.15.149:0/3353019715 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 0x7fe40c06f6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe40000c4e0 tx=0x7fe40000c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:36.010 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe3faffd700 1 -- 172.21.15.149:0/3353019715 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4000052d0 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.010 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe3faffd700 1 -- 172.21.15.149:0/3353019715 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe400005430 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.010 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe410955700 1 -- 172.21.15.149:0/3353019715 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40c0ccc30 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.011 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe3faffd700 1 -- 172.21.15.149:0/3353019715 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4000038c0 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.011 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe410955700 1 -- 172.21.15.149:0/3353019715 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40c0cc950 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.011 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe3faffd700 1 -- 172.21.15.149:0/3353019715 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fe400019030 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.011 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.001+0000 7fe3faffd700 1 -- 172.21.15.149:0/3353019715 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fe400011be0 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.011 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.002+0000 7fe410955700 1 -- 172.21.15.149:0/3353019715 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40c0ccf10 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.011 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.004+0000 7fe3faffd700 1 -- 172.21.15.149:0/3353019715 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fe400017070 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.039 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:36.039+0000 7f2d977bd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T12:58:36.066 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:36.065+0000 7fbbe43e2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T12:58:36.089 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.087+0000 7fe410955700 1 -- 172.21.15.149:0/3353019715 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4bb372c7-9bd8-4865-b4ce-b2e09840b087", "id": 2} v 0) v1 -- 0x7fe40c0ce650 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.097 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.095+0000 7fe3faffd700 1 -- 172.21.15.149:0/3353019715 <== mon.2 v2:172.21.15.149:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4bb372c7-9bd8-4865-b4ce-b2e09840b087", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe400011df0 con 0x7fe40c0cb2d0 2023-11-28T12:58:36.097 INFO:teuthology.orchestra.run.smithi149.stdout:2 2023-11-28T12:58:36.098 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.096+0000 7fe410955700 1 -- 172.21.15.149:0/3353019715 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 msgr2=0x7fe40c06f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.098 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.096+0000 7fe410955700 1 --2- 172.21.15.149:0/3353019715 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 0x7fe40c06f6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe40000c4e0 tx=0x7fe40000c7f0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.099 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.096+0000 7fe410955700 1 -- 172.21.15.149:0/3353019715 shutdown_connections 2023-11-28T12:58:36.099 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.096+0000 7fe410955700 1 --2- 172.21.15.149:0/3353019715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe40c15a5d0 0x7fe40c06fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.099 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.096+0000 7fe410955700 1 --2- 172.21.15.149:0/3353019715 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe40c0b0160 0x7fe40c06cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.099 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.096+0000 7fe410955700 1 --2- 172.21.15.149:0/3353019715 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe40c0cb2d0 0x7fe40c06f6c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.099 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.096+0000 7fe410955700 1 -- 172.21.15.149:0/3353019715 >> 172.21.15.149:0/3353019715 conn(0x7fe40c09a040 msgr2=0x7fe40c0ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:36.099 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.096+0000 7fe410955700 1 -- 172.21.15.149:0/3353019715 shutdown_connections 2023-11-28T12:58:36.099 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.096+0000 7fe410955700 1 -- 172.21.15.149:0/3353019715 wait complete. 2023-11-28T12:58:36.110 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new bfaea526-60c3-440f-9e13-9befd332e228 3 2023-11-28T12:58:36.114 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:36.114+0000 7f2d977bd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T12:58:36.148 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:36.146+0000 7fbbe43e2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T12:58:36.206 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:36.205+0000 7f2d977bd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T12:58:36.232 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.231+0000 7faee8809700 1 Processor -- start 2023-11-28T12:58:36.233 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.231+0000 7faee8809700 1 -- start start 2023-11-28T12:58:36.235 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee8809700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7faee4154b60 0x7faee415b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.235 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee8809700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faee406a980 0x7faee415df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.235 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee8809700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee406b490 con 0x7faee40cb830 2023-11-28T12:58:36.235 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee8809700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee40c9930 con 0x7faee406a980 2023-11-28T12:58:36.235 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee8809700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee40c9e80 con 0x7faee4154b60 2023-11-28T12:58:36.236 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee259c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7faee4154b60 0x7faee415b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.236 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee259c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7faee4154b60 0x7faee415b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:43094/0 (socket says 172.21.15.149:43094) 2023-11-28T12:58:36.236 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee259c700 1 -- 172.21.15.149:0/3526781486 learned_addr learned my addr 172.21.15.149:0/3526781486 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-11-28T12:58:36.236 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee2d9d700 1 --2- 172.21.15.149:0/3526781486 >> v2:172.21.15.111:3300/0 conn(0x7faee406a980 0x7faee415df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.236 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee259c700 1 -- 172.21.15.149:0/3526781486 >> v2:172.21.15.111:3300/0 conn(0x7faee406a980 msgr2=0x7faee415df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:36.236 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee259c700 1 --2- 172.21.15.149:0/3526781486 >> v2:172.21.15.111:3300/0 conn(0x7faee406a980 0x7faee415df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.236 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee259c700 1 -- 172.21.15.149:0/3526781486 >> v1:172.21.15.111:6789/0 conn(0x7faee40cb830 legacy=0x7faee406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.236 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee259c700 1 -- 172.21.15.149:0/3526781486 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee415b890 con 0x7faee4154b60 2023-11-28T12:58:36.236 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee259c700 1 --2- 172.21.15.149:0/3526781486 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee4154b60 0x7faee415b350 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faee406cbb0 tx=0x7faed4009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf1c693bc812f091 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:36.236 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee159a700 1 -- 172.21.15.149:0/3526781486 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed400b870 con 0x7faee4154b60 2023-11-28T12:58:36.236 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.232+0000 7faee159a700 1 -- 172.21.15.149:0/3526781486 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed400b9d0 con 0x7faee4154b60 2023-11-28T12:58:36.236 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.233+0000 7faee159a700 1 -- 172.21.15.149:0/3526781486 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed400bd60 con 0x7faee4154b60 2023-11-28T12:58:36.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.233+0000 7faee8809700 1 -- 172.21.15.149:0/3526781486 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee4154b60 msgr2=0x7faee415b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.233+0000 7faee8809700 1 --2- 172.21.15.149:0/3526781486 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee4154b60 0x7faee415b350 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faee406cbb0 tx=0x7faed4009bd0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.233+0000 7faee8809700 1 -- 172.21.15.149:0/3526781486 shutdown_connections 2023-11-28T12:58:36.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.233+0000 7faee8809700 1 --2- 172.21.15.149:0/3526781486 >> v2:172.21.15.111:3300/0 conn(0x7faee406a980 0x7faee415df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.233+0000 7faee8809700 1 --2- 172.21.15.149:0/3526781486 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee4154b60 0x7faee415b350 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.233+0000 7faee8809700 1 -- 172.21.15.149:0/3526781486 >> 172.21.15.149:0/3526781486 conn(0x7faee40cf2e0 msgr2=0x7faee40cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:36.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.233+0000 7faee8809700 1 -- 172.21.15.149:0/3526781486 shutdown_connections 2023-11-28T12:58:36.237 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.233+0000 7faee8809700 1 -- 172.21.15.149:0/3526781486 wait complete. 2023-11-28T12:58:36.239 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.234+0000 7faee8809700 1 Processor -- start 2023-11-28T12:58:36.240 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.235+0000 7faee8809700 1 -- start start 2023-11-28T12:58:36.240 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.235+0000 7faee8809700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee406a980 0x7faee4151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.240 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.235+0000 7faee8809700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee40cb830 0x7faee40cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.240 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.235+0000 7faee8809700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee4154b60 0x7faee41f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.240 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.235+0000 7faee8809700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee40ce0f0 con 0x7faee406a980 2023-11-28T12:58:36.240 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.235+0000 7faee8809700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee40aba70 con 0x7faee40cb830 2023-11-28T12:58:36.240 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.235+0000 7faee8809700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee40abd10 con 0x7faee4154b60 2023-11-28T12:58:36.240 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.235+0000 7faee1d9b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee40cb830 0x7faee40cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.240 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.235+0000 7faee2d9d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee4154b60 0x7faee41f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.240 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.235+0000 7faee1d9b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee40cb830 0x7faee40cd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:43102/0 (socket says 172.21.15.149:43102) 2023-11-28T12:58:36.240 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.235+0000 7faee1d9b700 1 -- 172.21.15.149:0/3760825444 learned_addr learned my addr 172.21.15.149:0/3760825444 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-11-28T12:58:36.241 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.235+0000 7faee259c700 1 --2- 172.21.15.149:0/3760825444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee406a980 0x7faee4151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.241 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.236+0000 7faee1d9b700 1 -- 172.21.15.149:0/3760825444 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee4154b60 msgr2=0x7faee41f5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.241 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.236+0000 7faee1d9b700 1 --2- 172.21.15.149:0/3760825444 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee4154b60 0x7faee41f5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.241 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.236+0000 7faee1d9b700 1 -- 172.21.15.149:0/3760825444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee406a980 msgr2=0x7faee4151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.241 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.236+0000 7faee1d9b700 1 --2- 172.21.15.149:0/3760825444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee406a980 0x7faee4151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.241 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.236+0000 7faee1d9b700 1 -- 172.21.15.149:0/3760825444 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed4009550 con 0x7faee40cb830 2023-11-28T12:58:36.241 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.236+0000 7faee1d9b700 1 --2- 172.21.15.149:0/3760825444 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee40cb830 0x7faee40cd8a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faed800c4e0 tx=0x7faed800c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:36.241 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.236+0000 7faed2ffd700 1 -- 172.21.15.149:0/3760825444 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed80052d0 con 0x7faee40cb830 2023-11-28T12:58:36.241 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.236+0000 7faed2ffd700 1 -- 172.21.15.149:0/3760825444 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed8005430 con 0x7faee40cb830 2023-11-28T12:58:36.241 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.236+0000 7faee8809700 1 -- 172.21.15.149:0/3760825444 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee406b9c0 con 0x7faee40cb830 2023-11-28T12:58:36.241 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.236+0000 7faed2ffd700 1 -- 172.21.15.149:0/3760825444 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed80038c0 con 0x7faee40cb830 2023-11-28T12:58:36.241 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.236+0000 7faee8809700 1 -- 172.21.15.149:0/3760825444 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee406b6e0 con 0x7faee40cb830 2023-11-28T12:58:36.242 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.236+0000 7faed2ffd700 1 -- 172.21.15.149:0/3760825444 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7faed8019030 con 0x7faee40cb830 2023-11-28T12:58:36.242 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.236+0000 7faed2ffd700 1 -- 172.21.15.149:0/3760825444 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7faed8011c00 con 0x7faee40cb830 2023-11-28T12:58:36.242 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.237+0000 7faee8809700 1 -- 172.21.15.149:0/3760825444 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee40c9b10 con 0x7faee40cb830 2023-11-28T12:58:36.242 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.239+0000 7faed2ffd700 1 -- 172.21.15.149:0/3760825444 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7faed8017070 con 0x7faee40cb830 2023-11-28T12:58:36.242 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:36.241+0000 7fbbe43e2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T12:58:36.324 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.322+0000 7faee8809700 1 -- 172.21.15.149:0/3760825444 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bfaea526-60c3-440f-9e13-9befd332e228", "id": 3} v 0) v1 -- 0x7faee4067170 con 0x7faee40cb830 2023-11-28T12:58:36.330 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.328+0000 7faed2ffd700 1 -- 172.21.15.149:0/3760825444 <== mon.1 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bfaea526-60c3-440f-9e13-9befd332e228", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7faed8011e60 con 0x7faee40cb830 2023-11-28T12:58:36.330 INFO:teuthology.orchestra.run.smithi149.stdout:3 2023-11-28T12:58:36.331 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.329+0000 7faee8809700 1 -- 172.21.15.149:0/3760825444 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee40cb830 msgr2=0x7faee40cd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.331 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.329+0000 7faee8809700 1 --2- 172.21.15.149:0/3760825444 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee40cb830 0x7faee40cd8a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faed800c4e0 tx=0x7faed800c7f0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.331 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.329+0000 7faee8809700 1 -- 172.21.15.149:0/3760825444 shutdown_connections 2023-11-28T12:58:36.331 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.329+0000 7faee8809700 1 --2- 172.21.15.149:0/3760825444 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee4154b60 0x7faee41f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.331 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.329+0000 7faee8809700 1 --2- 172.21.15.149:0/3760825444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee406a980 0x7faee4151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.332 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.329+0000 7faee8809700 1 --2- 172.21.15.149:0/3760825444 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee40cb830 0x7faee40cd8a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.332 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.329+0000 7faee8809700 1 -- 172.21.15.149:0/3760825444 >> 172.21.15.149:0/3760825444 conn(0x7faee40cf2e0 msgr2=0x7faee406e990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:36.332 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.329+0000 7faee8809700 1 -- 172.21.15.149:0/3760825444 shutdown_connections 2023-11-28T12:58:36.332 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.329+0000 7faee8809700 1 -- 172.21.15.149:0/3760825444 wait complete. 2023-11-28T12:58:36.341 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:36.341+0000 7f2d977bd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T12:58:36.342 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new e82f656b-9400-4bca-b566-10c0b17ad4d3 4 2023-11-28T12:58:36.389 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:36.388+0000 7fbbe43e2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T12:58:36.504 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.502+0000 7fdf66854700 1 Processor -- start 2023-11-28T12:58:36.504 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.502+0000 7fdf66854700 1 -- start start 2023-11-28T12:58:36.506 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.503+0000 7fdf66854700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdf600b2220 0x7fdf600cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.506 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.503+0000 7fdf66854700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fdf600cb380 0x7fdf600ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.506 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.503+0000 7fdf66854700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf600cf280 con 0x7fdf600cee40 2023-11-28T12:58:36.506 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.503+0000 7fdf66854700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf6008ae90 con 0x7fdf600b2220 2023-11-28T12:58:36.506 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.503+0000 7fdf66854700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf600cc7b0 con 0x7fdf600cb380 2023-11-28T12:58:36.506 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.503+0000 7fdf5f7fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fdf600cb380 0x7fdf600ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.506 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.503+0000 7fdf5f7fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fdf600cb380 0x7fdf600ce900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:43106/0 (socket says 172.21.15.149:43106) 2023-11-28T12:58:36.506 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.503+0000 7fdf5f7fe700 1 -- 172.21.15.149:0/1653394186 learned_addr learned my addr 172.21.15.149:0/1653394186 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-11-28T12:58:36.506 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.503+0000 7fdf5ffff700 1 --2- 172.21.15.149:0/1653394186 >> v2:172.21.15.111:3300/0 conn(0x7fdf600b2220 0x7fdf600cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.506 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.504+0000 7fdf5f7fe700 1 -- 172.21.15.149:0/1653394186 >> v2:172.21.15.111:3300/0 conn(0x7fdf600b2220 msgr2=0x7fdf600cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:36.507 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.504+0000 7fdf5f7fe700 1 --2- 172.21.15.149:0/1653394186 >> v2:172.21.15.111:3300/0 conn(0x7fdf600b2220 0x7fdf600cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.507 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.504+0000 7fdf5f7fe700 1 -- 172.21.15.149:0/1653394186 >> v1:172.21.15.111:6789/0 conn(0x7fdf600cee40 legacy=0x7fdf600cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.507 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.504+0000 7fdf5f7fe700 1 -- 172.21.15.149:0/1653394186 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf600b25f0 con 0x7fdf600cb380 2023-11-28T12:58:36.507 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.504+0000 7fdf5f7fe700 1 --2- 172.21.15.149:0/1653394186 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf600cb380 0x7fdf600ce900 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdf540098f0 tx=0x7fdf54009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35f1d7f9d96e16e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:36.507 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.504+0000 7fdf5effd700 1 -- 172.21.15.149:0/1653394186 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54011830 con 0x7fdf600cb380 2023-11-28T12:58:36.507 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.504+0000 7fdf5effd700 1 -- 172.21.15.149:0/1653394186 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf54011990 con 0x7fdf600cb380 2023-11-28T12:58:36.507 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.504+0000 7fdf5effd700 1 -- 172.21.15.149:0/1653394186 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54011d20 con 0x7fdf600cb380 2023-11-28T12:58:36.509 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.505+0000 7fdf66854700 1 -- 172.21.15.149:0/1653394186 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf600cb380 msgr2=0x7fdf600ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.509 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.505+0000 7fdf66854700 1 --2- 172.21.15.149:0/1653394186 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf600cb380 0x7fdf600ce900 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdf540098f0 tx=0x7fdf54009c00 comp rx=0 tx=0).stop 2023-11-28T12:58:36.509 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.505+0000 7fdf66854700 1 -- 172.21.15.149:0/1653394186 shutdown_connections 2023-11-28T12:58:36.509 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.505+0000 7fdf66854700 1 --2- 172.21.15.149:0/1653394186 >> v2:172.21.15.111:3300/0 conn(0x7fdf600b2220 0x7fdf600cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.509 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.505+0000 7fdf66854700 1 --2- 172.21.15.149:0/1653394186 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf600cb380 0x7fdf600ce900 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.510 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.505+0000 7fdf66854700 1 -- 172.21.15.149:0/1653394186 >> 172.21.15.149:0/1653394186 conn(0x7fdf6009a040 msgr2=0x7fdf600d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:36.510 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.505+0000 7fdf66854700 1 -- 172.21.15.149:0/1653394186 shutdown_connections 2023-11-28T12:58:36.510 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.505+0000 7fdf66854700 1 -- 172.21.15.149:0/1653394186 wait complete. 2023-11-28T12:58:36.510 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.506+0000 7fdf66854700 1 Processor -- start 2023-11-28T12:58:36.510 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.506+0000 7fdf66854700 1 -- start start 2023-11-28T12:58:36.510 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf66854700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf600b2220 0x7fdf60158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.510 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf66854700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf600cb380 0x7fdf600c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.510 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf66854700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf600cee40 0x7fdf600c7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.510 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf66854700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf601f6450 con 0x7fdf600cee40 2023-11-28T12:58:36.510 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf66854700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6006d010 con 0x7fdf600b2220 2023-11-28T12:58:36.510 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf66854700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf60158ea0 con 0x7fdf600cb380 2023-11-28T12:58:36.511 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf5ffff700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf600b2220 0x7fdf60158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.511 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf5f7fe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf600cb380 0x7fdf600c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.511 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf5ffff700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf600b2220 0x7fdf60158960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:43112/0 (socket says 172.21.15.149:43112) 2023-11-28T12:58:36.511 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf5ffff700 1 -- 172.21.15.149:0/2941668102 learned_addr learned my addr 172.21.15.149:0/2941668102 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-11-28T12:58:36.511 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf64df1700 1 --2- 172.21.15.149:0/2941668102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf600cee40 0x7fdf600c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.511 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf5f7fe700 1 -- 172.21.15.149:0/2941668102 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf600b2220 msgr2=0x7fdf60158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.511 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf5f7fe700 1 --2- 172.21.15.149:0/2941668102 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf600b2220 0x7fdf60158960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.511 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf5f7fe700 1 -- 172.21.15.149:0/2941668102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf600cee40 msgr2=0x7fdf600c7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.511 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf5f7fe700 1 --2- 172.21.15.149:0/2941668102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf600cee40 0x7fdf600c7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.511 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf5f7fe700 1 -- 172.21.15.149:0/2941668102 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf54009550 con 0x7fdf600cb380 2023-11-28T12:58:36.511 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf5ffff700 1 --2- 172.21.15.149:0/2941668102 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf600b2220 0x7fdf60158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:36.511 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf5f7fe700 1 --2- 172.21.15.149:0/2941668102 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf600cb380 0x7fdf600c6fc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdf54016000 tx=0x7fdf54004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:36.512 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.507+0000 7fdf5cff9700 1 -- 172.21.15.149:0/2941668102 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54004300 con 0x7fdf600cb380 2023-11-28T12:58:36.512 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.508+0000 7fdf5cff9700 1 -- 172.21.15.149:0/2941668102 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5401f070 con 0x7fdf600cb380 2023-11-28T12:58:36.512 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.508+0000 7fdf5cff9700 1 -- 172.21.15.149:0/2941668102 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf540036a0 con 0x7fdf600cb380 2023-11-28T12:58:36.512 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.508+0000 7fdf66854700 1 -- 172.21.15.149:0/2941668102 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf600cccf0 con 0x7fdf600cb380 2023-11-28T12:58:36.512 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.508+0000 7fdf66854700 1 -- 172.21.15.149:0/2941668102 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf600cca10 con 0x7fdf600cb380 2023-11-28T12:58:36.512 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.508+0000 7fdf5cff9700 1 -- 172.21.15.149:0/2941668102 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fdf5402a020 con 0x7fdf600cb380 2023-11-28T12:58:36.512 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.508+0000 7fdf5cff9700 1 -- 172.21.15.149:0/2941668102 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fdf5401aaf0 con 0x7fdf600cb380 2023-11-28T12:58:36.512 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.508+0000 7fdf66854700 1 -- 172.21.15.149:0/2941668102 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf600ccfd0 con 0x7fdf600cb380 2023-11-28T12:58:36.514 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.512+0000 7fdf5cff9700 1 -- 172.21.15.149:0/2941668102 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fdf54015020 con 0x7fdf600cb380 2023-11-28T12:58:36.597 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.595+0000 7fdf66854700 1 -- 172.21.15.149:0/2941668102 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e82f656b-9400-4bca-b566-10c0b17ad4d3", "id": 4} v 0) v1 -- 0x7fdf60067170 con 0x7fdf600cb380 2023-11-28T12:58:36.602 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.601+0000 7fdf5cff9700 1 -- 172.21.15.149:0/2941668102 <== mon.2 v2:172.21.15.149:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e82f656b-9400-4bca-b566-10c0b17ad4d3", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdf5401a5e0 con 0x7fdf600cb380 2023-11-28T12:58:36.603 INFO:teuthology.orchestra.run.smithi149.stdout:4 2023-11-28T12:58:36.604 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.602+0000 7fdf66854700 1 -- 172.21.15.149:0/2941668102 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf600cb380 msgr2=0x7fdf600c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.604 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.602+0000 7fdf66854700 1 --2- 172.21.15.149:0/2941668102 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf600cb380 0x7fdf600c6fc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdf54016000 tx=0x7fdf54004750 comp rx=0 tx=0).stop 2023-11-28T12:58:36.604 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.602+0000 7fdf66854700 1 -- 172.21.15.149:0/2941668102 shutdown_connections 2023-11-28T12:58:36.604 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.602+0000 7fdf66854700 1 --2- 172.21.15.149:0/2941668102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf600cee40 0x7fdf600c7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.604 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.602+0000 7fdf66854700 1 --2- 172.21.15.149:0/2941668102 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf600b2220 0x7fdf60158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.605 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.602+0000 7fdf66854700 1 --2- 172.21.15.149:0/2941668102 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf600cb380 0x7fdf600c6fc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.605 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.602+0000 7fdf66854700 1 -- 172.21.15.149:0/2941668102 >> 172.21.15.149:0/2941668102 conn(0x7fdf6009a040 msgr2=0x7fdf6006ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:36.605 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.602+0000 7fdf66854700 1 -- 172.21.15.149:0/2941668102 shutdown_connections 2023-11-28T12:58:36.605 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.602+0000 7fdf66854700 1 -- 172.21.15.149:0/2941668102 wait complete. 2023-11-28T12:58:36.616 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new 0b2c93fb-4843-4b7a-9407-e4f8fac2ff11 5 2023-11-28T12:58:36.726 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:36.725+0000 7f2d977bd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T12:58:36.778 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.776+0000 7f9ef178c700 1 Processor -- start 2023-11-28T12:58:36.778 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.777+0000 7f9ef178c700 1 -- start start 2023-11-28T12:58:36.782 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.777+0000 7f9ef178c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9eec0b0440 0x7f9eec0b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.782 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.777+0000 7f9ef178c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9eec0ca2f0 0x7f9eec0ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.782 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.777+0000 7f9ef178c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eec0cc750 con 0x7f9eec15c6c0 2023-11-28T12:58:36.782 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.777+0000 7f9ef178c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eec0b0260 con 0x7f9eec0b0440 2023-11-28T12:58:36.782 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.777+0000 7f9ef178c700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eec0b0e20 con 0x7f9eec0ca2f0 2023-11-28T12:58:36.783 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.777+0000 7f9eeb7fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9eec0ca2f0 0x7f9eec0ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.778+0000 7f9eeb7fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9eec0ca2f0 0x7f9eec0ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:43122/0 (socket says 172.21.15.149:43122) 2023-11-28T12:58:36.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.778+0000 7f9eeb7fe700 1 -- 172.21.15.149:0/1670517081 learned_addr learned my addr 172.21.15.149:0/1670517081 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-11-28T12:58:36.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.778+0000 7f9eea7fc700 1 --2- 172.21.15.149:0/1670517081 >> v2:172.21.15.111:3300/0 conn(0x7f9eec0b0440 0x7f9eec0b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.778+0000 7f9ee9ffb700 1 -- 172.21.15.149:0/1670517081 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852167612 0 0) 0x7f9eec0cc750 con 0x7f9eec15c6c0 2023-11-28T12:58:36.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.778+0000 7f9ee9ffb700 1 -- 172.21.15.149:0/1670517081 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ed40035b0 con 0x7f9eec15c6c0 2023-11-28T12:58:36.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.778+0000 7f9eeb7fe700 1 -- 172.21.15.149:0/1670517081 >> v2:172.21.15.111:3300/0 conn(0x7f9eec0b0440 msgr2=0x7f9eec0b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:36.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.778+0000 7f9eeb7fe700 1 --2- 172.21.15.149:0/1670517081 >> v2:172.21.15.111:3300/0 conn(0x7f9eec0b0440 0x7f9eec0b08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.778+0000 7f9eeb7fe700 1 -- 172.21.15.149:0/1670517081 >> v1:172.21.15.111:6789/0 conn(0x7f9eec15c6c0 legacy=0x7f9eec0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.778+0000 7f9eeb7fe700 1 -- 172.21.15.149:0/1670517081 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec0cca30 con 0x7f9eec0ca2f0 2023-11-28T12:58:36.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.778+0000 7f9eeb7fe700 1 --2- 172.21.15.149:0/1670517081 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9eec0ca2f0 0x7f9eec0ca750 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9eec071ae0 tx=0x7f9ee400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8c810912a52bdd3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:36.785 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.778+0000 7f9ee9ffb700 1 -- 172.21.15.149:0/1670517081 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee400e070 con 0x7f9eec0ca2f0 2023-11-28T12:58:36.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.778+0000 7f9ee9ffb700 1 -- 172.21.15.149:0/1670517081 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee400b910 con 0x7f9eec0ca2f0 2023-11-28T12:58:36.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.778+0000 7f9ee9ffb700 1 -- 172.21.15.149:0/1670517081 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee400bcc0 con 0x7f9eec0ca2f0 2023-11-28T12:58:36.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.779+0000 7f9ef178c700 1 -- 172.21.15.149:0/1670517081 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9eec0ca2f0 msgr2=0x7f9eec0ca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.779+0000 7f9ef178c700 1 --2- 172.21.15.149:0/1670517081 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9eec0ca2f0 0x7f9eec0ca750 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9eec071ae0 tx=0x7f9ee400b330 comp rx=0 tx=0).stop 2023-11-28T12:58:36.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.779+0000 7f9ef178c700 1 -- 172.21.15.149:0/1670517081 shutdown_connections 2023-11-28T12:58:36.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.779+0000 7f9ef178c700 1 --2- 172.21.15.149:0/1670517081 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9eec0ca2f0 0x7f9eec0ca750 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.779+0000 7f9ef178c700 1 --2- 172.21.15.149:0/1670517081 >> v2:172.21.15.111:3300/0 conn(0x7f9eec0b0440 0x7f9eec0b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.779+0000 7f9ef178c700 1 -- 172.21.15.149:0/1670517081 >> 172.21.15.149:0/1670517081 conn(0x7f9eec098ee0 msgr2=0x7f9eec06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:36.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.779+0000 7f9ef178c700 1 -- 172.21.15.149:0/1670517081 shutdown_connections 2023-11-28T12:58:36.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.779+0000 7f9ef178c700 1 -- 172.21.15.149:0/1670517081 wait complete. 2023-11-28T12:58:36.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.780+0000 7f9ef178c700 1 Processor -- start 2023-11-28T12:58:36.786 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.780+0000 7f9ef178c700 1 -- start start 2023-11-28T12:58:36.789 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9ef178c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9eec0b0440 0x7f9eec155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.789 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9ef178c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eec0ca2f0 0x7f9eec0cf850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.789 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9ef178c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9eec15c6c0 0x7f9eec0b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:36.789 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9ef178c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec155740 con 0x7f9eec0ca2f0 2023-11-28T12:58:36.789 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9ef178c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec1561b0 con 0x7f9eec0b0440 2023-11-28T12:58:36.789 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9ef178c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec156730 con 0x7f9eec15c6c0 2023-11-28T12:58:36.789 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9eeaffd700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9eec0b0440 0x7f9eec155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.789 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9eeaffd700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9eec0b0440 0x7f9eec155200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:43124/0 (socket says 172.21.15.149:43124) 2023-11-28T12:58:36.789 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9eeb7fe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9eec15c6c0 0x7f9eec0b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.789 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9eeaffd700 1 -- 172.21.15.149:0/3624264109 learned_addr learned my addr 172.21.15.149:0/3624264109 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-11-28T12:58:36.789 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9eea7fc700 1 --2- 172.21.15.149:0/3624264109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eec0ca2f0 0x7f9eec0cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:36.790 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9eeaffd700 1 -- 172.21.15.149:0/3624264109 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9eec15c6c0 msgr2=0x7f9eec0b4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.790 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9eeaffd700 1 --2- 172.21.15.149:0/3624264109 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9eec15c6c0 0x7f9eec0b4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.790 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9eeaffd700 1 -- 172.21.15.149:0/3624264109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eec0ca2f0 msgr2=0x7f9eec0cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.790 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9eeaffd700 1 --2- 172.21.15.149:0/3624264109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eec0ca2f0 0x7f9eec0cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.790 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9eeaffd700 1 -- 172.21.15.149:0/3624264109 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee4009b30 con 0x7f9eec0b0440 2023-11-28T12:58:36.790 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.781+0000 7f9eeb7fe700 1 --2- 172.21.15.149:0/3624264109 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9eec15c6c0 0x7f9eec0b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:36.790 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.782+0000 7f9eeaffd700 1 --2- 172.21.15.149:0/3624264109 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9eec0b0440 0x7f9eec155200 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c520 tx=0x7f9edc00c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:36.790 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.782+0000 7f9ed37fe700 1 -- 172.21.15.149:0/3624264109 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc0052d0 con 0x7f9eec0b0440 2023-11-28T12:58:36.790 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.782+0000 7f9ed37fe700 1 -- 172.21.15.149:0/3624264109 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9edc005430 con 0x7f9eec0b0440 2023-11-28T12:58:36.790 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.782+0000 7f9ef178c700 1 -- 172.21.15.149:0/3624264109 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eec0ccff0 con 0x7f9eec0b0440 2023-11-28T12:58:36.790 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.782+0000 7f9ed37fe700 1 -- 172.21.15.149:0/3624264109 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc0038c0 con 0x7f9eec0b0440 2023-11-28T12:58:36.790 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.782+0000 7f9ef178c700 1 -- 172.21.15.149:0/3624264109 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eec0ccd10 con 0x7f9eec0b0440 2023-11-28T12:58:36.790 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.782+0000 7f9ed37fe700 1 -- 172.21.15.149:0/3624264109 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f9edc019030 con 0x7f9eec0b0440 2023-11-28T12:58:36.791 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.782+0000 7f9ed37fe700 1 -- 172.21.15.149:0/3624264109 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f9edc0118b0 con 0x7f9eec0b0440 2023-11-28T12:58:36.791 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.783+0000 7f9ef178c700 1 -- 172.21.15.149:0/3624264109 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eec0cd2d0 con 0x7f9eec0b0440 2023-11-28T12:58:36.791 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.785+0000 7f9ed37fe700 1 -- 172.21.15.149:0/3624264109 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f9edc017070 con 0x7f9eec0b0440 2023-11-28T12:58:36.810 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:36.809+0000 7fbbe43e2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T12:58:36.872 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.870+0000 7f9ef178c700 1 -- 172.21.15.149:0/3624264109 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0b2c93fb-4843-4b7a-9407-e4f8fac2ff11", "id": 5} v 0) v1 -- 0x7f9eec0cdf80 con 0x7f9eec0b0440 2023-11-28T12:58:36.882 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.881+0000 7f9ed37fe700 1 -- 172.21.15.149:0/3624264109 <== mon.1 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0b2c93fb-4843-4b7a-9407-e4f8fac2ff11", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9edc011b50 con 0x7f9eec0b0440 2023-11-28T12:58:36.883 INFO:teuthology.orchestra.run.smithi149.stdout:5 2023-11-28T12:58:36.884 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.882+0000 7f9ef178c700 1 -- 172.21.15.149:0/3624264109 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9eec0b0440 msgr2=0x7f9eec155200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:36.884 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.882+0000 7f9ef178c700 1 --2- 172.21.15.149:0/3624264109 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9eec0b0440 0x7f9eec155200 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c520 tx=0x7f9edc00c830 comp rx=0 tx=0).stop 2023-11-28T12:58:36.884 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.882+0000 7f9ef178c700 1 -- 172.21.15.149:0/3624264109 shutdown_connections 2023-11-28T12:58:36.884 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.882+0000 7f9ef178c700 1 --2- 172.21.15.149:0/3624264109 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9eec15c6c0 0x7f9eec0b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.884 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.882+0000 7f9ef178c700 1 --2- 172.21.15.149:0/3624264109 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9eec0b0440 0x7f9eec155200 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.884 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.882+0000 7f9ef178c700 1 --2- 172.21.15.149:0/3624264109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9eec0ca2f0 0x7f9eec0cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:36.884 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.882+0000 7f9ef178c700 1 -- 172.21.15.149:0/3624264109 >> 172.21.15.149:0/3624264109 conn(0x7f9eec098ee0 msgr2=0x7f9eec158640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:36.884 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.882+0000 7f9ef178c700 1 -- 172.21.15.149:0/3624264109 shutdown_connections 2023-11-28T12:58:36.884 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:36.882+0000 7f9ef178c700 1 -- 172.21.15.149:0/3624264109 wait complete. 2023-11-28T12:58:36.895 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new a7aa32aa-5916-4040-9c3a-cadf59085b16 6 2023-11-28T12:58:37.017 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.015+0000 7f85ecf9d700 1 Processor -- start 2023-11-28T12:58:37.017 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.015+0000 7f85ecf9d700 1 -- start start 2023-11-28T12:58:37.019 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85ecf9d700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f85e815c8f0 0x7f85e80ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.019 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85ecf9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f85e80b1b90 0x7f85e80cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.020 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85ecf9d700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e80cabd0 con 0x7f85e80cd5d0 2023-11-28T12:58:37.020 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85ecf9d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e80cde00 con 0x7f85e80b1b90 2023-11-28T12:58:37.020 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85ecf9d700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e80cdb20 con 0x7f85e815c8f0 2023-11-28T12:58:37.020 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85e77fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f85e815c8f0 0x7f85e80ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.020 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85e6ffd700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f85e80cd5d0 0x7f85e80b1480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.149:40094/0 (socket says 172.21.15.149:40094) 2023-11-28T12:58:37.020 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85e6ffd700 1 -- 172.21.15.149:0/2308971561 learned_addr learned my addr 172.21.15.149:0/2308971561 (peer_addr_for_me v1:172.21.15.149:0/0) 2023-11-28T12:58:37.020 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85e7fff700 1 --2- 172.21.15.149:0/2308971561 >> v2:172.21.15.111:3300/0 conn(0x7f85e80b1b90 0x7f85e80cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.020 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85e77fe700 1 -- 172.21.15.149:0/2308971561 >> v2:172.21.15.111:3300/0 conn(0x7f85e80b1b90 msgr2=0x7f85e80cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:37.020 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85e77fe700 1 --2- 172.21.15.149:0/2308971561 >> v2:172.21.15.111:3300/0 conn(0x7f85e80b1b90 0x7f85e80cb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.020 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85e77fe700 1 -- 172.21.15.149:0/2308971561 >> v1:172.21.15.149:6789/0 conn(0x7f85e80cd5d0 legacy=0x7f85e80b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.020 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85e77fe700 1 -- 172.21.15.149:0/2308971561 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e80cae00 con 0x7f85e815c8f0 2023-11-28T12:58:37.020 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85e67fc700 1 -- 172.21.15.149:0/2308971561 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522076892 0 0) 0x7f85e80cabd0 con 0x7f85e80cd5d0 2023-11-28T12:58:37.020 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.016+0000 7f85e77fe700 1 --2- 172.21.15.149:0/2308971561 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85e815c8f0 0x7f85e80ce9f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f85e8071640 tx=0x7f85d8009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68ce06e7b918292a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:37.021 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.017+0000 7f85e67fc700 1 -- 172.21.15.149:0/2308971561 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d8011830 con 0x7f85e815c8f0 2023-11-28T12:58:37.021 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.017+0000 7f85e67fc700 1 -- 172.21.15.149:0/2308971561 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d8011990 con 0x7f85e815c8f0 2023-11-28T12:58:37.021 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.017+0000 7f85e67fc700 1 -- 172.21.15.149:0/2308971561 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d8011d20 con 0x7f85e815c8f0 2023-11-28T12:58:37.021 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.017+0000 7f85ecf9d700 1 -- 172.21.15.149:0/2308971561 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85e815c8f0 msgr2=0x7f85e80ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.021 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.017+0000 7f85ecf9d700 1 --2- 172.21.15.149:0/2308971561 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85e815c8f0 0x7f85e80ce9f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f85e8071640 tx=0x7f85d8009b20 comp rx=0 tx=0).stop 2023-11-28T12:58:37.021 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.017+0000 7f85ecf9d700 1 -- 172.21.15.149:0/2308971561 shutdown_connections 2023-11-28T12:58:37.021 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.017+0000 7f85ecf9d700 1 --2- 172.21.15.149:0/2308971561 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85e815c8f0 0x7f85e80ce9f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.021 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.017+0000 7f85ecf9d700 1 --2- 172.21.15.149:0/2308971561 >> v2:172.21.15.111:3300/0 conn(0x7f85e80b1b90 0x7f85e80cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.021 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.017+0000 7f85ecf9d700 1 -- 172.21.15.149:0/2308971561 >> 172.21.15.149:0/2308971561 conn(0x7f85e806bfb0 msgr2=0x7f85e806d950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:37.021 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.017+0000 7f85ecf9d700 1 -- 172.21.15.149:0/2308971561 shutdown_connections 2023-11-28T12:58:37.021 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.017+0000 7f85ecf9d700 1 -- 172.21.15.149:0/2308971561 wait complete. 2023-11-28T12:58:37.024 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.019+0000 7f85ecf9d700 1 Processor -- start 2023-11-28T12:58:37.024 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.019+0000 7f85ecf9d700 1 -- start start 2023-11-28T12:58:37.024 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.019+0000 7f85ecf9d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f85e80cd5d0 0x7f85e80b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.024 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.019+0000 7f85ecf9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85e815c8f0 0x7f85e8157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.024 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.019+0000 7f85ecf9d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85e80b5000 0x7f85e80c4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.024 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.019+0000 7f85ecf9d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e80ac030 con 0x7f85e815c8f0 2023-11-28T12:58:37.024 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.019+0000 7f85ecf9d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e80b4290 con 0x7f85e80b5000 2023-11-28T12:58:37.024 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.019+0000 7f85ecf9d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e80b4570 con 0x7f85e80cd5d0 2023-11-28T12:58:37.024 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85e7fff700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85e80b5000 0x7f85e80c4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.024 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85e77fe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f85e80cd5d0 0x7f85e80b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.024 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85e7fff700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85e80b5000 0x7f85e80c4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:43140/0 (socket says 172.21.15.149:43140) 2023-11-28T12:58:37.024 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85e7fff700 1 -- 172.21.15.149:0/4197243877 learned_addr learned my addr 172.21.15.149:0/4197243877 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-11-28T12:58:37.025 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85e6ffd700 1 --2- 172.21.15.149:0/4197243877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85e815c8f0 0x7f85e8157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.025 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85e77fe700 1 -- 172.21.15.149:0/4197243877 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85e80b5000 msgr2=0x7f85e80c4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.025 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85e77fe700 1 --2- 172.21.15.149:0/4197243877 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85e80b5000 0x7f85e80c4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.025 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85e77fe700 1 -- 172.21.15.149:0/4197243877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85e815c8f0 msgr2=0x7f85e8157080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.025 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85e77fe700 1 --2- 172.21.15.149:0/4197243877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85e815c8f0 0x7f85e8157080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.025 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85e77fe700 1 -- 172.21.15.149:0/4197243877 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d80094a0 con 0x7f85e80cd5d0 2023-11-28T12:58:37.025 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85e6ffd700 1 --2- 172.21.15.149:0/4197243877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85e815c8f0 0x7f85e8157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:37.025 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85e7fff700 1 --2- 172.21.15.149:0/4197243877 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85e80b5000 0x7f85e80c4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:37.025 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85e77fe700 1 --2- 172.21.15.149:0/4197243877 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f85e80cd5d0 0x7f85e80b39b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f85d8016000 tx=0x7f85d8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:37.025 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85cffff700 1 -- 172.21.15.149:0/4197243877 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d80036a0 con 0x7f85e80cd5d0 2023-11-28T12:58:37.025 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85cffff700 1 -- 172.21.15.149:0/4197243877 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d801e070 con 0x7f85e80cd5d0 2023-11-28T12:58:37.025 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85ecf9d700 1 -- 172.21.15.149:0/4197243877 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e80cb3c0 con 0x7f85e80cd5d0 2023-11-28T12:58:37.025 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85cffff700 1 -- 172.21.15.149:0/4197243877 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d8003a40 con 0x7f85e80cd5d0 2023-11-28T12:58:37.026 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.020+0000 7f85ecf9d700 1 -- 172.21.15.149:0/4197243877 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e80cb0e0 con 0x7f85e80cd5d0 2023-11-28T12:58:37.026 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.021+0000 7f85cffff700 1 -- 172.21.15.149:0/4197243877 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f85d800f050 con 0x7f85e80cd5d0 2023-11-28T12:58:37.026 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.021+0000 7f85cffff700 1 -- 172.21.15.149:0/4197243877 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f85d802de40 con 0x7f85e80cd5d0 2023-11-28T12:58:37.026 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.021+0000 7f85ecf9d700 1 -- 172.21.15.149:0/4197243877 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e815ccc0 con 0x7f85e80cd5d0 2023-11-28T12:58:37.026 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.023+0000 7f85cffff700 1 -- 172.21.15.149:0/4197243877 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f85d8020070 con 0x7f85e80cd5d0 2023-11-28T12:58:37.110 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.108+0000 7f85ecf9d700 1 -- 172.21.15.149:0/4197243877 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a7aa32aa-5916-4040-9c3a-cadf59085b16", "id": 6} v 0) v1 -- 0x7f85e804e620 con 0x7f85e80cd5d0 2023-11-28T12:58:37.118 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.116+0000 7f85cffff700 1 -- 172.21.15.149:0/4197243877 <== mon.2 v2:172.21.15.149:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a7aa32aa-5916-4040-9c3a-cadf59085b16", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f85d8015020 con 0x7f85e80cd5d0 2023-11-28T12:58:37.118 INFO:teuthology.orchestra.run.smithi149.stdout:6 2023-11-28T12:58:37.119 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.117+0000 7f85ecf9d700 1 -- 172.21.15.149:0/4197243877 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f85e80cd5d0 msgr2=0x7f85e80b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.119 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.117+0000 7f85ecf9d700 1 --2- 172.21.15.149:0/4197243877 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f85e80cd5d0 0x7f85e80b39b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f85d8016000 tx=0x7f85d8004750 comp rx=0 tx=0).stop 2023-11-28T12:58:37.119 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.117+0000 7f85ecf9d700 1 -- 172.21.15.149:0/4197243877 shutdown_connections 2023-11-28T12:58:37.120 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.117+0000 7f85ecf9d700 1 --2- 172.21.15.149:0/4197243877 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85e80b5000 0x7f85e80c4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.120 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.117+0000 7f85ecf9d700 1 --2- 172.21.15.149:0/4197243877 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f85e80cd5d0 0x7f85e80b39b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.120 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.117+0000 7f85ecf9d700 1 --2- 172.21.15.149:0/4197243877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85e815c8f0 0x7f85e8157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.120 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.117+0000 7f85ecf9d700 1 -- 172.21.15.149:0/4197243877 >> 172.21.15.149:0/4197243877 conn(0x7f85e80b1b90 msgr2=0x7f85e806d950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:37.120 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.117+0000 7f85ecf9d700 1 -- 172.21.15.149:0/4197243877 shutdown_connections 2023-11-28T12:58:37.120 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.117+0000 7f85ecf9d700 1 -- 172.21.15.149:0/4197243877 wait complete. 2023-11-28T12:58:37.131 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster ceph osd new 22bb9d02-3198-40c1-b318-120d1ca3224f 7 2023-11-28T12:58:37.267 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.266+0000 7ff27d58c700 1 Processor -- start 2023-11-28T12:58:37.270 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.266+0000 7ff27d58c700 1 -- start start 2023-11-28T12:58:37.270 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.267+0000 7ff27d58c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff2780c9ed0 0x7ff27806a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.270 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.267+0000 7ff27d58c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff2780cdee0 0x7ff27806c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.270 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.267+0000 7ff27d58c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27806ab80 con 0x7ff27806a270 2023-11-28T12:58:37.270 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.267+0000 7ff27d58c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2780ccb60 con 0x7ff2780c9ed0 2023-11-28T12:58:37.270 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.267+0000 7ff27d58c700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2780cce40 con 0x7ff2780cdee0 2023-11-28T12:58:37.270 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.267+0000 7ff2777fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff2780cdee0 0x7ff27806c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.271 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.267+0000 7ff2777fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff2780cdee0 0x7ff27806c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:43146/0 (socket says 172.21.15.149:43146) 2023-11-28T12:58:37.271 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.267+0000 7ff2777fe700 1 -- 172.21.15.149:0/1413834317 learned_addr learned my addr 172.21.15.149:0/1413834317 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-11-28T12:58:37.271 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.267+0000 7ff2777fe700 1 -- 172.21.15.149:0/1413834317 >> v2:172.21.15.111:3300/0 conn(0x7ff2780c9ed0 msgr2=0x7ff27806a640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T12:58:37.271 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.267+0000 7ff2777fe700 1 --2- 172.21.15.149:0/1413834317 >> v2:172.21.15.111:3300/0 conn(0x7ff2780c9ed0 0x7ff27806a640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.271 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.267+0000 7ff2777fe700 1 -- 172.21.15.149:0/1413834317 >> v1:172.21.15.111:6789/0 conn(0x7ff27806a270 legacy=0x7ff27815ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.271 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.268+0000 7ff2777fe700 1 -- 172.21.15.149:0/1413834317 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2780cd120 con 0x7ff2780cdee0 2023-11-28T12:58:37.271 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.268+0000 7ff2777fe700 1 --2- 172.21.15.149:0/1413834317 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff2780cdee0 0x7ff27806c6a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff2780cc4b0 tx=0x7ff27000bbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10c9e3d929f43a99 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:37.271 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.268+0000 7ff275ffb700 1 -- 172.21.15.149:0/1413834317 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff270011670 con 0x7ff2780cdee0 2023-11-28T12:58:37.271 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.268+0000 7ff275ffb700 1 -- 172.21.15.149:0/1413834317 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2700117d0 con 0x7ff2780cdee0 2023-11-28T12:58:37.271 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.268+0000 7ff275ffb700 1 -- 172.21.15.149:0/1413834317 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff270011b80 con 0x7ff2780cdee0 2023-11-28T12:58:37.272 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.269+0000 7ff27d58c700 1 -- 172.21.15.149:0/1413834317 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff2780cdee0 msgr2=0x7ff27806c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.272 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.269+0000 7ff27d58c700 1 --2- 172.21.15.149:0/1413834317 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff2780cdee0 0x7ff27806c6a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff2780cc4b0 tx=0x7ff27000bbc0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.272 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.269+0000 7ff27d58c700 1 -- 172.21.15.149:0/1413834317 shutdown_connections 2023-11-28T12:58:37.272 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.269+0000 7ff27d58c700 1 --2- 172.21.15.149:0/1413834317 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff2780cdee0 0x7ff27806c6a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.272 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.269+0000 7ff27d58c700 1 --2- 172.21.15.149:0/1413834317 >> v2:172.21.15.111:3300/0 conn(0x7ff2780c9ed0 0x7ff27806a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.272 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.269+0000 7ff27d58c700 1 -- 172.21.15.149:0/1413834317 >> 172.21.15.149:0/1413834317 conn(0x7ff2780cb490 msgr2=0x7ff2780cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:37.281 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:37.281+0000 7f2d977bd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T12:58:37.292 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.291+0000 7ff27d58c700 1 -- 172.21.15.149:0/1413834317 shutdown_connections 2023-11-28T12:58:37.305 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.302+0000 7ff27d58c700 1 -- 172.21.15.149:0/1413834317 wait complete. 2023-11-28T12:58:37.305 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.303+0000 7ff27d58c700 1 Processor -- start 2023-11-28T12:58:37.306 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.304+0000 7ff27d58c700 1 -- start start 2023-11-28T12:58:37.308 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.304+0000 7ff27d58c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff27806a270 0x7ff2781feef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.308 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff27d58c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff2780cdee0 0x7ff2780c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.308 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff27d58c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff2780c2490 0x7ff27806d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.308 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff27d58c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2781ff430 con 0x7ff27806a270 2023-11-28T12:58:37.308 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff27d58c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780c88e0 con 0x7ff2780c2490 2023-11-28T12:58:37.309 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff27d58c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780c3480 con 0x7ff2780cdee0 2023-11-28T12:58:37.309 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff2767fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff2780cdee0 0x7ff2780c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.309 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff2777fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff2780c2490 0x7ff27806d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.309 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff2777fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff2780c2490 0x7ff27806d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.149:43150/0 (socket says 172.21.15.149:43150) 2023-11-28T12:58:37.309 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff2767fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff2780cdee0 0x7ff2780c83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.149:60292/0 (socket says 172.21.15.149:60292) 2023-11-28T12:58:37.309 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff2777fe700 1 -- 172.21.15.149:0/2776910919 learned_addr learned my addr 172.21.15.149:0/2776910919 (peer_addr_for_me v2:172.21.15.149:0/0) 2023-11-28T12:58:37.309 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff276ffd700 1 --2- 172.21.15.149:0/2776910919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff27806a270 0x7ff2781feef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.309 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff2777fe700 1 -- 172.21.15.149:0/2776910919 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff2780cdee0 msgr2=0x7ff2780c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.309 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff2777fe700 1 --2- 172.21.15.149:0/2776910919 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff2780cdee0 0x7ff2780c83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.309 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff2777fe700 1 -- 172.21.15.149:0/2776910919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff27806a270 msgr2=0x7ff2781feef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.309 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff2777fe700 1 --2- 172.21.15.149:0/2776910919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff27806a270 0x7ff2781feef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff2777fe700 1 -- 172.21.15.149:0/2776910919 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff270009b30 con 0x7ff2780c2490 2023-11-28T12:58:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff2767fc700 1 --2- 172.21.15.149:0/2776910919 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff2780cdee0 0x7ff2780c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff2777fe700 1 --2- 172.21.15.149:0/2776910919 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff2780c2490 0x7ff27806d490 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff270009fd0 tx=0x7ff270004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff25f7fe700 1 -- 172.21.15.149:0/2776910919 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27000b270 con 0x7ff2780c2490 2023-11-28T12:58:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff25f7fe700 1 -- 172.21.15.149:0/2776910919 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27000b3d0 con 0x7ff2780c2490 2023-11-28T12:58:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff27d58c700 1 -- 172.21.15.149:0/2776910919 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2780cd6e0 con 0x7ff2780c2490 2023-11-28T12:58:37.310 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff25f7fe700 1 -- 172.21.15.149:0/2776910919 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2700099b0 con 0x7ff2780c2490 2023-11-28T12:58:37.343 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.305+0000 7ff27d58c700 1 -- 172.21.15.149:0/2776910919 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2780cd400 con 0x7ff2780c2490 2023-11-28T12:58:37.343 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.306+0000 7ff25f7fe700 1 -- 172.21.15.149:0/2776910919 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7ff27002b070 con 0x7ff2780c2490 2023-11-28T12:58:37.343 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.306+0000 7ff25f7fe700 1 -- 172.21.15.149:0/2776910919 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7ff27001aa80 con 0x7ff2780c2490 2023-11-28T12:58:37.343 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.306+0000 7ff27d58c700 1 -- 172.21.15.149:0/2776910919 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2780cd9c0 con 0x7ff2780c2490 2023-11-28T12:58:37.344 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.310+0000 7ff25f7fe700 1 -- 172.21.15.149:0/2776910919 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7ff270029060 con 0x7ff2780c2490 2023-11-28T12:58:37.350 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:37.350+0000 7f2d977bd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T12:58:37.395 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.394+0000 7ff27d58c700 1 -- 172.21.15.149:0/2776910919 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd new", "uuid": "22bb9d02-3198-40c1-b318-120d1ca3224f", "id": 7} v 0) v1 -- 0x7ff278067880 con 0x7ff2780c2490 2023-11-28T12:58:37.402 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:37.401+0000 7fbbe43e2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T12:58:37.406 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.405+0000 7ff25f7fe700 1 -- 172.21.15.149:0/2776910919 <== mon.1 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "22bb9d02-3198-40c1-b318-120d1ca3224f", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff27001ad20 con 0x7ff2780c2490 2023-11-28T12:58:37.406 INFO:teuthology.orchestra.run.smithi149.stdout:7 2023-11-28T12:58:37.407 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.406+0000 7ff27d58c700 1 -- 172.21.15.149:0/2776910919 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff2780c2490 msgr2=0x7ff27806d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.408 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.406+0000 7ff27d58c700 1 --2- 172.21.15.149:0/2776910919 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff2780c2490 0x7ff27806d490 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff270009fd0 tx=0x7ff270004ad0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.408 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.406+0000 7ff27d58c700 1 -- 172.21.15.149:0/2776910919 shutdown_connections 2023-11-28T12:58:37.408 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.406+0000 7ff27d58c700 1 --2- 172.21.15.149:0/2776910919 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff2780c2490 0x7ff27806d490 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.408 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.406+0000 7ff27d58c700 1 --2- 172.21.15.149:0/2776910919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff27806a270 0x7ff2781feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.408 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.406+0000 7ff27d58c700 1 --2- 172.21.15.149:0/2776910919 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff2780cdee0 0x7ff2780c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.408 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.406+0000 7ff27d58c700 1 -- 172.21.15.149:0/2776910919 >> 172.21.15.149:0/2776910919 conn(0x7ff2780cb490 msgr2=0x7ff2780cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:37.408 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.406+0000 7ff27d58c700 1 -- 172.21.15.149:0/2776910919 shutdown_connections 2023-11-28T12:58:37.408 INFO:teuthology.orchestra.run.smithi149.stderr:2023-11-28T12:58:37.406+0000 7ff27d58c700 1 -- 172.21.15.149:0/2776910919 wait complete. 2023-11-28T12:58:37.419 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-28T12:58:37.420 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-11-28T12:58:37.422 INFO:tasks.ceph.osd.0:Started 2023-11-28T12:58:37.422 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-28T12:58:37.422 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-11-28T12:58:37.424 INFO:tasks.ceph.osd.1:Started 2023-11-28T12:58:37.424 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-28T12:58:37.424 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-11-28T12:58:37.426 INFO:tasks.ceph.osd.2:Started 2023-11-28T12:58:37.426 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-28T12:58:37.426 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-11-28T12:58:37.428 INFO:tasks.ceph.osd.3:Started 2023-11-28T12:58:37.428 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-28T12:58:37.429 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-11-28T12:58:37.431 INFO:tasks.ceph.osd.4:Started 2023-11-28T12:58:37.431 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-28T12:58:37.431 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-11-28T12:58:37.433 INFO:tasks.ceph.osd.5:Started 2023-11-28T12:58:37.433 INFO:tasks.ceph.osd.6:Restarting daemon 2023-11-28T12:58:37.433 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-11-28T12:58:37.435 INFO:tasks.ceph.osd.6:Started 2023-11-28T12:58:37.435 INFO:tasks.ceph.osd.7:Restarting daemon 2023-11-28T12:58:37.435 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-11-28T12:58:37.439 INFO:tasks.ceph.osd.7:Started 2023-11-28T12:58:37.440 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:58:37.501 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:37.500+0000 7fbbe43e2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T12:58:37.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.576+0000 7ff631029700 1 Processor -- start 2023-11-28T12:58:37.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.576+0000 7ff631029700 1 -- start start 2023-11-28T12:58:37.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff631029700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff62c158cb0 0x7ff62c0ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff631029700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff62c159080 0x7ff62c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff631029700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62c06d870 con 0x7ff62c0c9300 2023-11-28T12:58:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff631029700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff62c06cc60 con 0x7ff62c159080 2023-11-28T12:58:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff631029700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c06cf40 con 0x7ff62c158cb0 2023-11-28T12:58:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff62b59e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff62c0c9300 0x7ff62c070910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:32786/0 (socket says 172.21.15.111:32786) 2023-11-28T12:58:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff62b59e700 1 -- 172.21.15.111:0/3585258448 learned_addr learned my addr 172.21.15.111:0/3585258448 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:58:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff62a59c700 1 --2- 172.21.15.111:0/3585258448 >> v2:172.21.15.149:3300/0 conn(0x7ff62c159080 0x7ff62c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff62ad9d700 1 --2- 172.21.15.111:0/3585258448 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff62c158cb0 0x7ff62c0ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff629d9b700 1 -- 172.21.15.111:0/3585258448 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022757145 0 0) 0x7ff62c06d870 con 0x7ff62c0c9300 2023-11-28T12:58:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff629d9b700 1 -- 172.21.15.111:0/3585258448 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6140035b0 con 0x7ff62c0c9300 2023-11-28T12:58:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff62ad9d700 1 -- 172.21.15.111:0/3585258448 >> v2:172.21.15.149:3300/0 conn(0x7ff62c159080 msgr2=0x7ff62c06dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff62ad9d700 1 --2- 172.21.15.111:0/3585258448 >> v2:172.21.15.149:3300/0 conn(0x7ff62c159080 0x7ff62c06dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff62ad9d700 1 -- 172.21.15.111:0/3585258448 >> v1:172.21.15.111:6789/0 conn(0x7ff62c0c9300 legacy=0x7ff62c070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff62ad9d700 1 -- 172.21.15.111:0/3585258448 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62c06d220 con 0x7ff62c158cb0 2023-11-28T12:58:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff62a59c700 1 --2- 172.21.15.111:0/3585258448 >> v2:172.21.15.149:3300/0 conn(0x7ff62c159080 0x7ff62c06dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.577+0000 7ff62ad9d700 1 --2- 172.21.15.111:0/3585258448 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff62c158cb0 0x7ff62c0ca1f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009840 tx=0x7ff61c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bcc75d4c9667d26 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.578+0000 7ff629d9b700 1 -- 172.21.15.111:0/3585258448 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c011830 con 0x7ff62c158cb0 2023-11-28T12:58:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.578+0000 7ff629d9b700 1 -- 172.21.15.111:0/3585258448 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61c011990 con 0x7ff62c158cb0 2023-11-28T12:58:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.578+0000 7ff629d9b700 1 -- 172.21.15.111:0/3585258448 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c011d20 con 0x7ff62c158cb0 2023-11-28T12:58:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.578+0000 7ff631029700 1 -- 172.21.15.111:0/3585258448 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff62c158cb0 msgr2=0x7ff62c0ca1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.578+0000 7ff631029700 1 --2- 172.21.15.111:0/3585258448 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff62c158cb0 0x7ff62c0ca1f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009840 tx=0x7ff61c009b50 comp rx=0 tx=0).stop 2023-11-28T12:58:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.578+0000 7ff631029700 1 -- 172.21.15.111:0/3585258448 shutdown_connections 2023-11-28T12:58:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.578+0000 7ff631029700 1 --2- 172.21.15.111:0/3585258448 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff62c158cb0 0x7ff62c0ca1f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.578+0000 7ff631029700 1 --2- 172.21.15.111:0/3585258448 >> v2:172.21.15.149:3300/0 conn(0x7ff62c159080 0x7ff62c06dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.578+0000 7ff631029700 1 -- 172.21.15.111:0/3585258448 >> 172.21.15.111:0/3585258448 conn(0x7ff62c0cb490 msgr2=0x7ff62c0cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:37.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.578+0000 7ff631029700 1 -- 172.21.15.111:0/3585258448 shutdown_connections 2023-11-28T12:58:37.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.578+0000 7ff631029700 1 -- 172.21.15.111:0/3585258448 wait complete. 2023-11-28T12:58:37.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.580+0000 7ff631029700 1 Processor -- start 2023-11-28T12:58:37.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.580+0000 7ff631029700 1 -- start start 2023-11-28T12:58:37.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.580+0000 7ff631029700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff62c0c9300 0x7ff62c1e1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.580+0000 7ff631029700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62c158cb0 0x7ff62c1578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.580+0000 7ff631029700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff62c1e1600 0x7ff62c0b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:37.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.580+0000 7ff631029700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c1e1da0 con 0x7ff62c158cb0 2023-11-28T12:58:37.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.580+0000 7ff631029700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c0b3ac0 con 0x7ff62c0c9300 2023-11-28T12:58:37.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.580+0000 7ff631029700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c1583c0 con 0x7ff62c1e1600 2023-11-28T12:58:37.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.580+0000 7ff62a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62c158cb0 0x7ff62c1578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff62a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62c158cb0 0x7ff62c1578a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60244/0 (socket says 172.21.15.111:60244) 2023-11-28T12:58:37.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff62a59c700 1 -- 172.21.15.111:0/3096341222 learned_addr learned my addr 172.21.15.111:0/3096341222 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:37.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff62ad9d700 1 --2- 172.21.15.111:0/3096341222 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff62c0c9300 0x7ff62c1e1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff62b59e700 1 --2- 172.21.15.111:0/3096341222 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff62c1e1600 0x7ff62c0b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:37.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff62a59c700 1 -- 172.21.15.111:0/3096341222 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff62c1e1600 msgr2=0x7ff62c0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff62a59c700 1 --2- 172.21.15.111:0/3096341222 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff62c1e1600 0x7ff62c0b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff62a59c700 1 -- 172.21.15.111:0/3096341222 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff62c0c9300 msgr2=0x7ff62c1e1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff62a59c700 1 --2- 172.21.15.111:0/3096341222 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff62c0c9300 0x7ff62c1e1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff62a59c700 1 -- 172.21.15.111:0/3096341222 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6200094a0 con 0x7ff62c158cb0 2023-11-28T12:58:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff62a59c700 1 --2- 172.21.15.111:0/3096341222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62c158cb0 0x7ff62c1578a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff62000ec10 tx=0x7ff62000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff6137fe700 1 -- 172.21.15.111:0/3096341222 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff620009760 con 0x7ff62c158cb0 2023-11-28T12:58:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff6137fe700 1 -- 172.21.15.111:0/3096341222 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62000c850 con 0x7ff62c158cb0 2023-11-28T12:58:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff631029700 1 -- 172.21.15.111:0/3096341222 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff61c0094a0 con 0x7ff62c158cb0 2023-11-28T12:58:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff6137fe700 1 -- 172.21.15.111:0/3096341222 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62000cbe0 con 0x7ff62c158cb0 2023-11-28T12:58:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff631029700 1 -- 172.21.15.111:0/3096341222 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62c071b20 con 0x7ff62c158cb0 2023-11-28T12:58:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff6137fe700 1 -- 172.21.15.111:0/3096341222 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7ff620019030 con 0x7ff62c158cb0 2023-11-28T12:58:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.581+0000 7ff6137fe700 1 -- 172.21.15.111:0/3096341222 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7ff620013470 con 0x7ff62c158cb0 2023-11-28T12:58:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.582+0000 7ff631029700 1 -- 172.21.15.111:0/3096341222 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62c06d500 con 0x7ff62c158cb0 2023-11-28T12:58:37.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.584+0000 7ff6137fe700 1 -- 172.21.15.111:0/3096341222 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7ff620021060 con 0x7ff62c158cb0 2023-11-28T12:58:37.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.654+0000 7ff631029700 1 -- 172.21.15.111:0/3096341222 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff62c071020 con 0x7ff62c158cb0 2023-11-28T12:58:37.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.655+0000 7ff6137fe700 1 -- 172.21.15.111:0/3096341222 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7ff620017070 con 0x7ff62c158cb0 2023-11-28T12:58:37.656 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:37.656 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":10,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:58:37.396611+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:58:37.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.656+0000 7ff631029700 1 -- 172.21.15.111:0/3096341222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62c158cb0 msgr2=0x7ff62c1578a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:37.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.656+0000 7ff631029700 1 --2- 172.21.15.111:0/3096341222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62c158cb0 0x7ff62c1578a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff62000ec10 tx=0x7ff62000ef20 comp rx=0 tx=0).stop 2023-11-28T12:58:37.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.656+0000 7ff631029700 1 -- 172.21.15.111:0/3096341222 shutdown_connections 2023-11-28T12:58:37.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.656+0000 7ff631029700 1 --2- 172.21.15.111:0/3096341222 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff62c1e1600 0x7ff62c0b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.656+0000 7ff631029700 1 --2- 172.21.15.111:0/3096341222 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff62c0c9300 0x7ff62c1e1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.656+0000 7ff631029700 1 --2- 172.21.15.111:0/3096341222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff62c158cb0 0x7ff62c1578a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:37.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.656+0000 7ff631029700 1 -- 172.21.15.111:0/3096341222 >> 172.21.15.111:0/3096341222 conn(0x7ff62c0cb490 msgr2=0x7ff62c0cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:37.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.656+0000 7ff631029700 1 -- 172.21.15.111:0/3096341222 shutdown_connections 2023-11-28T12:58:37.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:37.656+0000 7ff631029700 1 -- 172.21.15.111:0/3096341222 wait complete. 2023-11-28T12:58:37.669 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-28T12:58:37.669 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-28T12:58:38.051 INFO:tasks.ceph.osd.4.smithi149.stderr:2023-11-28T12:58:38.049+0000 7f8406344640 -1 Falling back to public interface 2023-11-28T12:58:40.756 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-11-28T12:58:38.054+0000 7f440cc92640 -1 Falling back to public interface 2023-11-28T12:58:40.756 INFO:tasks.ceph.osd.5.smithi149.stderr:2023-11-28T12:58:38.058+0000 7fe754c69640 -1 Falling back to public interface 2023-11-28T12:58:40.756 INFO:tasks.ceph.osd.3.smithi111.stderr:2023-11-28T12:58:38.058+0000 7f8ce3486640 -1 Falling back to public interface 2023-11-28T12:58:40.759 INFO:tasks.ceph.osd.6.smithi149.stderr:2023-11-28T12:58:38.065+0000 7f04e51cd640 -1 Falling back to public interface 2023-11-28T12:58:40.760 INFO:tasks.ceph.osd.2.smithi111.stderr:2023-11-28T12:58:38.059+0000 7f79d40af640 -1 Falling back to public interface 2023-11-28T12:58:40.760 INFO:tasks.ceph.osd.7.smithi149.stderr:2023-11-28T12:58:38.070+0000 7f518bd65640 -1 Falling back to public interface 2023-11-28T12:58:40.762 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-11-28T12:58:38.062+0000 7f23ab13a640 -1 Falling back to public interface 2023-11-28T12:58:40.764 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:38.209+0000 7fbbe43e2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T12:58:40.790 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:38.287+0000 7fbbe43e2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T12:58:40.790 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:38.431+0000 7fbbe43e2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T12:58:40.790 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:38.738+0000 7fbbe43e2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T12:58:40.791 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:38.871+0000 7fbbe43e2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T12:58:40.792 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:39.088+0000 7fbbe43e2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T12:58:40.792 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:39.246+0000 7fbbe43e2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T12:58:40.792 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:39.361+0000 7fbbe43e2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T12:58:40.793 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:39.690+0000 7fbbe43e2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T12:58:40.793 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:39.791+0000 7fbbe43e2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T12:58:40.802 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:39.977+0000 7fbbe43e2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T12:58:40.802 INFO:tasks.ceph.mgr.x.smithi149.stderr:2023-11-28T12:58:40.505+0000 7fbbe43e2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T12:58:40.802 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:38.077+0000 7f2d977bd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T12:58:40.802 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:38.154+0000 7f2d977bd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T12:58:40.802 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:38.291+0000 7f2d977bd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T12:58:40.802 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:38.587+0000 7f2d977bd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T12:58:40.802 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:38.719+0000 7f2d977bd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T12:58:40.802 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:38.934+0000 7f2d977bd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T12:58:40.803 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:39.087+0000 7f2d977bd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T12:58:40.803 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:39.201+0000 7f2d977bd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T12:58:40.803 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:39.518+0000 7f2d977bd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T12:58:40.803 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:39.615+0000 7f2d977bd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T12:58:40.803 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:39.797+0000 7f2d977bd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T12:58:40.803 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:40.313+0000 7f2d977bd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T12:58:41.156 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T12:58:41.267 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.265+0000 7f4e89480700 1 Processor -- start 2023-11-28T12:58:41.267 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.265+0000 7f4e89480700 1 -- start start 2023-11-28T12:58:41.271 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.266+0000 7f4e89480700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4e840d01a0 0x7f4e84160f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:41.271 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.266+0000 7f4e89480700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e840cf820 con 0x7f4e8415c860 2023-11-28T12:58:41.271 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.266+0000 7f4e89480700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e840cdbb0 con 0x7f4e840cd6d0 2023-11-28T12:58:41.271 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.266+0000 7f4e89480700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e84160490 con 0x7f4e840d01a0 2023-11-28T12:58:41.271 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.266+0000 7f4e82ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4e8415c860 0x7f4e840cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:32836/0 (socket says 172.21.15.111:32836) 2023-11-28T12:58:41.271 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.266+0000 7f4e82ffd700 1 -- 172.21.15.111:0/1175265616 learned_addr learned my addr 172.21.15.111:0/1175265616 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:58:41.271 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.266+0000 7f4e827fc700 1 --2- 172.21.15.111:0/1175265616 >> v2:172.21.15.149:3300/0 conn(0x7f4e840d01a0 0x7f4e84160f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:41.271 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.266+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044780927 0 0) 0x7f4e840cf820 con 0x7f4e8415c860 2023-11-28T12:58:41.271 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.266+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e6c0035b0 con 0x7f4e8415c860 2023-11-28T12:58:41.271 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 208756293 0 0) 0x7f4e6c0035b0 con 0x7f4e8415c860 2023-11-28T12:58:41.271 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 >> v2:172.21.15.149:3300/0 conn(0x7f4e840d01a0 msgr2=0x7f4e84160f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:41.272 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 --2- 172.21.15.111:0/1175265616 >> v2:172.21.15.149:3300/0 conn(0x7f4e840d01a0 0x7f4e84160f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:41.272 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 >> v1:172.21.15.149:6789/0 conn(0x7f4e840cd6d0 legacy=0x7f4e840d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:41.272 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e84160770 con 0x7f4e8415c860 2023-11-28T12:58:41.272 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f4e78001670 con 0x7f4e8415c860 2023-11-28T12:58:41.272 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 >> v1:172.21.15.111:6789/0 conn(0x7f4e8415c860 legacy=0x7f4e840cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:41.272 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 --2- 172.21.15.111:0/1175265616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e6c003a00 0x7f4e6c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:41.272 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 --2- 172.21.15.111:0/1175265616 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e6c0043f0 0x7f4e6c0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:41.272 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 --2- 172.21.15.111:0/1175265616 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e6c005150 0x7f4e6c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:41.272 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e84160490 con 0x7f4e6c003a00 2023-11-28T12:58:41.273 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c00c080 con 0x7f4e6c005150 2023-11-28T12:58:41.273 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c00c240 con 0x7f4e6c0043f0 2023-11-28T12:58:41.273 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e82ffd700 1 --2- 172.21.15.111:0/1175265616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e6c003a00 0x7f4e6c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:41.273 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e837fe700 1 --2- 172.21.15.111:0/1175265616 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e6c005150 0x7f4e6c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:41.273 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e827fc700 1 --2- 172.21.15.111:0/1175265616 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e6c0043f0 0x7f4e6c0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:41.273 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e82ffd700 1 -- 172.21.15.111:0/1175265616 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e6c0043f0 msgr2=0x7f4e6c0048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:41.273 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e82ffd700 1 --2- 172.21.15.111:0/1175265616 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e6c0043f0 0x7f4e6c0048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:41.273 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e82ffd700 1 -- 172.21.15.111:0/1175265616 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e6c005150 msgr2=0x7f4e6c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:41.273 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e82ffd700 1 --2- 172.21.15.111:0/1175265616 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e6c005150 0x7f4e6c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:41.273 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e837fe700 1 -- 172.21.15.111:0/1175265616 reap_dead start 2023-11-28T12:58:41.274 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e82ffd700 1 -- 172.21.15.111:0/1175265616 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c0035b0 con 0x7f4e6c003a00 2023-11-28T12:58:41.274 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e82ffd700 1 --2- 172.21.15.111:0/1175265616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e6c003a00 0x7f4e6c003dd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c240 tx=0x7f4e6c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9189d5ef1a5384ea server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:41.274 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7800cd80 con 0x7f4e6c003a00 2023-11-28T12:58:41.274 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e78005960 con 0x7f4e6c003a00 2023-11-28T12:58:41.274 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.267+0000 7f4e81ffb700 1 -- 172.21.15.111:0/1175265616 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78005d10 con 0x7f4e6c003a00 2023-11-28T12:58:41.274 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.268+0000 7f4e89480700 1 -- 172.21.15.111:0/1175265616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e6c003a00 msgr2=0x7f4e6c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:41.274 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.268+0000 7f4e89480700 1 --2- 172.21.15.111:0/1175265616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e6c003a00 0x7f4e6c003dd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c240 tx=0x7f4e6c005880 comp rx=0 tx=0).stop 2023-11-28T12:58:41.274 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.268+0000 7f4e89480700 1 -- 172.21.15.111:0/1175265616 shutdown_connections 2023-11-28T12:58:41.274 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.268+0000 7f4e89480700 1 --2- 172.21.15.111:0/1175265616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e6c003a00 0x7f4e6c003dd0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:41.274 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.268+0000 7f4e89480700 1 -- 172.21.15.111:0/1175265616 >> 172.21.15.111:0/1175265616 conn(0x7f4e840665b0 msgr2=0x7f4e84071950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:41.274 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.268+0000 7f4e89480700 1 -- 172.21.15.111:0/1175265616 shutdown_connections 2023-11-28T12:58:41.274 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.268+0000 7f4e89480700 1 -- 172.21.15.111:0/1175265616 wait complete. 2023-11-28T12:58:41.275 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.269+0000 7f4e89480700 1 Processor -- start 2023-11-28T12:58:41.275 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.269+0000 7f4e89480700 1 -- start start 2023-11-28T12:58:41.277 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e89480700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e6c003a00 0x7f4e841f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:41.277 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e89480700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e84156290 0x7f4e84156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:41.277 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e89480700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e841572e0 0x7f4e841f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:41.277 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e89480700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e84159e70 con 0x7f4e84156290 2023-11-28T12:58:41.277 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e89480700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e841f2370 con 0x7f4e6c003a00 2023-11-28T12:58:41.277 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e89480700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e84157780 con 0x7f4e841572e0 2023-11-28T12:58:41.278 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e827fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e84156290 0x7f4e84156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:41.278 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e82ffd700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e6c003a00 0x7f4e841f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:41.278 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e827fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e84156290 0x7f4e84156da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60304/0 (socket says 172.21.15.111:60304) 2023-11-28T12:58:41.278 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e827fc700 1 -- 172.21.15.111:0/1999765881 learned_addr learned my addr 172.21.15.111:0/1999765881 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:41.278 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e837fe700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e841572e0 0x7f4e841f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:41.278 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e827fc700 1 -- 172.21.15.111:0/1999765881 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e841572e0 msgr2=0x7f4e841f5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:41.278 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e827fc700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e841572e0 0x7f4e841f5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:41.278 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e827fc700 1 -- 172.21.15.111:0/1999765881 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e6c003a00 msgr2=0x7f4e841f1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:41.278 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e827fc700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e6c003a00 0x7f4e841f1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:41.278 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e827fc700 1 -- 172.21.15.111:0/1999765881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8415cc30 con 0x7f4e84156290 2023-11-28T12:58:41.278 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.270+0000 7f4e82ffd700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e6c003a00 0x7f4e841f1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:41.279 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.271+0000 7f4e827fc700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e84156290 0x7f4e84156da0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4e740108a0 tx=0x7f4e74010bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:41.279 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.271+0000 7f4e6b7fe700 1 -- 172.21.15.111:0/1999765881 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74004240 con 0x7f4e84156290 2023-11-28T12:58:41.279 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.271+0000 7f4e6b7fe700 1 -- 172.21.15.111:0/1999765881 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e740043a0 con 0x7f4e84156290 2023-11-28T12:58:41.279 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.271+0000 7f4e6b7fe700 1 -- 172.21.15.111:0/1999765881 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e740036a0 con 0x7f4e84156290 2023-11-28T12:58:41.279 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.271+0000 7f4e89480700 1 -- 172.21.15.111:0/1999765881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e84160a50 con 0x7f4e84156290 2023-11-28T12:58:41.279 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.271+0000 7f4e89480700 1 -- 172.21.15.111:0/1999765881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e840cde90 con 0x7f4e84156290 2023-11-28T12:58:41.279 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.271+0000 7f4e6b7fe700 1 -- 172.21.15.111:0/1999765881 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 2) v1 ==== 43190+0+0 (secure 0 0 0) 0x7f4e74003800 con 0x7f4e84156290 2023-11-28T12:58:41.279 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.272+0000 7f4e89480700 1 -- 172.21.15.111:0/1999765881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8404e700 con 0x7f4e84156290 2023-11-28T12:58:41.279 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.272+0000 7f4e6b7fe700 1 -- 172.21.15.111:0/1999765881 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f4e74047db0 con 0x7f4e84156290 2023-11-28T12:58:41.279 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.274+0000 7f4e6b7fe700 1 -- 172.21.15.111:0/1999765881 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f4e74014470 con 0x7f4e84156290 2023-11-28T12:58:41.337 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.335+0000 7f4e6b7fe700 1 -- 172.21.15.111:0/1999765881 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 3) v1 ==== 86295+0+0 (secure 0 0 0) 0x7f4e74003a90 con 0x7f4e84156290 2023-11-28T12:58:41.337 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.335+0000 7f4e6b7fe700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4e70068910 0x7f4e7006acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:41.337 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.336+0000 7f4e82ffd700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4e70068910 0x7f4e7006acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:41.338 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.336+0000 7f4e82ffd700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4e70068910 0x7f4e7006acf0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4e78015040 tx=0x7f4e7800a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:41.339 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:41.338+0000 7f2d4e9eb700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-28T12:58:41.339 INFO:tasks.ceph.mgr.y.smithi111.stderr:2023-11-28T12:58:41.338+0000 7f2d4e9eb700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-28T12:58:41.356 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.354+0000 7f4e89480700 1 -- 172.21.15.111:0/1999765881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e8415bda0 con 0x7f4e84156290 2023-11-28T12:58:41.356 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.355+0000 7f4e6b7fe700 1 -- 172.21.15.111:0/1999765881 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f4e74055180 con 0x7f4e84156290 2023-11-28T12:58:41.357 INFO:teuthology.misc.health.smithi111.stdout: 2023-11-28T12:58:41.357 INFO:teuthology.misc.health.smithi111.stdout:{"epoch":10,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:58:37.396611+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:58:41.358 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.356+0000 7f4e89480700 1 -- 172.21.15.111:0/1999765881 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4e70068910 msgr2=0x7f4e7006acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:41.358 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.356+0000 7f4e89480700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4e70068910 0x7f4e7006acf0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4e78015040 tx=0x7f4e7800a000 comp rx=0 tx=0).stop 2023-11-28T12:58:41.358 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.356+0000 7f4e89480700 1 -- 172.21.15.111:0/1999765881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e84156290 msgr2=0x7f4e84156da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:41.358 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.356+0000 7f4e89480700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e84156290 0x7f4e84156da0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4e740108a0 tx=0x7f4e74010bb0 comp rx=0 tx=0).stop 2023-11-28T12:58:41.358 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.356+0000 7f4e89480700 1 -- 172.21.15.111:0/1999765881 shutdown_connections 2023-11-28T12:58:41.358 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.356+0000 7f4e89480700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e841572e0 0x7f4e841f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:41.358 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.356+0000 7f4e89480700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4e70068910 0x7f4e7006acf0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:41.358 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.356+0000 7f4e89480700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e6c003a00 0x7f4e841f1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:41.358 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.356+0000 7f4e89480700 1 --2- 172.21.15.111:0/1999765881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e84156290 0x7f4e84156da0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:41.358 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.356+0000 7f4e89480700 1 -- 172.21.15.111:0/1999765881 >> 172.21.15.111:0/1999765881 conn(0x7f4e840665b0 msgr2=0x7f4e8415f980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:41.359 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.356+0000 7f4e89480700 1 -- 172.21.15.111:0/1999765881 shutdown_connections 2023-11-28T12:58:41.359 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:41.356+0000 7f4e89480700 1 -- 172.21.15.111:0/1999765881 wait complete. 2023-11-28T12:58:41.368 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-28T12:58:41.583 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-11-28T12:58:41.581+0000 7f440cc92640 -1 osd.0 0 log_to_monitors true 2023-11-28T12:58:41.594 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-11-28T12:58:41.593+0000 7f23ab13a640 -1 osd.1 0 log_to_monitors true 2023-11-28T12:58:41.594 INFO:tasks.ceph.osd.2.smithi111.stderr:2023-11-28T12:58:41.593+0000 7f79d40af640 -1 osd.2 0 log_to_monitors true 2023-11-28T12:58:41.604 INFO:tasks.ceph.osd.3.smithi111.stderr:2023-11-28T12:58:41.602+0000 7f8ce3486640 -1 osd.3 0 log_to_monitors true 2023-11-28T12:58:41.616 INFO:tasks.ceph.osd.6.smithi149.stderr:2023-11-28T12:58:41.613+0000 7f04e51cd640 -1 osd.6 0 log_to_monitors true 2023-11-28T12:58:41.618 INFO:tasks.ceph.osd.5.smithi149.stderr:2023-11-28T12:58:41.616+0000 7fe754c69640 -1 osd.5 0 log_to_monitors true 2023-11-28T12:58:41.622 INFO:tasks.ceph.osd.4.smithi149.stderr:2023-11-28T12:58:41.621+0000 7f8406344640 -1 osd.4 0 log_to_monitors true 2023-11-28T12:58:41.625 INFO:tasks.ceph.osd.7.smithi149.stderr:2023-11-28T12:58:41.624+0000 7f518bd65640 -1 osd.7 0 log_to_monitors true 2023-11-28T12:58:45.161 INFO:tasks.ceph.osd.2.smithi111.stderr:2023-11-28T12:58:45.158+0000 7f79c8467700 -1 osd.2 0 waiting for initial osdmap 2023-11-28T12:58:45.171 INFO:tasks.ceph.osd.2.smithi111.stderr:2023-11-28T12:58:45.170+0000 7f79c0041700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T12:58:45.294 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-11-28T12:58:45.289+0000 7f440104a700 -1 osd.0 0 waiting for initial osdmap 2023-11-28T12:58:45.304 INFO:tasks.ceph.osd.3.smithi111.stderr:2023-11-28T12:58:45.302+0000 7f8cd783e700 -1 osd.3 0 waiting for initial osdmap 2023-11-28T12:58:45.304 INFO:tasks.ceph.osd.0.smithi111.stderr:2023-11-28T12:58:45.303+0000 7f43f8c24700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T12:58:45.318 INFO:tasks.ceph.osd.3.smithi111.stderr:2023-11-28T12:58:45.314+0000 7f8ccf418700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T12:58:45.333 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-11-28T12:58:45.332+0000 7f239f4f2700 -1 osd.1 0 waiting for initial osdmap 2023-11-28T12:58:45.348 INFO:tasks.ceph.osd.1.smithi111.stderr:2023-11-28T12:58:45.347+0000 7f23970cc700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T12:58:45.386 INFO:tasks.ceph.osd.5.smithi149.stderr:2023-11-28T12:58:45.385+0000 7fe749021700 -1 osd.5 0 waiting for initial osdmap 2023-11-28T12:58:45.391 INFO:tasks.ceph.osd.5.smithi149.stderr:2023-11-28T12:58:45.390+0000 7fe740bfb700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T12:58:45.400 INFO:tasks.ceph.osd.6.smithi149.stderr:2023-11-28T12:58:45.399+0000 7f04d9585700 -1 osd.6 0 waiting for initial osdmap 2023-11-28T12:58:45.408 INFO:tasks.ceph.osd.4.smithi149.stderr:2023-11-28T12:58:45.407+0000 7f83fa6fc700 -1 osd.4 0 waiting for initial osdmap 2023-11-28T12:58:45.414 INFO:tasks.ceph.osd.7.smithi149.stderr:2023-11-28T12:58:45.413+0000 7f518011d700 -1 osd.7 0 waiting for initial osdmap 2023-11-28T12:58:45.415 INFO:tasks.ceph.osd.6.smithi149.stderr:2023-11-28T12:58:45.414+0000 7f04d115f700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T12:58:45.420 INFO:tasks.ceph.osd.4.smithi149.stderr:2023-11-28T12:58:45.419+0000 7f83f22d6700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T12:58:45.428 INFO:tasks.ceph.osd.7.smithi149.stderr:2023-11-28T12:58:45.427+0000 7f5177cf7700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T12:58:48.175 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T12:58:48.294 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.292+0000 7f9af7f39700 1 Processor -- start 2023-11-28T12:58:48.294 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.293+0000 7f9af7f39700 1 -- start start 2023-11-28T12:58:48.301 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.293+0000 7f9af7f39700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af006a6c0 0x7f9af00d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.301 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.293+0000 7f9af7f39700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af00ce4d0 con 0x7f9af006a220 2023-11-28T12:58:48.301 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.293+0000 7f9af7f39700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af00b18c0 con 0x7f9af015c9b0 2023-11-28T12:58:48.301 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.293+0000 7f9af7f39700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00b1ba0 con 0x7f9af006a6c0 2023-11-28T12:58:48.301 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af54d4700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9af006a220 0x7f9af00cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33094/0 (socket says 172.21.15.111:33094) 2023-11-28T12:58:48.301 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af54d4700 1 -- 172.21.15.111:0/730735243 learned_addr learned my addr 172.21.15.111:0/730735243 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:58:48.301 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af64d6700 1 --2- 172.21.15.111:0/730735243 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af006a6c0 0x7f9af00d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.301 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883730286 0 0) 0x7f9af00ce4d0 con 0x7f9af006a220 2023-11-28T12:58:48.301 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad80035b0 con 0x7f9af006a220 2023-11-28T12:58:48.301 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087007032 0 0) 0x7f9af00b18c0 con 0x7f9af015c9b0 2023-11-28T12:58:48.301 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af00ce4d0 con 0x7f9af015c9b0 2023-11-28T12:58:48.302 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2716877783 0 0) 0x7f9ad80035b0 con 0x7f9af006a220 2023-11-28T12:58:48.302 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af006a6c0 msgr2=0x7f9af00d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:48.302 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 --2- 172.21.15.111:0/730735243 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af006a6c0 0x7f9af00d0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.302 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 >> v1:172.21.15.149:6789/0 conn(0x7f9af015c9b0 legacy=0x7f9af00cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:48.302 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af00b1e80 con 0x7f9af006a220 2023-11-28T12:58:48.302 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f9ae4001670 con 0x7f9af006a220 2023-11-28T12:58:48.302 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af64d6700 1 --2- 172.21.15.111:0/730735243 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af006a6c0 0x7f9af00d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:48.302 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 >> v1:172.21.15.111:6789/0 conn(0x7f9af006a220 legacy=0x7f9af00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:48.302 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 --2- 172.21.15.111:0/730735243 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ad8003bc0 0x7f9ad8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.302 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 --2- 172.21.15.111:0/730735243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ad8004850 0x7f9ad8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.303 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 --2- 172.21.15.111:0/730735243 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af006a6c0 0x7f9ad8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.303 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00b1ba0 con 0x7f9ad8004850 2023-11-28T12:58:48.303 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad800ba40 con 0x7f9ad8003bc0 2023-11-28T12:58:48.303 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad800bc00 con 0x7f9af006a6c0 2023-11-28T12:58:48.303 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af54d4700 1 --2- 172.21.15.111:0/730735243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ad8004850 0x7f9ad8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.303 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af5cd5700 1 --2- 172.21.15.111:0/730735243 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ad8003bc0 0x7f9ad8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.303 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.294+0000 7f9af64d6700 1 --2- 172.21.15.111:0/730735243 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af006a6c0 0x7f9ad8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.303 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af54d4700 1 -- 172.21.15.111:0/730735243 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af006a6c0 msgr2=0x7f9ad8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:48.303 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af54d4700 1 --2- 172.21.15.111:0/730735243 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af006a6c0 0x7f9ad8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.303 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af54d4700 1 -- 172.21.15.111:0/730735243 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ad8003bc0 msgr2=0x7f9ad8003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:48.304 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af54d4700 1 --2- 172.21.15.111:0/730735243 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ad8003bc0 0x7f9ad8003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.304 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af54d4700 1 -- 172.21.15.111:0/730735243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae80094a0 con 0x7f9ad8004850 2023-11-28T12:58:48.304 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af54d4700 1 --2- 172.21.15.111:0/730735243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ad8004850 0x7f9ad8004cc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400c210 tx=0x7f9ad8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce2e1461adc32ca9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:48.304 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae400cc90 con 0x7f9ad8004850 2023-11-28T12:58:48.304 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae400ce10 con 0x7f9ad8004850 2023-11-28T12:58:48.304 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af4cd3700 1 -- 172.21.15.111:0/730735243 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4005ba0 con 0x7f9ad8004850 2023-11-28T12:58:48.304 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af7f39700 1 -- 172.21.15.111:0/730735243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ad8004850 msgr2=0x7f9ad8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:48.304 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af7f39700 1 --2- 172.21.15.111:0/730735243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ad8004850 0x7f9ad8004cc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400c210 tx=0x7f9ad8005880 comp rx=0 tx=0).stop 2023-11-28T12:58:48.304 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af64d6700 1 -- 172.21.15.111:0/730735243 reap_dead start 2023-11-28T12:58:48.304 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af7f39700 1 -- 172.21.15.111:0/730735243 shutdown_connections 2023-11-28T12:58:48.304 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.295+0000 7f9af7f39700 1 -- 172.21.15.111:0/730735243 >> 172.21.15.111:0/730735243 conn(0x7f9af006dfe0 msgr2=0x7f9af006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:48.305 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.296+0000 7f9af7f39700 1 -- 172.21.15.111:0/730735243 shutdown_connections 2023-11-28T12:58:48.305 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.296+0000 7f9af7f39700 1 -- 172.21.15.111:0/730735243 wait complete. 2023-11-28T12:58:48.305 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.297+0000 7f9af7f39700 1 Processor -- start 2023-11-28T12:58:48.305 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.297+0000 7f9af7f39700 1 -- start start 2023-11-28T12:58:48.305 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.297+0000 7f9af7f39700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ad8004850 0x7f9af01554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.305 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.297+0000 7f9af7f39700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af00b4d80 0x7f9af00b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.305 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.297+0000 7f9af7f39700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af00b2d20 0x7f9af00b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.305 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.297+0000 7f9af7f39700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af0155a00 con 0x7f9af00b4d80 2023-11-28T12:58:48.305 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.297+0000 7f9af7f39700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00b5730 con 0x7f9ad8004850 2023-11-28T12:58:48.305 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.297+0000 7f9af7f39700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00b36e0 con 0x7f9af00b2d20 2023-11-28T12:58:48.305 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.297+0000 7f9af54d4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af00b4d80 0x7f9af00b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.305 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9af54d4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af00b4d80 0x7f9af00b51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60560/0 (socket says 172.21.15.111:60560) 2023-11-28T12:58:48.305 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9af54d4700 1 -- 172.21.15.111:0/2036360587 learned_addr learned my addr 172.21.15.111:0/2036360587 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:48.306 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9af64d6700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af00b2d20 0x7f9af00b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.306 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9af5cd5700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ad8004850 0x7f9af01554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.306 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9af54d4700 1 -- 172.21.15.111:0/2036360587 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af00b2d20 msgr2=0x7f9af00b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:48.306 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9af54d4700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af00b2d20 0x7f9af00b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.306 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9af54d4700 1 -- 172.21.15.111:0/2036360587 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ad8004850 msgr2=0x7f9af01554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:48.306 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9af54d4700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ad8004850 0x7f9af01554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.306 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9af54d4700 1 -- 172.21.15.111:0/2036360587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af006aae0 con 0x7f9af00b4d80 2023-11-28T12:58:48.306 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9af64d6700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af00b2d20 0x7f9af00b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:58:48.307 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9af54d4700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af00b4d80 0x7f9af00b51f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800ba10 tx=0x7f9ae4001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:48.308 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9aee7fc700 1 -- 172.21.15.111:0/2036360587 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae40072d0 con 0x7f9af00b4d80 2023-11-28T12:58:48.308 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9aee7fc700 1 -- 172.21.15.111:0/2036360587 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae4007430 con 0x7f9af00b4d80 2023-11-28T12:58:48.308 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9aee7fc700 1 -- 172.21.15.111:0/2036360587 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae40077c0 con 0x7f9af00b4d80 2023-11-28T12:58:48.308 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9af7f39700 1 -- 172.21.15.111:0/2036360587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af00b2160 con 0x7f9af00b4d80 2023-11-28T12:58:48.308 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.298+0000 7f9af7f39700 1 -- 172.21.15.111:0/2036360587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af015dc90 con 0x7f9af00b4d80 2023-11-28T12:58:48.308 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.299+0000 7f9af7f39700 1 -- 172.21.15.111:0/2036360587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af00c4c60 con 0x7f9af00b4d80 2023-11-28T12:58:48.308 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.300+0000 7f9aee7fc700 1 -- 172.21.15.111:0/2036360587 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7f9ae4024030 con 0x7f9af00b4d80 2023-11-28T12:58:48.309 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.300+0000 7f9aee7fc700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9adc0682a0 0x7f9adc06a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.309 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.300+0000 7f9aee7fc700 1 -- 172.21.15.111:0/2036360587 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9ae4086c10 con 0x7f9af00b4d80 2023-11-28T12:58:48.309 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.300+0000 7f9af5cd5700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9adc0682a0 0x7f9adc06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.309 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.300+0000 7f9af5cd5700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9adc0682a0 0x7f9adc06a750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0002230 tx=0x7f9ae0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:48.309 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.305+0000 7f9aee7fc700 1 -- 172.21.15.111:0/2036360587 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ae40538f0 con 0x7f9af00b4d80 2023-11-28T12:58:48.491 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.489+0000 7f9af7f39700 1 -- 172.21.15.111:0/2036360587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9af004e700 con 0x7f9af00b4d80 2023-11-28T12:58:48.491 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.490+0000 7f9aee7fc700 1 -- 172.21.15.111:0/2036360587 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+12539 (secure 0 0 0) 0x7f9ae4056f10 con 0x7f9af00b4d80 2023-11-28T12:58:48.492 INFO:teuthology.misc.health.smithi111.stdout: 2023-11-28T12:58:48.492 INFO:teuthology.misc.health.smithi111.stdout:{"epoch":15,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:58:48.174368+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:58:48.494 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.492+0000 7f9af7f39700 1 -- 172.21.15.111:0/2036360587 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9adc0682a0 msgr2=0x7f9adc06a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:48.494 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.492+0000 7f9af7f39700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9adc0682a0 0x7f9adc06a750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0002230 tx=0x7f9ae0009040 comp rx=0 tx=0).stop 2023-11-28T12:58:48.494 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.492+0000 7f9af7f39700 1 -- 172.21.15.111:0/2036360587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af00b4d80 msgr2=0x7f9af00b51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:48.494 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.492+0000 7f9af7f39700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af00b4d80 0x7f9af00b51f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800ba10 tx=0x7f9ae4001980 comp rx=0 tx=0).stop 2023-11-28T12:58:48.495 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.492+0000 7f9af7f39700 1 -- 172.21.15.111:0/2036360587 shutdown_connections 2023-11-28T12:58:48.495 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.492+0000 7f9af7f39700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9af00b2d20 0x7f9af00b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.495 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.492+0000 7f9af7f39700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9adc0682a0 0x7f9adc06a750 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.495 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.492+0000 7f9af7f39700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ad8004850 0x7f9af01554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.495 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.492+0000 7f9af7f39700 1 --2- 172.21.15.111:0/2036360587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af00b4d80 0x7f9af00b51f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.495 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.492+0000 7f9af7f39700 1 -- 172.21.15.111:0/2036360587 >> 172.21.15.111:0/2036360587 conn(0x7f9af006dfe0 msgr2=0x7f9af015f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:48.495 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.492+0000 7f9af7f39700 1 -- 172.21.15.111:0/2036360587 shutdown_connections 2023-11-28T12:58:48.495 INFO:teuthology.misc.health.smithi111.stderr:2023-11-28T12:58:48.493+0000 7f9af7f39700 1 -- 172.21.15.111:0/2036360587 wait complete. 2023-11-28T12:58:48.505 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-11-28T12:58:48.505 INFO:tasks.ceph:Creating RBD pool 2023-11-28T12:58:48.505 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-28T12:58:48.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.635+0000 7f957bafe700 1 Processor -- start 2023-11-28T12:58:48.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.636+0000 7f957bafe700 1 -- start start 2023-11-28T12:58:48.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.636+0000 7f957bafe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9574142e80 0x7f957406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.636+0000 7f957bafe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f957406a7b0 0x7f95740c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.636+0000 7f957bafe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f95740ca430 0x7f957406d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.636+0000 7f957bafe700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95740c9380 con 0x7f95740ca430 2023-11-28T12:58:48.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.636+0000 7f957bafe700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f957406ac00 con 0x7f957406a7b0 2023-11-28T12:58:48.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.636+0000 7f957bafe700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9574159b10 con 0x7f9574142e80 2023-11-28T12:58:48.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.636+0000 7f957a09b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f95740ca430 0x7f957406d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.637+0000 7f957a09b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f95740ca430 0x7f957406d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60588/0 (socket says 172.21.15.111:60588) 2023-11-28T12:58:48.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.637+0000 7f957a09b700 1 -- 172.21.15.111:0/1571608477 learned_addr learned my addr 172.21.15.111:0/1571608477 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:48.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.637+0000 7f957989a700 1 --2- 172.21.15.111:0/1571608477 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9574142e80 0x7f957406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.637+0000 7f9579099700 1 --2- 172.21.15.111:0/1571608477 >> v2:172.21.15.149:3300/0 conn(0x7f957406a7b0 0x7f95740c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.637+0000 7f957a09b700 1 -- 172.21.15.111:0/1571608477 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9574142e80 msgr2=0x7f957406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:48.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.637+0000 7f957a09b700 1 --2- 172.21.15.111:0/1571608477 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9574142e80 0x7f957406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.637+0000 7f957a09b700 1 -- 172.21.15.111:0/1571608477 >> v2:172.21.15.149:3300/0 conn(0x7f957406a7b0 msgr2=0x7f95740c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:48.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.637+0000 7f957a09b700 1 --2- 172.21.15.111:0/1571608477 >> v2:172.21.15.149:3300/0 conn(0x7f957406a7b0 0x7f95740c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.637+0000 7f957a09b700 1 -- 172.21.15.111:0/1571608477 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95740ca890 con 0x7f95740ca430 2023-11-28T12:58:48.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.637+0000 7f957a09b700 1 --2- 172.21.15.111:0/1571608477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95740ca430 0x7f957406d2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f957406eee0 tx=0x7f956c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8590a823a204973 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:48.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.637+0000 7f9578898700 1 -- 172.21.15.111:0/1571608477 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c010070 con 0x7f95740ca430 2023-11-28T12:58:48.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.637+0000 7f9578898700 1 -- 172.21.15.111:0/1571608477 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956c00dce0 con 0x7f95740ca430 2023-11-28T12:58:48.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.637+0000 7f9578898700 1 -- 172.21.15.111:0/1571608477 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c00b310 con 0x7f95740ca430 2023-11-28T12:58:48.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.638+0000 7f957bafe700 1 -- 172.21.15.111:0/1571608477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95740ca430 msgr2=0x7f957406d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:48.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.638+0000 7f957bafe700 1 --2- 172.21.15.111:0/1571608477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95740ca430 0x7f957406d2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f957406eee0 tx=0x7f956c00d430 comp rx=0 tx=0).stop 2023-11-28T12:58:48.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.638+0000 7f957bafe700 1 -- 172.21.15.111:0/1571608477 shutdown_connections 2023-11-28T12:58:48.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.638+0000 7f957bafe700 1 --2- 172.21.15.111:0/1571608477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95740ca430 0x7f957406d2d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.638+0000 7f957bafe700 1 --2- 172.21.15.111:0/1571608477 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9574142e80 0x7f957406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.638+0000 7f957bafe700 1 --2- 172.21.15.111:0/1571608477 >> v2:172.21.15.149:3300/0 conn(0x7f957406a7b0 0x7f95740c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.638+0000 7f957bafe700 1 -- 172.21.15.111:0/1571608477 >> 172.21.15.111:0/1571608477 conn(0x7f95740cb490 msgr2=0x7f95740cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:48.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.638+0000 7f957bafe700 1 -- 172.21.15.111:0/1571608477 shutdown_connections 2023-11-28T12:58:48.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.638+0000 7f957bafe700 1 -- 172.21.15.111:0/1571608477 wait complete. 2023-11-28T12:58:48.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.640+0000 7f957bafe700 1 Processor -- start 2023-11-28T12:58:48.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.640+0000 7f957bafe700 1 -- start start 2023-11-28T12:58:48.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.640+0000 7f957bafe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f957406a7b0 0x7f95740b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.640+0000 7f957bafe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95740ca430 0x7f95740c4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.640+0000 7f957bafe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9574142e80 0x7f95740c1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.640+0000 7f957bafe700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95740b2b80 con 0x7f95740ca430 2023-11-28T12:58:48.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.640+0000 7f957bafe700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95740c5360 con 0x7f9574142e80 2023-11-28T12:58:48.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.640+0000 7f957bafe700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95740c1cb0 con 0x7f957406a7b0 2023-11-28T12:58:48.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f9579099700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95740ca430 0x7f95740c4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f9579099700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95740ca430 0x7f95740c4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60592/0 (socket says 172.21.15.111:60592) 2023-11-28T12:58:48.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f9579099700 1 -- 172.21.15.111:0/2826887181 learned_addr learned my addr 172.21.15.111:0/2826887181 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:48.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f957a09b700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9574142e80 0x7f95740c1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f957989a700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f957406a7b0 0x7f95740b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f9579099700 1 -- 172.21.15.111:0/2826887181 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f957406a7b0 msgr2=0x7f95740b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:48.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f9579099700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f957406a7b0 0x7f95740b1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f9579099700 1 -- 172.21.15.111:0/2826887181 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9574142e80 msgr2=0x7f95740c1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:48.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f9579099700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9574142e80 0x7f95740c1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:48.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f9579099700 1 -- 172.21.15.111:0/2826887181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c00bd40 con 0x7f95740ca430 2023-11-28T12:58:48.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f9579099700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95740ca430 0x7f95740c4e20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f956400e9b0 tx=0x7f956400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:48.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f956a7fc700 1 -- 172.21.15.111:0/2826887181 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9564009710 con 0x7f95740ca430 2023-11-28T12:58:48.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f956a7fc700 1 -- 172.21.15.111:0/2826887181 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956400c850 con 0x7f95740ca430 2023-11-28T12:58:48.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f957bafe700 1 -- 172.21.15.111:0/2826887181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95740cab00 con 0x7f95740ca430 2023-11-28T12:58:48.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f956a7fc700 1 -- 172.21.15.111:0/2826887181 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956400cbc0 con 0x7f95740ca430 2023-11-28T12:58:48.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.641+0000 7f957bafe700 1 -- 172.21.15.111:0/2826887181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9574143250 con 0x7f95740ca430 2023-11-28T12:58:48.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.642+0000 7f957bafe700 1 -- 172.21.15.111:0/2826887181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95740cc5f0 con 0x7f95740ca430 2023-11-28T12:58:48.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.643+0000 7f956a7fc700 1 -- 172.21.15.111:0/2826887181 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7f9564019030 con 0x7f95740ca430 2023-11-28T12:58:48.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.643+0000 7f956a7fc700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95580682d0 0x7f955806a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:48.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.643+0000 7f957989a700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95580682d0 0x7f955806a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:48.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.643+0000 7f956a7fc700 1 -- 172.21.15.111:0/2826887181 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f95640856e0 con 0x7f95740ca430 2023-11-28T12:58:48.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.643+0000 7f957989a700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95580682d0 0x7f955806a780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9570009990 tx=0x7f9570008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:48.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.647+0000 7f956a7fc700 1 -- 172.21.15.111:0/2826887181 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9564052440 con 0x7f95740ca430 2023-11-28T12:58:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:48.846+0000 7f957bafe700 1 -- 172.21.15.111:0/2826887181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f95740b2cc0 con 0x7f95740ca430 2023-11-28T12:58:49.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.197+0000 7f956a7fc700 1 -- 172.21.15.111:0/2826887181 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) v1 ==== 109+0+0 (secure 0 0 0) 0x7f95740b2cc0 con 0x7f95740ca430 2023-11-28T12:58:49.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.200+0000 7f957bafe700 1 -- 172.21.15.111:0/2826887181 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95580682d0 msgr2=0x7f955806a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:49.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.200+0000 7f957bafe700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95580682d0 0x7f955806a780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9570009990 tx=0x7f9570008040 comp rx=0 tx=0).stop 2023-11-28T12:58:49.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.200+0000 7f957bafe700 1 -- 172.21.15.111:0/2826887181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95740ca430 msgr2=0x7f95740c4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:49.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.200+0000 7f957bafe700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95740ca430 0x7f95740c4e20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f956400e9b0 tx=0x7f956400ed70 comp rx=0 tx=0).stop 2023-11-28T12:58:49.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.200+0000 7f957bafe700 1 -- 172.21.15.111:0/2826887181 shutdown_connections 2023-11-28T12:58:49.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.200+0000 7f957bafe700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95580682d0 0x7f955806a780 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9570009990 tx=0x7f9570008040 comp rx=0 tx=0).stop 2023-11-28T12:58:49.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.200+0000 7f957bafe700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9574142e80 0x7f95740c1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:49.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.200+0000 7f957bafe700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f957406a7b0 0x7f95740b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:49.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.200+0000 7f957bafe700 1 --2- 172.21.15.111:0/2826887181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95740ca430 0x7f95740c4e20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:49.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.200+0000 7f957bafe700 1 -- 172.21.15.111:0/2826887181 >> 172.21.15.111:0/2826887181 conn(0x7f95740cb490 msgr2=0x7f95740cd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:49.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.200+0000 7f957bafe700 1 -- 172.21.15.111:0/2826887181 shutdown_connections 2023-11-28T12:58:49.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.200+0000 7f957bafe700 1 -- 172.21.15.111:0/2826887181 wait complete. 2023-11-28T12:58:49.203 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'rbd' created 2023-11-28T12:58:49.213 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-28T12:58:49.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.334+0000 7f0e7e35a700 1 Processor -- start 2023-11-28T12:58:49.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.334+0000 7f0e7e35a700 1 -- start start 2023-11-28T12:58:49.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.335+0000 7f0e7e35a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0e78159960 0x7f0e780b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:49.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.335+0000 7f0e7e35a700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f0e780aecc0 0x7f0e780cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:49.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.335+0000 7f0e7e35a700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e780ad420 con 0x7f0e780cdbc0 2023-11-28T12:58:49.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.335+0000 7f0e7e35a700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e780ce040 con 0x7f0e780aecc0 2023-11-28T12:58:49.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.335+0000 7f0e7e35a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e78159d30 con 0x7f0e78159960 2023-11-28T12:58:49.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.335+0000 7f0e777fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f0e780aecc0 0x7f0e780cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:49.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.335+0000 7f0e77fff700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0e78159960 0x7f0e780b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:49.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.335+0000 7f0e77fff700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0e78159960 0x7f0e780b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:54130/0 (socket says 172.21.15.111:54130) 2023-11-28T12:58:49.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.335+0000 7f0e777fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f0e780aecc0 0x7f0e780cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:50284/0 (socket says 172.21.15.111:50284) 2023-11-28T12:58:49.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.335+0000 7f0e7c8f7700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f0e780cdbc0 0x7f0e780afa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:41706/0 (socket says 172.21.15.111:41706) 2023-11-28T12:58:49.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.335+0000 7f0e77fff700 1 -- 172.21.15.111:0/1167800388 learned_addr learned my addr 172.21.15.111:0/1167800388 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:49.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.336+0000 7f0e77fff700 1 -- 172.21.15.111:0/1167800388 >> v2:172.21.15.149:3300/0 conn(0x7f0e780aecc0 msgr2=0x7f0e780cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:49.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.336+0000 7f0e77fff700 1 --2- 172.21.15.111:0/1167800388 >> v2:172.21.15.149:3300/0 conn(0x7f0e780aecc0 0x7f0e780cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:49.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.336+0000 7f0e77fff700 1 -- 172.21.15.111:0/1167800388 >> v1:172.21.15.149:6789/0 conn(0x7f0e780cdbc0 legacy=0x7f0e780afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:49.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.336+0000 7f0e77fff700 1 -- 172.21.15.111:0/1167800388 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e780ac810 con 0x7f0e78159960 2023-11-28T12:58:49.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.336+0000 7f0e76ffd700 1 -- 172.21.15.111:0/1167800388 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888539642 0 0) 0x7f0e780ad420 con 0x7f0e780cdbc0 2023-11-28T12:58:49.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.336+0000 7f0e777fe700 1 --2- 172.21.15.111:0/1167800388 >> v2:172.21.15.149:3300/0 conn(0x7f0e780aecc0 0x7f0e780cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:49.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.336+0000 7f0e77fff700 1 --2- 172.21.15.111:0/1167800388 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0e78159960 0x7f0e780b1fd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0e68006fd0 tx=0x7f0e68009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbf5dcd825176e13 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:49.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.336+0000 7f0e76ffd700 1 -- 172.21.15.111:0/1167800388 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e68005e30 con 0x7f0e78159960 2023-11-28T12:58:49.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.336+0000 7f0e76ffd700 1 -- 172.21.15.111:0/1167800388 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e68004030 con 0x7f0e78159960 2023-11-28T12:58:49.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.336+0000 7f0e76ffd700 1 -- 172.21.15.111:0/1167800388 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e680043e0 con 0x7f0e78159960 2023-11-28T12:58:49.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.337+0000 7f0e7e35a700 1 -- 172.21.15.111:0/1167800388 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0e78159960 msgr2=0x7f0e780b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:49.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.337+0000 7f0e7e35a700 1 --2- 172.21.15.111:0/1167800388 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0e78159960 0x7f0e780b1fd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0e68006fd0 tx=0x7f0e68009bd0 comp rx=0 tx=0).stop 2023-11-28T12:58:49.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.337+0000 7f0e7e35a700 1 -- 172.21.15.111:0/1167800388 shutdown_connections 2023-11-28T12:58:49.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.337+0000 7f0e7e35a700 1 --2- 172.21.15.111:0/1167800388 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0e78159960 0x7f0e780b1fd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:49.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.337+0000 7f0e7e35a700 1 --2- 172.21.15.111:0/1167800388 >> v2:172.21.15.149:3300/0 conn(0x7f0e780aecc0 0x7f0e780cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:49.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.337+0000 7f0e7e35a700 1 -- 172.21.15.111:0/1167800388 >> 172.21.15.111:0/1167800388 conn(0x7f0e78098ee0 msgr2=0x7f0e7806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:49.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.337+0000 7f0e7e35a700 1 -- 172.21.15.111:0/1167800388 shutdown_connections 2023-11-28T12:58:49.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.337+0000 7f0e7e35a700 1 -- 172.21.15.111:0/1167800388 wait complete. 2023-11-28T12:58:49.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.338+0000 7f0e7e35a700 1 Processor -- start 2023-11-28T12:58:49.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.338+0000 7f0e7e35a700 1 -- start start 2023-11-28T12:58:49.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e7e35a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0e780aecc0 0x7f0e7814c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:49.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e7e35a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e780cdbc0 0x7f0e781f0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:49.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e7e35a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0e78159960 0x7f0e780b3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:49.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e7e35a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7814c760 con 0x7f0e780cdbc0 2023-11-28T12:58:49.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e7e35a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e781f0d90 con 0x7f0e78159960 2023-11-28T12:58:49.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e7e35a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e781f1070 con 0x7f0e780aecc0 2023-11-28T12:58:49.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e777fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e780cdbc0 0x7f0e781f0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:49.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e7c8f7700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0e78159960 0x7f0e780b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:49.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e77fff700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0e780aecc0 0x7f0e7814c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:49.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e777fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e780cdbc0 0x7f0e781f0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60622/0 (socket says 172.21.15.111:60622) 2023-11-28T12:58:49.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e777fe700 1 -- 172.21.15.111:0/4139176262 learned_addr learned my addr 172.21.15.111:0/4139176262 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:49.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e7c8f7700 1 -- 172.21.15.111:0/4139176262 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0e780aecc0 msgr2=0x7f0e7814c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:49.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e7c8f7700 1 --2- 172.21.15.111:0/4139176262 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0e780aecc0 0x7f0e7814c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:49.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e7c8f7700 1 -- 172.21.15.111:0/4139176262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e780cdbc0 msgr2=0x7f0e781f0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:49.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e7c8f7700 1 --2- 172.21.15.111:0/4139176262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e780cdbc0 0x7f0e781f0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:49.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e7c8f7700 1 -- 172.21.15.111:0/4139176262 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c0094a0 con 0x7f0e78159960 2023-11-28T12:58:49.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.339+0000 7f0e77fff700 1 --2- 172.21.15.111:0/4139176262 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0e780aecc0 0x7f0e7814c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:49.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.340+0000 7f0e7c8f7700 1 --2- 172.21.15.111:0/4139176262 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0e78159960 0x7f0e780b3210 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000cbe0 tx=0x7f0e7000cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:49.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.340+0000 7f0e74ff9700 1 -- 172.21.15.111:0/4139176262 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7000aa40 con 0x7f0e78159960 2023-11-28T12:58:49.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.340+0000 7f0e74ff9700 1 -- 172.21.15.111:0/4139176262 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7000abc0 con 0x7f0e78159960 2023-11-28T12:58:49.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.340+0000 7f0e7e35a700 1 -- 172.21.15.111:0/4139176262 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e68009500 con 0x7f0e78159960 2023-11-28T12:58:49.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.340+0000 7f0e74ff9700 1 -- 172.21.15.111:0/4139176262 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e700070d0 con 0x7f0e78159960 2023-11-28T12:58:49.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.340+0000 7f0e7e35a700 1 -- 172.21.15.111:0/4139176262 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e780acdd0 con 0x7f0e78159960 2023-11-28T12:58:49.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.341+0000 7f0e7e35a700 1 -- 172.21.15.111:0/4139176262 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e780acaf0 con 0x7f0e78159960 2023-11-28T12:58:49.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.341+0000 7f0e74ff9700 1 -- 172.21.15.111:0/4139176262 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7f0e7001a030 con 0x7f0e78159960 2023-11-28T12:58:49.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.341+0000 7f0e74ff9700 1 --2- 172.21.15.111:0/4139176262 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0e640682d0 0x7f0e6406a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:49.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.342+0000 7f0e74ff9700 1 -- 172.21.15.111:0/4139176262 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f0e70085450 con 0x7f0e78159960 2023-11-28T12:58:49.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.342+0000 7f0e77fff700 1 --2- 172.21.15.111:0/4139176262 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0e640682d0 0x7f0e6406a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:49.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.342+0000 7f0e77fff700 1 --2- 172.21.15.111:0/4139176262 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0e640682d0 0x7f0e6406a780 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0e68006fd0 tx=0x7f0e68008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:49.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.346+0000 7f0e74ff9700 1 -- 172.21.15.111:0/4139176262 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0e70052000 con 0x7f0e78159960 2023-11-28T12:58:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:49.535+0000 7f0e7e35a700 1 -- 172.21.15.111:0/4139176262 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0e780ad0b0 con 0x7f0e78159960 2023-11-28T12:58:50.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.199+0000 7f0e74ff9700 1 -- 172.21.15.111:0/4139176262 <== mon.1 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v17) v1 ==== 173+0+0 (secure 0 0 0) 0x7f0e70055620 con 0x7f0e78159960 2023-11-28T12:58:50.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.202+0000 7f0e7e35a700 1 -- 172.21.15.111:0/4139176262 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0e640682d0 msgr2=0x7f0e6406a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.202+0000 7f0e7e35a700 1 --2- 172.21.15.111:0/4139176262 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0e640682d0 0x7f0e6406a780 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0e68006fd0 tx=0x7f0e68008070 comp rx=0 tx=0).stop 2023-11-28T12:58:50.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.202+0000 7f0e7e35a700 1 -- 172.21.15.111:0/4139176262 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0e78159960 msgr2=0x7f0e780b3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.202+0000 7f0e7e35a700 1 --2- 172.21.15.111:0/4139176262 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0e78159960 0x7f0e780b3210 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000cbe0 tx=0x7f0e7000cef0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.202+0000 7f0e7e35a700 1 -- 172.21.15.111:0/4139176262 shutdown_connections 2023-11-28T12:58:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.202+0000 7f0e7e35a700 1 --2- 172.21.15.111:0/4139176262 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0e640682d0 0x7f0e6406a780 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.202+0000 7f0e7e35a700 1 --2- 172.21.15.111:0/4139176262 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0e78159960 0x7f0e780b3210 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.202+0000 7f0e7e35a700 1 --2- 172.21.15.111:0/4139176262 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0e780aecc0 0x7f0e7814c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.202+0000 7f0e7e35a700 1 --2- 172.21.15.111:0/4139176262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e780cdbc0 0x7f0e781f0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.203+0000 7f0e7e35a700 1 -- 172.21.15.111:0/4139176262 >> 172.21.15.111:0/4139176262 conn(0x7f0e78098ee0 msgr2=0x7f0e78155ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:50.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.203+0000 7f0e7e35a700 1 -- 172.21.15.111:0/4139176262 shutdown_connections 2023-11-28T12:58:50.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.203+0000 7f0e7e35a700 1 -- 172.21.15.111:0/4139176262 wait complete. 2023-11-28T12:58:50.208 INFO:teuthology.orchestra.run.smithi111.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-28T12:58:50.221 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-28T12:58:50.221 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T12:58:50.221 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T12:58:50.224 INFO:tasks.ceph.mds.a:Started 2023-11-28T12:58:50.224 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T12:58:50.224 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T12:58:50.226 INFO:tasks.ceph.mds.c:Started 2023-11-28T12:58:50.227 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T12:58:50.227 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T12:58:50.229 INFO:tasks.ceph.mds.b:Started 2023-11-28T12:58:50.229 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T12:58:50.229 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T12:58:50.231 INFO:tasks.ceph.mds.d:Started 2023-11-28T12:58:50.231 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-11-28T12:58:50.232 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:58:50.307 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-11-28T12:58:50.307 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-11-28T12:58:50.309 INFO:tasks.ceph.mds.b.smithi149.stdout:starting mds.b at 2023-11-28T12:58:50.309 INFO:tasks.ceph.mds.d.smithi149.stdout:starting mds.d at 2023-11-28T12:58:50.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.361+0000 7fd41d7cf700 1 Processor -- start 2023-11-28T12:58:50.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.362+0000 7fd41d7cf700 1 -- start start 2023-11-28T12:58:50.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd41d7cf700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd418159960 0x7fd4180b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:50.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd41d7cf700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd4180cdd90 0x7fd4180ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:50.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd41d7cf700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4180ad420 con 0x7fd4180aecc0 2023-11-28T12:58:50.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd41d7cf700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd418159d30 con 0x7fd4180cdd90 2023-11-28T12:58:50.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd41d7cf700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4180ac810 con 0x7fd418159960 2023-11-28T12:58:50.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd4177fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd4180cdd90 0x7fd4180ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:50.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd4167fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd4180aecc0 0x7fd4180cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33206/0 (socket says 172.21.15.111:33206) 2023-11-28T12:58:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd4177fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd4180cdd90 0x7fd4180ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60662/0 (socket says 172.21.15.111:60662) 2023-11-28T12:58:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd4167fc700 1 -- 172.21.15.111:0/2521696959 learned_addr learned my addr 172.21.15.111:0/2521696959 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:58:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd416ffd700 1 --2- 172.21.15.111:0/2521696959 >> v2:172.21.15.149:3300/0 conn(0x7fd418159960 0x7fd4180b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd4177fe700 1 -- 172.21.15.111:0/2521696959 >> v2:172.21.15.149:3300/0 conn(0x7fd418159960 msgr2=0x7fd4180b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd4177fe700 1 --2- 172.21.15.111:0/2521696959 >> v2:172.21.15.149:3300/0 conn(0x7fd418159960 0x7fd4180b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd4177fe700 1 -- 172.21.15.111:0/2521696959 >> v1:172.21.15.111:6789/0 conn(0x7fd4180aecc0 legacy=0x7fd4180cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd4177fe700 1 -- 172.21.15.111:0/2521696959 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4180acaf0 con 0x7fd4180cdd90 2023-11-28T12:58:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd415ffb700 1 -- 172.21.15.111:0/2521696959 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573075381 0 0) 0x7fd4180ad420 con 0x7fd4180aecc0 2023-11-28T12:58:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd4177fe700 1 --2- 172.21.15.111:0/2521696959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4180cdd90 0x7fd4180ad8b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd41806c6a0 tx=0x7fd41000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=404a9a58c28d2fa0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd415ffb700 1 -- 172.21.15.111:0/2521696959 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41000e070 con 0x7fd4180cdd90 2023-11-28T12:58:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd415ffb700 1 -- 172.21.15.111:0/2521696959 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41000bb20 con 0x7fd4180cdd90 2023-11-28T12:58:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.363+0000 7fd415ffb700 1 -- 172.21.15.111:0/2521696959 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd410012430 con 0x7fd4180cdd90 2023-11-28T12:58:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.364+0000 7fd41d7cf700 1 -- 172.21.15.111:0/2521696959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4180cdd90 msgr2=0x7fd4180ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.364+0000 7fd41d7cf700 1 --2- 172.21.15.111:0/2521696959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4180cdd90 0x7fd4180ad8b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd41806c6a0 tx=0x7fd41000b330 comp rx=0 tx=0).stop 2023-11-28T12:58:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.364+0000 7fd41d7cf700 1 -- 172.21.15.111:0/2521696959 shutdown_connections 2023-11-28T12:58:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.364+0000 7fd41d7cf700 1 --2- 172.21.15.111:0/2521696959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4180cdd90 0x7fd4180ad8b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.364+0000 7fd41d7cf700 1 --2- 172.21.15.111:0/2521696959 >> v2:172.21.15.149:3300/0 conn(0x7fd418159960 0x7fd4180b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.364+0000 7fd41d7cf700 1 -- 172.21.15.111:0/2521696959 >> 172.21.15.111:0/2521696959 conn(0x7fd418098ee0 msgr2=0x7fd41806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.364+0000 7fd41d7cf700 1 -- 172.21.15.111:0/2521696959 shutdown_connections 2023-11-28T12:58:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.364+0000 7fd41d7cf700 1 -- 172.21.15.111:0/2521696959 wait complete. 2023-11-28T12:58:50.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.365+0000 7fd41d7cf700 1 Processor -- start 2023-11-28T12:58:50.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.365+0000 7fd41d7cf700 1 -- start start 2023-11-28T12:58:50.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd41d7cf700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4180aecc0 0x7fd418153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:50.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd41d7cf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4180cdd90 0x7fd4180c57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:50.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd41d7cf700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd418159960 0x7fd4180c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:50.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd41d7cf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd418153e40 con 0x7fd4180cdd90 2023-11-28T12:58:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd41d7cf700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41815a350 con 0x7fd4180aecc0 2023-11-28T12:58:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd41d7cf700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd41815a630 con 0x7fd418159960 2023-11-28T12:58:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd4167fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4180cdd90 0x7fd4180c57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd4167fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4180cdd90 0x7fd4180c57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60664/0 (socket says 172.21.15.111:60664) 2023-11-28T12:58:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd4167fc700 1 -- 172.21.15.111:0/4019599657 learned_addr learned my addr 172.21.15.111:0/4019599657 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd416ffd700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4180aecc0 0x7fd418153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd4177fe700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd418159960 0x7fd4180c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd4167fc700 1 -- 172.21.15.111:0/4019599657 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd418159960 msgr2=0x7fd4180c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd4167fc700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd418159960 0x7fd4180c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd4167fc700 1 -- 172.21.15.111:0/4019599657 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4180aecc0 msgr2=0x7fd418153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd4167fc700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4180aecc0 0x7fd418153900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.366+0000 7fd4167fc700 1 -- 172.21.15.111:0/4019599657 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd410009a70 con 0x7fd4180cdd90 2023-11-28T12:58:50.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.367+0000 7fd4167fc700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4180cdd90 0x7fd4180c57f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd4080011e0 tx=0x7fd40800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:50.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.367+0000 7fd3ff7fe700 1 -- 172.21.15.111:0/4019599657 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd408004c80 con 0x7fd4180cdd90 2023-11-28T12:58:50.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.367+0000 7fd3ff7fe700 1 -- 172.21.15.111:0/4019599657 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd408004e00 con 0x7fd4180cdd90 2023-11-28T12:58:50.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.367+0000 7fd3ff7fe700 1 -- 172.21.15.111:0/4019599657 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd408007310 con 0x7fd4180cdd90 2023-11-28T12:58:50.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.367+0000 7fd41d7cf700 1 -- 172.21.15.111:0/4019599657 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4180ad0b0 con 0x7fd4180cdd90 2023-11-28T12:58:50.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.367+0000 7fd41d7cf700 1 -- 172.21.15.111:0/4019599657 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4180acdd0 con 0x7fd4180cdd90 2023-11-28T12:58:50.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.368+0000 7fd41d7cf700 1 -- 172.21.15.111:0/4019599657 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4180addf0 con 0x7fd4180cdd90 2023-11-28T12:58:50.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.368+0000 7fd3ff7fe700 1 -- 172.21.15.111:0/4019599657 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7fd40801a020 con 0x7fd4180cdd90 2023-11-28T12:58:50.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.369+0000 7fd3ff7fe700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4040682a0 0x7fd40406a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:50.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.369+0000 7fd3ff7fe700 1 -- 172.21.15.111:0/4019599657 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd408085370 con 0x7fd4180cdd90 2023-11-28T12:58:50.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.369+0000 7fd416ffd700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4040682a0 0x7fd40406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:50.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.369+0000 7fd416ffd700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4040682a0 0x7fd40406a750 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd40c009b20 tx=0x7fd40c0091c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:50.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.373+0000 7fd3ff7fe700 1 -- 172.21.15.111:0/4019599657 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd408051f70 con 0x7fd4180cdd90 2023-11-28T12:58:50.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.558+0000 7fd41d7cf700 1 -- 172.21.15.111:0/4019599657 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd418067170 con 0x7fd4180cdd90 2023-11-28T12:58:50.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.559+0000 7fd3ff7fe700 1 -- 172.21.15.111:0/4019599657 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+14325 (secure 0 0 0) 0x7fd408055590 con 0x7fd4180cdd90 2023-11-28T12:58:50.560 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:50.560 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":17,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:58:50.192182+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:58:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.560+0000 7fd41d7cf700 1 -- 172.21.15.111:0/4019599657 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4040682a0 msgr2=0x7fd40406a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.560+0000 7fd41d7cf700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4040682a0 0x7fd40406a750 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd40c009b20 tx=0x7fd40c0091c0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.560+0000 7fd41d7cf700 1 -- 172.21.15.111:0/4019599657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4180cdd90 msgr2=0x7fd4180c57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.560+0000 7fd41d7cf700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4180cdd90 0x7fd4180c57f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd4080011e0 tx=0x7fd40800ca30 comp rx=0 tx=0).stop 2023-11-28T12:58:50.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.561+0000 7fd41d7cf700 1 -- 172.21.15.111:0/4019599657 shutdown_connections 2023-11-28T12:58:50.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.561+0000 7fd41d7cf700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd418159960 0x7fd4180c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.561+0000 7fd41d7cf700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4040682a0 0x7fd40406a750 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.561+0000 7fd41d7cf700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4180aecc0 0x7fd418153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.561+0000 7fd41d7cf700 1 --2- 172.21.15.111:0/4019599657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4180cdd90 0x7fd4180c57f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.561+0000 7fd41d7cf700 1 -- 172.21.15.111:0/4019599657 >> 172.21.15.111:0/4019599657 conn(0x7fd418098ee0 msgr2=0x7fd418155c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:50.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.561+0000 7fd41d7cf700 1 -- 172.21.15.111:0/4019599657 shutdown_connections 2023-11-28T12:58:50.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.561+0000 7fd41d7cf700 1 -- 172.21.15.111:0/4019599657 wait complete. 2023-11-28T12:58:50.576 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T12:58:50.577 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:58:50.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.698+0000 7f3b199b8700 1 Processor -- start 2023-11-28T12:58:50.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.698+0000 7f3b199b8700 1 -- start start 2023-11-28T12:58:50.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b199b8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3b140b8510 0x7f3b1406b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:50.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b199b8700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f3b140cd460 0x7f3b1406beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:50.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b199b8700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b140b88e0 con 0x7f3b1406c3f0 2023-11-28T12:58:50.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b199b8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b140cdbd0 con 0x7f3b140b8510 2023-11-28T12:58:50.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b199b8700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b140d02d0 con 0x7f3b140cd460 2023-11-28T12:58:50.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b12ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3b140b8510 0x7f3b1406b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:50.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b137fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3b1406c3f0 0x7f3b1415cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33222/0 (socket says 172.21.15.111:33222) 2023-11-28T12:58:50.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b137fe700 1 -- 172.21.15.111:0/1299564794 learned_addr learned my addr 172.21.15.111:0/1299564794 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:58:50.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b127fc700 1 --2- 172.21.15.111:0/1299564794 >> v2:172.21.15.149:3300/0 conn(0x7f3b140cd460 0x7f3b1406beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:50.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b12ffd700 1 -- 172.21.15.111:0/1299564794 >> v2:172.21.15.149:3300/0 conn(0x7f3b140cd460 msgr2=0x7f3b1406beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:50.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b12ffd700 1 --2- 172.21.15.111:0/1299564794 >> v2:172.21.15.149:3300/0 conn(0x7f3b140cd460 0x7f3b1406beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b12ffd700 1 -- 172.21.15.111:0/1299564794 >> v1:172.21.15.111:6789/0 conn(0x7f3b1406c3f0 legacy=0x7f3b1415cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b12ffd700 1 -- 172.21.15.111:0/1299564794 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b140cd8f0 con 0x7f3b140b8510 2023-11-28T12:58:50.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.699+0000 7f3b11ffb700 1 -- 172.21.15.111:0/1299564794 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580901302 0 0) 0x7f3b140b88e0 con 0x7f3b1406c3f0 2023-11-28T12:58:50.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.700+0000 7f3b12ffd700 1 --2- 172.21.15.111:0/1299564794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b140b8510 0x7f3b1406b970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3b08009870 tx=0x7f3b08009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36375efc01db32b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:50.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.700+0000 7f3b11ffb700 1 -- 172.21.15.111:0/1299564794 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b08011830 con 0x7f3b140b8510 2023-11-28T12:58:50.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.700+0000 7f3b11ffb700 1 -- 172.21.15.111:0/1299564794 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b08011990 con 0x7f3b140b8510 2023-11-28T12:58:50.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.700+0000 7f3b11ffb700 1 -- 172.21.15.111:0/1299564794 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b08011d20 con 0x7f3b140b8510 2023-11-28T12:58:50.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.700+0000 7f3b199b8700 1 -- 172.21.15.111:0/1299564794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b140b8510 msgr2=0x7f3b1406b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.700+0000 7f3b199b8700 1 --2- 172.21.15.111:0/1299564794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b140b8510 0x7f3b1406b970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3b08009870 tx=0x7f3b08009b80 comp rx=0 tx=0).stop 2023-11-28T12:58:50.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.700+0000 7f3b199b8700 1 -- 172.21.15.111:0/1299564794 shutdown_connections 2023-11-28T12:58:50.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.700+0000 7f3b199b8700 1 --2- 172.21.15.111:0/1299564794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b140b8510 0x7f3b1406b970 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.700+0000 7f3b199b8700 1 --2- 172.21.15.111:0/1299564794 >> v2:172.21.15.149:3300/0 conn(0x7f3b140cd460 0x7f3b1406beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.700+0000 7f3b199b8700 1 -- 172.21.15.111:0/1299564794 >> 172.21.15.111:0/1299564794 conn(0x7f3b140719a0 msgr2=0x7f3b14071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:50.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.700+0000 7f3b199b8700 1 -- 172.21.15.111:0/1299564794 shutdown_connections 2023-11-28T12:58:50.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.700+0000 7f3b199b8700 1 -- 172.21.15.111:0/1299564794 wait complete. 2023-11-28T12:58:50.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.701+0000 7f3b199b8700 1 Processor -- start 2023-11-28T12:58:50.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.702+0000 7f3b199b8700 1 -- start start 2023-11-28T12:58:50.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.702+0000 7f3b199b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b1406c3f0 0x7f3b141578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:50.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.702+0000 7f3b199b8700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3b140b8510 0x7f3b141f1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:50.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.702+0000 7f3b199b8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3b140cd460 0x7f3b140c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:50.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.702+0000 7f3b199b8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b14158e70 con 0x7f3b1406c3f0 2023-11-28T12:58:50.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.702+0000 7f3b199b8700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b14158250 con 0x7f3b140b8510 2023-11-28T12:58:50.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.702+0000 7f3b199b8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b14158530 con 0x7f3b140cd460 2023-11-28T12:58:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.702+0000 7f3b12ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b1406c3f0 0x7f3b141578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.702+0000 7f3b12ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b1406c3f0 0x7f3b141578e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60678/0 (socket says 172.21.15.111:60678) 2023-11-28T12:58:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3b12ffd700 1 -- 172.21.15.111:0/1324731169 learned_addr learned my addr 172.21.15.111:0/1324731169 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3b127fc700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3b140b8510 0x7f3b141f1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3b137fe700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3b140cd460 0x7f3b140c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3b12ffd700 1 -- 172.21.15.111:0/1324731169 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3b140cd460 msgr2=0x7f3b140c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3b12ffd700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3b140cd460 0x7f3b140c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3b12ffd700 1 -- 172.21.15.111:0/1324731169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3b140b8510 msgr2=0x7f3b141f1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3b12ffd700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3b140b8510 0x7f3b141f1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3b12ffd700 1 -- 172.21.15.111:0/1324731169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b080094a0 con 0x7f3b1406c3f0 2023-11-28T12:58:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3b12ffd700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b1406c3f0 0x7f3b141578e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3b08000c00 tx=0x7f3b08004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3afbfff700 1 -- 172.21.15.111:0/1324731169 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b080036a0 con 0x7f3b1406c3f0 2023-11-28T12:58:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3afbfff700 1 -- 172.21.15.111:0/1324731169 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0801f070 con 0x7f3b1406c3f0 2023-11-28T12:58:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3b199b8700 1 -- 172.21.15.111:0/1324731169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b140cfff0 con 0x7f3b1406c3f0 2023-11-28T12:58:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3afbfff700 1 -- 172.21.15.111:0/1324731169 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b08003a60 con 0x7f3b1406c3f0 2023-11-28T12:58:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.703+0000 7f3b199b8700 1 -- 172.21.15.111:0/1324731169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b140cfd10 con 0x7f3b1406c3f0 2023-11-28T12:58:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.704+0000 7f3b199b8700 1 -- 172.21.15.111:0/1324731169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b14158a50 con 0x7f3b1406c3f0 2023-11-28T12:58:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.704+0000 7f3afbfff700 1 -- 172.21.15.111:0/1324731169 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7f3b0800f050 con 0x7f3b1406c3f0 2023-11-28T12:58:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.705+0000 7f3afbfff700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3b000682d0 0x7f3b0006a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.705+0000 7f3afbfff700 1 -- 172.21.15.111:0/1324731169 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3b08021070 con 0x7f3b1406c3f0 2023-11-28T12:58:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.705+0000 7f3b127fc700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3b000682d0 0x7f3b0006a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.705+0000 7f3b127fc700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3b000682d0 0x7f3b0006a780 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3b140d1e00 tx=0x7f3b04009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.709+0000 7f3afbfff700 1 -- 172.21.15.111:0/1324731169 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3b080b85e0 con 0x7f3b1406c3f0 2023-11-28T12:58:50.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.892+0000 7f3b199b8700 1 -- 172.21.15.111:0/1324731169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3b14158c40 con 0x7f3b1406c3f0 2023-11-28T12:58:50.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.893+0000 7f3afbfff700 1 -- 172.21.15.111:0/1324731169 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3b14158c40 con 0x7f3b1406c3f0 2023-11-28T12:58:50.895 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:58:50.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.895+0000 7f3b199b8700 1 -- 172.21.15.111:0/1324731169 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3b000682d0 msgr2=0x7f3b0006a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.895+0000 7f3b199b8700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3b000682d0 0x7f3b0006a780 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3b140d1e00 tx=0x7f3b04009210 comp rx=0 tx=0).stop 2023-11-28T12:58:50.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.895+0000 7f3b199b8700 1 -- 172.21.15.111:0/1324731169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b1406c3f0 msgr2=0x7f3b141578e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:50.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.895+0000 7f3b199b8700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b1406c3f0 0x7f3b141578e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3b08000c00 tx=0x7f3b08004750 comp rx=0 tx=0).stop 2023-11-28T12:58:50.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.895+0000 7f3b199b8700 1 -- 172.21.15.111:0/1324731169 shutdown_connections 2023-11-28T12:58:50.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.895+0000 7f3b199b8700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3b140cd460 0x7f3b140c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.895+0000 7f3b199b8700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b1406c3f0 0x7f3b141578e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.895+0000 7f3b199b8700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3b000682d0 0x7f3b0006a780 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.895+0000 7f3b199b8700 1 --2- 172.21.15.111:0/1324731169 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3b140b8510 0x7f3b141f1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:50.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.895+0000 7f3b199b8700 1 -- 172.21.15.111:0/1324731169 >> 172.21.15.111:0/1324731169 conn(0x7f3b140719a0 msgr2=0x7f3b140b41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:50.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.895+0000 7f3b199b8700 1 -- 172.21.15.111:0/1324731169 shutdown_connections 2023-11-28T12:58:50.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:50.895+0000 7f3b199b8700 1 -- 172.21.15.111:0/1324731169 wait complete. 2023-11-28T12:58:50.908 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:58:51.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.030+0000 7f12b87d9700 1 Processor -- start 2023-11-28T12:58:51.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.030+0000 7f12b87d9700 1 -- start start 2023-11-28T12:58:51.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.030+0000 7f12b87d9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f12b015a5d0 0x7f12b00ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.030+0000 7f12b87d9700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f12b00cb2d0 0x7f12b00ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b87d9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b00ae8b0 con 0x7f12b00b0160 2023-11-28T12:58:51.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b87d9700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b00cc1b0 con 0x7f12b00cb2d0 2023-11-28T12:58:51.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b87d9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b00cc390 con 0x7f12b015a5d0 2023-11-28T12:58:51.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b5d74700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f12b00b0160 0x7f12b015e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33232/0 (socket says 172.21.15.111:33232) 2023-11-28T12:58:51.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b5d74700 1 -- 172.21.15.111:0/3378150975 learned_addr learned my addr 172.21.15.111:0/3378150975 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:58:51.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b6d76700 1 --2- 172.21.15.111:0/3378150975 >> v2:172.21.15.149:3300/0 conn(0x7f12b00cb2d0 0x7f12b00ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b6575700 1 --2- 172.21.15.111:0/3378150975 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f12b015a5d0 0x7f12b00ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b5573700 1 -- 172.21.15.111:0/3378150975 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895217117 0 0) 0x7f12b00ae8b0 con 0x7f12b00b0160 2023-11-28T12:58:51.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b5573700 1 -- 172.21.15.111:0/3378150975 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f129c0035b0 con 0x7f12b00b0160 2023-11-28T12:58:51.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b6575700 1 -- 172.21.15.111:0/3378150975 >> v2:172.21.15.149:3300/0 conn(0x7f12b00cb2d0 msgr2=0x7f12b00ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:51.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b6575700 1 --2- 172.21.15.111:0/3378150975 >> v2:172.21.15.149:3300/0 conn(0x7f12b00cb2d0 0x7f12b00ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b6575700 1 -- 172.21.15.111:0/3378150975 >> v1:172.21.15.111:6789/0 conn(0x7f12b00b0160 legacy=0x7f12b015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b6575700 1 -- 172.21.15.111:0/3378150975 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b00cc670 con 0x7f12b015a5d0 2023-11-28T12:58:51.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b6d76700 1 --2- 172.21.15.111:0/3378150975 >> v2:172.21.15.149:3300/0 conn(0x7f12b00cb2d0 0x7f12b00ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:51.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b6575700 1 --2- 172.21.15.111:0/3378150975 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f12b015a5d0 0x7f12b00ce850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009820 tx=0x7f12ac009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98b80cbed3bbd468 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:51.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b5573700 1 -- 172.21.15.111:0/3378150975 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac005e30 con 0x7f12b015a5d0 2023-11-28T12:58:51.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b5573700 1 -- 172.21.15.111:0/3378150975 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ac006030 con 0x7f12b015a5d0 2023-11-28T12:58:51.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.031+0000 7f12b5573700 1 -- 172.21.15.111:0/3378150975 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac0063a0 con 0x7f12b015a5d0 2023-11-28T12:58:51.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.032+0000 7f12b87d9700 1 -- 172.21.15.111:0/3378150975 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f12b015a5d0 msgr2=0x7f12b00ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.032+0000 7f12b87d9700 1 --2- 172.21.15.111:0/3378150975 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f12b015a5d0 0x7f12b00ce850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009820 tx=0x7f12ac009b30 comp rx=0 tx=0).stop 2023-11-28T12:58:51.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.032+0000 7f12b87d9700 1 -- 172.21.15.111:0/3378150975 shutdown_connections 2023-11-28T12:58:51.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.032+0000 7f12b87d9700 1 --2- 172.21.15.111:0/3378150975 >> v2:172.21.15.149:3300/0 conn(0x7f12b00cb2d0 0x7f12b00ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.032+0000 7f12b87d9700 1 --2- 172.21.15.111:0/3378150975 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f12b015a5d0 0x7f12b00ce850 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.032+0000 7f12b87d9700 1 -- 172.21.15.111:0/3378150975 >> 172.21.15.111:0/3378150975 conn(0x7f12b009a040 msgr2=0x7f12b00d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:51.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.032+0000 7f12b87d9700 1 -- 172.21.15.111:0/3378150975 shutdown_connections 2023-11-28T12:58:51.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.032+0000 7f12b87d9700 1 -- 172.21.15.111:0/3378150975 wait complete. 2023-11-28T12:58:51.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.033+0000 7f12b87d9700 1 Processor -- start 2023-11-28T12:58:51.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.033+0000 7f12b87d9700 1 -- start start 2023-11-28T12:58:51.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b87d9700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f12b00b0160 0x7f12b01554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b87d9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f12b00cb2d0 0x7f12b00ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b87d9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b015a5d0 0x7f12b01e1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b87d9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b00c4790 con 0x7f12b015a5d0 2023-11-28T12:58:51.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b87d9700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b00c4a30 con 0x7f12b00b0160 2023-11-28T12:58:51.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b87d9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b00c4d10 con 0x7f12b00cb2d0 2023-11-28T12:58:51.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b6d76700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b015a5d0 0x7f12b01e1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b6d76700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b015a5d0 0x7f12b01e1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60696/0 (socket says 172.21.15.111:60696) 2023-11-28T12:58:51.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b5d74700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f12b00cb2d0 0x7f12b00ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b6d76700 1 -- 172.21.15.111:0/255054660 learned_addr learned my addr 172.21.15.111:0/255054660 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:51.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b6575700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f12b00b0160 0x7f12b01554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b6d76700 1 -- 172.21.15.111:0/255054660 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f12b00cb2d0 msgr2=0x7f12b00ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b6d76700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f12b00cb2d0 0x7f12b00ca210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b6d76700 1 -- 172.21.15.111:0/255054660 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f12b00b0160 msgr2=0x7f12b01554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b6d76700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f12b00b0160 0x7f12b01554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b6d76700 1 -- 172.21.15.111:0/255054660 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a8009a70 con 0x7f12b015a5d0 2023-11-28T12:58:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.034+0000 7f12b6575700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f12b00b0160 0x7f12b01554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.035+0000 7f12b6d76700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b015a5d0 0x7f12b01e1110 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f12a800efb0 tx=0x7f12a800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.035+0000 7f12a6ffd700 1 -- 172.21.15.111:0/255054660 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a800cc90 con 0x7f12b015a5d0 2023-11-28T12:58:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.035+0000 7f12a6ffd700 1 -- 172.21.15.111:0/255054660 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a800f040 con 0x7f12b015a5d0 2023-11-28T12:58:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.035+0000 7f12b87d9700 1 -- 172.21.15.111:0/255054660 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ac009430 con 0x7f12b015a5d0 2023-11-28T12:58:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.035+0000 7f12a6ffd700 1 -- 172.21.15.111:0/255054660 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8014480 con 0x7f12b015a5d0 2023-11-28T12:58:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.035+0000 7f12b87d9700 1 -- 172.21.15.111:0/255054660 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b00ccc30 con 0x7f12b015a5d0 2023-11-28T12:58:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.036+0000 7f12b87d9700 1 -- 172.21.15.111:0/255054660 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b00cc950 con 0x7f12b015a5d0 2023-11-28T12:58:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.036+0000 7f12a6ffd700 1 -- 172.21.15.111:0/255054660 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7f12a80145e0 con 0x7f12b015a5d0 2023-11-28T12:58:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.036+0000 7f12a6ffd700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f12940682a0 0x7f129406a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.036+0000 7f12b6575700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f12940682a0 0x7f129406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.037+0000 7f12a6ffd700 1 -- 172.21.15.111:0/255054660 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f12a80866a0 con 0x7f12b015a5d0 2023-11-28T12:58:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.037+0000 7f12b6575700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f12940682a0 0x7f129406a750 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12ac000c00 tx=0x7f12ac008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.041+0000 7f12a6ffd700 1 -- 172.21.15.111:0/255054660 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f12a80532a0 con 0x7f12b015a5d0 2023-11-28T12:58:51.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.222+0000 7f12b87d9700 1 -- 172.21.15.111:0/255054660 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f12b00ccf10 con 0x7f12b015a5d0 2023-11-28T12:58:51.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.223+0000 7f12a6ffd700 1 -- 172.21.15.111:0/255054660 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v18) v1 ==== 92+0+10 (secure 0 0 0) 0x7f12a80180a0 con 0x7f12b015a5d0 2023-11-28T12:58:51.224 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:58:51.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.225+0000 7f12b87d9700 1 -- 172.21.15.111:0/255054660 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f12940682a0 msgr2=0x7f129406a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.225+0000 7f12b87d9700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f12940682a0 0x7f129406a750 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12ac000c00 tx=0x7f12ac008070 comp rx=0 tx=0).stop 2023-11-28T12:58:51.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.225+0000 7f12b87d9700 1 -- 172.21.15.111:0/255054660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b015a5d0 msgr2=0x7f12b01e1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.225+0000 7f12b87d9700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b015a5d0 0x7f12b01e1110 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f12a800efb0 tx=0x7f12a800c3a0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.225+0000 7f12b87d9700 1 -- 172.21.15.111:0/255054660 shutdown_connections 2023-11-28T12:58:51.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.225+0000 7f12b87d9700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b015a5d0 0x7f12b01e1110 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.225+0000 7f12b87d9700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f12940682a0 0x7f129406a750 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.225+0000 7f12b87d9700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f12b00b0160 0x7f12b01554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.225+0000 7f12b87d9700 1 --2- 172.21.15.111:0/255054660 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f12b00cb2d0 0x7f12b00ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.225+0000 7f12b87d9700 1 -- 172.21.15.111:0/255054660 >> 172.21.15.111:0/255054660 conn(0x7f12b009a040 msgr2=0x7f12b00d0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:51.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.225+0000 7f12b87d9700 1 -- 172.21.15.111:0/255054660 shutdown_connections 2023-11-28T12:58:51.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.225+0000 7f12b87d9700 1 -- 172.21.15.111:0/255054660 wait complete. 2023-11-28T12:58:51.238 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T12:58:51.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.361+0000 7fd827ae8700 1 Processor -- start 2023-11-28T12:58:51.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.361+0000 7fd827ae8700 1 -- start start 2023-11-28T12:58:51.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.362+0000 7fd827ae8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd8200b0d70 0x7fd8201604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.362+0000 7fd827ae8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd8200b15b0 0x7fd8200cc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.362+0000 7fd827ae8700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd82015cd90 con 0x7fd8200b1140 2023-11-28T12:58:51.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.362+0000 7fd827ae8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8200cedd0 con 0x7fd8200b15b0 2023-11-28T12:58:51.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.362+0000 7fd827ae8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8201613e0 con 0x7fd8200b0d70 2023-11-28T12:58:51.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.362+0000 7fd826085700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd8200b15b0 0x7fd8200cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.362+0000 7fd826085700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd8200b15b0 0x7fd8200cc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60706/0 (socket says 172.21.15.111:60706) 2023-11-28T12:58:51.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.362+0000 7fd826085700 1 -- 172.21.15.111:0/1441077962 learned_addr learned my addr 172.21.15.111:0/1441077962 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:51.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.363+0000 7fd825884700 1 --2- 172.21.15.111:0/1441077962 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd8200b0d70 0x7fd8201604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.363+0000 7fd826085700 1 -- 172.21.15.111:0/1441077962 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd8200b0d70 msgr2=0x7fd8201604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:51.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.363+0000 7fd826085700 1 --2- 172.21.15.111:0/1441077962 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd8200b0d70 0x7fd8201604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.363+0000 7fd826085700 1 -- 172.21.15.111:0/1441077962 >> v1:172.21.15.149:6789/0 conn(0x7fd8200b1140 legacy=0x7fd8200ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:51.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.363+0000 7fd826085700 1 -- 172.21.15.111:0/1441077962 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd820160b40 con 0x7fd8200b15b0 2023-11-28T12:58:51.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.363+0000 7fd825884700 1 --2- 172.21.15.111:0/1441077962 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd8200b0d70 0x7fd8201604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:58:51.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.363+0000 7fd826085700 1 --2- 172.21.15.111:0/1441077962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8200b15b0 0x7fd8200cc550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd818009e50 tx=0x7fd81800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86c586b40946288 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:51.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.363+0000 7fd824882700 1 -- 172.21.15.111:0/1441077962 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81800e070 con 0x7fd8200b15b0 2023-11-28T12:58:51.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.363+0000 7fd824882700 1 -- 172.21.15.111:0/1441077962 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd818003a20 con 0x7fd8200b15b0 2023-11-28T12:58:51.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.363+0000 7fd824882700 1 -- 172.21.15.111:0/1441077962 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd818003dd0 con 0x7fd8200b15b0 2023-11-28T12:58:51.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.363+0000 7fd827ae8700 1 -- 172.21.15.111:0/1441077962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8200b15b0 msgr2=0x7fd8200cc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.363+0000 7fd827ae8700 1 --2- 172.21.15.111:0/1441077962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8200b15b0 0x7fd8200cc550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd818009e50 tx=0x7fd81800b330 comp rx=0 tx=0).stop 2023-11-28T12:58:51.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.364+0000 7fd827ae8700 1 -- 172.21.15.111:0/1441077962 shutdown_connections 2023-11-28T12:58:51.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.364+0000 7fd827ae8700 1 --2- 172.21.15.111:0/1441077962 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd8200b0d70 0x7fd8201604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.364+0000 7fd827ae8700 1 --2- 172.21.15.111:0/1441077962 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8200b15b0 0x7fd8200cc550 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.364+0000 7fd827ae8700 1 -- 172.21.15.111:0/1441077962 >> 172.21.15.111:0/1441077962 conn(0x7fd820098ee0 msgr2=0x7fd82006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:51.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.364+0000 7fd827ae8700 1 -- 172.21.15.111:0/1441077962 shutdown_connections 2023-11-28T12:58:51.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.364+0000 7fd827ae8700 1 -- 172.21.15.111:0/1441077962 wait complete. 2023-11-28T12:58:51.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.365+0000 7fd827ae8700 1 Processor -- start 2023-11-28T12:58:51.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.365+0000 7fd827ae8700 1 -- start start 2023-11-28T12:58:51.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.365+0000 7fd827ae8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8200b0d70 0x7fd8201f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.365+0000 7fd827ae8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd8200b1140 0x7fd8200cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.365+0000 7fd827ae8700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd8200b15b0 0x7fd8200cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.365+0000 7fd827ae8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8200b39a0 con 0x7fd8200b0d70 2023-11-28T12:58:51.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd827ae8700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8200b5740 con 0x7fd8200b15b0 2023-11-28T12:58:51.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd827ae8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82015e1e0 con 0x7fd8200b1140 2023-11-28T12:58:51.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd825884700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8200b0d70 0x7fd8201f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd825884700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8200b0d70 0x7fd8201f1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60708/0 (socket says 172.21.15.111:60708) 2023-11-28T12:58:51.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd825884700 1 -- 172.21.15.111:0/540215955 learned_addr learned my addr 172.21.15.111:0/540215955 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:51.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd826085700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd8200b15b0 0x7fd8200cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd825083700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd8200b1140 0x7fd8200cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd825884700 1 -- 172.21.15.111:0/540215955 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd8200b1140 msgr2=0x7fd8200cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd825884700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd8200b1140 0x7fd8200cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd825884700 1 -- 172.21.15.111:0/540215955 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd8200b15b0 msgr2=0x7fd8200cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd825884700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd8200b15b0 0x7fd8200cc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd825884700 1 -- 172.21.15.111:0/540215955 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd818009b30 con 0x7fd8200b0d70 2023-11-28T12:58:51.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd826085700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd8200b15b0 0x7fd8200cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:58:51.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd825884700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8200b0d70 0x7fd8201f1c30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00c7f0 tx=0x7fd81c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:51.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd8167fc700 1 -- 172.21.15.111:0/540215955 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c005c20 con 0x7fd8200b0d70 2023-11-28T12:58:51.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd8167fc700 1 -- 172.21.15.111:0/540215955 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81c005d80 con 0x7fd8200b0d70 2023-11-28T12:58:51.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd827ae8700 1 -- 172.21.15.111:0/540215955 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd820161100 con 0x7fd8200b0d70 2023-11-28T12:58:51.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd8167fc700 1 -- 172.21.15.111:0/540215955 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c011820 con 0x7fd8200b0d70 2023-11-28T12:58:51.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.366+0000 7fd827ae8700 1 -- 172.21.15.111:0/540215955 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd820160e20 con 0x7fd8200b0d70 2023-11-28T12:58:51.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.367+0000 7fd827ae8700 1 -- 172.21.15.111:0/540215955 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8200af340 con 0x7fd8200b0d70 2023-11-28T12:58:51.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.367+0000 7fd8167fc700 1 -- 172.21.15.111:0/540215955 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd81c020070 con 0x7fd8200b0d70 2023-11-28T12:58:51.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.368+0000 7fd8167fc700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd8040682f0 0x7fd80406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.368+0000 7fd8167fc700 1 -- 172.21.15.111:0/540215955 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd81c085600 con 0x7fd8200b0d70 2023-11-28T12:58:51.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.368+0000 7fd825083700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd8040682f0 0x7fd80406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.368+0000 7fd825083700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd8040682f0 0x7fd80406a7a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd82015b790 tx=0x7fd810009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:51.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.372+0000 7fd8167fc700 1 -- 172.21.15.111:0/540215955 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd81c0521b0 con 0x7fd8200b0d70 2023-11-28T12:58:51.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.600+0000 7fd827ae8700 1 -- 172.21.15.111:0/540215955 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd820067170 con 0x7fd8200b0d70 2023-11-28T12:58:51.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.601+0000 7fd8167fc700 1 -- 172.21.15.111:0/540215955 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3536 (secure 0 0 0) 0x7fd81c01e090 con 0x7fd8200b0d70 2023-11-28T12:58:51.606 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:51.606 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2513950989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2513950989},{"type":"v1","addr":"172.21.15.149:6834","nonce":2513950989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1258891714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1258891714},{"type":"v1","addr":"172.21.15.149:6835","nonce":1258891714}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/3923909288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3923909288},{"type":"v1","addr":"172.21.15.111:6836","nonce":3923909288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3180164366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":3180164366},{"type":"v1","addr":"172.21.15.111:6837","nonce":3180164366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-11-28T12:58:51.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.603+0000 7fd827ae8700 1 -- 172.21.15.111:0/540215955 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd8040682f0 msgr2=0x7fd80406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.603+0000 7fd827ae8700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd8040682f0 0x7fd80406a7a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd82015b790 tx=0x7fd810009040 comp rx=0 tx=0).stop 2023-11-28T12:58:51.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.603+0000 7fd827ae8700 1 -- 172.21.15.111:0/540215955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8200b0d70 msgr2=0x7fd8201f1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.603+0000 7fd827ae8700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8200b0d70 0x7fd8201f1c30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00c7f0 tx=0x7fd81c00cbb0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.603+0000 7fd827ae8700 1 -- 172.21.15.111:0/540215955 shutdown_connections 2023-11-28T12:58:51.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.603+0000 7fd827ae8700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd8040682f0 0x7fd80406a7a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.603+0000 7fd827ae8700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd8200b15b0 0x7fd8200cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.603+0000 7fd827ae8700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8200b0d70 0x7fd8201f1c30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.603+0000 7fd827ae8700 1 --2- 172.21.15.111:0/540215955 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd8200b1140 0x7fd8200cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.603+0000 7fd827ae8700 1 -- 172.21.15.111:0/540215955 >> 172.21.15.111:0/540215955 conn(0x7fd820098ee0 msgr2=0x7fd82006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:51.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.603+0000 7fd827ae8700 1 -- 172.21.15.111:0/540215955 shutdown_connections 2023-11-28T12:58:51.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.603+0000 7fd827ae8700 1 -- 172.21.15.111:0/540215955 wait complete. 2023-11-28T12:58:51.610 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 3 2023-11-28T12:58:51.618 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:58:51.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.736+0000 7ff8b9695700 1 Processor -- start 2023-11-28T12:58:51.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.736+0000 7ff8b9695700 1 -- start start 2023-11-28T12:58:51.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b9695700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff8b40b8510 0x7ff8b406b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b9695700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b40b88e0 con 0x7ff8b40d0d60 2023-11-28T12:58:51.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b9695700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b40cd8f0 con 0x7ff8b40cd460 2023-11-28T12:58:51.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b9695700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b40cdbd0 con 0x7ff8b40b8510 2023-11-28T12:58:51.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b37fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff8b40d0d60 0x7ff8b40d1200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33262/0 (socket says 172.21.15.111:33262) 2023-11-28T12:58:51.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b37fe700 1 -- 172.21.15.111:0/3908828774 learned_addr learned my addr 172.21.15.111:0/3908828774 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:58:51.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b2ffd700 1 --2- 172.21.15.111:0/3908828774 >> v2:172.21.15.149:3300/0 conn(0x7ff8b40b8510 0x7ff8b406b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050142526 0 0) 0x7ff8b40b88e0 con 0x7ff8b40d0d60 2023-11-28T12:58:51.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff89c0035b0 con 0x7ff8b40d0d60 2023-11-28T12:58:51.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 836441157 0 0) 0x7ff89c0035b0 con 0x7ff8b40d0d60 2023-11-28T12:58:51.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 >> v2:172.21.15.149:3300/0 conn(0x7ff8b40b8510 msgr2=0x7ff8b406b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:51.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b1ffb700 1 --2- 172.21.15.111:0/3908828774 >> v2:172.21.15.149:3300/0 conn(0x7ff8b40b8510 0x7ff8b406b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 >> v1:172.21.15.149:6789/0 conn(0x7ff8b40cd460 legacy=0x7ff8b406beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b40cfd10 con 0x7ff8b40d0d60 2023-11-28T12:58:51.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7ff8ac0019f0 con 0x7ff8b40d0d60 2023-11-28T12:58:51.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 >> v1:172.21.15.111:6789/0 conn(0x7ff8b40d0d60 legacy=0x7ff8b40d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b1ffb700 1 --2- 172.21.15.111:0/3908828774 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff89c003a00 0x7ff89c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b1ffb700 1 --2- 172.21.15.111:0/3908828774 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff89c0043f0 0x7ff89c0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.737+0000 7ff8b1ffb700 1 --2- 172.21.15.111:0/3908828774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c0050b0 0x7ff89c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b40cdbd0 con 0x7ff89c0050b0 2023-11-28T12:58:51.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c00c000 con 0x7ff89c0043f0 2023-11-28T12:58:51.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c00c1c0 con 0x7ff89c003a00 2023-11-28T12:58:51.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b2ffd700 1 --2- 172.21.15.111:0/3908828774 >> v2:172.21.15.149:3300/0 conn(0x7ff8b40b8510 0x7ff8b406b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:51.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b37fe700 1 --2- 172.21.15.111:0/3908828774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c0050b0 0x7ff89c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b27fc700 1 --2- 172.21.15.111:0/3908828774 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff89c0043f0 0x7ff89c0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b2ffd700 1 --2- 172.21.15.111:0/3908828774 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff89c003a00 0x7ff89c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b37fe700 1 -- 172.21.15.111:0/3908828774 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff89c003a00 msgr2=0x7ff89c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:51.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b37fe700 1 --2- 172.21.15.111:0/3908828774 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff89c003a00 0x7ff89c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b37fe700 1 -- 172.21.15.111:0/3908828774 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff89c0043f0 msgr2=0x7ff89c0048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:51.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b37fe700 1 --2- 172.21.15.111:0/3908828774 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff89c0043f0 0x7ff89c0048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b37fe700 1 -- 172.21.15.111:0/3908828774 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89c0035b0 con 0x7ff89c0050b0 2023-11-28T12:58:51.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b37fe700 1 -- 172.21.15.111:0/3908828774 reap_dead start 2023-11-28T12:58:51.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b37fe700 1 --2- 172.21.15.111:0/3908828774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c0050b0 0x7ff89c00ba10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00a830 tx=0x7ff89c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6cfe7ef3bd4df7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:51.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac012600 con 0x7ff89c0050b0 2023-11-28T12:58:51.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ac012780 con 0x7ff89c0050b0 2023-11-28T12:58:51.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.738+0000 7ff8b1ffb700 1 -- 172.21.15.111:0/3908828774 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac012b30 con 0x7ff89c0050b0 2023-11-28T12:58:51.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.739+0000 7ff8b9695700 1 -- 172.21.15.111:0/3908828774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c0050b0 msgr2=0x7ff89c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.739+0000 7ff8b9695700 1 --2- 172.21.15.111:0/3908828774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c0050b0 0x7ff89c00ba10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00a830 tx=0x7ff89c005880 comp rx=0 tx=0).stop 2023-11-28T12:58:51.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.739+0000 7ff8b9695700 1 -- 172.21.15.111:0/3908828774 shutdown_connections 2023-11-28T12:58:51.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.739+0000 7ff8b9695700 1 --2- 172.21.15.111:0/3908828774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c0050b0 0x7ff89c00ba10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.739+0000 7ff8b9695700 1 -- 172.21.15.111:0/3908828774 >> 172.21.15.111:0/3908828774 conn(0x7ff8b40719a0 msgr2=0x7ff8b4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:51.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.739+0000 7ff8b9695700 1 -- 172.21.15.111:0/3908828774 shutdown_connections 2023-11-28T12:58:51.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.739+0000 7ff8b9695700 1 -- 172.21.15.111:0/3908828774 wait complete. 2023-11-28T12:58:51.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.740+0000 7ff8b9695700 1 Processor -- start 2023-11-28T12:58:51.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.740+0000 7ff8b9695700 1 -- start start 2023-11-28T12:58:51.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b9695700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c0050b0 0x7ff8b4158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b9695700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8b40b76c0 0x7ff8b41f49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b9695700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b41f19c0 0x7ff8b40cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b9695700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b4158a00 con 0x7ff89c0050b0 2023-11-28T12:58:51.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b9695700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b4158ce0 con 0x7ff8b41f19c0 2023-11-28T12:58:51.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b9695700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b4158fc0 con 0x7ff8b40b76c0 2023-11-28T12:58:51.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c0050b0 0x7ff8b4158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c0050b0 0x7ff8b4158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60726/0 (socket says 172.21.15.111:60726) 2023-11-28T12:58:51.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b2ffd700 1 -- 172.21.15.111:0/249586684 learned_addr learned my addr 172.21.15.111:0/249586684 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:51.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b27fc700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8b40b76c0 0x7ff8b41f49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b37fe700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b41f19c0 0x7ff8b40cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b2ffd700 1 -- 172.21.15.111:0/249586684 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8b40b76c0 msgr2=0x7ff8b41f49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b2ffd700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8b40b76c0 0x7ff8b41f49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b2ffd700 1 -- 172.21.15.111:0/249586684 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b41f19c0 msgr2=0x7ff8b40cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b2ffd700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b41f19c0 0x7ff8b40cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b2ffd700 1 -- 172.21.15.111:0/249586684 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b40d29c0 con 0x7ff89c0050b0 2023-11-28T12:58:51.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b37fe700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b41f19c0 0x7ff8b40cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:58:51.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b2ffd700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c0050b0 0x7ff8b4158420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8010a30 tx=0x7ff8a8010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:51.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff89b7fe700 1 -- 172.21.15.111:0/249586684 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a800e870 con 0x7ff89c0050b0 2023-11-28T12:58:51.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff89b7fe700 1 -- 172.21.15.111:0/249586684 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a800e9d0 con 0x7ff89c0050b0 2023-11-28T12:58:51.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.741+0000 7ff8b9695700 1 -- 172.21.15.111:0/249586684 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b40cfff0 con 0x7ff89c0050b0 2023-11-28T12:58:51.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.742+0000 7ff89b7fe700 1 -- 172.21.15.111:0/249586684 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a800ed80 con 0x7ff89c0050b0 2023-11-28T12:58:51.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.742+0000 7ff8b9695700 1 -- 172.21.15.111:0/249586684 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b40d2b50 con 0x7ff89c0050b0 2023-11-28T12:58:51.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.742+0000 7ff8b9695700 1 -- 172.21.15.111:0/249586684 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b41598f0 con 0x7ff89c0050b0 2023-11-28T12:58:51.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.743+0000 7ff89b7fe700 1 -- 172.21.15.111:0/249586684 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff8a801c030 con 0x7ff89c0050b0 2023-11-28T12:58:51.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.743+0000 7ff89b7fe700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8a00682f0 0x7ff8a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:51.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.743+0000 7ff89b7fe700 1 -- 172.21.15.111:0/249586684 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff8a8085ce0 con 0x7ff89c0050b0 2023-11-28T12:58:51.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.743+0000 7ff8b27fc700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8a00682f0 0x7ff8a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:51.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.744+0000 7ff8b27fc700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8a00682f0 0x7ff8a006a7a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40b7650 tx=0x7ff8a4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:51.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.747+0000 7ff89b7fe700 1 -- 172.21.15.111:0/249586684 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff8a8052830 con 0x7ff89c0050b0 2023-11-28T12:58:51.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.933+0000 7ff8b9695700 1 -- 172.21.15.111:0/249586684 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8b4067880 con 0x7ff89c0050b0 2023-11-28T12:58:51.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.933+0000 7ff89b7fe700 1 -- 172.21.15.111:0/249586684 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+14312 (secure 0 0 0) 0x7ff8a8009300 con 0x7ff89c0050b0 2023-11-28T12:58:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:51.935 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":18,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:58:51.203559+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:58:51.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.935+0000 7ff8b9695700 1 -- 172.21.15.111:0/249586684 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8a00682f0 msgr2=0x7ff8a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.935+0000 7ff8b9695700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8a00682f0 0x7ff8a006a7a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40b7650 tx=0x7ff8a4009040 comp rx=0 tx=0).stop 2023-11-28T12:58:51.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.935+0000 7ff8b9695700 1 -- 172.21.15.111:0/249586684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c0050b0 msgr2=0x7ff8b4158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:51.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.935+0000 7ff8b9695700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c0050b0 0x7ff8b4158420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8010a30 tx=0x7ff8a8010d40 comp rx=0 tx=0).stop 2023-11-28T12:58:51.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.935+0000 7ff8b9695700 1 -- 172.21.15.111:0/249586684 shutdown_connections 2023-11-28T12:58:51.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.935+0000 7ff8b9695700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8a00682f0 0x7ff8a006a7a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.935+0000 7ff8b9695700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b41f19c0 0x7ff8b40cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.935+0000 7ff8b9695700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c0050b0 0x7ff8b4158420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.935+0000 7ff8b9695700 1 --2- 172.21.15.111:0/249586684 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8b40b76c0 0x7ff8b41f49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:51.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.935+0000 7ff8b9695700 1 -- 172.21.15.111:0/249586684 >> 172.21.15.111:0/249586684 conn(0x7ff8b40719a0 msgr2=0x7ff8b40b3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:51.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.936+0000 7ff8b9695700 1 -- 172.21.15.111:0/249586684 shutdown_connections 2023-11-28T12:58:51.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:51.936+0000 7ff8b9695700 1 -- 172.21.15.111:0/249586684 wait complete. 2023-11-28T12:58:51.949 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T12:58:51.950 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:58:52.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.070+0000 7f01a6d54700 1 Processor -- start 2023-11-28T12:58:52.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.070+0000 7f01a6d54700 1 -- start start 2023-11-28T12:58:52.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.070+0000 7f01a6d54700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01a00b0960 0x7f01a00ac670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.070+0000 7f01a6d54700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a00af650 con 0x7f01a015a9e0 2023-11-28T12:58:52.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.070+0000 7f01a6d54700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a00cb6d0 con 0x7f01a00b15c0 2023-11-28T12:58:52.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.070+0000 7f01a6d54700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00cb9b0 con 0x7f01a00b0960 2023-11-28T12:58:52.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f01a4af0700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f01a015a9e0 0x7f01a00caef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33280/0 (socket says 172.21.15.111:33280) 2023-11-28T12:58:52.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f01a4af0700 1 -- 172.21.15.111:0/2881727414 learned_addr learned my addr 172.21.15.111:0/2881727414 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:58:52.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626656215 0 0) 0x7f01a00af650 con 0x7f01a015a9e0 2023-11-28T12:58:52.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f01a52f1700 1 --2- 172.21.15.111:0/2881727414 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01a00b0960 0x7f01a00ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01900035b0 con 0x7f01a015a9e0 2023-11-28T12:58:52.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 286018607 0 0) 0x7f01900035b0 con 0x7f01a015a9e0 2023-11-28T12:58:52.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01a00b0960 msgr2=0x7f01a00ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:52.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 --2- 172.21.15.111:0/2881727414 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01a00b0960 0x7f01a00ac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 >> v1:172.21.15.149:6789/0 conn(0x7f01a00b15c0 legacy=0x7f01a00b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a00cbc90 con 0x7f01a015a9e0 2023-11-28T12:58:52.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f01940017f0 con 0x7f01a015a9e0 2023-11-28T12:58:52.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 >> v1:172.21.15.111:6789/0 conn(0x7f01a015a9e0 legacy=0x7f01a00caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 --2- 172.21.15.111:0/2881727414 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01a00b0960 0x7f0190003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 --2- 172.21.15.111:0/2881727414 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0190003fb0 0x7f0190004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 --2- 172.21.15.111:0/2881727414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0190004d90 0x7f0190005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00cb9b0 con 0x7f0190004d90 2023-11-28T12:58:52.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019000baf0 con 0x7f0190003fb0 2023-11-28T12:58:52.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019000bcb0 con 0x7f01a00b0960 2023-11-28T12:58:52.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f01a52f1700 1 --2- 172.21.15.111:0/2881727414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0190004d90 0x7f0190005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f01a4af0700 1 --2- 172.21.15.111:0/2881727414 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01a00b0960 0x7f0190003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.071+0000 7f019ffff700 1 --2- 172.21.15.111:0/2881727414 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0190003fb0 0x7f0190004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f01a52f1700 1 -- 172.21.15.111:0/2881727414 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01a00b0960 msgr2=0x7f0190003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:52.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f01a52f1700 1 --2- 172.21.15.111:0/2881727414 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01a00b0960 0x7f0190003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f01a52f1700 1 -- 172.21.15.111:0/2881727414 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0190003fb0 msgr2=0x7f0190004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:52.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f01a52f1700 1 --2- 172.21.15.111:0/2881727414 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0190003fb0 0x7f0190004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f01a52f1700 1 -- 172.21.15.111:0/2881727414 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01900035b0 con 0x7f0190004d90 2023-11-28T12:58:52.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f01a52f1700 1 --2- 172.21.15.111:0/2881727414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0190004d90 0x7f0190005200 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0198009f90 tx=0x7f0190005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0f24aab027f12b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:52.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019800ec30 con 0x7f0190004d90 2023-11-28T12:58:52.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019800f040 con 0x7f0190004d90 2023-11-28T12:58:52.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f019f7fe700 1 -- 172.21.15.111:0/2881727414 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0198014480 con 0x7f0190004d90 2023-11-28T12:58:52.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f01a6d54700 1 -- 172.21.15.111:0/2881727414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0190004d90 msgr2=0x7f0190005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f01a6d54700 1 --2- 172.21.15.111:0/2881727414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0190004d90 0x7f0190005200 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0198009f90 tx=0x7f0190005880 comp rx=0 tx=0).stop 2023-11-28T12:58:52.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f01a52f1700 1 -- 172.21.15.111:0/2881727414 reap_dead start 2023-11-28T12:58:52.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f01a6d54700 1 -- 172.21.15.111:0/2881727414 shutdown_connections 2023-11-28T12:58:52.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f01a6d54700 1 -- 172.21.15.111:0/2881727414 >> 172.21.15.111:0/2881727414 conn(0x7f01a0098ee0 msgr2=0x7f01a006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:52.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f01a6d54700 1 -- 172.21.15.111:0/2881727414 shutdown_connections 2023-11-28T12:58:52.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.072+0000 7f01a6d54700 1 -- 172.21.15.111:0/2881727414 wait complete. 2023-11-28T12:58:52.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.074+0000 7f01a6d54700 1 Processor -- start 2023-11-28T12:58:52.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.074+0000 7f01a6d54700 1 -- start start 2023-11-28T12:58:52.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f01a6d54700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01a01491e0 0x7f01a00c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f01a6d54700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01a01565a0 0x7f01a00afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f01a6d54700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01a00b0030 0x7f01a00c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f01a6d54700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a015de40 con 0x7f01a01565a0 2023-11-28T12:58:52.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f01a6d54700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00b0480 con 0x7f01a00b0030 2023-11-28T12:58:52.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f01a6d54700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a015d270 con 0x7f01a01491e0 2023-11-28T12:58:52.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f019ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01a01565a0 0x7f01a00afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f019ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01a01565a0 0x7f01a00afaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60746/0 (socket says 172.21.15.111:60746) 2023-11-28T12:58:52.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f019ffff700 1 -- 172.21.15.111:0/3440964157 learned_addr learned my addr 172.21.15.111:0/3440964157 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:52.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f01a52f1700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01a00b0030 0x7f01a00c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f01a4af0700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01a01491e0 0x7f01a00c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f019ffff700 1 -- 172.21.15.111:0/3440964157 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01a01491e0 msgr2=0x7f01a00c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f019ffff700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01a01491e0 0x7f01a00c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f019ffff700 1 -- 172.21.15.111:0/3440964157 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01a00b0030 msgr2=0x7f01a00c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f019ffff700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01a00b0030 0x7f01a00c7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.075+0000 7f019ffff700 1 -- 172.21.15.111:0/3440964157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a00cc250 con 0x7f01a01565a0 2023-11-28T12:58:52.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.076+0000 7f019ffff700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01a01565a0 0x7f01a00afaf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f018c00f820 tx=0x7f018c00fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:52.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.076+0000 7f019d7fa700 1 -- 172.21.15.111:0/3440964157 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c0052e0 con 0x7f01a01565a0 2023-11-28T12:58:52.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.076+0000 7f019d7fa700 1 -- 172.21.15.111:0/3440964157 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018c005440 con 0x7f01a01565a0 2023-11-28T12:58:52.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.076+0000 7f01a6d54700 1 -- 172.21.15.111:0/3440964157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a00cbf70 con 0x7f01a01565a0 2023-11-28T12:58:52.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.076+0000 7f019d7fa700 1 -- 172.21.15.111:0/3440964157 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c0038c0 con 0x7f01a01565a0 2023-11-28T12:58:52.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.076+0000 7f01a6d54700 1 -- 172.21.15.111:0/3440964157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a00ac670 con 0x7f01a01565a0 2023-11-28T12:58:52.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.077+0000 7f01a6d54700 1 -- 172.21.15.111:0/3440964157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a0067170 con 0x7f01a01565a0 2023-11-28T12:58:52.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.077+0000 7f019d7fa700 1 -- 172.21.15.111:0/3440964157 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f018c003a20 con 0x7f01a01565a0 2023-11-28T12:58:52.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.077+0000 7f019d7fa700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01840682f0 0x7f018406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.077+0000 7f019d7fa700 1 -- 172.21.15.111:0/3440964157 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f018c085830 con 0x7f01a01565a0 2023-11-28T12:58:52.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.077+0000 7f01a4af0700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01840682f0 0x7f018406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.078+0000 7f01a4af0700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01840682f0 0x7f018406a7a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0194000ef0 tx=0x7f0194009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:52.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.082+0000 7f019d7fa700 1 -- 172.21.15.111:0/3440964157 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f018c0522f0 con 0x7f01a01565a0 2023-11-28T12:58:52.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.266+0000 7f01a6d54700 1 -- 172.21.15.111:0/3440964157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f01a00717d0 con 0x7f01a01565a0 2023-11-28T12:58:52.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.266+0000 7f019d7fa700 1 -- 172.21.15.111:0/3440964157 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f018c003cb0 con 0x7f01a01565a0 2023-11-28T12:58:52.268 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:58:52.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.268+0000 7f01a6d54700 1 -- 172.21.15.111:0/3440964157 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01840682f0 msgr2=0x7f018406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.268+0000 7f01a6d54700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01840682f0 0x7f018406a7a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0194000ef0 tx=0x7f0194009040 comp rx=0 tx=0).stop 2023-11-28T12:58:52.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.268+0000 7f01a6d54700 1 -- 172.21.15.111:0/3440964157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01a01565a0 msgr2=0x7f01a00afaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.268+0000 7f01a6d54700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01a01565a0 0x7f01a00afaf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f018c00f820 tx=0x7f018c00fb30 comp rx=0 tx=0).stop 2023-11-28T12:58:52.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.268+0000 7f01a6d54700 1 -- 172.21.15.111:0/3440964157 shutdown_connections 2023-11-28T12:58:52.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.268+0000 7f01a6d54700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01840682f0 0x7f018406a7a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.268+0000 7f01a6d54700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01a00b0030 0x7f01a00c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.268+0000 7f01a6d54700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01a01491e0 0x7f01a00c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.268+0000 7f01a6d54700 1 --2- 172.21.15.111:0/3440964157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01a01565a0 0x7f01a00afaf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.268+0000 7f01a6d54700 1 -- 172.21.15.111:0/3440964157 >> 172.21.15.111:0/3440964157 conn(0x7f01a0098ee0 msgr2=0x7f01a00aeca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:52.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.268+0000 7f01a6d54700 1 -- 172.21.15.111:0/3440964157 shutdown_connections 2023-11-28T12:58:52.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.268+0000 7f01a6d54700 1 -- 172.21.15.111:0/3440964157 wait complete. 2023-11-28T12:58:52.281 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:58:52.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.402+0000 7fa7c1349700 1 Processor -- start 2023-11-28T12:58:52.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.402+0000 7fa7c1349700 1 -- start start 2023-11-28T12:58:52.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7c1349700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fa7bc15cbf0 0x7fa7bc0b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7c1349700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa7bc0b40a0 0x7fa7bc06b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7c1349700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc0cfdb0 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7c1349700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7bc0cc740 con 0x7fa7bc0b40a0 2023-11-28T12:58:52.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7c1349700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7bc0b35e0 con 0x7fa7bc15cbf0 2023-11-28T12:58:52.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7ba7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa7bc0b40a0 0x7fa7bc06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7ba7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa7bc0b40a0 0x7fa7bc06b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60752/0 (socket says 172.21.15.111:60752) 2023-11-28T12:58:52.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7ba7fc700 1 -- 172.21.15.111:0/3574841180 learned_addr learned my addr 172.21.15.111:0/3574841180 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:52.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7baffd700 1 --2- 172.21.15.111:0/3574841180 >> v2:172.21.15.149:3300/0 conn(0x7fa7bc15cbf0 0x7fa7bc0b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7ba7fc700 1 -- 172.21.15.111:0/3574841180 >> v2:172.21.15.149:3300/0 conn(0x7fa7bc15cbf0 msgr2=0x7fa7bc0b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:52.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7ba7fc700 1 --2- 172.21.15.111:0/3574841180 >> v2:172.21.15.149:3300/0 conn(0x7fa7bc15cbf0 0x7fa7bc0b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7ba7fc700 1 -- 172.21.15.111:0/3574841180 >> v1:172.21.15.149:6789/0 conn(0x7fa7bc0cffd0 legacy=0x7fa7bc0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:52.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7ba7fc700 1 -- 172.21.15.111:0/3574841180 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc0cbe10 con 0x7fa7bc0b40a0 2023-11-28T12:58:52.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7baffd700 1 --2- 172.21.15.111:0/3574841180 >> v2:172.21.15.149:3300/0 conn(0x7fa7bc15cbf0 0x7fa7bc0b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:58:52.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7ba7fc700 1 --2- 172.21.15.111:0/3574841180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7bc0b40a0 0x7fa7bc06b380 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0d1d80 tx=0x7fa7a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=686d1b81849eeb4b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:52.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7b9ffb700 1 -- 172.21.15.111:0/3574841180 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4004230 con 0x7fa7bc0b40a0 2023-11-28T12:58:52.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7b9ffb700 1 -- 172.21.15.111:0/3574841180 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a4004390 con 0x7fa7bc0b40a0 2023-11-28T12:58:52.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.403+0000 7fa7b9ffb700 1 -- 172.21.15.111:0/3574841180 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a40036a0 con 0x7fa7bc0b40a0 2023-11-28T12:58:52.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.404+0000 7fa7c1349700 1 -- 172.21.15.111:0/3574841180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7bc0b40a0 msgr2=0x7fa7bc06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.404+0000 7fa7c1349700 1 --2- 172.21.15.111:0/3574841180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7bc0b40a0 0x7fa7bc06b380 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0d1d80 tx=0x7fa7a4009bd0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.404+0000 7fa7c1349700 1 -- 172.21.15.111:0/3574841180 shutdown_connections 2023-11-28T12:58:52.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.404+0000 7fa7c1349700 1 --2- 172.21.15.111:0/3574841180 >> v2:172.21.15.149:3300/0 conn(0x7fa7bc15cbf0 0x7fa7bc0b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.404+0000 7fa7c1349700 1 --2- 172.21.15.111:0/3574841180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7bc0b40a0 0x7fa7bc06b380 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.404+0000 7fa7c1349700 1 -- 172.21.15.111:0/3574841180 >> 172.21.15.111:0/3574841180 conn(0x7fa7bc09a040 msgr2=0x7fa7bc0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:52.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.404+0000 7fa7c1349700 1 -- 172.21.15.111:0/3574841180 shutdown_connections 2023-11-28T12:58:52.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.404+0000 7fa7c1349700 1 -- 172.21.15.111:0/3574841180 wait complete. 2023-11-28T12:58:52.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.405+0000 7fa7c1349700 1 Processor -- start 2023-11-28T12:58:52.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.405+0000 7fa7c1349700 1 -- start start 2023-11-28T12:58:52.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7c1349700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa7bc0b40a0 0x7fa7bc0c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7c1349700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7bc0cffd0 0x7fa7bc1f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7c1349700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa7bc15cbf0 0x7fa7bc15a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7c1349700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc1f3220 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7c1349700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0c97e0 con 0x7fa7bc0b40a0 2023-11-28T12:58:52.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7c1349700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0c9220 con 0x7fa7bc15cbf0 2023-11-28T12:58:52.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7ba7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7bc0cffd0 0x7fa7bc1f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7ba7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7bc0cffd0 0x7fa7bc1f2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60760/0 (socket says 172.21.15.111:60760) 2023-11-28T12:58:52.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7ba7fc700 1 -- 172.21.15.111:0/688774287 learned_addr learned my addr 172.21.15.111:0/688774287 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:52.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7baffd700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa7bc0b40a0 0x7fa7bc0c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7bb7fe700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa7bc15cbf0 0x7fa7bc15a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7ba7fc700 1 -- 172.21.15.111:0/688774287 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa7bc15cbf0 msgr2=0x7fa7bc15a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7ba7fc700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa7bc15cbf0 0x7fa7bc15a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7ba7fc700 1 -- 172.21.15.111:0/688774287 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa7bc0b40a0 msgr2=0x7fa7bc0c8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.406+0000 7fa7ba7fc700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa7bc0b40a0 0x7fa7bc0c8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.407+0000 7fa7ba7fc700 1 -- 172.21.15.111:0/688774287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a4009550 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.407+0000 7fa7ba7fc700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7bc0cffd0 0x7fa7bc1f2ce0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc06d960 tx=0x7fa7a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.407+0000 7fa7b37fe700 1 -- 172.21.15.111:0/688774287 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4003cd0 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.407+0000 7fa7b37fe700 1 -- 172.21.15.111:0/688774287 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a401f070 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.407+0000 7fa7c1349700 1 -- 172.21.15.111:0/688774287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7bc0cc3d0 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.407+0000 7fa7b37fe700 1 -- 172.21.15.111:0/688774287 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4011870 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.407+0000 7fa7c1349700 1 -- 172.21.15.111:0/688774287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7bc0cc0f0 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.408+0000 7fa7c1349700 1 -- 172.21.15.111:0/688774287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7bc0d06f0 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.408+0000 7fa7b37fe700 1 -- 172.21.15.111:0/688774287 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa7a4017020 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.409+0000 7fa7b37fe700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa79c0682f0 0x7fa79c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.409+0000 7fa7b37fe700 1 -- 172.21.15.111:0/688774287 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa7a40870c0 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.409+0000 7fa7baffd700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa79c0682f0 0x7fa79c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.409+0000 7fa7baffd700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa79c0682f0 0x7fa79c06a7a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac006fd0 tx=0x7fa7ac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:52.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.413+0000 7fa7b37fe700 1 -- 172.21.15.111:0/688774287 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa7a4053c10 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.594+0000 7fa7c1349700 1 -- 172.21.15.111:0/688774287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa7bc067880 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.594+0000 7fa7b37fe700 1 -- 172.21.15.111:0/688774287 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v18) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa7a4057230 con 0x7fa7bc0cffd0 2023-11-28T12:58:52.596 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:58:52.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.596+0000 7fa7c1349700 1 -- 172.21.15.111:0/688774287 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa79c0682f0 msgr2=0x7fa79c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.596+0000 7fa7c1349700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa79c0682f0 0x7fa79c06a7a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac006fd0 tx=0x7fa7ac008040 comp rx=0 tx=0).stop 2023-11-28T12:58:52.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.596+0000 7fa7c1349700 1 -- 172.21.15.111:0/688774287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7bc0cffd0 msgr2=0x7fa7bc1f2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.596+0000 7fa7c1349700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7bc0cffd0 0x7fa7bc1f2ce0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc06d960 tx=0x7fa7a4004750 comp rx=0 tx=0).stop 2023-11-28T12:58:52.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.596+0000 7fa7c1349700 1 -- 172.21.15.111:0/688774287 shutdown_connections 2023-11-28T12:58:52.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.596+0000 7fa7c1349700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa7bc15cbf0 0x7fa7bc15a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.596+0000 7fa7c1349700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa79c0682f0 0x7fa79c06a7a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.596+0000 7fa7c1349700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa7bc0b40a0 0x7fa7bc0c8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.596+0000 7fa7c1349700 1 --2- 172.21.15.111:0/688774287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7bc0cffd0 0x7fa7bc1f2ce0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.596+0000 7fa7c1349700 1 -- 172.21.15.111:0/688774287 >> 172.21.15.111:0/688774287 conn(0x7fa7bc09a040 msgr2=0x7fa7bc06b380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:52.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.597+0000 7fa7c1349700 1 -- 172.21.15.111:0/688774287 shutdown_connections 2023-11-28T12:58:52.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.597+0000 7fa7c1349700 1 -- 172.21.15.111:0/688774287 wait complete. 2023-11-28T12:58:52.610 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T12:58:52.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.727+0000 7f2858e3c700 1 Processor -- start 2023-11-28T12:58:52.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.727+0000 7f2858e3c700 1 -- start start 2023-11-28T12:58:52.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f2858e3c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f28540cef60 0x7f28540ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f2858e3c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28540b0160 0x7f28540adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f2858e3c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28540cc390 con 0x7f285415a5d0 2023-11-28T12:58:52.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f2858e3c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28540cc1b0 con 0x7f28540b0160 2023-11-28T12:58:52.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f2858e3c700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28540cc670 con 0x7f28540cef60 2023-11-28T12:58:52.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f2852d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28540b0160 0x7f28540adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f2852d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28540b0160 0x7f28540adde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60770/0 (socket says 172.21.15.111:60770) 2023-11-28T12:58:52.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f2852d9d700 1 -- 172.21.15.111:0/3812076447 learned_addr learned my addr 172.21.15.111:0/3812076447 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:52.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f2852d9d700 1 -- 172.21.15.111:0/3812076447 >> v2:172.21.15.149:3300/0 conn(0x7f28540cef60 msgr2=0x7f28540ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:52.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f2852d9d700 1 --2- 172.21.15.111:0/3812076447 >> v2:172.21.15.149:3300/0 conn(0x7f28540cef60 0x7f28540ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f2852d9d700 1 -- 172.21.15.111:0/3812076447 >> v1:172.21.15.149:6789/0 conn(0x7f285415a5d0 legacy=0x7f28540ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:52.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f2852d9d700 1 -- 172.21.15.111:0/3812076447 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28540cc950 con 0x7f28540b0160 2023-11-28T12:58:52.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f2852d9d700 1 --2- 172.21.15.111:0/3812076447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b0160 0x7f28540adde0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f285408b560 tx=0x7f284c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a7107a49b97187a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:52.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f285159a700 1 -- 172.21.15.111:0/3812076447 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c00e070 con 0x7f28540b0160 2023-11-28T12:58:52.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f285159a700 1 -- 172.21.15.111:0/3812076447 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284c003a20 con 0x7f28540b0160 2023-11-28T12:58:52.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.728+0000 7f285159a700 1 -- 172.21.15.111:0/3812076447 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c003dd0 con 0x7f28540b0160 2023-11-28T12:58:52.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.729+0000 7f2858e3c700 1 -- 172.21.15.111:0/3812076447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b0160 msgr2=0x7f28540adde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.729+0000 7f2858e3c700 1 --2- 172.21.15.111:0/3812076447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b0160 0x7f28540adde0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f285408b560 tx=0x7f284c00b330 comp rx=0 tx=0).stop 2023-11-28T12:58:52.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.729+0000 7f2858e3c700 1 -- 172.21.15.111:0/3812076447 shutdown_connections 2023-11-28T12:58:52.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.729+0000 7f2858e3c700 1 --2- 172.21.15.111:0/3812076447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b0160 0x7f28540adde0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.729+0000 7f2858e3c700 1 --2- 172.21.15.111:0/3812076447 >> v2:172.21.15.149:3300/0 conn(0x7f28540cef60 0x7f28540ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.729+0000 7f2858e3c700 1 -- 172.21.15.111:0/3812076447 >> 172.21.15.111:0/3812076447 conn(0x7f285409a040 msgr2=0x7f28540d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:52.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.729+0000 7f2858e3c700 1 -- 172.21.15.111:0/3812076447 shutdown_connections 2023-11-28T12:58:52.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.729+0000 7f2858e3c700 1 -- 172.21.15.111:0/3812076447 wait complete. 2023-11-28T12:58:52.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.730+0000 7f2858e3c700 1 Processor -- start 2023-11-28T12:58:52.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.730+0000 7f2858e3c700 1 -- start start 2023-11-28T12:58:52.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2858e3c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28540b0160 0x7f28540c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2858e3c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540cef60 0x7f2854148220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2858e3c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f285415a5d0 0x7f28540c4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2858e3c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28540c7a30 con 0x7f28540cef60 2023-11-28T12:58:52.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2858e3c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28540c5090 con 0x7f28540b0160 2023-11-28T12:58:52.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2858e3c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28540c5370 con 0x7f285415a5d0 2023-11-28T12:58:52.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2851d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540cef60 0x7f2854148220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2851d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540cef60 0x7f2854148220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60774/0 (socket says 172.21.15.111:60774) 2023-11-28T12:58:52.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2851d9b700 1 -- 172.21.15.111:0/3555962265 learned_addr learned my addr 172.21.15.111:0/3555962265 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:52.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f285259c700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28540b0160 0x7f28540c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2852d9d700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f285415a5d0 0x7f28540c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2851d9b700 1 -- 172.21.15.111:0/3555962265 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f285415a5d0 msgr2=0x7f28540c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2851d9b700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f285415a5d0 0x7f28540c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2851d9b700 1 -- 172.21.15.111:0/3555962265 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28540b0160 msgr2=0x7f28540c6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2851d9b700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28540b0160 0x7f28540c6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2851d9b700 1 -- 172.21.15.111:0/3555962265 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284c009b30 con 0x7f28540cef60 2023-11-28T12:58:52.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.731+0000 7f2851d9b700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540cef60 0x7f2854148220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f284800b500 tx=0x7f284800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:52.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.732+0000 7f2842ffd700 1 -- 172.21.15.111:0/3555962265 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848011830 con 0x7f28540cef60 2023-11-28T12:58:52.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.732+0000 7f2842ffd700 1 -- 172.21.15.111:0/3555962265 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2848011990 con 0x7f28540cef60 2023-11-28T12:58:52.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.732+0000 7f2858e3c700 1 -- 172.21.15.111:0/3555962265 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28540ccf10 con 0x7f28540cef60 2023-11-28T12:58:52.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.732+0000 7f2842ffd700 1 -- 172.21.15.111:0/3555962265 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848011d40 con 0x7f28540cef60 2023-11-28T12:58:52.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.732+0000 7f2858e3c700 1 -- 172.21.15.111:0/3555962265 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28540ccc30 con 0x7f28540cef60 2023-11-28T12:58:52.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.733+0000 7f2858e3c700 1 -- 172.21.15.111:0/3555962265 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285415a9a0 con 0x7f28540cef60 2023-11-28T12:58:52.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.733+0000 7f2842ffd700 1 -- 172.21.15.111:0/3555962265 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2848021070 con 0x7f28540cef60 2023-11-28T12:58:52.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.733+0000 7f2842ffd700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28340682f0 0x7f283406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:52.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.733+0000 7f2842ffd700 1 -- 172.21.15.111:0/3555962265 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2848085850 con 0x7f28540cef60 2023-11-28T12:58:52.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.733+0000 7f285259c700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28340682f0 0x7f283406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:52.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.734+0000 7f285259c700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28340682f0 0x7f283406a7a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2844006fd0 tx=0x7f2844008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:52.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.737+0000 7f2842ffd700 1 -- 172.21.15.111:0/3555962265 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2848052400 con 0x7f28540cef60 2023-11-28T12:58:52.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.925+0000 7f2858e3c700 1 -- 172.21.15.111:0/3555962265 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f28540c7ba0 con 0x7f28540cef60 2023-11-28T12:58:52.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.925+0000 7f2842ffd700 1 -- 172.21.15.111:0/3555962265 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+126 (secure 0 0 0) 0x7f2848015020 con 0x7f28540cef60 2023-11-28T12:58:52.927 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:58:52.927 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-11-28T12:58:52.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T12:58:52.927 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-11-28T12:58:52.927 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-11-28T12:58:52.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T12:58:52.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T12:58:52.927 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 2, 2023-11-28T12:58:52.927 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": "rbd" 2023-11-28T12:58:52.928 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T12:58:52.928 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-11-28T12:58:52.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.927+0000 7f2858e3c700 1 -- 172.21.15.111:0/3555962265 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28340682f0 msgr2=0x7f283406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.927+0000 7f2858e3c700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28340682f0 0x7f283406a7a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2844006fd0 tx=0x7f2844008040 comp rx=0 tx=0).stop 2023-11-28T12:58:52.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.927+0000 7f2858e3c700 1 -- 172.21.15.111:0/3555962265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540cef60 msgr2=0x7f2854148220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:52.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.927+0000 7f2858e3c700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540cef60 0x7f2854148220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f284800b500 tx=0x7f284800b8c0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.927+0000 7f2858e3c700 1 -- 172.21.15.111:0/3555962265 shutdown_connections 2023-11-28T12:58:52.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.927+0000 7f2858e3c700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f285415a5d0 0x7f28540c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.927+0000 7f2858e3c700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28340682f0 0x7f283406a7a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.927+0000 7f2858e3c700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28540b0160 0x7f28540c6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.927+0000 7f2858e3c700 1 --2- 172.21.15.111:0/3555962265 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540cef60 0x7f2854148220 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:52.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.927+0000 7f2858e3c700 1 -- 172.21.15.111:0/3555962265 >> 172.21.15.111:0/3555962265 conn(0x7f285409a040 msgr2=0x7f2854159740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:52.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.927+0000 7f2858e3c700 1 -- 172.21.15.111:0/3555962265 shutdown_connections 2023-11-28T12:58:52.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:52.927+0000 7f2858e3c700 1 -- 172.21.15.111:0/3555962265 wait complete. 2023-11-28T12:58:52.940 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T12:58:52.941 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T12:58:53.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.062+0000 7facfea83700 1 Processor -- start 2023-11-28T12:58:53.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.062+0000 7facfea83700 1 -- start start 2023-11-28T12:58:53.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.062+0000 7facfea83700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7facf80b0d70 0x7facf81604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:53.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.062+0000 7facfea83700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7facf80cc550 0x7facf80cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:53.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.062+0000 7facfea83700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf815cd90 con 0x7facf80b1140 2023-11-28T12:58:53.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.062+0000 7facfea83700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf80cedd0 con 0x7facf80b0d70 2023-11-28T12:58:53.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.062+0000 7facfea83700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf80cced0 con 0x7facf80cc550 2023-11-28T12:58:53.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.063+0000 7facfc81f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7facf80b0d70 0x7facf81604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:53.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.063+0000 7facf7fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7facf80b1140 0x7facf80ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33326/0 (socket says 172.21.15.111:33326) 2023-11-28T12:58:53.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.063+0000 7facf7fff700 1 -- 172.21.15.111:0/3830605890 learned_addr learned my addr 172.21.15.111:0/3830605890 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:58:53.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.063+0000 7facfd020700 1 --2- 172.21.15.111:0/3830605890 >> v2:172.21.15.149:3300/0 conn(0x7facf80cc550 0x7facf80cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:53.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.063+0000 7facfc81f700 1 -- 172.21.15.111:0/3830605890 >> v2:172.21.15.149:3300/0 conn(0x7facf80cc550 msgr2=0x7facf80cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:53.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.063+0000 7facfc81f700 1 --2- 172.21.15.111:0/3830605890 >> v2:172.21.15.149:3300/0 conn(0x7facf80cc550 0x7facf80cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:53.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.063+0000 7facfc81f700 1 -- 172.21.15.111:0/3830605890 >> v1:172.21.15.111:6789/0 conn(0x7facf80b1140 legacy=0x7facf80ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:53.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.063+0000 7facfc81f700 1 -- 172.21.15.111:0/3830605890 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf81613e0 con 0x7facf80b0d70 2023-11-28T12:58:53.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.063+0000 7facf77fe700 1 -- 172.21.15.111:0/3830605890 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065585006 0 0) 0x7face4001110 con 0x7facf80b1140 2023-11-28T12:58:53.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.063+0000 7facfc81f700 1 --2- 172.21.15.111:0/3830605890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facf80b0d70 0x7facf81604b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7facf8070310 tx=0x7facec009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece86f146c0b5e92 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:53.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.063+0000 7facf77fe700 1 -- 172.21.15.111:0/3830605890 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec005e30 con 0x7facf80b0d70 2023-11-28T12:58:53.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.063+0000 7facf77fe700 1 -- 172.21.15.111:0/3830605890 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facec006030 con 0x7facf80b0d70 2023-11-28T12:58:53.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.063+0000 7facf77fe700 1 -- 172.21.15.111:0/3830605890 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec0063c0 con 0x7facf80b0d70 2023-11-28T12:58:53.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.064+0000 7facfea83700 1 -- 172.21.15.111:0/3830605890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facf80b0d70 msgr2=0x7facf81604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:53.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.064+0000 7facfea83700 1 --2- 172.21.15.111:0/3830605890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facf80b0d70 0x7facf81604b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7facf8070310 tx=0x7facec009b00 comp rx=0 tx=0).stop 2023-11-28T12:58:53.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.064+0000 7facfea83700 1 -- 172.21.15.111:0/3830605890 shutdown_connections 2023-11-28T12:58:53.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.064+0000 7facfea83700 1 --2- 172.21.15.111:0/3830605890 >> v2:172.21.15.149:3300/0 conn(0x7facf80cc550 0x7facf80cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:53.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.064+0000 7facfea83700 1 --2- 172.21.15.111:0/3830605890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facf80b0d70 0x7facf81604b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:53.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.064+0000 7facfea83700 1 -- 172.21.15.111:0/3830605890 >> 172.21.15.111:0/3830605890 conn(0x7facf8098ee0 msgr2=0x7facf806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:53.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.064+0000 7facfea83700 1 -- 172.21.15.111:0/3830605890 shutdown_connections 2023-11-28T12:58:53.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.064+0000 7facfea83700 1 -- 172.21.15.111:0/3830605890 wait complete. 2023-11-28T12:58:53.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.065+0000 7facfea83700 1 Processor -- start 2023-11-28T12:58:53.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.065+0000 7facfea83700 1 -- start start 2023-11-28T12:58:53.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.066+0000 7facfea83700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facf80b0d70 0x7facf8156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:53.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.066+0000 7facfea83700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7facf80b1140 0x7facf815b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:53.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.066+0000 7facfea83700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7facf80cc550 0x7facf815b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:53.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.066+0000 7facfea83700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf80c63c0 con 0x7facf80b0d70 2023-11-28T12:58:53.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.066+0000 7facfea83700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf80c5840 con 0x7facf80cc550 2023-11-28T12:58:53.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.066+0000 7facfea83700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf80c5b20 con 0x7facf80b1140 2023-11-28T12:58:53.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.066+0000 7facfc81f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facf80b0d70 0x7facf8156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:53.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.066+0000 7facfc81f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facf80b0d70 0x7facf8156570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60786/0 (socket says 172.21.15.111:60786) 2023-11-28T12:58:53.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.066+0000 7facfc81f700 1 -- 172.21.15.111:0/3491715756 learned_addr learned my addr 172.21.15.111:0/3491715756 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:53.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.067+0000 7facfc81f700 1 -- 172.21.15.111:0/3491715756 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7facf80b1140 msgr2=0x7facf815b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:53.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.067+0000 7facfc81f700 1 --2- 172.21.15.111:0/3491715756 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7facf80b1140 0x7facf815b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:53.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.067+0000 7facfc81f700 1 -- 172.21.15.111:0/3491715756 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7facf80cc550 msgr2=0x7facf815b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:53.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.067+0000 7facfc81f700 1 --2- 172.21.15.111:0/3491715756 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7facf80cc550 0x7facf815b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:53.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.067+0000 7facfc81f700 1 -- 172.21.15.111:0/3491715756 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec009430 con 0x7facf80b0d70 2023-11-28T12:58:53.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.067+0000 7facfc81f700 1 --2- 172.21.15.111:0/3491715756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facf80b0d70 0x7facf8156570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7facec009fd0 tx=0x7facec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:53.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.067+0000 7facf57fa700 1 -- 172.21.15.111:0/3491715756 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec004040 con 0x7facf80b0d70 2023-11-28T12:58:53.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.067+0000 7facf57fa700 1 -- 172.21.15.111:0/3491715756 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facec020070 con 0x7facf80b0d70 2023-11-28T12:58:53.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.067+0000 7facf57fa700 1 -- 172.21.15.111:0/3491715756 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec004400 con 0x7facf80b0d70 2023-11-28T12:58:53.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.067+0000 7facfea83700 1 -- 172.21.15.111:0/3491715756 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf8160e20 con 0x7facf80b0d70 2023-11-28T12:58:53.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.067+0000 7facfea83700 1 -- 172.21.15.111:0/3491715756 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf8160b40 con 0x7facf80b0d70 2023-11-28T12:58:53.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.068+0000 7facfea83700 1 -- 172.21.15.111:0/3491715756 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf8161100 con 0x7facf80b0d70 2023-11-28T12:58:53.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.068+0000 7facf57fa700 1 -- 172.21.15.111:0/3491715756 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7facec00f050 con 0x7facf80b0d70 2023-11-28T12:58:53.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.069+0000 7facf57fa700 1 --2- 172.21.15.111:0/3491715756 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7facdc0682f0 0x7facdc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:53.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.069+0000 7facf57fa700 1 -- 172.21.15.111:0/3491715756 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7facec0870e0 con 0x7facf80b0d70 2023-11-28T12:58:53.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.069+0000 7facf7fff700 1 --2- 172.21.15.111:0/3491715756 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7facdc0682f0 0x7facdc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:53.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.069+0000 7facf7fff700 1 --2- 172.21.15.111:0/3491715756 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7facdc0682f0 0x7facdc06a7a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7facf8156ab0 tx=0x7face4001310 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:53.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.073+0000 7facf57fa700 1 -- 172.21.15.111:0/3491715756 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7facec053c90 con 0x7facf80b0d70 2023-11-28T12:58:53.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:53.256+0000 7facfea83700 1 -- 172.21.15.111:0/3491715756 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7facf8067170 con 0x7facf80b0d70 2023-11-28T12:58:54.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.220+0000 7facf57fa700 1 -- 172.21.15.111:0/3491715756 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7facec02a0b0 con 0x7facf80b0d70 2023-11-28T12:58:54.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.224+0000 7facfea83700 1 -- 172.21.15.111:0/3491715756 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7facdc0682f0 msgr2=0x7facdc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:54.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.224+0000 7facfea83700 1 --2- 172.21.15.111:0/3491715756 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7facdc0682f0 0x7facdc06a7a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7facf8156ab0 tx=0x7face4001310 comp rx=0 tx=0).stop 2023-11-28T12:58:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.224+0000 7facfea83700 1 -- 172.21.15.111:0/3491715756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facf80b0d70 msgr2=0x7facf8156570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.224+0000 7facfea83700 1 --2- 172.21.15.111:0/3491715756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facf80b0d70 0x7facf8156570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7facec009fd0 tx=0x7facec004750 comp rx=0 tx=0).stop 2023-11-28T12:58:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.224+0000 7facfea83700 1 -- 172.21.15.111:0/3491715756 shutdown_connections 2023-11-28T12:58:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.224+0000 7facfea83700 1 --2- 172.21.15.111:0/3491715756 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7facdc0682f0 0x7facdc06a7a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.224+0000 7facfea83700 1 --2- 172.21.15.111:0/3491715756 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7facf80cc550 0x7facf815b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.224+0000 7facfea83700 1 --2- 172.21.15.111:0/3491715756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facf80b0d70 0x7facf8156570 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.224+0000 7facfea83700 1 --2- 172.21.15.111:0/3491715756 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7facf80b1140 0x7facf815b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:54.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.224+0000 7facfea83700 1 -- 172.21.15.111:0/3491715756 >> 172.21.15.111:0/3491715756 conn(0x7facf8098ee0 msgr2=0x7facf80af700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:54.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.224+0000 7facfea83700 1 -- 172.21.15.111:0/3491715756 shutdown_connections 2023-11-28T12:58:54.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.224+0000 7facfea83700 1 -- 172.21.15.111:0/3491715756 wait complete. 2023-11-28T12:58:54.229 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-11-28T12:58:54.242 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-11-28T12:58:54.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.374+0000 7f861663d700 1 Processor -- start 2023-11-28T12:58:54.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.374+0000 7f861663d700 1 -- start start 2023-11-28T12:58:54.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.375+0000 7f861663d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f86100cde60 0x7f861015d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:54.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.375+0000 7f861663d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8610161100 con 0x7f8610160ca0 2023-11-28T12:58:54.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.375+0000 7f861663d700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86100b1d50 con 0x7f861006bff0 2023-11-28T12:58:54.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.375+0000 7f861663d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861015cd40 con 0x7f86100cde60 2023-11-28T12:58:54.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.375+0000 7f8614bda700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8610160ca0 0x7f861006a700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33344/0 (socket says 172.21.15.111:33344) 2023-11-28T12:58:54.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.375+0000 7f8614bda700 1 -- 172.21.15.111:0/2331154143 learned_addr learned my addr 172.21.15.111:0/2331154143 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:58:54.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860f7fe700 1 --2- 172.21.15.111:0/2331154143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f86100cde60 0x7f861015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:54.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336162348 0 0) 0x7f8610161100 con 0x7f8610160ca0 2023-11-28T12:58:54.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f80035b0 con 0x7f8610160ca0 2023-11-28T12:58:54.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514637344 0 0) 0x7f86100b1d50 con 0x7f861006bff0 2023-11-28T12:58:54.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8610161100 con 0x7f861006bff0 2023-11-28T12:58:54.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 583975578 0 0) 0x7f85f80035b0 con 0x7f8610160ca0 2023-11-28T12:58:54.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f86100cde60 msgr2=0x7f861015d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:54.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 --2- 172.21.15.111:0/2331154143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f86100cde60 0x7f861015d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:54.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 >> v1:172.21.15.149:6789/0 conn(0x7f861006bff0 legacy=0x7f86100cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:54.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861015d020 con 0x7f8610160ca0 2023-11-28T12:58:54.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f86080019f0 con 0x7f8610160ca0 2023-11-28T12:58:54.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 >> v1:172.21.15.111:6789/0 conn(0x7f8610160ca0 legacy=0x7f861006a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:54.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 --2- 172.21.15.111:0/2331154143 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85f8003bc0 0x7f85f8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:54.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 --2- 172.21.15.111:0/2331154143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f8004850 0x7f85f8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:54.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 --2- 172.21.15.111:0/2331154143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f86100cde60 0x7f85f8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:54.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861015cd40 con 0x7f85f8004850 2023-11-28T12:58:54.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f800baf0 con 0x7f85f8003bc0 2023-11-28T12:58:54.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f800bcb0 con 0x7f86100cde60 2023-11-28T12:58:54.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860f7fe700 1 --2- 172.21.15.111:0/2331154143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f8004850 0x7f85f8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:54.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860ffff700 1 --2- 172.21.15.111:0/2331154143 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85f8003bc0 0x7f85f8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:54.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f8614bda700 1 --2- 172.21.15.111:0/2331154143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f86100cde60 0x7f85f8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:54.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860f7fe700 1 -- 172.21.15.111:0/2331154143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f86100cde60 msgr2=0x7f85f8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:54.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860f7fe700 1 --2- 172.21.15.111:0/2331154143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f86100cde60 0x7f85f8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:54.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860f7fe700 1 -- 172.21.15.111:0/2331154143 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85f8003bc0 msgr2=0x7f85f8003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:54.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860f7fe700 1 --2- 172.21.15.111:0/2331154143 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f85f8003bc0 0x7f85f8003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:54.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.376+0000 7f860f7fe700 1 -- 172.21.15.111:0/2331154143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f80035b0 con 0x7f85f8004850 2023-11-28T12:58:54.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.377+0000 7f860f7fe700 1 --2- 172.21.15.111:0/2331154143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f8004850 0x7f85f8004cc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8600009b30 tx=0x7f85f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1214a249c92ce85 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:54.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.377+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860000e6c0 con 0x7f85f8004850 2023-11-28T12:58:54.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.377+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860000e820 con 0x7f85f8004850 2023-11-28T12:58:54.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.377+0000 7f860effd700 1 -- 172.21.15.111:0/2331154143 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860000eb90 con 0x7f85f8004850 2023-11-28T12:58:54.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.377+0000 7f861663d700 1 -- 172.21.15.111:0/2331154143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f8004850 msgr2=0x7f85f8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:54.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.377+0000 7f861663d700 1 --2- 172.21.15.111:0/2331154143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85f8004850 0x7f85f8004cc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8600009b30 tx=0x7f85f8005880 comp rx=0 tx=0).stop 2023-11-28T12:58:54.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.377+0000 7f8614bda700 1 -- 172.21.15.111:0/2331154143 reap_dead start 2023-11-28T12:58:54.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.377+0000 7f861663d700 1 -- 172.21.15.111:0/2331154143 shutdown_connections 2023-11-28T12:58:54.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.377+0000 7f861663d700 1 -- 172.21.15.111:0/2331154143 >> 172.21.15.111:0/2331154143 conn(0x7f861006e790 msgr2=0x7f86100cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:54.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.377+0000 7f861663d700 1 -- 172.21.15.111:0/2331154143 shutdown_connections 2023-11-28T12:58:54.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.377+0000 7f861663d700 1 -- 172.21.15.111:0/2331154143 wait complete. 2023-11-28T12:58:54.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.378+0000 7f861663d700 1 Processor -- start 2023-11-28T12:58:54.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.379+0000 7f861663d700 1 -- start start 2023-11-28T12:58:54.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.379+0000 7f861663d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f85f8004850 0x7f861014dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:54.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.379+0000 7f861663d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86100b29b0 0x7f86101e7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:54.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.379+0000 7f861663d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f861015f1f0 0x7f861015f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:54.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.379+0000 7f861663d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861015fbb0 con 0x7f86100b29b0 2023-11-28T12:58:54.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.379+0000 7f861663d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86100b3ac0 con 0x7f861015f1f0 2023-11-28T12:58:54.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.379+0000 7f861663d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86100b2eb0 con 0x7f85f8004850 2023-11-28T12:58:54.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.379+0000 7f860f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86100b29b0 0x7f86101e7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:54.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.379+0000 7f860f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86100b29b0 0x7f86101e7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60806/0 (socket says 172.21.15.111:60806) 2023-11-28T12:58:54.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.379+0000 7f860f7fe700 1 -- 172.21.15.111:0/720737793 learned_addr learned my addr 172.21.15.111:0/720737793 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:54.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f8614bda700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f861015f1f0 0x7f861015f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:54.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f860ffff700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f85f8004850 0x7f861014dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:54.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f860f7fe700 1 -- 172.21.15.111:0/720737793 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f85f8004850 msgr2=0x7f861014dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:54.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f860f7fe700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f85f8004850 0x7f861014dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:54.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f860f7fe700 1 -- 172.21.15.111:0/720737793 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f861015f1f0 msgr2=0x7f861015f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:54.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f860f7fe700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f861015f1f0 0x7f861015f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:54.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f860f7fe700 1 -- 172.21.15.111:0/720737793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86100b2030 con 0x7f86100b29b0 2023-11-28T12:58:54.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f8614bda700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f861015f1f0 0x7f861015f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:58:54.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f860f7fe700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86100b29b0 0x7f86101e7cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f860000e4b0 tx=0x7f86000047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:54.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f860cff9700 1 -- 172.21.15.111:0/720737793 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860000e6c0 con 0x7f86100b29b0 2023-11-28T12:58:54.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f860cff9700 1 -- 172.21.15.111:0/720737793 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8600003ad0 con 0x7f86100b29b0 2023-11-28T12:58:54.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f861663d700 1 -- 172.21.15.111:0/720737793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861015d300 con 0x7f86100b29b0 2023-11-28T12:58:54.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f860cff9700 1 -- 172.21.15.111:0/720737793 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600003e80 con 0x7f86100b29b0 2023-11-28T12:58:54.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.380+0000 7f861663d700 1 -- 172.21.15.111:0/720737793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f861015d820 con 0x7f86100b29b0 2023-11-28T12:58:54.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.381+0000 7f861663d700 1 -- 172.21.15.111:0/720737793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8610067170 con 0x7f86100b29b0 2023-11-28T12:58:54.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.382+0000 7f860cff9700 1 -- 172.21.15.111:0/720737793 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8600025070 con 0x7f86100b29b0 2023-11-28T12:58:54.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.382+0000 7f860cff9700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f85fc0682f0 0x7f85fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:54.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.382+0000 7f860cff9700 1 -- 172.21.15.111:0/720737793 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5580+0+0 (secure 0 0 0) 0x7f86000872f0 con 0x7f86100b29b0 2023-11-28T12:58:54.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.382+0000 7f860ffff700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f85fc0682f0 0x7f85fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:54.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.382+0000 7f860ffff700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f85fc0682f0 0x7f85fc06a7a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8604007fd0 tx=0x7f8604009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:54.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.386+0000 7f860cff9700 1 -- 172.21.15.111:0/720737793 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8600053d30 con 0x7f86100b29b0 2023-11-28T12:58:54.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:54.569+0000 7f861663d700 1 -- 172.21.15.111:0/720737793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f86100cd1d0 con 0x7f86100b29b0 2023-11-28T12:58:55.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.236+0000 7f860cff9700 1 -- 172.21.15.111:0/720737793 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v20) v1 ==== 171+0+0 (secure 0 0 0) 0x7f8600057350 con 0x7f86100b29b0 2023-11-28T12:58:55.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.243+0000 7f861663d700 1 -- 172.21.15.111:0/720737793 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f85fc0682f0 msgr2=0x7f85fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:55.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.243+0000 7f861663d700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f85fc0682f0 0x7f85fc06a7a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8604007fd0 tx=0x7f8604009040 comp rx=0 tx=0).stop 2023-11-28T12:58:55.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.243+0000 7f861663d700 1 -- 172.21.15.111:0/720737793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86100b29b0 msgr2=0x7f86101e7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:55.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.243+0000 7f861663d700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86100b29b0 0x7f86101e7cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f860000e4b0 tx=0x7f86000047c0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.243+0000 7f861663d700 1 -- 172.21.15.111:0/720737793 shutdown_connections 2023-11-28T12:58:55.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.243+0000 7f861663d700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f85fc0682f0 0x7f85fc06a7a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.243+0000 7f861663d700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f861015f1f0 0x7f861015f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.243+0000 7f861663d700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f85f8004850 0x7f861014dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.243+0000 7f861663d700 1 --2- 172.21.15.111:0/720737793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86100b29b0 0x7f86101e7cf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.243+0000 7f861663d700 1 -- 172.21.15.111:0/720737793 >> 172.21.15.111:0/720737793 conn(0x7f861006e790 msgr2=0x7f861006a700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:55.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.246+0000 7f861663d700 1 -- 172.21.15.111:0/720737793 shutdown_connections 2023-11-28T12:58:55.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.247+0000 7f861663d700 1 -- 172.21.15.111:0/720737793 wait complete. 2023-11-28T12:58:55.251 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-11-28T12:58:55.266 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T12:58:55.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.392+0000 7fcf18eb4700 1 Processor -- start 2023-11-28T12:58:55.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.392+0000 7fcf18eb4700 1 -- start start 2023-11-28T12:58:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.393+0000 7fcf18eb4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcf140cd950 0x7fcf140b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.393+0000 7fcf18eb4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf140cdd20 0x7fcf140cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.393+0000 7fcf18eb4700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf140cb530 con 0x7fcf140cbf00 2023-11-28T12:58:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.393+0000 7fcf18eb4700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1406bd80 con 0x7fcf140cd950 2023-11-28T12:58:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.393+0000 7fcf18eb4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1406b500 con 0x7fcf140cdd20 2023-11-28T12:58:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.393+0000 7fcf1259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcf140cd950 0x7fcf140b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.393+0000 7fcf1259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcf140cd950 0x7fcf140b1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60814/0 (socket says 172.21.15.111:60814) 2023-11-28T12:58:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.393+0000 7fcf1259c700 1 -- 172.21.15.111:0/4022297506 learned_addr learned my addr 172.21.15.111:0/4022297506 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.394+0000 7fcf11d9b700 1 --2- 172.21.15.111:0/4022297506 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf140cdd20 0x7fcf140cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.394+0000 7fcf1259c700 1 -- 172.21.15.111:0/4022297506 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf140cdd20 msgr2=0x7fcf140cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.394+0000 7fcf1259c700 1 --2- 172.21.15.111:0/4022297506 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf140cdd20 0x7fcf140cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.394+0000 7fcf1259c700 1 -- 172.21.15.111:0/4022297506 >> v1:172.21.15.149:6789/0 conn(0x7fcf140cbf00 legacy=0x7fcf140cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.394+0000 7fcf1259c700 1 -- 172.21.15.111:0/4022297506 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1406b7c0 con 0x7fcf140cd950 2023-11-28T12:58:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.394+0000 7fcf1259c700 1 --2- 172.21.15.111:0/4022297506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf140cd950 0x7fcf140b1fe0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcf080098f0 tx=0x7fcf08009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501c58c629a05401 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:55.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.394+0000 7fcf1159a700 1 -- 172.21.15.111:0/4022297506 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0800b8a0 con 0x7fcf140cd950 2023-11-28T12:58:55.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.394+0000 7fcf1159a700 1 -- 172.21.15.111:0/4022297506 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0800ba00 con 0x7fcf140cd950 2023-11-28T12:58:55.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.394+0000 7fcf1159a700 1 -- 172.21.15.111:0/4022297506 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0800bd90 con 0x7fcf140cd950 2023-11-28T12:58:55.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.394+0000 7fcf18eb4700 1 -- 172.21.15.111:0/4022297506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf140cd950 msgr2=0x7fcf140b1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:55.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.394+0000 7fcf18eb4700 1 --2- 172.21.15.111:0/4022297506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf140cd950 0x7fcf140b1fe0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcf080098f0 tx=0x7fcf08009c00 comp rx=0 tx=0).stop 2023-11-28T12:58:55.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.395+0000 7fcf18eb4700 1 -- 172.21.15.111:0/4022297506 shutdown_connections 2023-11-28T12:58:55.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.395+0000 7fcf18eb4700 1 --2- 172.21.15.111:0/4022297506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf140cd950 0x7fcf140b1fe0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.395+0000 7fcf18eb4700 1 --2- 172.21.15.111:0/4022297506 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf140cdd20 0x7fcf140cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.395+0000 7fcf18eb4700 1 -- 172.21.15.111:0/4022297506 >> 172.21.15.111:0/4022297506 conn(0x7fcf1409a020 msgr2=0x7fcf140d0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:55.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.395+0000 7fcf18eb4700 1 -- 172.21.15.111:0/4022297506 shutdown_connections 2023-11-28T12:58:55.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.395+0000 7fcf18eb4700 1 -- 172.21.15.111:0/4022297506 wait complete. 2023-11-28T12:58:55.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.396+0000 7fcf18eb4700 1 Processor -- start 2023-11-28T12:58:55.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.396+0000 7fcf18eb4700 1 -- start start 2023-11-28T12:58:55.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf18eb4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf140cbf00 0x7fcf1415e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:55.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf18eb4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf140cd950 0x7fcf141f57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:55.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf18eb4700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf140cdd20 0x7fcf141f5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:55.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf18eb4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1415ef10 con 0x7fcf140cd950 2023-11-28T12:58:55.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf18eb4700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf140b0470 con 0x7fcf140cdd20 2023-11-28T12:58:55.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf18eb4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf140b0750 con 0x7fcf140cbf00 2023-11-28T12:58:55.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf11d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf140cd950 0x7fcf141f57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:55.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf11d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf140cd950 0x7fcf141f57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60822/0 (socket says 172.21.15.111:60822) 2023-11-28T12:58:55.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf11d9b700 1 -- 172.21.15.111:0/1971700797 learned_addr learned my addr 172.21.15.111:0/1971700797 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:55.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf12d9d700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf140cdd20 0x7fcf141f5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:55.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf1259c700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf140cbf00 0x7fcf1415e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:55.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf11d9b700 1 -- 172.21.15.111:0/1971700797 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf140cbf00 msgr2=0x7fcf1415e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:55.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf11d9b700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf140cbf00 0x7fcf1415e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf11d9b700 1 -- 172.21.15.111:0/1971700797 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf140cdd20 msgr2=0x7fcf141f5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:55.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf11d9b700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf140cdd20 0x7fcf141f5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf11d9b700 1 -- 172.21.15.111:0/1971700797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf08009550 con 0x7fcf140cd950 2023-11-28T12:58:55.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf1259c700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf140cbf00 0x7fcf1415e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:58:55.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf11d9b700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf140cd950 0x7fcf141f57a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400e810 tx=0x7fcf0400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:55.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf02ffd700 1 -- 172.21.15.111:0/1971700797 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf04013610 con 0x7fcf140cd950 2023-11-28T12:58:55.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf02ffd700 1 -- 172.21.15.111:0/1971700797 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf04013770 con 0x7fcf140cd950 2023-11-28T12:58:55.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf18eb4700 1 -- 172.21.15.111:0/1971700797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf140b2520 con 0x7fcf140cd950 2023-11-28T12:58:55.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf02ffd700 1 -- 172.21.15.111:0/1971700797 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf04013b20 con 0x7fcf140cd950 2023-11-28T12:58:55.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.397+0000 7fcf18eb4700 1 -- 172.21.15.111:0/1971700797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1406baa0 con 0x7fcf140cd950 2023-11-28T12:58:55.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.398+0000 7fcf18eb4700 1 -- 172.21.15.111:0/1971700797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1409a420 con 0x7fcf140cd950 2023-11-28T12:58:55.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.399+0000 7fcf02ffd700 1 -- 172.21.15.111:0/1971700797 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcf04023070 con 0x7fcf140cd950 2023-11-28T12:58:55.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.399+0000 7fcf02ffd700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcef40682f0 0x7fcef406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:55.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.399+0000 7fcf02ffd700 1 -- 172.21.15.111:0/1971700797 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5962+0+0 (secure 0 0 0) 0x7fcf040859b0 con 0x7fcf140cd950 2023-11-28T12:58:55.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.399+0000 7fcf1259c700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcef40682f0 0x7fcef406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:55.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.399+0000 7fcf1259c700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcef40682f0 0x7fcef406a7a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcf140b34b0 tx=0x7fcf08008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:55.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.403+0000 7fcf02ffd700 1 -- 172.21.15.111:0/1971700797 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcf040521f0 con 0x7fcf140cd950 2023-11-28T12:58:55.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.631+0000 7fcf18eb4700 1 -- 172.21.15.111:0/1971700797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcf140ce6f0 con 0x7fcf140cd950 2023-11-28T12:58:55.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.638+0000 7fcf02ffd700 1 -- 172.21.15.111:0/1971700797 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fcf04055810 con 0x7fcf140cd950 2023-11-28T12:58:55.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.641+0000 7fcf18eb4700 1 -- 172.21.15.111:0/1971700797 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcef40682f0 msgr2=0x7fcef406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:55.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.642+0000 7fcf18eb4700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcef40682f0 0x7fcef406a7a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcf140b34b0 tx=0x7fcf08008030 comp rx=0 tx=0).stop 2023-11-28T12:58:55.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.642+0000 7fcf18eb4700 1 -- 172.21.15.111:0/1971700797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf140cd950 msgr2=0x7fcf141f57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:55.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.642+0000 7fcf18eb4700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf140cd950 0x7fcf141f57a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400e810 tx=0x7fcf0400ebd0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.642+0000 7fcf18eb4700 1 -- 172.21.15.111:0/1971700797 shutdown_connections 2023-11-28T12:58:55.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.642+0000 7fcf18eb4700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcef40682f0 0x7fcef406a7a0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcf140b34b0 tx=0x7fcf08008030 comp rx=0 tx=0).stop 2023-11-28T12:58:55.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.642+0000 7fcf18eb4700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf140cdd20 0x7fcf141f5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.642+0000 7fcf18eb4700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf140cbf00 0x7fcf1415e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.642+0000 7fcf18eb4700 1 --2- 172.21.15.111:0/1971700797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf140cd950 0x7fcf141f57a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.642+0000 7fcf18eb4700 1 -- 172.21.15.111:0/1971700797 >> 172.21.15.111:0/1971700797 conn(0x7fcf1409a020 msgr2=0x7fcf140d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:55.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.642+0000 7fcf18eb4700 1 -- 172.21.15.111:0/1971700797 shutdown_connections 2023-11-28T12:58:55.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.642+0000 7fcf18eb4700 1 -- 172.21.15.111:0/1971700797 wait complete. 2023-11-28T12:58:55.648 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 3 and data pool 4 2023-11-28T12:58:55.667 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-11-28T12:58:55.667 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-11-28T12:58:55.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.790+0000 7eff2ada6700 1 Processor -- start 2023-11-28T12:58:55.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.790+0000 7eff2ada6700 1 -- start start 2023-11-28T12:58:55.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff2ada6700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7eff240cef60 0x7eff240ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:55.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff2ada6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7eff240b0160 0x7eff240adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:55.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff2ada6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff240cc390 con 0x7eff2415a5d0 2023-11-28T12:58:55.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff2ada6700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff240cc1b0 con 0x7eff240b0160 2023-11-28T12:58:55.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff2ada6700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff240cc670 con 0x7eff240cef60 2023-11-28T12:58:55.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff29343700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7eff240b0160 0x7eff240adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:55.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff28b42700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7eff2415a5d0 0x7eff240ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33386/0 (socket says 172.21.15.111:33386) 2023-11-28T12:58:55.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff28b42700 1 -- 172.21.15.111:0/3337659203 learned_addr learned my addr 172.21.15.111:0/3337659203 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:58:55.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff23fff700 1 --2- 172.21.15.111:0/3337659203 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7eff240cef60 0x7eff240ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:55.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff29343700 1 -- 172.21.15.111:0/3337659203 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7eff240cef60 msgr2=0x7eff240ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:55.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff29343700 1 --2- 172.21.15.111:0/3337659203 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7eff240cef60 0x7eff240ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff29343700 1 -- 172.21.15.111:0/3337659203 >> v1:172.21.15.111:6789/0 conn(0x7eff2415a5d0 legacy=0x7eff240ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:55.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff29343700 1 -- 172.21.15.111:0/3337659203 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff240cc950 con 0x7eff240b0160 2023-11-28T12:58:55.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff29343700 1 --2- 172.21.15.111:0/3337659203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff240b0160 0x7eff240adde0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7eff1c009d90 tx=0x7eff1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db2d13bb6c29fdf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:55.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.791+0000 7eff237fe700 1 -- 172.21.15.111:0/3337659203 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c00e070 con 0x7eff240b0160 2023-11-28T12:58:55.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.792+0000 7eff237fe700 1 -- 172.21.15.111:0/3337659203 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c00bad0 con 0x7eff240b0160 2023-11-28T12:58:55.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.792+0000 7eff237fe700 1 -- 172.21.15.111:0/3337659203 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c00be80 con 0x7eff240b0160 2023-11-28T12:58:55.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.792+0000 7eff2ada6700 1 -- 172.21.15.111:0/3337659203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff240b0160 msgr2=0x7eff240adde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:55.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.792+0000 7eff2ada6700 1 --2- 172.21.15.111:0/3337659203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff240b0160 0x7eff240adde0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7eff1c009d90 tx=0x7eff1c00b330 comp rx=0 tx=0).stop 2023-11-28T12:58:55.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.792+0000 7eff2ada6700 1 -- 172.21.15.111:0/3337659203 shutdown_connections 2023-11-28T12:58:55.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.792+0000 7eff2ada6700 1 --2- 172.21.15.111:0/3337659203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff240b0160 0x7eff240adde0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.792+0000 7eff2ada6700 1 --2- 172.21.15.111:0/3337659203 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7eff240cef60 0x7eff240ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.792+0000 7eff2ada6700 1 -- 172.21.15.111:0/3337659203 >> 172.21.15.111:0/3337659203 conn(0x7eff2409a040 msgr2=0x7eff240d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:55.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.792+0000 7eff2ada6700 1 -- 172.21.15.111:0/3337659203 shutdown_connections 2023-11-28T12:58:55.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.792+0000 7eff2ada6700 1 -- 172.21.15.111:0/3337659203 wait complete. 2023-11-28T12:58:55.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.793+0000 7eff2ada6700 1 Processor -- start 2023-11-28T12:58:55.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.793+0000 7eff2ada6700 1 -- start start 2023-11-28T12:58:55.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.794+0000 7eff2ada6700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7eff240b0160 0x7eff241ef540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:55.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.794+0000 7eff2ada6700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7eff240cef60 0x7eff2406f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:55.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.794+0000 7eff2ada6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff2415a5d0 0x7eff2406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:55.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.794+0000 7eff2ada6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff240c74d0 con 0x7eff2415a5d0 2023-11-28T12:58:55.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.794+0000 7eff2ada6700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff240b28e0 con 0x7eff240b0160 2023-11-28T12:58:55.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.794+0000 7eff2ada6700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff240b1cd0 con 0x7eff240cef60 2023-11-28T12:58:55.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.794+0000 7eff29343700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff2415a5d0 0x7eff2406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:55.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.794+0000 7eff29343700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff2415a5d0 0x7eff2406fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60850/0 (socket says 172.21.15.111:60850) 2023-11-28T12:58:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.794+0000 7eff29343700 1 -- 172.21.15.111:0/593224839 learned_addr learned my addr 172.21.15.111:0/593224839 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.794+0000 7eff28b42700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7eff240b0160 0x7eff241ef540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.794+0000 7eff23fff700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7eff240cef60 0x7eff2406f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.795+0000 7eff29343700 1 -- 172.21.15.111:0/593224839 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7eff240cef60 msgr2=0x7eff2406f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.795+0000 7eff29343700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7eff240cef60 0x7eff2406f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.795+0000 7eff29343700 1 -- 172.21.15.111:0/593224839 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7eff240b0160 msgr2=0x7eff241ef540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.795+0000 7eff29343700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7eff240b0160 0x7eff241ef540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.795+0000 7eff29343700 1 -- 172.21.15.111:0/593224839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1c009a70 con 0x7eff2415a5d0 2023-11-28T12:58:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.795+0000 7eff29343700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff2415a5d0 0x7eff2406fd20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7eff2406d8f0 tx=0x7eff1c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.795+0000 7eff217fa700 1 -- 172.21.15.111:0/593224839 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c009410 con 0x7eff2415a5d0 2023-11-28T12:58:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.795+0000 7eff217fa700 1 -- 172.21.15.111:0/593224839 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c012430 con 0x7eff2415a5d0 2023-11-28T12:58:55.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.795+0000 7eff2ada6700 1 -- 172.21.15.111:0/593224839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff240ccf10 con 0x7eff2415a5d0 2023-11-28T12:58:55.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.795+0000 7eff217fa700 1 -- 172.21.15.111:0/593224839 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0127e0 con 0x7eff2415a5d0 2023-11-28T12:58:55.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.795+0000 7eff2ada6700 1 -- 172.21.15.111:0/593224839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff240ccc30 con 0x7eff2415a5d0 2023-11-28T12:58:55.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.796+0000 7eff2ada6700 1 -- 172.21.15.111:0/593224839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff2415a9a0 con 0x7eff2415a5d0 2023-11-28T12:58:55.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.796+0000 7eff217fa700 1 -- 172.21.15.111:0/593224839 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7eff1c012940 con 0x7eff2415a5d0 2023-11-28T12:58:55.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.797+0000 7eff217fa700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7eff080682f0 0x7eff0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:55.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.797+0000 7eff217fa700 1 -- 172.21.15.111:0/593224839 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6062+0+0 (secure 0 0 0) 0x7eff1c088210 con 0x7eff2415a5d0 2023-11-28T12:58:55.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.797+0000 7eff28b42700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7eff080682f0 0x7eff0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:55.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.797+0000 7eff28b42700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7eff080682f0 0x7eff0806a7a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7eff18000ef0 tx=0x7eff18009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:55.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.801+0000 7eff217fa700 1 -- 172.21.15.111:0/593224839 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7eff1c054a70 con 0x7eff2415a5d0 2023-11-28T12:58:55.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:55.993+0000 7eff2ada6700 1 -- 172.21.15.111:0/593224839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7eff240b2a20 con 0x7eff2415a5d0 2023-11-28T12:58:56.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.645+0000 7eff217fa700 1 -- 172.21.15.111:0/593224839 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v22) v1 ==== 157+0+0 (secure 0 0 0) 0x7eff1c058090 con 0x7eff2415a5d0 2023-11-28T12:58:56.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.648+0000 7eff2ada6700 1 -- 172.21.15.111:0/593224839 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7eff080682f0 msgr2=0x7eff0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:56.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.648+0000 7eff2ada6700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7eff080682f0 0x7eff0806a7a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7eff18000ef0 tx=0x7eff18009040 comp rx=0 tx=0).stop 2023-11-28T12:58:56.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.648+0000 7eff2ada6700 1 -- 172.21.15.111:0/593224839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff2415a5d0 msgr2=0x7eff2406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:56.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.648+0000 7eff2ada6700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff2415a5d0 0x7eff2406fd20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7eff2406d8f0 tx=0x7eff1c004ad0 comp rx=0 tx=0).stop 2023-11-28T12:58:56.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.648+0000 7eff2ada6700 1 -- 172.21.15.111:0/593224839 shutdown_connections 2023-11-28T12:58:56.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.648+0000 7eff2ada6700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff2415a5d0 0x7eff2406fd20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:56.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.648+0000 7eff2ada6700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7eff080682f0 0x7eff0806a7a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:56.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.648+0000 7eff2ada6700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7eff240b0160 0x7eff241ef540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:56.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.648+0000 7eff2ada6700 1 --2- 172.21.15.111:0/593224839 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7eff240cef60 0x7eff2406f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:56.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.648+0000 7eff2ada6700 1 -- 172.21.15.111:0/593224839 >> 172.21.15.111:0/593224839 conn(0x7eff2409a040 msgr2=0x7eff24070610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:56.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.648+0000 7eff2ada6700 1 -- 172.21.15.111:0/593224839 shutdown_connections 2023-11-28T12:58:56.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.648+0000 7eff2ada6700 1 -- 172.21.15.111:0/593224839 wait complete. 2023-11-28T12:58:56.663 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T12:58:56.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.792+0000 7facc6e3f700 1 Processor -- start 2023-11-28T12:58:56.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.792+0000 7facc6e3f700 1 -- start start 2023-11-28T12:58:56.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.792+0000 7facc6e3f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7facc015e0c0 0x7facc01604a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:56.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.792+0000 7facc6e3f700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7facc01609e0 0x7facc0162dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:56.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.792+0000 7facc6e3f700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc00ca260 con 0x7facc01596f0 2023-11-28T12:58:56.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.792+0000 7facc6e3f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc00ca540 con 0x7facc015e0c0 2023-11-28T12:58:56.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.792+0000 7facc6e3f700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc00ca820 con 0x7facc01609e0 2023-11-28T12:58:56.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.793+0000 7facbffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7facc015e0c0 0x7facc01604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:56.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.793+0000 7facbffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7facc015e0c0 0x7facc01604a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60856/0 (socket says 172.21.15.111:60856) 2023-11-28T12:58:56.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.793+0000 7facbffff700 1 -- 172.21.15.111:0/3238848987 learned_addr learned my addr 172.21.15.111:0/3238848987 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:56.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.793+0000 7facc53dc700 1 --2- 172.21.15.111:0/3238848987 >> v2:172.21.15.149:3300/0 conn(0x7facc01609e0 0x7facc0162dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:56.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.793+0000 7facbffff700 1 -- 172.21.15.111:0/3238848987 >> v2:172.21.15.149:3300/0 conn(0x7facc01609e0 msgr2=0x7facc0162dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:56.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.793+0000 7facbffff700 1 --2- 172.21.15.111:0/3238848987 >> v2:172.21.15.149:3300/0 conn(0x7facc01609e0 0x7facc0162dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:56.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.793+0000 7facbffff700 1 -- 172.21.15.111:0/3238848987 >> v1:172.21.15.149:6789/0 conn(0x7facc01596f0 legacy=0x7facc00cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:56.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.793+0000 7facbffff700 1 -- 172.21.15.111:0/3238848987 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc00cab00 con 0x7facc015e0c0 2023-11-28T12:58:56.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.793+0000 7facbffff700 1 --2- 172.21.15.111:0/3238848987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facc015e0c0 0x7facc01604a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7facc00cec20 tx=0x7faca8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b32bed174510cab server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:56.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.793+0000 7facbf7fe700 1 -- 172.21.15.111:0/3238848987 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca800b870 con 0x7facc015e0c0 2023-11-28T12:58:56.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.793+0000 7facbf7fe700 1 -- 172.21.15.111:0/3238848987 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca800b9d0 con 0x7facc015e0c0 2023-11-28T12:58:56.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.793+0000 7facbf7fe700 1 -- 172.21.15.111:0/3238848987 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca800bd60 con 0x7facc015e0c0 2023-11-28T12:58:56.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.794+0000 7facc6e3f700 1 -- 172.21.15.111:0/3238848987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facc015e0c0 msgr2=0x7facc01604a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:56.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.794+0000 7facc6e3f700 1 --2- 172.21.15.111:0/3238848987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facc015e0c0 0x7facc01604a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7facc00cec20 tx=0x7faca8009bd0 comp rx=0 tx=0).stop 2023-11-28T12:58:56.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.794+0000 7facc6e3f700 1 -- 172.21.15.111:0/3238848987 shutdown_connections 2023-11-28T12:58:56.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.794+0000 7facc6e3f700 1 --2- 172.21.15.111:0/3238848987 >> v2:172.21.15.149:3300/0 conn(0x7facc01609e0 0x7facc0162dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:56.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.794+0000 7facc6e3f700 1 --2- 172.21.15.111:0/3238848987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facc015e0c0 0x7facc01604a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:56.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.794+0000 7facc6e3f700 1 -- 172.21.15.111:0/3238848987 >> 172.21.15.111:0/3238848987 conn(0x7facc0098f00 msgr2=0x7facc006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:56.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.794+0000 7facc6e3f700 1 -- 172.21.15.111:0/3238848987 shutdown_connections 2023-11-28T12:58:56.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.794+0000 7facc6e3f700 1 -- 172.21.15.111:0/3238848987 wait complete. 2023-11-28T12:58:56.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.795+0000 7facc6e3f700 1 Processor -- start 2023-11-28T12:58:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.796+0000 7facc6e3f700 1 -- start start 2023-11-28T12:58:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.796+0000 7facc6e3f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7facc01596f0 0x7facc0146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.796+0000 7facc6e3f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7facc015e0c0 0x7facc00b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.796+0000 7facc6e3f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facc01609e0 0x7facc00b50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.796+0000 7facc6e3f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc00b55f0 con 0x7facc01609e0 2023-11-28T12:58:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.796+0000 7facc6e3f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc00b5890 con 0x7facc01596f0 2023-11-28T12:58:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.796+0000 7facc6e3f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc0159ee0 con 0x7facc015e0c0 2023-11-28T12:58:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facc53dc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facc01609e0 0x7facc00b50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facc53dc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facc01609e0 0x7facc00b50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60864/0 (socket says 172.21.15.111:60864) 2023-11-28T12:58:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facc53dc700 1 -- 172.21.15.111:0/3977099646 learned_addr learned my addr 172.21.15.111:0/3977099646 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facbffff700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7facc015e0c0 0x7facc00b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facc4bdb700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7facc01596f0 0x7facc0146f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:56.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facc53dc700 1 -- 172.21.15.111:0/3977099646 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7facc015e0c0 msgr2=0x7facc00b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:56.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facc53dc700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7facc015e0c0 0x7facc00b2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:56.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facc53dc700 1 -- 172.21.15.111:0/3977099646 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7facc01596f0 msgr2=0x7facc0146f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:56.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facc53dc700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7facc01596f0 0x7facc0146f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:56.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facc53dc700 1 -- 172.21.15.111:0/3977099646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca8009550 con 0x7facc01609e0 2023-11-28T12:58:56.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facbffff700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7facc015e0c0 0x7facc00b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:58:56.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facc4bdb700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7facc01596f0 0x7facc0146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:58:56.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facc53dc700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facc01609e0 0x7facc00b50b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7facb000d5d0 tx=0x7facb000d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:56.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facbd7fa700 1 -- 172.21.15.111:0/3977099646 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00125f0 con 0x7facc01609e0 2023-11-28T12:58:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facbd7fa700 1 -- 172.21.15.111:0/3977099646 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb0012750 con 0x7facc01609e0 2023-11-28T12:58:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facc6e3f700 1 -- 172.21.15.111:0/3977099646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc00cddc0 con 0x7facc01609e0 2023-11-28T12:58:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facbd7fa700 1 -- 172.21.15.111:0/3977099646 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb0012b00 con 0x7facc01609e0 2023-11-28T12:58:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.797+0000 7facc6e3f700 1 -- 172.21.15.111:0/3977099646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc00ce170 con 0x7facc01609e0 2023-11-28T12:58:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.798+0000 7facc6e3f700 1 -- 172.21.15.111:0/3977099646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc00ae6d0 con 0x7facc01609e0 2023-11-28T12:58:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.798+0000 7facbd7fa700 1 -- 172.21.15.111:0/3977099646 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7facb0012c60 con 0x7facc01609e0 2023-11-28T12:58:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.799+0000 7facbd7fa700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faca00682f0 0x7faca006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.799+0000 7facc4bdb700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faca00682f0 0x7faca006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.799+0000 7facbd7fa700 1 -- 172.21.15.111:0/3977099646 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6288+0+0 (secure 0 0 0) 0x7facb0085fb0 con 0x7facc01609e0 2023-11-28T12:58:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.799+0000 7facc4bdb700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faca00682f0 0x7faca006a7a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7facb80012e0 tx=0x7facb8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:56.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.803+0000 7facbd7fa700 1 -- 172.21.15.111:0/3977099646 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7facb0052730 con 0x7facc01609e0 2023-11-28T12:58:56.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:56.984+0000 7facc6e3f700 1 -- 172.21.15.111:0/3977099646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7facc006c4d0 con 0x7facc01609e0 2023-11-28T12:58:58.442 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T12:58:58.445 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T12:58:58.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.652+0000 7facbd7fa700 1 -- 172.21.15.111:0/3977099646 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v24) v1 ==== 228+0+0 (secure 0 0 0) 0x7facb0055d50 con 0x7facc01609e0 2023-11-28T12:58:58.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.657+0000 7facc6e3f700 1 -- 172.21.15.111:0/3977099646 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faca00682f0 msgr2=0x7faca006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:58.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.657+0000 7facc6e3f700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faca00682f0 0x7faca006a7a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7facb80012e0 tx=0x7facb8009040 comp rx=0 tx=0).stop 2023-11-28T12:58:58.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.657+0000 7facc6e3f700 1 -- 172.21.15.111:0/3977099646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facc01609e0 msgr2=0x7facc00b50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:58.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.657+0000 7facc6e3f700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facc01609e0 0x7facc00b50b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7facb000d5d0 tx=0x7facb000d990 comp rx=0 tx=0).stop 2023-11-28T12:58:58.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.657+0000 7facc6e3f700 1 -- 172.21.15.111:0/3977099646 shutdown_connections 2023-11-28T12:58:58.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.657+0000 7facc6e3f700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facc01609e0 0x7facc00b50b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:58.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.657+0000 7facc6e3f700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faca00682f0 0x7faca006a7a0 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7facb80012e0 tx=0x7facb8009040 comp rx=0 tx=0).stop 2023-11-28T12:58:58.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.657+0000 7facc6e3f700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7facc01596f0 0x7facc0146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:58.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.657+0000 7facc6e3f700 1 --2- 172.21.15.111:0/3977099646 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7facc015e0c0 0x7facc00b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:58.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.657+0000 7facc6e3f700 1 -- 172.21.15.111:0/3977099646 >> 172.21.15.111:0/3977099646 conn(0x7facc0098f00 msgr2=0x7facc00ad050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:58.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.657+0000 7facc6e3f700 1 -- 172.21.15.111:0/3977099646 shutdown_connections 2023-11-28T12:58:58.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.657+0000 7facc6e3f700 1 -- 172.21.15.111:0/3977099646 wait complete. 2023-11-28T12:58:58.661 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data_ec' created 2023-11-28T12:58:58.677 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-11-28T12:58:58.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.804+0000 7f416e79b700 1 Processor -- start 2023-11-28T12:58:58.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.804+0000 7f416e79b700 1 -- start start 2023-11-28T12:58:58.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.805+0000 7f416e79b700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f41680b0d70 0x7f41681604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:58.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.805+0000 7f416e79b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41680cc550 0x7f41680cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:58.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.805+0000 7f416e79b700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416815cd90 con 0x7f41680b1140 2023-11-28T12:58:58.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.805+0000 7f416e79b700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41680cedd0 con 0x7f41680b0d70 2023-11-28T12:58:58.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.805+0000 7f416e79b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41680ccf10 con 0x7f41680cc550 2023-11-28T12:58:58.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.805+0000 7f4167fff700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f41680b0d70 0x7f41681604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:58.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.805+0000 7f416cd38700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41680cc550 0x7f41680cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:58.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.805+0000 7f41677fe700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f41680b1140 0x7f41680ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:41956/0 (socket says 172.21.15.111:41956) 2023-11-28T12:58:58.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.805+0000 7f41677fe700 1 -- 172.21.15.111:0/156854302 learned_addr learned my addr 172.21.15.111:0/156854302 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:58:58.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.805+0000 7f4167fff700 1 -- 172.21.15.111:0/156854302 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41680cc550 msgr2=0x7f41680cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:58.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.805+0000 7f4167fff700 1 --2- 172.21.15.111:0/156854302 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41680cc550 0x7f41680cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:58.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.805+0000 7f4167fff700 1 -- 172.21.15.111:0/156854302 >> v1:172.21.15.149:6789/0 conn(0x7f41680b1140 legacy=0x7f41680ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:58.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.805+0000 7f4167fff700 1 -- 172.21.15.111:0/156854302 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41681613e0 con 0x7f41680b0d70 2023-11-28T12:58:58.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.806+0000 7f416cd38700 1 --2- 172.21.15.111:0/156854302 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41680cc550 0x7f41680cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:58:58.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.806+0000 7f4167fff700 1 --2- 172.21.15.111:0/156854302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41680b0d70 0x7f41681604b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f415c009820 tx=0x7f415c009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d482a500484e377 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:58.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.806+0000 7f4166ffd700 1 -- 172.21.15.111:0/156854302 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c005e30 con 0x7f41680b0d70 2023-11-28T12:58:58.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.806+0000 7f4166ffd700 1 -- 172.21.15.111:0/156854302 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415c006030 con 0x7f41680b0d70 2023-11-28T12:58:58.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.806+0000 7f4166ffd700 1 -- 172.21.15.111:0/156854302 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c0063c0 con 0x7f41680b0d70 2023-11-28T12:58:58.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.806+0000 7f416e79b700 1 -- 172.21.15.111:0/156854302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41680b0d70 msgr2=0x7f41681604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:58.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.806+0000 7f416e79b700 1 --2- 172.21.15.111:0/156854302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41680b0d70 0x7f41681604b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f415c009820 tx=0x7f415c009b30 comp rx=0 tx=0).stop 2023-11-28T12:58:58.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.806+0000 7f416e79b700 1 -- 172.21.15.111:0/156854302 shutdown_connections 2023-11-28T12:58:58.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.806+0000 7f416e79b700 1 --2- 172.21.15.111:0/156854302 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41680cc550 0x7f41680cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:58.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.806+0000 7f416e79b700 1 --2- 172.21.15.111:0/156854302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41680b0d70 0x7f41681604b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:58.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.806+0000 7f416e79b700 1 -- 172.21.15.111:0/156854302 >> 172.21.15.111:0/156854302 conn(0x7f4168098ee0 msgr2=0x7f416806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:58.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.806+0000 7f416e79b700 1 -- 172.21.15.111:0/156854302 shutdown_connections 2023-11-28T12:58:58.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.807+0000 7f416e79b700 1 -- 172.21.15.111:0/156854302 wait complete. 2023-11-28T12:58:58.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.808+0000 7f416e79b700 1 Processor -- start 2023-11-28T12:58:58.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.808+0000 7f416e79b700 1 -- start start 2023-11-28T12:58:58.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f416e79b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41680b0d70 0x7f416815b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:58.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f416e79b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680b1140 0x7f41681553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:58.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f416e79b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41680cc550 0x7f4168155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:58.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f416e79b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41680b4790 con 0x7f41680b1140 2023-11-28T12:58:58.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f416e79b700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41680b36f0 con 0x7f41680b0d70 2023-11-28T12:58:58.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f416e79b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41680b39d0 con 0x7f41680cc550 2023-11-28T12:58:58.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f41677fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680b1140 0x7f41681553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:58.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f41677fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680b1140 0x7f41681553f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60874/0 (socket says 172.21.15.111:60874) 2023-11-28T12:58:58.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f41677fe700 1 -- 172.21.15.111:0/1240210162 learned_addr learned my addr 172.21.15.111:0/1240210162 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:58.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f4167fff700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41680b0d70 0x7f416815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:58.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f416cd38700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41680cc550 0x7f4168155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:58.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f41677fe700 1 -- 172.21.15.111:0/1240210162 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41680cc550 msgr2=0x7f4168155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:58.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f41677fe700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41680cc550 0x7f4168155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:58.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f41677fe700 1 -- 172.21.15.111:0/1240210162 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41680b0d70 msgr2=0x7f416815b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:58.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f41677fe700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41680b0d70 0x7f416815b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:58.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f41677fe700 1 -- 172.21.15.111:0/1240210162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4160009a70 con 0x7f41680b1140 2023-11-28T12:58:58.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f41677fe700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680b1140 0x7f41681553f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f415800c490 tx=0x7f415800c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:58.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f4164ff9700 1 -- 172.21.15.111:0/1240210162 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158005d70 con 0x7f41680b1140 2023-11-28T12:58:58.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f4164ff9700 1 -- 172.21.15.111:0/1240210162 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415800ce90 con 0x7f41680b1140 2023-11-28T12:58:58.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f416e79b700 1 -- 172.21.15.111:0/1240210162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415c009430 con 0x7f41680b1140 2023-11-28T12:58:58.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f4164ff9700 1 -- 172.21.15.111:0/1240210162 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41580072f0 con 0x7f41680b1140 2023-11-28T12:58:58.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.809+0000 7f416e79b700 1 -- 172.21.15.111:0/1240210162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4168160e20 con 0x7f41680b1140 2023-11-28T12:58:58.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.810+0000 7f416e79b700 1 -- 172.21.15.111:0/1240210162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4168160b40 con 0x7f41680b1140 2023-11-28T12:58:58.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.811+0000 7f4164ff9700 1 -- 172.21.15.111:0/1240210162 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4158019030 con 0x7f41680b1140 2023-11-28T12:58:58.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.811+0000 7f4164ff9700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41540682f0 0x7f415406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:58.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.811+0000 7f4164ff9700 1 -- 172.21.15.111:0/1240210162 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6781+0+0 (secure 0 0 0) 0x7f4158085000 con 0x7f41680b1140 2023-11-28T12:58:58.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.811+0000 7f4167fff700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41540682f0 0x7f415406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:58.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.811+0000 7f4167fff700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41540682f0 0x7f415406a7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f415c009b70 tx=0x7f415c008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:58.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.815+0000 7f4164ff9700 1 -- 172.21.15.111:0/1240210162 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4158051590 con 0x7f41680b1140 2023-11-28T12:58:58.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:58.997+0000 7f416e79b700 1 -- 172.21.15.111:0/1240210162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f4168161100 con 0x7f41680b1140 2023-11-28T12:58:59.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.655+0000 7f4164ff9700 1 -- 172.21.15.111:0/1240210162 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 5 allow_ec_overwrites to true v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4158017020 con 0x7f41680b1140 2023-11-28T12:58:59.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.660+0000 7f416e79b700 1 -- 172.21.15.111:0/1240210162 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41540682f0 msgr2=0x7f415406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:59.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.660+0000 7f416e79b700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41540682f0 0x7f415406a7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f415c009b70 tx=0x7f415c008070 comp rx=0 tx=0).stop 2023-11-28T12:58:59.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.660+0000 7f416e79b700 1 -- 172.21.15.111:0/1240210162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680b1140 msgr2=0x7f41681553f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:59.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.660+0000 7f416e79b700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680b1140 0x7f41681553f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f415800c490 tx=0x7f415800c7a0 comp rx=0 tx=0).stop 2023-11-28T12:58:59.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.661+0000 7f416e79b700 1 -- 172.21.15.111:0/1240210162 shutdown_connections 2023-11-28T12:58:59.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.661+0000 7f416e79b700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41680cc550 0x7f4168155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:59.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.661+0000 7f416e79b700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41540682f0 0x7f415406a7a0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f415c009b70 tx=0x7f415c008070 comp rx=0 tx=0).stop 2023-11-28T12:58:59.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.661+0000 7f416e79b700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41680b0d70 0x7f416815b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:59.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.661+0000 7f416e79b700 1 --2- 172.21.15.111:0/1240210162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680b1140 0x7f41681553f0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f415800c490 tx=0x7f415800c7a0 comp rx=0 tx=0).stop 2023-11-28T12:58:59.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.661+0000 7f416e79b700 1 -- 172.21.15.111:0/1240210162 >> 172.21.15.111:0/1240210162 conn(0x7f4168098ee0 msgr2=0x7f41680af760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:59.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.661+0000 7f416e79b700 1 -- 172.21.15.111:0/1240210162 shutdown_connections 2023-11-28T12:58:59.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.661+0000 7f416e79b700 1 -- 172.21.15.111:0/1240210162 wait complete. 2023-11-28T12:58:59.668 INFO:teuthology.orchestra.run.smithi111.stderr:set pool 5 allow_ec_overwrites to true 2023-11-28T12:58:59.691 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-11-28T12:58:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.813+0000 7fc7dff56700 1 Processor -- start 2023-11-28T12:58:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.813+0000 7fc7dff56700 1 -- start start 2023-11-28T12:58:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.814+0000 7fc7dff56700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc7d80b15c0 0x7fc7d80ac670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.814+0000 7fc7dff56700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d80af650 con 0x7fc7d80b0250 2023-11-28T12:58:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.814+0000 7fc7dff56700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d80cb6d0 con 0x7fc7d815a9e0 2023-11-28T12:58:59.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.814+0000 7fc7dff56700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d80cb9b0 con 0x7fc7d80b15c0 2023-11-28T12:58:59.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.814+0000 7fc7dd4f1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc7d80b15c0 0x7fc7d80ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:59.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.814+0000 7fc7dd4f1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc7d80b15c0 0x7fc7d80ac670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60882/0 (socket says 172.21.15.111:60882) 2023-11-28T12:58:59.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.814+0000 7fc7de4f3700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc7d80b0250 0x7fc7d80cc770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33430/0 (socket says 172.21.15.111:33430) 2023-11-28T12:58:59.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.814+0000 7fc7dd4f1700 1 -- 172.21.15.111:0/1025028660 learned_addr learned my addr 172.21.15.111:0/1025028660 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:59.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.815+0000 7fc7dd4f1700 1 -- 172.21.15.111:0/1025028660 >> v1:172.21.15.149:6789/0 conn(0x7fc7d815a9e0 legacy=0x7fc7d80caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:58:59.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.815+0000 7fc7dd4f1700 1 -- 172.21.15.111:0/1025028660 >> v1:172.21.15.111:6789/0 conn(0x7fc7d80b0250 legacy=0x7fc7d80cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:59.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.815+0000 7fc7dd4f1700 1 -- 172.21.15.111:0/1025028660 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d80cbc90 con 0x7fc7d80b15c0 2023-11-28T12:58:59.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.815+0000 7fc7dd4f1700 1 --2- 172.21.15.111:0/1025028660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d80b15c0 0x7fc7d80ac670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc7c80098f0 tx=0x7fc7c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c072dc3c6a7ab3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:59.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.815+0000 7fc7dccf0700 1 -- 172.21.15.111:0/1025028660 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8004230 con 0x7fc7d80b15c0 2023-11-28T12:58:59.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.815+0000 7fc7dccf0700 1 -- 172.21.15.111:0/1025028660 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c8004390 con 0x7fc7d80b15c0 2023-11-28T12:58:59.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.815+0000 7fc7dccf0700 1 -- 172.21.15.111:0/1025028660 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c80036a0 con 0x7fc7d80b15c0 2023-11-28T12:58:59.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.815+0000 7fc7dff56700 1 -- 172.21.15.111:0/1025028660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d80b15c0 msgr2=0x7fc7d80ac670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:59.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.815+0000 7fc7dff56700 1 --2- 172.21.15.111:0/1025028660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d80b15c0 0x7fc7d80ac670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc7c80098f0 tx=0x7fc7c8009c00 comp rx=0 tx=0).stop 2023-11-28T12:58:59.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.815+0000 7fc7dff56700 1 -- 172.21.15.111:0/1025028660 shutdown_connections 2023-11-28T12:58:59.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.815+0000 7fc7dff56700 1 --2- 172.21.15.111:0/1025028660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d80b15c0 0x7fc7d80ac670 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:59.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.815+0000 7fc7dff56700 1 -- 172.21.15.111:0/1025028660 >> 172.21.15.111:0/1025028660 conn(0x7fc7d8098ee0 msgr2=0x7fc7d806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:58:59.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.815+0000 7fc7dff56700 1 -- 172.21.15.111:0/1025028660 shutdown_connections 2023-11-28T12:58:59.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.816+0000 7fc7dff56700 1 -- 172.21.15.111:0/1025028660 wait complete. 2023-11-28T12:58:59.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.817+0000 7fc7dff56700 1 Processor -- start 2023-11-28T12:58:59.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.817+0000 7fc7dff56700 1 -- start start 2023-11-28T12:58:59.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7dff56700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7d80b0250 0x7fc7d80c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:59.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7dff56700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7d80b15c0 0x7fc7d81544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:59.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7dff56700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d815a9e0 0x7fc7d80b3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:59.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7dff56700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d80b4490 con 0x7fc7d815a9e0 2023-11-28T12:58:59.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7dff56700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d815c490 con 0x7fc7d80b15c0 2023-11-28T12:58:59.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7dff56700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d80c70c0 con 0x7fc7d80b0250 2023-11-28T12:58:59.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7de4f3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d815a9e0 0x7fc7d80b3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:59.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7de4f3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d815a9e0 0x7fc7d80b3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60890/0 (socket says 172.21.15.111:60890) 2023-11-28T12:58:59.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7de4f3700 1 -- 172.21.15.111:0/2930909822 learned_addr learned my addr 172.21.15.111:0/2930909822 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:58:59.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7ddcf2700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7d80b0250 0x7fc7d80c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:59.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7dd4f1700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7d80b15c0 0x7fc7d81544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:59.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7de4f3700 1 -- 172.21.15.111:0/2930909822 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7d80b0250 msgr2=0x7fc7d80c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:59.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7de4f3700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7d80b0250 0x7fc7d80c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:59.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7de4f3700 1 -- 172.21.15.111:0/2930909822 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7d80b15c0 msgr2=0x7fc7d81544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:58:59.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7de4f3700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7d80b15c0 0x7fc7d81544c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:58:59.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7de4f3700 1 -- 172.21.15.111:0/2930909822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c8009550 con 0x7fc7d815a9e0 2023-11-28T12:58:59.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.818+0000 7fc7de4f3700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d815a9e0 0x7fc7d80b3840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000cbd0 tx=0x7fc7d000cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:59.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.819+0000 7fc7ce7fc700 1 -- 172.21.15.111:0/2930909822 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d000aaa0 con 0x7fc7d815a9e0 2023-11-28T12:58:59.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.819+0000 7fc7ce7fc700 1 -- 172.21.15.111:0/2930909822 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d000ac00 con 0x7fc7d815a9e0 2023-11-28T12:58:59.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.819+0000 7fc7dff56700 1 -- 172.21.15.111:0/2930909822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d80cc250 con 0x7fc7d815a9e0 2023-11-28T12:58:59.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.819+0000 7fc7ce7fc700 1 -- 172.21.15.111:0/2930909822 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0007070 con 0x7fc7d815a9e0 2023-11-28T12:58:59.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.819+0000 7fc7dff56700 1 -- 172.21.15.111:0/2930909822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d80cbf70 con 0x7fc7d815a9e0 2023-11-28T12:58:59.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.820+0000 7fc7dff56700 1 -- 172.21.15.111:0/2930909822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d80b06d0 con 0x7fc7d815a9e0 2023-11-28T12:58:59.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.820+0000 7fc7ce7fc700 1 -- 172.21.15.111:0/2930909822 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc7d0019030 con 0x7fc7d815a9e0 2023-11-28T12:58:59.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.820+0000 7fc7ce7fc700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7bc0682f0 0x7fc7bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:58:59.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.821+0000 7fc7ce7fc700 1 -- 172.21.15.111:0/2930909822 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6781+0+0 (secure 0 0 0) 0x7fc7d00859f0 con 0x7fc7d815a9e0 2023-11-28T12:58:59.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.821+0000 7fc7ddcf2700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7bc0682f0 0x7fc7bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:58:59.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.821+0000 7fc7ddcf2700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7bc0682f0 0x7fc7bc06a7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7d80aea00 tx=0x7fc7d40090e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:58:59.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:58:59.825+0000 7fc7ce7fc700 1 -- 172.21.15.111:0/2930909822 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc7d004b5e0 con 0x7fc7d815a9e0 2023-11-28T12:59:00.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.052+0000 7fc7dff56700 1 -- 172.21.15.111:0/2930909822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fc7d8067170 con 0x7fc7d815a9e0 2023-11-28T12:59:00.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.058+0000 7fc7ce7fc700 1 -- 172.21.15.111:0/2930909822 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 5 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7fc7d00170b0 con 0x7fc7d815a9e0 2023-11-28T12:59:00.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.060+0000 7fc7dff56700 1 -- 172.21.15.111:0/2930909822 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7bc0682f0 msgr2=0x7fc7bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.060+0000 7fc7dff56700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7bc0682f0 0x7fc7bc06a7a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7d80aea00 tx=0x7fc7d40090e0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.060+0000 7fc7dff56700 1 -- 172.21.15.111:0/2930909822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d815a9e0 msgr2=0x7fc7d80b3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.060+0000 7fc7dff56700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d815a9e0 0x7fc7d80b3840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000cbd0 tx=0x7fc7d000cee0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.060+0000 7fc7dff56700 1 -- 172.21.15.111:0/2930909822 shutdown_connections 2023-11-28T12:59:00.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.060+0000 7fc7dff56700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7d815a9e0 0x7fc7d80b3840 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.060+0000 7fc7dff56700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7d80b0250 0x7fc7d80c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.060+0000 7fc7dff56700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7bc0682f0 0x7fc7bc06a7a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.060+0000 7fc7dff56700 1 --2- 172.21.15.111:0/2930909822 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7d80b15c0 0x7fc7d81544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.060+0000 7fc7dff56700 1 -- 172.21.15.111:0/2930909822 >> 172.21.15.111:0/2930909822 conn(0x7fc7d8098ee0 msgr2=0x7fc7d806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:00.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.061+0000 7fc7dff56700 1 -- 172.21.15.111:0/2930909822 shutdown_connections 2023-11-28T12:59:00.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.061+0000 7fc7dff56700 1 -- 172.21.15.111:0/2930909822 wait complete. 2023-11-28T12:59:00.065 INFO:teuthology.orchestra.run.smithi111.stderr:added data pool 5 to fsmap 2023-11-28T12:59:00.074 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T12:59:00.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.193+0000 7fe19eff7700 1 Processor -- start 2023-11-28T12:59:00.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.193+0000 7fe19eff7700 1 -- start start 2023-11-28T12:59:00.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.193+0000 7fe19eff7700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe1980b0d70 0x7fe1981604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe19eff7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1980cc550 0x7fe1980cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe19eff7700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe19815cd90 con 0x7fe1980b1140 2023-11-28T12:59:00.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe19eff7700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1980cedd0 con 0x7fe1980b0d70 2023-11-28T12:59:00.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe19eff7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980ccf10 con 0x7fe1980cc550 2023-11-28T12:59:00.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe197fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe1980b1140 0x7fe1980ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33440/0 (socket says 172.21.15.111:33440) 2023-11-28T12:59:00.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe197fff700 1 -- 172.21.15.111:0/3981379804 learned_addr learned my addr 172.21.15.111:0/3981379804 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:00.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe19cd93700 1 --2- 172.21.15.111:0/3981379804 >> v2:172.21.15.149:3300/0 conn(0x7fe1980b0d70 0x7fe1981604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe19d594700 1 --2- 172.21.15.111:0/3981379804 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1980cc550 0x7fe1980cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742525312 0 0) 0x7fe184001110 con 0x7fe1980b1140 2023-11-28T12:59:00.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1880035b0 con 0x7fe1980b1140 2023-11-28T12:59:00.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3971305850 0 0) 0x7fe1880035b0 con 0x7fe1980b1140 2023-11-28T12:59:00.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1980cc550 msgr2=0x7fe1980cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:00.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 --2- 172.21.15.111:0/3981379804 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1980cc550 0x7fe1980cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 >> v2:172.21.15.149:3300/0 conn(0x7fe1980b0d70 msgr2=0x7fe1981604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:00.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 --2- 172.21.15.111:0/3981379804 >> v2:172.21.15.149:3300/0 conn(0x7fe1980b0d70 0x7fe1981604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1981613e0 con 0x7fe1980b1140 2023-11-28T12:59:00.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fe184001800 con 0x7fe1980b1140 2023-11-28T12:59:00.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 >> v1:172.21.15.111:6789/0 conn(0x7fe1980b1140 legacy=0x7fe1980ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 --2- 172.21.15.111:0/3981379804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe188003a00 0x7fe188003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 --2- 172.21.15.111:0/3981379804 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1980cc550 0x7fe188004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 --2- 172.21.15.111:0/3981379804 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe188004d90 0x7fe188005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980cedd0 con 0x7fe188003a00 2023-11-28T12:59:00.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980ccf10 con 0x7fe188004d90 2023-11-28T12:59:00.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18800ba10 con 0x7fe1980cc550 2023-11-28T12:59:00.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe19cd93700 1 --2- 172.21.15.111:0/3981379804 >> v2:172.21.15.149:3300/0 conn(0x7fe1980b0d70 0x7fe1981604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:00.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe197fff700 1 --2- 172.21.15.111:0/3981379804 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1980cc550 0x7fe188004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe19cd93700 1 --2- 172.21.15.111:0/3981379804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe188003a00 0x7fe188003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.194+0000 7fe19d594700 1 --2- 172.21.15.111:0/3981379804 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe188004d90 0x7fe188005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe19cd93700 1 -- 172.21.15.111:0/3981379804 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1980cc550 msgr2=0x7fe188004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:00.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe19cd93700 1 --2- 172.21.15.111:0/3981379804 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1980cc550 0x7fe188004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe19cd93700 1 -- 172.21.15.111:0/3981379804 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe188004d90 msgr2=0x7fe188005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:00.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe19cd93700 1 --2- 172.21.15.111:0/3981379804 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe188004d90 0x7fe188005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe19cd93700 1 -- 172.21.15.111:0/3981379804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1880035b0 con 0x7fe188003a00 2023-11-28T12:59:00.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe197fff700 1 --2- 172.21.15.111:0/3981379804 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1980cc550 0x7fe188004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:00.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe19cd93700 1 --2- 172.21.15.111:0/3981379804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe188003a00 0x7fe188003e10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009cf0 tx=0x7fe188005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44869a5cafe74120 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:00.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c004030 con 0x7fe188003a00 2023-11-28T12:59:00.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18c004190 con 0x7fe188003a00 2023-11-28T12:59:00.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe1977fe700 1 -- 172.21.15.111:0/3981379804 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c004500 con 0x7fe188003a00 2023-11-28T12:59:00.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe19eff7700 1 -- 172.21.15.111:0/3981379804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe188003a00 msgr2=0x7fe188003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe19eff7700 1 --2- 172.21.15.111:0/3981379804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe188003a00 0x7fe188003e10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009cf0 tx=0x7fe188005880 comp rx=0 tx=0).stop 2023-11-28T12:59:00.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe19d594700 1 -- 172.21.15.111:0/3981379804 reap_dead start 2023-11-28T12:59:00.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.195+0000 7fe19eff7700 1 -- 172.21.15.111:0/3981379804 shutdown_connections 2023-11-28T12:59:00.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.196+0000 7fe19eff7700 1 -- 172.21.15.111:0/3981379804 >> 172.21.15.111:0/3981379804 conn(0x7fe198098ee0 msgr2=0x7fe19806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:00.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.196+0000 7fe19eff7700 1 -- 172.21.15.111:0/3981379804 shutdown_connections 2023-11-28T12:59:00.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.196+0000 7fe19eff7700 1 -- 172.21.15.111:0/3981379804 wait complete. 2023-11-28T12:59:00.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.197+0000 7fe19eff7700 1 Processor -- start 2023-11-28T12:59:00.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.197+0000 7fe19eff7700 1 -- start start 2023-11-28T12:59:00.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19eff7700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe188003a00 0x7fe1981e7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19eff7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe198157c90 0x7fe1981e67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19eff7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1981e6d20 0x7fe1981e71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19eff7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980b4080 con 0x7fe1981e6d20 2023-11-28T12:59:00.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19eff7700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1981e7fd0 con 0x7fe188003a00 2023-11-28T12:59:00.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19eff7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1980c99a0 con 0x7fe198157c90 2023-11-28T12:59:00.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19d594700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1981e6d20 0x7fe1981e71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19d594700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1981e6d20 0x7fe1981e71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60908/0 (socket says 172.21.15.111:60908) 2023-11-28T12:59:00.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19d594700 1 -- 172.21.15.111:0/581022527 learned_addr learned my addr 172.21.15.111:0/581022527 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:00.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19cd93700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe188003a00 0x7fe1981e7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe197fff700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe198157c90 0x7fe1981e67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19d594700 1 -- 172.21.15.111:0/581022527 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe198157c90 msgr2=0x7fe1981e67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19d594700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe198157c90 0x7fe1981e67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19d594700 1 -- 172.21.15.111:0/581022527 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe188003a00 msgr2=0x7fe1981e7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19d594700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe188003a00 0x7fe1981e7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19d594700 1 -- 172.21.15.111:0/581022527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe198160e20 con 0x7fe1981e6d20 2023-11-28T12:59:00.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19cd93700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe188003a00 0x7fe1981e7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:00.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19d594700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1981e6d20 0x7fe1981e71c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe19806c470 tx=0x7fe190011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:00.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe1957fa700 1 -- 172.21.15.111:0/581022527 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19000e040 con 0x7fe1981e6d20 2023-11-28T12:59:00.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe1957fa700 1 -- 172.21.15.111:0/581022527 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19000f040 con 0x7fe1981e6d20 2023-11-28T12:59:00.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19eff7700 1 -- 172.21.15.111:0/581022527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe198160b40 con 0x7fe1981e6d20 2023-11-28T12:59:00.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe1957fa700 1 -- 172.21.15.111:0/581022527 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19000fdb0 con 0x7fe1981e6d20 2023-11-28T12:59:00.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.198+0000 7fe19eff7700 1 -- 172.21.15.111:0/581022527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe198161100 con 0x7fe1981e6d20 2023-11-28T12:59:00.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.199+0000 7fe19eff7700 1 -- 172.21.15.111:0/581022527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1980b41c0 con 0x7fe1981e6d20 2023-11-28T12:59:00.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.200+0000 7fe1957fa700 1 -- 172.21.15.111:0/581022527 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe190015430 con 0x7fe1981e6d20 2023-11-28T12:59:00.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.200+0000 7fe1957fa700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe17c0682f0 0x7fe17c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.200+0000 7fe1957fa700 1 -- 172.21.15.111:0/581022527 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe1900867c0 con 0x7fe1981e6d20 2023-11-28T12:59:00.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.200+0000 7fe19cd93700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe17c0682f0 0x7fe17c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.200+0000 7fe19cd93700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe17c0682f0 0x7fe17c06a7a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009c10 tx=0x7fe18c01e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:00.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.204+0000 7fe1957fa700 1 -- 172.21.15.111:0/581022527 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe1980b41c0 con 0x7fe1981e6d20 2023-11-28T12:59:00.320 INFO:tasks.ceph.mds.c.smithi111.stderr:2023-11-28T12:59:00.319+0000 7f1d180c4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T12:59:00.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.434+0000 7fe19eff7700 1 -- 172.21.15.111:0/581022527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe19815cd90 con 0x7fe1981e6d20 2023-11-28T12:59:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.434+0000 7fe1957fa700 1 -- 172.21.15.111:0/581022527 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4656 (secure 0 0 0) 0x7fe19815cd90 con 0x7fe1981e6d20 2023-11-28T12:59:00.436 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:00.436 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2513950989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2513950989},{"type":"v1","addr":"172.21.15.149:6834","nonce":2513950989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1258891714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1258891714},{"type":"v1","addr":"172.21.15.149:6835","nonce":1258891714}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/3923909288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3923909288},{"type":"v1","addr":"172.21.15.111:6836","nonce":3923909288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T12:58:55.633507+0000","modified":"2023-11-28T12:59:00.054172+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/3180164366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":3180164366},{"type":"v1","addr":"172.21.15.111:6837","nonce":3180164366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-11-28T12:59:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.436+0000 7fe19eff7700 1 -- 172.21.15.111:0/581022527 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe17c0682f0 msgr2=0x7fe17c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.436+0000 7fe19eff7700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe17c0682f0 0x7fe17c06a7a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009c10 tx=0x7fe18c01e040 comp rx=0 tx=0).stop 2023-11-28T12:59:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.436+0000 7fe19eff7700 1 -- 172.21.15.111:0/581022527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1981e6d20 msgr2=0x7fe1981e71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.436+0000 7fe19eff7700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1981e6d20 0x7fe1981e71c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe19806c470 tx=0x7fe190011f80 comp rx=0 tx=0).stop 2023-11-28T12:59:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.436+0000 7fe19eff7700 1 -- 172.21.15.111:0/581022527 shutdown_connections 2023-11-28T12:59:00.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.436+0000 7fe19eff7700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1981e6d20 0x7fe1981e71c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.436+0000 7fe19eff7700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe17c0682f0 0x7fe17c06a7a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.436+0000 7fe19eff7700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe188003a00 0x7fe1981e7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.436+0000 7fe19eff7700 1 --2- 172.21.15.111:0/581022527 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe198157c90 0x7fe1981e67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.436+0000 7fe19eff7700 1 -- 172.21.15.111:0/581022527 >> 172.21.15.111:0/581022527 conn(0x7fe198098ee0 msgr2=0x7fe198072cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.436+0000 7fe19eff7700 1 -- 172.21.15.111:0/581022527 shutdown_connections 2023-11-28T12:59:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.437+0000 7fe19eff7700 1 -- 172.21.15.111:0/581022527 wait complete. 2023-11-28T12:59:00.439 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 7 2023-11-28T12:59:00.450 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:00.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.568+0000 7f9037063700 1 Processor -- start 2023-11-28T12:59:00.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.568+0000 7f9037063700 1 -- start start 2023-11-28T12:59:00.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.569+0000 7f9037063700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9030072460 0x7f903006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.569+0000 7f9037063700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f90300712e0 0x7f9030071760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.569+0000 7f9037063700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f903006b610 con 0x7f9030159440 2023-11-28T12:59:00.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.569+0000 7f9037063700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9030071ca0 con 0x7f90300712e0 2023-11-28T12:59:00.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.569+0000 7f9037063700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90300cb830 con 0x7f9030072460 2023-11-28T12:59:00.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.569+0000 7f9034dff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9030159440 0x7f90300cafb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33458/0 (socket says 172.21.15.111:33458) 2023-11-28T12:59:00.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.569+0000 7f9034dff700 1 -- 172.21.15.111:0/1892935925 learned_addr learned my addr 172.21.15.111:0/1892935925 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:00.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.569+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413685174 0 0) 0x7f9024001110 con 0x7f9030159440 2023-11-28T12:59:00.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.569+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90200035b0 con 0x7f9030159440 2023-11-28T12:59:00.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.569+0000 7f9035600700 1 --2- 172.21.15.111:0/1892935925 >> v2:172.21.15.149:3300/0 conn(0x7f90300712e0 0x7f9030071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.569+0000 7f902ffff700 1 --2- 172.21.15.111:0/1892935925 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9030072460 0x7f903006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 451131192 0 0) 0x7f90200035b0 con 0x7f9030159440 2023-11-28T12:59:00.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9030072460 msgr2=0x7f903006b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:00.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 --2- 172.21.15.111:0/1892935925 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9030072460 0x7f903006b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 >> v2:172.21.15.149:3300/0 conn(0x7f90300712e0 msgr2=0x7f9030071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:00.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 --2- 172.21.15.111:0/1892935925 >> v2:172.21.15.149:3300/0 conn(0x7f90300712e0 0x7f9030071760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903006a310 con 0x7f9030159440 2023-11-28T12:59:00.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f9024001800 con 0x7f9030159440 2023-11-28T12:59:00.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 >> v1:172.21.15.111:6789/0 conn(0x7f9030159440 legacy=0x7f90300cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 --2- 172.21.15.111:0/1892935925 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9020003a00 0x7f9020003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 --2- 172.21.15.111:0/1892935925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9020004430 0x7f90200048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 --2- 172.21.15.111:0/1892935925 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9030072460 0x7f9020004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9030071ca0 con 0x7f9020004430 2023-11-28T12:59:00.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90300cb830 con 0x7f9020003a00 2023-11-28T12:59:00.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902000dc20 con 0x7f9030072460 2023-11-28T12:59:00.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902ffff700 1 --2- 172.21.15.111:0/1892935925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9020004430 0x7f90200048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f9034dff700 1 --2- 172.21.15.111:0/1892935925 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9020003a00 0x7f9020003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f9035600700 1 --2- 172.21.15.111:0/1892935925 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9030072460 0x7f9020004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902ffff700 1 -- 172.21.15.111:0/1892935925 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9030072460 msgr2=0x7f9020004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:00.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902ffff700 1 --2- 172.21.15.111:0/1892935925 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9030072460 0x7f9020004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902ffff700 1 -- 172.21.15.111:0/1892935925 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9020003a00 msgr2=0x7f9020003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:00.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902ffff700 1 --2- 172.21.15.111:0/1892935925 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9020003a00 0x7f9020003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902ffff700 1 -- 172.21.15.111:0/1892935925 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90200035b0 con 0x7f9020004430 2023-11-28T12:59:00.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902ffff700 1 --2- 172.21.15.111:0/1892935925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9020004430 0x7f90200048f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f901c009b60 tx=0x7f9020005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=841cb6abe1c8e321 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:00.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c00e6c0 con 0x7f9020004430 2023-11-28T12:59:00.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901c00e820 con 0x7f9020004430 2023-11-28T12:59:00.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.570+0000 7f902f7fe700 1 -- 172.21.15.111:0/1892935925 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c00eb90 con 0x7f9020004430 2023-11-28T12:59:00.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.571+0000 7f9037063700 1 -- 172.21.15.111:0/1892935925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9020004430 msgr2=0x7f90200048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.571+0000 7f9037063700 1 --2- 172.21.15.111:0/1892935925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9020004430 0x7f90200048f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f901c009b60 tx=0x7f9020005880 comp rx=0 tx=0).stop 2023-11-28T12:59:00.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.571+0000 7f9035600700 1 -- 172.21.15.111:0/1892935925 reap_dead start 2023-11-28T12:59:00.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.571+0000 7f9037063700 1 -- 172.21.15.111:0/1892935925 shutdown_connections 2023-11-28T12:59:00.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.571+0000 7f9037063700 1 -- 172.21.15.111:0/1892935925 >> 172.21.15.111:0/1892935925 conn(0x7f90300b0db0 msgr2=0x7f90300b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:00.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.571+0000 7f9037063700 1 -- 172.21.15.111:0/1892935925 shutdown_connections 2023-11-28T12:59:00.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.571+0000 7f9037063700 1 -- 172.21.15.111:0/1892935925 wait complete. 2023-11-28T12:59:00.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.572+0000 7f9037063700 1 Processor -- start 2023-11-28T12:59:00.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.572+0000 7f9037063700 1 -- start start 2023-11-28T12:59:00.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f9037063700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9020004430 0x7f90300b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f9037063700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90300b20f0 0x7f90301f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f9037063700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90301f6410 0x7f90301f68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f9037063700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90300b26b0 con 0x7f90300b20f0 2023-11-28T12:59:00.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f9037063700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90300ad2f0 con 0x7f90301f6410 2023-11-28T12:59:00.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f9037063700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90300b5070 con 0x7f9020004430 2023-11-28T12:59:00.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f902ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90300b20f0 0x7f90301f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f902ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90300b20f0 0x7f90301f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60926/0 (socket says 172.21.15.111:60926) 2023-11-28T12:59:00.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f902ffff700 1 -- 172.21.15.111:0/1987933650 learned_addr learned my addr 172.21.15.111:0/1987933650 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:00.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f9034dff700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9020004430 0x7f90300b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f9035600700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90301f6410 0x7f90301f68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f902ffff700 1 -- 172.21.15.111:0/1987933650 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9020004430 msgr2=0x7f90300b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f902ffff700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9020004430 0x7f90300b4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f902ffff700 1 -- 172.21.15.111:0/1987933650 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90301f6410 msgr2=0x7f90301f68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f902ffff700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90301f6410 0x7f90301f68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.573+0000 7f902ffff700 1 -- 172.21.15.111:0/1987933650 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903006a8d0 con 0x7f90300b20f0 2023-11-28T12:59:00.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.574+0000 7f902ffff700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90300b20f0 0x7f90301f3240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f901c000c00 tx=0x7f901c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:00.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.574+0000 7f902d7fa700 1 -- 172.21.15.111:0/1987933650 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c00e6c0 con 0x7f90300b20f0 2023-11-28T12:59:00.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.574+0000 7f902d7fa700 1 -- 172.21.15.111:0/1987933650 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901c003ad0 con 0x7f90300b20f0 2023-11-28T12:59:00.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.574+0000 7f9037063700 1 -- 172.21.15.111:0/1987933650 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903006a5f0 con 0x7f90300b20f0 2023-11-28T12:59:00.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.574+0000 7f902d7fa700 1 -- 172.21.15.111:0/1987933650 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c0045d0 con 0x7f90300b20f0 2023-11-28T12:59:00.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.574+0000 7f9037063700 1 -- 172.21.15.111:0/1987933650 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903006abb0 con 0x7f90300b20f0 2023-11-28T12:59:00.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.575+0000 7f9037063700 1 -- 172.21.15.111:0/1987933650 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9030067170 con 0x7f90300b20f0 2023-11-28T12:59:00.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.575+0000 7f902d7fa700 1 -- 172.21.15.111:0/1987933650 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f901c025070 con 0x7f90300b20f0 2023-11-28T12:59:00.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.575+0000 7f902d7fa700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90140682f0 0x7f901406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.576+0000 7f902d7fa700 1 -- 172.21.15.111:0/1987933650 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f901c0877d0 con 0x7f90300b20f0 2023-11-28T12:59:00.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.576+0000 7f9034dff700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90140682f0 0x7f901406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.576+0000 7f9034dff700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90140682f0 0x7f901406a7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9024000fd0 tx=0x7f9024009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:00.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.580+0000 7f902d7fa700 1 -- 172.21.15.111:0/1987933650 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f901c053d40 con 0x7f90300b20f0 2023-11-28T12:59:00.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.765+0000 7f9037063700 1 -- 172.21.15.111:0/1987933650 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f903006b3b0 con 0x7f90300b20f0 2023-11-28T12:59:00.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.766+0000 7f902d7fa700 1 -- 172.21.15.111:0/1987933650 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19796 (secure 0 0 0) 0x7f901c057360 con 0x7f90300b20f0 2023-11-28T12:59:00.767 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:00.768 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:00.053943+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:00.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.768+0000 7f9037063700 1 -- 172.21.15.111:0/1987933650 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90140682f0 msgr2=0x7f901406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.768+0000 7f9037063700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90140682f0 0x7f901406a7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9024000fd0 tx=0x7f9024009040 comp rx=0 tx=0).stop 2023-11-28T12:59:00.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.768+0000 7f9037063700 1 -- 172.21.15.111:0/1987933650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90300b20f0 msgr2=0x7f90301f3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.768+0000 7f9037063700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90300b20f0 0x7f90301f3240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f901c000c00 tx=0x7f901c0047c0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.768+0000 7f9037063700 1 -- 172.21.15.111:0/1987933650 shutdown_connections 2023-11-28T12:59:00.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.768+0000 7f9037063700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90140682f0 0x7f901406a7a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.768+0000 7f9037063700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90301f6410 0x7f90301f68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.768+0000 7f9037063700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9020004430 0x7f90300b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.768+0000 7f9037063700 1 --2- 172.21.15.111:0/1987933650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90300b20f0 0x7f90301f3240 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.768+0000 7f9037063700 1 -- 172.21.15.111:0/1987933650 >> 172.21.15.111:0/1987933650 conn(0x7f90300b0db0 msgr2=0x7f90300cce70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:00.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.768+0000 7f9037063700 1 -- 172.21.15.111:0/1987933650 shutdown_connections 2023-11-28T12:59:00.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.768+0000 7f9037063700 1 -- 172.21.15.111:0/1987933650 wait complete. 2023-11-28T12:59:00.782 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:00.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.901+0000 7fe900147700 1 Processor -- start 2023-11-28T12:59:00.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.901+0000 7fe900147700 1 -- start start 2023-11-28T12:59:00.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.902+0000 7fe900147700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe8f80cdd90 0x7fe8f80b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.902+0000 7fe900147700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8f80aecc0 0x7fe8f80ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.902+0000 7fe900147700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f80ad420 con 0x7fe8f8159960 2023-11-28T12:59:00.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.902+0000 7fe900147700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f8159d30 con 0x7fe8f80aecc0 2023-11-28T12:59:00.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.902+0000 7fe900147700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80ac810 con 0x7fe8f80cdd90 2023-11-28T12:59:00.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.902+0000 7fe8fe6e4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8f80aecc0 0x7fe8f80ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.902+0000 7fe8fe6e4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8f80aecc0 0x7fe8f80ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60936/0 (socket says 172.21.15.111:60936) 2023-11-28T12:59:00.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.902+0000 7fe8fe6e4700 1 -- 172.21.15.111:0/2864100921 learned_addr learned my addr 172.21.15.111:0/2864100921 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:00.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.902+0000 7fe8fd6e2700 1 --2- 172.21.15.111:0/2864100921 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe8f80cdd90 0x7fe8f80b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.902+0000 7fe8fe6e4700 1 -- 172.21.15.111:0/2864100921 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe8f80cdd90 msgr2=0x7fe8f80b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:00.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.902+0000 7fe8fe6e4700 1 --2- 172.21.15.111:0/2864100921 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe8f80cdd90 0x7fe8f80b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.902+0000 7fe8fe6e4700 1 -- 172.21.15.111:0/2864100921 >> v1:172.21.15.149:6789/0 conn(0x7fe8f8159960 legacy=0x7fe8f80cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.902+0000 7fe8fe6e4700 1 -- 172.21.15.111:0/2864100921 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f80acaf0 con 0x7fe8f80aecc0 2023-11-28T12:59:00.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.903+0000 7fe8fe6e4700 1 --2- 172.21.15.111:0/2864100921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8f80aecc0 0x7fe8f80ad8b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe8f0009e50 tx=0x7fe8f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b707f3b7fa06641f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:00.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.903+0000 7fe8fcee1700 1 -- 172.21.15.111:0/2864100921 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f000e070 con 0x7fe8f80aecc0 2023-11-28T12:59:00.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.903+0000 7fe8fcee1700 1 -- 172.21.15.111:0/2864100921 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f000b9c0 con 0x7fe8f80aecc0 2023-11-28T12:59:00.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.903+0000 7fe8fcee1700 1 -- 172.21.15.111:0/2864100921 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f000bd70 con 0x7fe8f80aecc0 2023-11-28T12:59:00.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.903+0000 7fe900147700 1 -- 172.21.15.111:0/2864100921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8f80aecc0 msgr2=0x7fe8f80ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.903+0000 7fe900147700 1 --2- 172.21.15.111:0/2864100921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8f80aecc0 0x7fe8f80ad8b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe8f0009e50 tx=0x7fe8f000b330 comp rx=0 tx=0).stop 2023-11-28T12:59:00.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.903+0000 7fe900147700 1 -- 172.21.15.111:0/2864100921 shutdown_connections 2023-11-28T12:59:00.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.903+0000 7fe900147700 1 --2- 172.21.15.111:0/2864100921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8f80aecc0 0x7fe8f80ad8b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.903+0000 7fe900147700 1 --2- 172.21.15.111:0/2864100921 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe8f80cdd90 0x7fe8f80b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.903+0000 7fe900147700 1 -- 172.21.15.111:0/2864100921 >> 172.21.15.111:0/2864100921 conn(0x7fe8f8098ee0 msgr2=0x7fe8f806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:00.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.903+0000 7fe900147700 1 -- 172.21.15.111:0/2864100921 shutdown_connections 2023-11-28T12:59:00.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.903+0000 7fe900147700 1 -- 172.21.15.111:0/2864100921 wait complete. 2023-11-28T12:59:00.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.904+0000 7fe900147700 1 Processor -- start 2023-11-28T12:59:00.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.904+0000 7fe900147700 1 -- start start 2023-11-28T12:59:00.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.905+0000 7fe900147700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8f80aecc0 0x7fe8f8151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.905+0000 7fe900147700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8f80cdd90 0x7fe8f8158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.905+0000 7fe900147700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe8f8159960 0x7fe8f81e62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.905+0000 7fe900147700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80c6360 con 0x7fe8f80cdd90 2023-11-28T12:59:00.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.905+0000 7fe900147700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80c54b0 con 0x7fe8f80aecc0 2023-11-28T12:59:00.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.905+0000 7fe900147700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f8151db0 con 0x7fe8f8159960 2023-11-28T12:59:00.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.905+0000 7fe8fd6e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8f80cdd90 0x7fe8f8158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.905+0000 7fe8fd6e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8f80cdd90 0x7fe8f8158760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60942/0 (socket says 172.21.15.111:60942) 2023-11-28T12:59:00.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.905+0000 7fe8fd6e2700 1 -- 172.21.15.111:0/3602757032 learned_addr learned my addr 172.21.15.111:0/3602757032 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:00.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.905+0000 7fe8fe6e4700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe8f8159960 0x7fe8f81e62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.905+0000 7fe8fdee3700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8f80aecc0 0x7fe8f8151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.906+0000 7fe8fd6e2700 1 -- 172.21.15.111:0/3602757032 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe8f8159960 msgr2=0x7fe8f81e62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.906+0000 7fe8fd6e2700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe8f8159960 0x7fe8f81e62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.906+0000 7fe8fd6e2700 1 -- 172.21.15.111:0/3602757032 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8f80aecc0 msgr2=0x7fe8f8151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:00.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.906+0000 7fe8fd6e2700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8f80aecc0 0x7fe8f8151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:00.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.906+0000 7fe8fd6e2700 1 -- 172.21.15.111:0/3602757032 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f0009b30 con 0x7fe8f80cdd90 2023-11-28T12:59:00.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.906+0000 7fe8fd6e2700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8f80cdd90 0x7fe8f8158760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400e810 tx=0x7fe8f400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:00.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.906+0000 7fe8ee7fc700 1 -- 172.21.15.111:0/3602757032 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f400c070 con 0x7fe8f80cdd90 2023-11-28T12:59:00.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.906+0000 7fe8ee7fc700 1 -- 172.21.15.111:0/3602757032 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f4013610 con 0x7fe8f80cdd90 2023-11-28T12:59:00.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.906+0000 7fe900147700 1 -- 172.21.15.111:0/3602757032 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f80ad0b0 con 0x7fe8f80cdd90 2023-11-28T12:59:00.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.906+0000 7fe8ee7fc700 1 -- 172.21.15.111:0/3602757032 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f40139a0 con 0x7fe8f80cdd90 2023-11-28T12:59:00.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.906+0000 7fe900147700 1 -- 172.21.15.111:0/3602757032 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f80acdd0 con 0x7fe8f80cdd90 2023-11-28T12:59:00.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.907+0000 7fe900147700 1 -- 172.21.15.111:0/3602757032 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f80addf0 con 0x7fe8f80cdd90 2023-11-28T12:59:00.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.907+0000 7fe8ee7fc700 1 -- 172.21.15.111:0/3602757032 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe8f4013b00 con 0x7fe8f80cdd90 2023-11-28T12:59:00.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.907+0000 7fe8ee7fc700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8dc0682f0 0x7fe8dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:00.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.908+0000 7fe8ee7fc700 1 -- 172.21.15.111:0/3602757032 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe8f4085c00 con 0x7fe8f80cdd90 2023-11-28T12:59:00.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.908+0000 7fe8fdee3700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8dc0682f0 0x7fe8dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:00.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.908+0000 7fe8fdee3700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8dc0682f0 0x7fe8dc06a7a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe8f80b5860 tx=0x7fe8e8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:00.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:00.912+0000 7fe8ee7fc700 1 -- 172.21.15.111:0/3602757032 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8f408b020 con 0x7fe8f80cdd90 2023-11-28T12:59:01.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.097+0000 7fe900147700 1 -- 172.21.15.111:0/3602757032 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8f80c6780 con 0x7fe8f80cdd90 2023-11-28T12:59:01.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.098+0000 7fe8ee7fc700 1 -- 172.21.15.111:0/3602757032 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19796 (secure 0 0 0) 0x7fe8f4052170 con 0x7fe8f80cdd90 2023-11-28T12:59:01.103 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:01.104 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:00.053943+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:01.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.100+0000 7fe900147700 1 -- 172.21.15.111:0/3602757032 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8dc0682f0 msgr2=0x7fe8dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.100+0000 7fe900147700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8dc0682f0 0x7fe8dc06a7a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe8f80b5860 tx=0x7fe8e8009040 comp rx=0 tx=0).stop 2023-11-28T12:59:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.100+0000 7fe900147700 1 -- 172.21.15.111:0/3602757032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8f80cdd90 msgr2=0x7fe8f8158760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.100+0000 7fe900147700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8f80cdd90 0x7fe8f8158760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400e810 tx=0x7fe8f400eb20 comp rx=0 tx=0).stop 2023-11-28T12:59:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.100+0000 7fe900147700 1 -- 172.21.15.111:0/3602757032 shutdown_connections 2023-11-28T12:59:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.100+0000 7fe900147700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe8f8159960 0x7fe8f81e62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.100+0000 7fe900147700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8dc0682f0 0x7fe8dc06a7a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.100+0000 7fe900147700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8f80aecc0 0x7fe8f8151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.100+0000 7fe900147700 1 --2- 172.21.15.111:0/3602757032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8f80cdd90 0x7fe8f8158760 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.100+0000 7fe900147700 1 -- 172.21.15.111:0/3602757032 >> 172.21.15.111:0/3602757032 conn(0x7fe8f8098ee0 msgr2=0x7fe8f806dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.100+0000 7fe900147700 1 -- 172.21.15.111:0/3602757032 shutdown_connections 2023-11-28T12:59:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.100+0000 7fe900147700 1 -- 172.21.15.111:0/3602757032 wait complete. 2023-11-28T12:59:01.128 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T12:59:01.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.247+0000 7fdc30ec3700 1 Processor -- start 2023-11-28T12:59:01.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.247+0000 7fdc30ec3700 1 -- start start 2023-11-28T12:59:01.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc30ec3700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fdc2c0af8f0 0x7fdc2c0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:01.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc30ec3700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdc2c0b0940 0x7fdc2c0ca230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:01.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc30ec3700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c0cc1a0 con 0x7fdc2c15cad0 2023-11-28T12:59:01.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc30ec3700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2c0b0270 con 0x7fdc2c0af8f0 2023-11-28T12:59:01.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc30ec3700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0b0dc0 con 0x7fdc2c0b0940 2023-11-28T12:59:01.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc2ad9d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdc2c0b0940 0x7fdc2c0ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:01.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc29d9b700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fdc2c0af8f0 0x7fdc2c0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:01.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc2ad9d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdc2c0b0940 0x7fdc2c0ca230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:54466/0 (socket says 172.21.15.111:54466) 2023-11-28T12:59:01.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc29d9b700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fdc2c0af8f0 0x7fdc2c0b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:50616/0 (socket says 172.21.15.111:50616) 2023-11-28T12:59:01.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc2ad9d700 1 -- 172.21.15.111:0/1056278433 learned_addr learned my addr 172.21.15.111:0/1056278433 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:01.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc2ad9d700 1 -- 172.21.15.111:0/1056278433 >> v2:172.21.15.149:3300/0 conn(0x7fdc2c0af8f0 msgr2=0x7fdc2c0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:01.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc2ad9d700 1 --2- 172.21.15.111:0/1056278433 >> v2:172.21.15.149:3300/0 conn(0x7fdc2c0af8f0 0x7fdc2c0b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc2ad9d700 1 -- 172.21.15.111:0/1056278433 >> v1:172.21.15.149:6789/0 conn(0x7fdc2c15cad0 legacy=0x7fdc2c0ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:01.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc2ad9d700 1 -- 172.21.15.111:0/1056278433 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c0cf400 con 0x7fdc2c0b0940 2023-11-28T12:59:01.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc29d9b700 1 --2- 172.21.15.111:0/1056278433 >> v2:172.21.15.149:3300/0 conn(0x7fdc2c0af8f0 0x7fdc2c0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:01.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.248+0000 7fdc2ad9d700 1 --2- 172.21.15.111:0/1056278433 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdc2c0b0940 0x7fdc2c0ca230 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdc24009e50 tx=0x7fdc2400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dca9953410627183 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:01.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.249+0000 7fdc2959a700 1 -- 172.21.15.111:0/1056278433 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2400e070 con 0x7fdc2c0b0940 2023-11-28T12:59:01.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.249+0000 7fdc2959a700 1 -- 172.21.15.111:0/1056278433 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2400b910 con 0x7fdc2c0b0940 2023-11-28T12:59:01.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.249+0000 7fdc2959a700 1 -- 172.21.15.111:0/1056278433 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2400bcc0 con 0x7fdc2c0b0940 2023-11-28T12:59:01.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.249+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1056278433 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdc2c0b0940 msgr2=0x7fdc2c0ca230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:01.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.249+0000 7fdc30ec3700 1 --2- 172.21.15.111:0/1056278433 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdc2c0b0940 0x7fdc2c0ca230 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdc24009e50 tx=0x7fdc2400b330 comp rx=0 tx=0).stop 2023-11-28T12:59:01.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.249+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1056278433 shutdown_connections 2023-11-28T12:59:01.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.249+0000 7fdc30ec3700 1 --2- 172.21.15.111:0/1056278433 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdc2c0b0940 0x7fdc2c0ca230 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.249+0000 7fdc30ec3700 1 --2- 172.21.15.111:0/1056278433 >> v2:172.21.15.149:3300/0 conn(0x7fdc2c0af8f0 0x7fdc2c0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.249+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1056278433 >> 172.21.15.111:0/1056278433 conn(0x7fdc2c06bca0 msgr2=0x7fdc2c0cfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:01.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.249+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1056278433 shutdown_connections 2023-11-28T12:59:01.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.249+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1056278433 wait complete. 2023-11-28T12:59:01.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.250+0000 7fdc30ec3700 1 Processor -- start 2023-11-28T12:59:01.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.251+0000 7fdc30ec3700 1 -- start start 2023-11-28T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.251+0000 7fdc30ec3700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc2c0af8f0 0x7fdc2c0b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.251+0000 7fdc30ec3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc2c0b0940 0x7fdc2c1e5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.251+0000 7fdc30ec3700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdc2c15cad0 0x7fdc2c0c4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.251+0000 7fdc30ec3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c1e6850 con 0x7fdc2c0b0940 2023-11-28T12:59:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.251+0000 7fdc30ec3700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c1e4040 con 0x7fdc2c0af8f0 2023-11-28T12:59:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.251+0000 7fdc30ec3700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c1e4320 con 0x7fdc2c15cad0 2023-11-28T12:59:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.251+0000 7fdc29d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc2c0b0940 0x7fdc2c1e5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.251+0000 7fdc29d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc2c0b0940 0x7fdc2c1e5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60954/0 (socket says 172.21.15.111:60954) 2023-11-28T12:59:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.251+0000 7fdc29d9b700 1 -- 172.21.15.111:0/1054117386 learned_addr learned my addr 172.21.15.111:0/1054117386 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.251+0000 7fdc2ad9d700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdc2c15cad0 0x7fdc2c0c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.251+0000 7fdc2a59c700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc2c0af8f0 0x7fdc2c0b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:01.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.252+0000 7fdc29d9b700 1 -- 172.21.15.111:0/1054117386 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdc2c15cad0 msgr2=0x7fdc2c0c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:01.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.252+0000 7fdc29d9b700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdc2c15cad0 0x7fdc2c0c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.252+0000 7fdc29d9b700 1 -- 172.21.15.111:0/1054117386 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc2c0af8f0 msgr2=0x7fdc2c0b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:01.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.252+0000 7fdc29d9b700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc2c0af8f0 0x7fdc2c0b5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.252+0000 7fdc29d9b700 1 -- 172.21.15.111:0/1054117386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc200094a0 con 0x7fdc2c0b0940 2023-11-28T12:59:01.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.252+0000 7fdc29d9b700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc2c0b0940 0x7fdc2c1e5ef0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdc2000ebe0 tx=0x7fdc20009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:01.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.252+0000 7fdc1affd700 1 -- 172.21.15.111:0/1054117386 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2000c910 con 0x7fdc2c0b0940 2023-11-28T12:59:01.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.252+0000 7fdc1affd700 1 -- 172.21.15.111:0/1054117386 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2000ca70 con 0x7fdc2c0b0940 2023-11-28T12:59:01.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.252+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1054117386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc24009b30 con 0x7fdc2c0b0940 2023-11-28T12:59:01.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.252+0000 7fdc1affd700 1 -- 172.21.15.111:0/1054117386 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2000cde0 con 0x7fdc2c0b0940 2023-11-28T12:59:01.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.252+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1054117386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c15cea0 con 0x7fdc2c0b0940 2023-11-28T12:59:01.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.253+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1054117386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2c0ce860 con 0x7fdc2c0b0940 2023-11-28T12:59:01.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.253+0000 7fdc1affd700 1 -- 172.21.15.111:0/1054117386 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdc20019030 con 0x7fdc2c0b0940 2023-11-28T12:59:01.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.254+0000 7fdc1affd700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdc0c0682f0 0x7fdc0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:01.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.254+0000 7fdc1affd700 1 -- 172.21.15.111:0/1054117386 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fdc20086010 con 0x7fdc2c0b0940 2023-11-28T12:59:01.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.254+0000 7fdc2a59c700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdc0c0682f0 0x7fdc0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:01.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.254+0000 7fdc2a59c700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdc0c0682f0 0x7fdc0c06a7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c000f60 tx=0x7fdc1c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:01.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.258+0000 7fdc1affd700 1 -- 172.21.15.111:0/1054117386 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdc20052580 con 0x7fdc2c0b0940 2023-11-28T12:59:01.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.485+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1054117386 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc2c0cead0 con 0x7fdc2c0b0940 2023-11-28T12:59:01.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.486+0000 7fdc1affd700 1 -- 172.21.15.111:0/1054117386 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4656 (secure 0 0 0) 0x7fdc20017090 con 0x7fdc2c0b0940 2023-11-28T12:59:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:01.488 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2513950989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2513950989},{"type":"v1","addr":"172.21.15.149:6834","nonce":2513950989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1258891714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1258891714},{"type":"v1","addr":"172.21.15.149:6835","nonce":1258891714}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/3923909288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3923909288},{"type":"v1","addr":"172.21.15.111:6836","nonce":3923909288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T12:58:55.633507+0000","modified":"2023-11-28T12:59:00.054172+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/3180164366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":3180164366},{"type":"v1","addr":"172.21.15.111:6837","nonce":3180164366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-11-28T12:59:01.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.487+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1054117386 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdc0c0682f0 msgr2=0x7fdc0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:01.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.487+0000 7fdc30ec3700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdc0c0682f0 0x7fdc0c06a7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c000f60 tx=0x7fdc1c009040 comp rx=0 tx=0).stop 2023-11-28T12:59:01.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.487+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1054117386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc2c0b0940 msgr2=0x7fdc2c1e5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:01.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.487+0000 7fdc30ec3700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc2c0b0940 0x7fdc2c1e5ef0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdc2000ebe0 tx=0x7fdc20009ec0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.487+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1054117386 shutdown_connections 2023-11-28T12:59:01.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.487+0000 7fdc30ec3700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdc2c15cad0 0x7fdc2c0c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.487+0000 7fdc30ec3700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdc0c0682f0 0x7fdc0c06a7a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.487+0000 7fdc30ec3700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdc2c0af8f0 0x7fdc2c0b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.487+0000 7fdc30ec3700 1 --2- 172.21.15.111:0/1054117386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc2c0b0940 0x7fdc2c1e5ef0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:01.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.487+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1054117386 >> 172.21.15.111:0/1054117386 conn(0x7fdc2c06bca0 msgr2=0x7fdc2c06efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:01.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.488+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1054117386 shutdown_connections 2023-11-28T12:59:01.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:01.488+0000 7fdc30ec3700 1 -- 172.21.15.111:0/1054117386 wait complete. 2023-11-28T12:59:01.491 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 7 2023-11-28T12:59:01.904 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T12:58:55.633507+0000', 'modified': '2023-11-28T12:59:00.054172+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4305}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4305': {'gid': 4305, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6837/3180164366', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6835', 'nonce': 3180164366}, {'type': 'v1', 'addr': '172.21.15.111:6837', 'nonce': 3180164366}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-11-28T12:59:01.904 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T12:59:01.905 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T12:59:02.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.028+0000 7fbbe7e66700 1 Processor -- start 2023-11-28T12:59:02.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.029+0000 7fbbe7e66700 1 -- start start 2023-11-28T12:59:02.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe7e66700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbe01641b0 0x7fbbe00ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe7e66700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fbbe00c3ce0 0x7fbbe00cf390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe7e66700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbbe00cf8d0 0x7fbbe00cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe7e66700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe00cef00 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe7e66700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe00ce8f0 con 0x7fbbe00c3ce0 2023-11-28T12:59:02.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe7e66700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe00c4250 con 0x7fbbe01641b0 2023-11-28T12:59:02.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe6403700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbbe00cf8d0 0x7fbbe00cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe6403700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbbe00cf8d0 0x7fbbe00cfd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60960/0 (socket says 172.21.15.111:60960) 2023-11-28T12:59:02.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe6403700 1 -- 172.21.15.111:0/1577607223 learned_addr learned my addr 172.21.15.111:0/1577607223 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:02.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/1577607223 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbe01641b0 0x7fbbe00ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe5401700 1 --2- 172.21.15.111:0/1577607223 >> v2:172.21.15.149:3300/0 conn(0x7fbbe00c3ce0 0x7fbbe00cf390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe6403700 1 -- 172.21.15.111:0/1577607223 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbe01641b0 msgr2=0x7fbbe00ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:02.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe6403700 1 --2- 172.21.15.111:0/1577607223 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbe01641b0 0x7fbbe00ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe6403700 1 -- 172.21.15.111:0/1577607223 >> v2:172.21.15.149:3300/0 conn(0x7fbbe00c3ce0 msgr2=0x7fbbe00cf390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:02.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe6403700 1 --2- 172.21.15.111:0/1577607223 >> v2:172.21.15.149:3300/0 conn(0x7fbbe00c3ce0 0x7fbbe00cf390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe6403700 1 -- 172.21.15.111:0/1577607223 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe00ceb90 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe6403700 1 --2- 172.21.15.111:0/1577607223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00cf8d0 0x7fbbe00cfd50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00723f0 tx=0x7fbbd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce356210eca06fec server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:02.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.030+0000 7fbbe4c00700 1 -- 172.21.15.111:0/1577607223 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd800e070 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.031+0000 7fbbe4c00700 1 -- 172.21.15.111:0/1577607223 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd800b9c0 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.031+0000 7fbbe4c00700 1 -- 172.21.15.111:0/1577607223 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd800bd70 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.031+0000 7fbbe7e66700 1 -- 172.21.15.111:0/1577607223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00cf8d0 msgr2=0x7fbbe00cfd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.031+0000 7fbbe7e66700 1 --2- 172.21.15.111:0/1577607223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00cf8d0 0x7fbbe00cfd50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00723f0 tx=0x7fbbd800b330 comp rx=0 tx=0).stop 2023-11-28T12:59:02.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.031+0000 7fbbe7e66700 1 -- 172.21.15.111:0/1577607223 shutdown_connections 2023-11-28T12:59:02.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.031+0000 7fbbe7e66700 1 --2- 172.21.15.111:0/1577607223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00cf8d0 0x7fbbe00cfd50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.031+0000 7fbbe7e66700 1 --2- 172.21.15.111:0/1577607223 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbe01641b0 0x7fbbe00ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.031+0000 7fbbe7e66700 1 --2- 172.21.15.111:0/1577607223 >> v2:172.21.15.149:3300/0 conn(0x7fbbe00c3ce0 0x7fbbe00cf390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.031+0000 7fbbe7e66700 1 -- 172.21.15.111:0/1577607223 >> 172.21.15.111:0/1577607223 conn(0x7fbbe009a040 msgr2=0x7fbbe00d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:02.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.031+0000 7fbbe7e66700 1 -- 172.21.15.111:0/1577607223 shutdown_connections 2023-11-28T12:59:02.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.031+0000 7fbbe7e66700 1 -- 172.21.15.111:0/1577607223 wait complete. 2023-11-28T12:59:02.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.033+0000 7fbbe7e66700 1 Processor -- start 2023-11-28T12:59:02.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.033+0000 7fbbe7e66700 1 -- start start 2023-11-28T12:59:02.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.033+0000 7fbbe7e66700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbe00c3ce0 0x7fbbe01f9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.033+0000 7fbbe7e66700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00cf8d0 0x7fbbe0156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.033+0000 7fbbe7e66700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbe01641b0 0x7fbbe006fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.033+0000 7fbbe7e66700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe00b5280 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.033+0000 7fbbe7e66700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe0157300 con 0x7fbbe01641b0 2023-11-28T12:59:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.033+0000 7fbbe7e66700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe01fa130 con 0x7fbbe00c3ce0 2023-11-28T12:59:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.033+0000 7fbbe5401700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00cf8d0 0x7fbbe0156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.033+0000 7fbbe5401700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00cf8d0 0x7fbbe0156a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60966/0 (socket says 172.21.15.111:60966) 2023-11-28T12:59:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.033+0000 7fbbe5401700 1 -- 172.21.15.111:0/974210526 learned_addr learned my addr 172.21.15.111:0/974210526 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.034+0000 7fbbe6403700 1 --2- 172.21.15.111:0/974210526 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbe01641b0 0x7fbbe006fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.034+0000 7fbbe5401700 1 -- 172.21.15.111:0/974210526 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbe00c3ce0 msgr2=0x7fbbe01f9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.034+0000 7fbbe5401700 1 --2- 172.21.15.111:0/974210526 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbe00c3ce0 0x7fbbe01f9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.034+0000 7fbbe5401700 1 -- 172.21.15.111:0/974210526 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbe01641b0 msgr2=0x7fbbe006fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.034+0000 7fbbe5401700 1 --2- 172.21.15.111:0/974210526 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbe01641b0 0x7fbbe006fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.034+0000 7fbbe5401700 1 -- 172.21.15.111:0/974210526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd8009b30 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.034+0000 7fbbe5401700 1 --2- 172.21.15.111:0/974210526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00cf8d0 0x7fbbe0156a70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00d800 tx=0x7fbbdc00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.034+0000 7fbbd67fc700 1 -- 172.21.15.111:0/974210526 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc0125f0 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.034+0000 7fbbd67fc700 1 -- 172.21.15.111:0/974210526 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbdc012750 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.034+0000 7fbbe7e66700 1 -- 172.21.15.111:0/974210526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe00c4530 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.034+0000 7fbbd67fc700 1 -- 172.21.15.111:0/974210526 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc012ae0 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.034+0000 7fbbe7e66700 1 -- 172.21.15.111:0/974210526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe0164580 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.035+0000 7fbbd67fc700 1 -- 172.21.15.111:0/974210526 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbbdc012c40 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.036+0000 7fbbd67fc700 1 --2- 172.21.15.111:0/974210526 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbbc40682f0 0x7fbbc406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.036+0000 7fbbd67fc700 1 -- 172.21.15.111:0/974210526 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fbbdc086450 con 0x7fbbe00cf8d0 2023-11-28T12:59:02.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.036+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/974210526 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbbc40682f0 0x7fbbc406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.036+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/974210526 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbbc40682f0 0x7fbbc406a7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0006eb0 tx=0x7fbbd0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:02.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.038+0000 7fbbf3fefb80 1 Processor -- start 2023-11-28T12:59:02.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.038+0000 7fbbf3fefb80 1 -- start start 2023-11-28T12:59:02.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbf3fefb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54e810 0x5573bc524a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbf3fefb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54c130 0x5573bc54ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbf3fefb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54f120 0x5573bc56a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbf3fefb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bc40ebc0 con 0x5573bc54e810 2023-11-28T12:59:02.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbf3fefb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bc47d350 con 0x5573bc54f120 2023-11-28T12:59:02.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbf3fefb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5573bc476880 con 0x5573bc54c130 2023-11-28T12:59:02.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbe5c02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54e810 0x5573bc524a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbe5c02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54e810 0x5573bc524a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60972/0 (socket says 172.21.15.111:60972) 2023-11-28T12:59:02.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbe5c02700 1 -- 172.21.15.111:0/657178192 learned_addr learned my addr 172.21.15.111:0/657178192 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:02.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbe5c02700 1 -- 172.21.15.111:0/657178192 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54c130 msgr2=0x5573bc54ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:02.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/657178192 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54c130 0x5573bc54ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbe5c02700 1 -- 172.21.15.111:0/657178192 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54f120 msgr2=0x5573bc56a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:02.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/657178192 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54f120 0x5573bc56a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbe5c02700 1 -- 172.21.15.111:0/657178192 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573bc2f85e0 con 0x5573bc54e810 2023-11-28T12:59:02.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/657178192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54e810 0x5573bc524a20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbbd00158a0 tx=0x7fbbd0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8813f9540e5e6568 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:02.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbc37fe700 1 -- 172.21.15.111:0/657178192 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd00136f0 con 0x5573bc54e810 2023-11-28T12:59:02.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbc37fe700 1 -- 172.21.15.111:0/657178192 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd0013850 con 0x5573bc54e810 2023-11-28T12:59:02.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.039+0000 7fbbc37fe700 1 -- 172.21.15.111:0/657178192 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0013c00 con 0x5573bc54e810 2023-11-28T12:59:02.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.040+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/657178192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54e810 msgr2=0x5573bc524a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.040+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/657178192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54e810 0x5573bc524a20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbbd00158a0 tx=0x7fbbd0011000 comp rx=0 tx=0).stop 2023-11-28T12:59:02.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.040+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/657178192 shutdown_connections 2023-11-28T12:59:02.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.040+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/657178192 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54f120 0x5573bc56a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.040+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/657178192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54e810 0x5573bc524a20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.040+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/657178192 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54c130 0x5573bc54ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.040+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/657178192 >> 172.21.15.111:0/657178192 conn(0x5573bc521d60 msgr2=0x5573bc5226f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:02.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.040+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/657178192 shutdown_connections 2023-11-28T12:59:02.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.040+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/657178192 wait complete. 2023-11-28T12:59:02.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.041+0000 7fbbf3fefb80 1 -- start start 2023-11-28T12:59:02.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.041+0000 7fbbf3fefb80 1 Processor -- start 2023-11-28T12:59:02.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbf3fefb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54c130 0x5573bc5409d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbf3fefb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54e810 0x5573bc52b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbf3fefb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 0x5573bc54d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbf3fefb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bc2fa8d0 con 0x5573bc54f120 2023-11-28T12:59:02.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbf3fefb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bc52c460 con 0x5573bc54c130 2023-11-28T12:59:02.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbf3fefb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5573bc54c5b0 con 0x5573bc54e810 2023-11-28T12:59:02.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbf3fefb80 10 client.0 _ll_register_callbacks cb 0x5573bc31c240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T12:59:02.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbe6403700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 0x5573bc54d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbe6403700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 0x5573bc54d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60982/0 (socket says 172.21.15.111:60982) 2023-11-28T12:59:02.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbe6403700 1 -- 172.21.15.111:0/4138705436 learned_addr learned my addr 172.21.15.111:0/4138705436 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:02.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbe5401700 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54e810 0x5573bc52b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54c130 0x5573bc5409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbe6403700 1 -- 172.21.15.111:0/4138705436 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54e810 msgr2=0x5573bc52b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:02.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbe6403700 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54e810 0x5573bc52b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbe6403700 1 -- 172.21.15.111:0/4138705436 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54c130 msgr2=0x5573bc5409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:02.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbe6403700 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54c130 0x5573bc5409d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbe6403700 1 -- 172.21.15.111:0/4138705436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd00155b0 con 0x5573bc54f120 2023-11-28T12:59:02.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.042+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54c130 0x5573bc5409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:02.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbe6403700 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 0x5573bc54d8a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800b730 tx=0x7fbbd8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e615935f908060 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:02.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbc0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T12:59:02.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbc0ff9700 1 -- 172.21.15.111:0/4138705436 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8004450 con 0x5573bc54f120 2023-11-28T12:59:02.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbc0ff9700 1 -- 172.21.15.111:0/4138705436 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fbbd80045b0 con 0x5573bc54f120 2023-11-28T12:59:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/4138705436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5573bc444140 con 0x5573bc54f120 2023-11-28T12:59:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbc0ff9700 1 -- 172.21.15.111:0/4138705436 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbd8025ea0 con 0x5573bc54f120 2023-11-28T12:59:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbf3fefb80 10 client.4446 fetch_fsmap learned FSMap version 7 2023-11-28T12:59:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/4138705436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5573bc2a4d90 con 0x5573bc54f120 2023-11-28T12:59:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbc0ff9700 1 -- 172.21.15.111:0/4138705436 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5573bc444140 con 0x5573bc54f120 2023-11-28T12:59:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbc0ff9700 1 -- 172.21.15.111:0/4138705436 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 7) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fbbd8019050 con 0x5573bc54f120 2023-11-28T12:59:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbf3fefb80 10 client.4446 fetch_fsmap finished waiting for FSMap version 7 2023-11-28T12:59:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbf3fefb80 10 client.4446 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T12:59:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbf3fefb80 20 client.4446 populate_metadata read hostname 'smithi111' 2023-11-28T12:59:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x5573bc560940 0x5573bc52d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbf3fefb80 4 client.4446 mds_command: new command op to 4305 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-28T12:59:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/4138705436 --> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5573bc2faf10 con 0x5573bc560940 2023-11-28T12:59:02.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.043+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x5573bc560940 0x5573bc52d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbc0ff9700 10 client.4446 ms_handle_connect on v2:172.21.15.111:6835/3180164366 2023-11-28T12:59:02.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x5573bc560940 0x5573bc52d9c0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8d3e4785b2c7f1c server_cookie=76aa6a5b81c2c4 in_seq=0 out_seq=0 2023-11-28T12:59:02.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbc0ff9700 1 -- 172.21.15.111:0/4138705436 <== mds.0 v2:172.21.15.111:6835/3180164366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5573bc2faf10 con 0x5573bc560940 2023-11-28T12:59:02.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbc0ff9700 10 client.4446 handle_command_reply: tid=0 2023-11-28T12:59:02.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbf3fefb80 1 client.4446 shutdown 2023-11-28T12:59:02.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/4138705436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 msgr2=0x5573bc54d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 0x5573bc54d8a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800b730 tx=0x7fbbd8004ad0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/4138705436 shutdown_connections 2023-11-28T12:59:02.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 0x5573bc54d8a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54c130 0x5573bc5409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x5573bc560940 0x5573bc52d9c0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/4138705436 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54e810 0x5573bc52b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbc0ff9700 0 client.4446 ms_handle_reset on v2:172.21.15.111:6835/3180164366 2023-11-28T12:59:02.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/4138705436 >> 172.21.15.111:0/4138705436 conn(0x5573bc521d60 msgr2=0x5573bc524a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:02.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/4138705436 shutdown_connections 2023-11-28T12:59:02.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.044+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/4138705436 wait complete. 2023-11-28T12:59:02.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.045+0000 7fbbf3fefb80 20 client.4446 trim_cache size 0 max 16384 2023-11-28T12:59:02.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.062+0000 7fbbf3fefb80 1 Processor -- start 2023-11-28T12:59:02.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.062+0000 7fbbf3fefb80 1 -- start start 2023-11-28T12:59:02.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.062+0000 7fbbf3fefb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54c130 0x5573bc53fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.062+0000 7fbbf3fefb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54e810 0x5573bc57b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.062+0000 7fbbf3fefb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54f120 0x5573bc57b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.062+0000 7fbbf3fefb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bc54bbd0 con 0x5573bc54c130 2023-11-28T12:59:02.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.062+0000 7fbbf3fefb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bc54fb90 con 0x5573bc54f120 2023-11-28T12:59:02.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.062+0000 7fbbf3fefb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5573bc561db0 con 0x5573bc54e810 2023-11-28T12:59:02.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.062+0000 7fbbe5c02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54c130 0x5573bc53fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.062+0000 7fbbe5c02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54c130 0x5573bc53fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60986/0 (socket says 172.21.15.111:60986) 2023-11-28T12:59:02.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.062+0000 7fbbe5c02700 1 -- 172.21.15.111:0/2421195318 learned_addr learned my addr 172.21.15.111:0/2421195318 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:02.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.063+0000 7fbbe5c02700 1 -- 172.21.15.111:0/2421195318 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54e810 msgr2=0x5573bc57b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:02.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.063+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/2421195318 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54e810 0x5573bc57b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.063+0000 7fbbe5c02700 1 -- 172.21.15.111:0/2421195318 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54f120 msgr2=0x5573bc57b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:02.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.063+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/2421195318 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54f120 0x5573bc57b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.063+0000 7fbbe5c02700 1 -- 172.21.15.111:0/2421195318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573bc43d140 con 0x5573bc54c130 2023-11-28T12:59:02.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.063+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/2421195318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54c130 0x5573bc53fa60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0023040 tx=0x7fbbd001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377fc6166d4d823b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:02.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.063+0000 7fbbc3fff700 1 -- 172.21.15.111:0/2421195318 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0030450 con 0x5573bc54c130 2023-11-28T12:59:02.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.063+0000 7fbbc3fff700 1 -- 172.21.15.111:0/2421195318 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd0005a20 con 0x5573bc54c130 2023-11-28T12:59:02.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.063+0000 7fbbc3fff700 1 -- 172.21.15.111:0/2421195318 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0005db0 con 0x5573bc54c130 2023-11-28T12:59:02.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.063+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/2421195318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54c130 msgr2=0x5573bc53fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.063+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/2421195318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54c130 0x5573bc53fa60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0023040 tx=0x7fbbd001d000 comp rx=0 tx=0).stop 2023-11-28T12:59:02.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.064+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/2421195318 shutdown_connections 2023-11-28T12:59:02.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.064+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/2421195318 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54f120 0x5573bc57b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.064+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/2421195318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54c130 0x5573bc53fa60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.064+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/2421195318 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54e810 0x5573bc57b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.064+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/2421195318 >> 172.21.15.111:0/2421195318 conn(0x5573bc521d60 msgr2=0x5573bc522380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:02.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.064+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/2421195318 shutdown_connections 2023-11-28T12:59:02.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.064+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/2421195318 wait complete. 2023-11-28T12:59:02.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.064+0000 7fbbf3fefb80 1 -- start start 2023-11-28T12:59:02.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.064+0000 7fbbf3fefb80 1 Processor -- start 2023-11-28T12:59:02.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbf3fefb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54c130 0x5573bc55fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbf3fefb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54e810 0x5573bc5ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbf3fefb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 0x5573bc523510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbf3fefb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bc526030 con 0x5573bc54f120 2023-11-28T12:59:02.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbf3fefb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bc583340 con 0x5573bc54e810 2023-11-28T12:59:02.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbf3fefb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5573bc523e10 con 0x5573bc54c130 2023-11-28T12:59:02.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbf3fefb80 10 client.0 _ll_register_callbacks cb 0x5573bc31c240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T12:59:02.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbe6403700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 0x5573bc523510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbe6403700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 0x5573bc523510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60996/0 (socket says 172.21.15.111:60996) 2023-11-28T12:59:02.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbe6403700 1 -- 172.21.15.111:0/664975526 learned_addr learned my addr 172.21.15.111:0/664975526 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:02.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbe5401700 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54e810 0x5573bc5ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54c130 0x5573bc55fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbe6403700 1 -- 172.21.15.111:0/664975526 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54c130 msgr2=0x5573bc55fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:02.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbe6403700 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54c130 0x5573bc55fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbe6403700 1 -- 172.21.15.111:0/664975526 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54e810 msgr2=0x5573bc5ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:02.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbe6403700 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54e810 0x5573bc5ce3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbe6403700 1 -- 172.21.15.111:0/664975526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573bc2f85e0 con 0x5573bc54f120 2023-11-28T12:59:02.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54c130 0x5573bc55fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:02.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbe6403700 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 0x5573bc523510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8002230 tx=0x7fbbd8009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547a64c88cf518b5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:02.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.065+0000 7fbbc0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T12:59:02.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbc0ff9700 1 -- 172.21.15.111:0/664975526 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd800e040 con 0x5573bc54f120 2023-11-28T12:59:02.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbc0ff9700 1 -- 172.21.15.111:0/664975526 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fbbd800b7a0 con 0x5573bc54f120 2023-11-28T12:59:02.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/664975526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5573bc434160 con 0x5573bc54f120 2023-11-28T12:59:02.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbc0ff9700 1 -- 172.21.15.111:0/664975526 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbd8003db0 con 0x5573bc54f120 2023-11-28T12:59:02.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbf3fefb80 10 client.4452 fetch_fsmap learned FSMap version 7 2023-11-28T12:59:02.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/664975526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5573bc2cb200 con 0x5573bc54f120 2023-11-28T12:59:02.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbc0ff9700 1 -- 172.21.15.111:0/664975526 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5573bc434160 con 0x5573bc54f120 2023-11-28T12:59:02.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbc0ff9700 1 -- 172.21.15.111:0/664975526 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 7) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fbbd8019070 con 0x5573bc54f120 2023-11-28T12:59:02.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbf3fefb80 10 client.4452 fetch_fsmap finished waiting for FSMap version 7 2023-11-28T12:59:02.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbf3fefb80 10 client.4452 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T12:59:02.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbf3fefb80 20 client.4452 populate_metadata read hostname 'smithi111' 2023-11-28T12:59:02.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x5573bc582830 0x5573bc57b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbf3fefb80 4 client.4452 mds_command: new command op to 4305 tid=0[{"prefix": "status"}] 2023-11-28T12:59:02.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/664975526 --> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5573bc41e170 con 0x5573bc582830 2023-11-28T12:59:02.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x5573bc582830 0x5573bc57b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.066+0000 7fbbc0ff9700 10 client.4452 ms_handle_connect on v2:172.21.15.111:6835/3180164366 2023-11-28T12:59:02.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbe5c02700 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x5573bc582830 0x5573bc57b270 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3922ac1ac2184eb6 server_cookie=e1912c56fdab0fa9 in_seq=0 out_seq=0 2023-11-28T12:59:02.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbc0ff9700 1 -- 172.21.15.111:0/664975526 <== mds.0 v2:172.21.15.111:6835/3180164366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x5573bc41e170 con 0x5573bc582830 2023-11-28T12:59:02.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbc0ff9700 10 client.4452 handle_command_reply: tid=0 2023-11-28T12:59:02.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbf3fefb80 1 client.4452 shutdown 2023-11-28T12:59:02.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/664975526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 msgr2=0x5573bc523510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 0x5573bc523510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8002230 tx=0x7fbbd8009810 comp rx=0 tx=0).stop 2023-11-28T12:59:02.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/664975526 shutdown_connections 2023-11-28T12:59:02.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5573bc54f120 0x5573bc523510 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x5573bc582830 0x5573bc57b270 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5573bc54c130 0x5573bc55fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbf3fefb80 1 --2- 172.21.15.111:0/664975526 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5573bc54e810 0x5573bc5ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbc0ff9700 0 client.4452 ms_handle_reset on v2:172.21.15.111:6835/3180164366 2023-11-28T12:59:02.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/664975526 >> 172.21.15.111:0/664975526 conn(0x5573bc521d60 msgr2=0x5573bc524540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:02.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/664975526 shutdown_connections 2023-11-28T12:59:02.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbf3fefb80 1 -- 172.21.15.111:0/664975526 wait complete. 2023-11-28T12:59:02.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbf3fefb80 20 client.4452 trim_cache size 0 max 16384 2023-11-28T12:59:02.077 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T12:59:02.077 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "a1b2d00c-aeec-4b5d-93e3-41820c4189ba", 2023-11-28T12:59:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-11-28T12:59:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 4305, 2023-11-28T12:59:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-11-28T12:59:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-11-28T12:59:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-11-28T12:59:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 6.4330852610000004, 2023-11-28T12:59:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 7, 2023-11-28T12:59:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 26, 2023-11-28T12:59:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 25, 2023-11-28T12:59:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 11.759042494999999 2023-11-28T12:59:02.078 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T12:59:02.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbe7e66700 1 -- 172.21.15.111:0/974210526 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbbc40682f0 msgr2=0x7fbbc406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbe7e66700 1 --2- 172.21.15.111:0/974210526 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbbc40682f0 0x7fbbc406a7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0006eb0 tx=0x7fbbd0008040 comp rx=0 tx=0).stop 2023-11-28T12:59:02.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.067+0000 7fbbe7e66700 1 -- 172.21.15.111:0/974210526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00cf8d0 msgr2=0x7fbbe0156a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.068+0000 7fbbe7e66700 1 --2- 172.21.15.111:0/974210526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00cf8d0 0x7fbbe0156a70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00d800 tx=0x7fbbdc00dbc0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.068+0000 7fbbe7e66700 1 -- 172.21.15.111:0/974210526 shutdown_connections 2023-11-28T12:59:02.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.068+0000 7fbbe7e66700 1 --2- 172.21.15.111:0/974210526 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbbc40682f0 0x7fbbc406a7a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.068+0000 7fbbe7e66700 1 --2- 172.21.15.111:0/974210526 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbe01641b0 0x7fbbe006fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.068+0000 7fbbe7e66700 1 --2- 172.21.15.111:0/974210526 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbe00c3ce0 0x7fbbe01f9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.068+0000 7fbbe7e66700 1 --2- 172.21.15.111:0/974210526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00cf8d0 0x7fbbe0156a70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.068+0000 7fbbe7e66700 1 -- 172.21.15.111:0/974210526 >> 172.21.15.111:0/974210526 conn(0x7fbbe009a040 msgr2=0x7fbbe006c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:02.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.068+0000 7fbbe7e66700 1 -- 172.21.15.111:0/974210526 shutdown_connections 2023-11-28T12:59:02.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.068+0000 7fbbe7e66700 1 -- 172.21.15.111:0/974210526 wait complete. 2023-11-28T12:59:02.081 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T12:59:02.082 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-11-28T12:59:02.082 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T12:59:02.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.200+0000 7f42331d5700 1 Processor -- start 2023-11-28T12:59:02.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.200+0000 7f42331d5700 1 -- start start 2023-11-28T12:59:02.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.200+0000 7f42331d5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f422c1604b0 0x7f422c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.200+0000 7f42331d5700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f422c0cc550 0x7f422c0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.200+0000 7f42331d5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422c15cd90 con 0x7f422c0b0d70 2023-11-28T12:59:02.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.200+0000 7f42331d5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c160950 con 0x7f422c1604b0 2023-11-28T12:59:02.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.200+0000 7f42331d5700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c0cced0 con 0x7f422c0cc550 2023-11-28T12:59:02.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f422bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f422c1604b0 0x7f422c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f4230f71700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f422c0b0d70 0x7f422c0b1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33546/0 (socket says 172.21.15.111:33546) 2023-11-28T12:59:02.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f422bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f422c1604b0 0x7f422c0ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32770/0 (socket says 172.21.15.111:32770) 2023-11-28T12:59:02.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f4230f71700 1 -- 172.21.15.111:0/2099317183 learned_addr learned my addr 172.21.15.111:0/2099317183 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:02.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f4231772700 1 --2- 172.21.15.111:0/2099317183 >> v2:172.21.15.149:3300/0 conn(0x7f422c0cc550 0x7f422c0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f422bfff700 1 -- 172.21.15.111:0/2099317183 >> v2:172.21.15.149:3300/0 conn(0x7f422c0cc550 msgr2=0x7f422c0cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:02.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f422bfff700 1 --2- 172.21.15.111:0/2099317183 >> v2:172.21.15.149:3300/0 conn(0x7f422c0cc550 0x7f422c0cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f422bfff700 1 -- 172.21.15.111:0/2099317183 >> v1:172.21.15.111:6789/0 conn(0x7f422c0b0d70 legacy=0x7f422c0b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f422bfff700 1 -- 172.21.15.111:0/2099317183 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c1613e0 con 0x7f422c1604b0 2023-11-28T12:59:02.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f422b7fe700 1 -- 172.21.15.111:0/2099317183 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994515317 0 0) 0x7f4220001290 con 0x7f422c0b0d70 2023-11-28T12:59:02.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f4231772700 1 --2- 172.21.15.111:0/2099317183 >> v2:172.21.15.149:3300/0 conn(0x7f422c0cc550 0x7f422c0cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:02.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f422bfff700 1 --2- 172.21.15.111:0/2099317183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f422c1604b0 0x7f422c0ce6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f422c070310 tx=0x7f4218009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f4f7d19f9ff3bc1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:02.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f422b7fe700 1 -- 172.21.15.111:0/2099317183 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4218011830 con 0x7f422c1604b0 2023-11-28T12:59:02.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f422b7fe700 1 -- 172.21.15.111:0/2099317183 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4218011990 con 0x7f422c1604b0 2023-11-28T12:59:02.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.201+0000 7f422b7fe700 1 -- 172.21.15.111:0/2099317183 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4218011d20 con 0x7f422c1604b0 2023-11-28T12:59:02.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.202+0000 7f42331d5700 1 -- 172.21.15.111:0/2099317183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f422c1604b0 msgr2=0x7f422c0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.202+0000 7f42331d5700 1 --2- 172.21.15.111:0/2099317183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f422c1604b0 0x7f422c0ce6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f422c070310 tx=0x7f4218009b20 comp rx=0 tx=0).stop 2023-11-28T12:59:02.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.202+0000 7f42331d5700 1 -- 172.21.15.111:0/2099317183 shutdown_connections 2023-11-28T12:59:02.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.202+0000 7f42331d5700 1 --2- 172.21.15.111:0/2099317183 >> v2:172.21.15.149:3300/0 conn(0x7f422c0cc550 0x7f422c0cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.202+0000 7f42331d5700 1 --2- 172.21.15.111:0/2099317183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f422c1604b0 0x7f422c0ce6c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.202+0000 7f42331d5700 1 -- 172.21.15.111:0/2099317183 >> 172.21.15.111:0/2099317183 conn(0x7f422c098ee0 msgr2=0x7f422c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:02.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.202+0000 7f42331d5700 1 -- 172.21.15.111:0/2099317183 shutdown_connections 2023-11-28T12:59:02.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.202+0000 7f42331d5700 1 -- 172.21.15.111:0/2099317183 wait complete. 2023-11-28T12:59:02.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.203+0000 7f42331d5700 1 Processor -- start 2023-11-28T12:59:02.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.203+0000 7f42331d5700 1 -- start start 2023-11-28T12:59:02.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f42331d5700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f422c0b0d70 0x7f422c0c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f42331d5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f422c0cc550 0x7f422c0b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f42331d5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f422c1604b0 0x7f422c0c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f42331d5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c15a290 con 0x7f422c0cc550 2023-11-28T12:59:02.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f42331d5700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c0c7800 con 0x7f422c0b0d70 2023-11-28T12:59:02.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f42331d5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c0c7ae0 con 0x7f422c1604b0 2023-11-28T12:59:02.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f422bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f422c0cc550 0x7f422c0b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f422bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f422c0cc550 0x7f422c0b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32778/0 (socket says 172.21.15.111:32778) 2023-11-28T12:59:02.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f422bfff700 1 -- 172.21.15.111:0/1968496474 learned_addr learned my addr 172.21.15.111:0/1968496474 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:02.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f4230f71700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f422c0b0d70 0x7f422c0c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f4231772700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f422c1604b0 0x7f422c0c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f422bfff700 1 -- 172.21.15.111:0/1968496474 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f422c1604b0 msgr2=0x7f422c0c35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f422bfff700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f422c1604b0 0x7f422c0c35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f422bfff700 1 -- 172.21.15.111:0/1968496474 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f422c0b0d70 msgr2=0x7f422c0c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f422bfff700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f422c0b0d70 0x7f422c0c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f422bfff700 1 -- 172.21.15.111:0/1968496474 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42180094a0 con 0x7f422c0cc550 2023-11-28T12:59:02.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f4230f71700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f422c0b0d70 0x7f422c0c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:02.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f422bfff700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f422c0cc550 0x7f422c0b51a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f42180090b0 tx=0x7f4218004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:02.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.204+0000 7f42297fa700 1 -- 172.21.15.111:0/1968496474 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42180038a0 con 0x7f422c0cc550 2023-11-28T12:59:02.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.205+0000 7f42297fa700 1 -- 172.21.15.111:0/1968496474 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4218003a00 con 0x7f422c0cc550 2023-11-28T12:59:02.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.205+0000 7f42331d5700 1 -- 172.21.15.111:0/1968496474 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c160e20 con 0x7f422c0cc550 2023-11-28T12:59:02.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.205+0000 7f42297fa700 1 -- 172.21.15.111:0/1968496474 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4218003d90 con 0x7f422c0cc550 2023-11-28T12:59:02.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.205+0000 7f42331d5700 1 -- 172.21.15.111:0/1968496474 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c160b40 con 0x7f422c0cc550 2023-11-28T12:59:02.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.205+0000 7f42331d5700 1 -- 172.21.15.111:0/1968496474 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422c161100 con 0x7f422c0cc550 2023-11-28T12:59:02.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.206+0000 7f42297fa700 1 -- 172.21.15.111:0/1968496474 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f421800f050 con 0x7f422c0cc550 2023-11-28T12:59:02.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.206+0000 7f42297fa700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42100682f0 0x7f421006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:02.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.206+0000 7f42297fa700 1 -- 172.21.15.111:0/1968496474 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f42180873f0 con 0x7f422c0cc550 2023-11-28T12:59:02.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.206+0000 7f4230f71700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42100682f0 0x7f421006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:02.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.207+0000 7f4230f71700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42100682f0 0x7f421006a7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f422c15c700 tx=0x7f4220009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:02.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.210+0000 7f42297fa700 1 -- 172.21.15.111:0/1968496474 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4218053960 con 0x7f422c0cc550 2023-11-28T12:59:02.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.437+0000 7f42331d5700 1 -- 172.21.15.111:0/1968496474 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f422c067170 con 0x7f422c0cc550 2023-11-28T12:59:02.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.437+0000 7f42297fa700 1 -- 172.21.15.111:0/1968496474 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f4218015070 con 0x7f422c0cc550 2023-11-28T12:59:02.439 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-28T12:59:02.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.440+0000 7f42331d5700 1 -- 172.21.15.111:0/1968496474 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42100682f0 msgr2=0x7f421006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.440+0000 7f42331d5700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42100682f0 0x7f421006a7a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f422c15c700 tx=0x7f4220009040 comp rx=0 tx=0).stop 2023-11-28T12:59:02.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.440+0000 7f42331d5700 1 -- 172.21.15.111:0/1968496474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f422c0cc550 msgr2=0x7f422c0b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:02.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.440+0000 7f42331d5700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f422c0cc550 0x7f422c0b51a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f42180090b0 tx=0x7f4218004750 comp rx=0 tx=0).stop 2023-11-28T12:59:02.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.440+0000 7f42331d5700 1 -- 172.21.15.111:0/1968496474 shutdown_connections 2023-11-28T12:59:02.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.440+0000 7f42331d5700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f422c1604b0 0x7f422c0c35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.440+0000 7f42331d5700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42100682f0 0x7f421006a7a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.440+0000 7f42331d5700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f422c0b0d70 0x7f422c0c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.440+0000 7f42331d5700 1 --2- 172.21.15.111:0/1968496474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f422c0cc550 0x7f422c0b51a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:02.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.440+0000 7f42331d5700 1 -- 172.21.15.111:0/1968496474 >> 172.21.15.111:0/1968496474 conn(0x7f422c098ee0 msgr2=0x7f422c06e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:02.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.440+0000 7f42331d5700 1 -- 172.21.15.111:0/1968496474 shutdown_connections 2023-11-28T12:59:02.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:02.440+0000 7f42331d5700 1 -- 172.21.15.111:0/1968496474 wait complete. 2023-11-28T12:59:02.453 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T12:59:02.453 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T12:59:02.453 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-11-28T12:59:02.453 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-11-28T12:59:02.453 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-11-28T12:59:02.453 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T12:59:02.453 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T12:59:02.454 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T12:59:02.454 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-11-28T12:59:02.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:02.454 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-11-28T12:59:02.471 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T12:59:02.471 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T12:59:02.471 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T12:59:02.471 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:02.471 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-11-28T12:59:02.472 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:02.472 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:02.472 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:02.472 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:02.472 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:02.472 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:02.472 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:02.472 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T12:59:02.472 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:02.472 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T12:59:02.472 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 42384sec preferred_lft 42384sec 2023-11-28T12:59:02.473 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::9195:85a0:8e3c:2da4/64 scope link noprefixroute 2023-11-28T12:59:02.473 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:02.473 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-28T12:59:02.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:02.473 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:02.474 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:02.474 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add name ceph-brx type bridge 2023-11-28T12:59:02.474 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr flush dev ceph-brx 2023-11-28T12:59:02.474 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set ceph-brx up 2023-11-28T12:59:02.474 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-28T12:59:02.474 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:02.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:02.639 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T12:59:02.668 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-11-28T12:59:02.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:02.670 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-11-28T12:59:02.743 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-11-28T12:59:02.743 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T12:59:02.743 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T12:59:02.743 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T12:59:02.743 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T12:59:02.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:02.744 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:02.744 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:02.744 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T12:59:02.744 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T12:59:02.744 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T12:59:02.744 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:02.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:02.992 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-11-28T12:59:03.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:03.009 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list-id 2023-11-28T12:59:03.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:03.065 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:03.065 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:03.066 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-28T12:59:03.066 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-11-28T12:59:03.066 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:03.160 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-11-28T12:59:03.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:03.160 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:03.160 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:03.160 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-11-28T12:59:03.160 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-28T12:59:03.160 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-11-28T12:59:03.160 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-11-28T12:59:03.161 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-11-28T12:59:03.161 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:03.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:03.361 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:03.361 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:03.361 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-11-28T12:59:03.362 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set dev brx.0 master ceph-brx 2023-11-28T12:59:03.362 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:03.463 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-11-28T12:59:03.463 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T12:59:03.463 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-11-28T12:59:03.484 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-11-28T12:59:03.484 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T12:59:03.484 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-11-28T12:59:03.541 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-11-28T12:59:03.608 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-28T12:59:03.608 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-28T12:59:03.675 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-28T12:59:03.676 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-28T12:59:03.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:03.677 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-11-28T12:59:03.735 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-28T12:59:03.735 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_mountpoint=/ --client_fs=cephfs) 2023-11-28T12:59:03.779 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-11-28T12:59:03.805 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-28T12:59:03.805 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-28T12:59:03.871 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-28T12:59:03.872 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-28T12:59:03.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:03.872 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-11-28T12:59:03.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi111.stderr:2023-11-28T12:59:03.879+0000 7f83c671e5c0 -1 init, newargv = 0x55e5a1027ef0 newargc=15 2023-11-28T12:59:03.881 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi111.stderr:ceph-fuse[111241]: starting ceph client 2023-11-28T12:59:03.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi111.stderr:ceph-fuse[111241]: starting fuse 2023-11-28T12:59:03.897 INFO:teuthology.orchestra.run.smithi111.stdout:47 2023-11-28T12:59:03.897 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-11-28T12:59:03.897 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-11-28T12:59:03.897 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-11-28T12:59:03.897 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-11-28T12:59:03.897 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-11-28T12:59:03.897 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-11-28T12:59:03.897 DEBUG:teuthology.orchestra.run.smithi111:> 2023-11-28T12:59:03.898 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-11-28T12:59:03.898 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-11-28T12:59:03.898 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-11-28T12:59:03.898 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-11-28T12:59:03.898 DEBUG:teuthology.orchestra.run.smithi111:> 2023-11-28T12:59:03.898 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-11-28T12:59:03.898 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-11-28T12:59:03.898 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-11-28T12:59:03.898 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-11-28T12:59:03.898 DEBUG:teuthology.orchestra.run.smithi111:> 2023-11-28T12:59:03.898 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-11-28T12:59:03.898 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-28T12:59:03.899 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-28T12:59:03.899 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-28T12:59:03.899 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-11-28T12:59:03.899 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-11-28T12:59:03.899 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-11-28T12:59:03.899 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-28T12:59:03.899 DEBUG:teuthology.orchestra.run.smithi111:> 2023-11-28T12:59:03.899 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.admin")) 2023-11-28T12:59:03.899 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-11-28T12:59:04.020 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.admin.111241.asok 2023-11-28T12:59:04.202 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.111241.asok 2023-11-28T12:59:04.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:04.202 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.111241.asok status 2023-11-28T12:59:04.358 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T12:59:04.358 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-11-28T12:59:04.358 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "2f5f6028b893ad41357cd06e827373f0fe2a1232", 2023-11-28T12:59:04.358 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 18.0.0-2697-g2f5f6028 (2f5f6028b893ad41357cd06e827373f0fe2a1232) reef (dev)", 2023-11-28T12:59:04.358 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "admin", 2023-11-28T12:59:04.358 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-11-28T12:59:04.358 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-11-28T12:59:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "111241", 2023-11-28T12:59:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-11-28T12:59:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T12:59:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-11-28T12:59:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-11-28T12:59:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 4470, 2023-11-28T12:59:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-11-28T12:59:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-11-28T12:59:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-11-28T12:59:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 4470 2023-11-28T12:59:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T12:59:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-11-28T12:59:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-11-28T12:59:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-11-28T12:59:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1518909820 2023-11-28T12:59:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T12:59:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T12:59:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-11-28T12:59:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-11-28T12:59:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-11-28T12:59:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1518909820 2023-11-28T12:59:04.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T12:59:04.361 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.4470 192.168.0.1:0/1518909820", 2023-11-28T12:59:04.361 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/1518909820", 2023-11-28T12:59:04.361 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-11-28T12:59:04.361 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 7, 2023-11-28T12:59:04.361 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 26, 2023-11-28T12:59:04.361 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-11-28T12:59:04.361 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-11-28T12:59:04.361 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-11-28T12:59:04.361 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T12:59:04.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:04.368 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-11-28T12:59:04.368 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-11-28T12:59:04.385 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-11-28T12:59:04.385 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-11-28T12:59:04.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:04.385 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-11-28T12:59:04.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:04.453 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-11-28T12:59:04.557 INFO:teuthology.orchestra.run.smithi111.stdout:# file: . 2023-11-28T12:59:04.558 INFO:teuthology.orchestra.run.smithi111.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-11-28T12:59:04.558 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:04.558 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:59:04.558 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:04.575 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:04.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.734+0000 7fefcdb47700 1 Processor -- start 2023-11-28T12:59:04.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.734+0000 7fefcdb47700 1 -- start start 2023-11-28T12:59:04.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.735+0000 7fefcdb47700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fefc8142e80 0x7fefc806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:04.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.735+0000 7fefcdb47700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fefc806a7b0 0x7fefc80c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:04.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.735+0000 7fefcdb47700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fefc80ca430 0x7fefc806d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.735+0000 7fefcdb47700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefc80c9380 con 0x7fefc8142e80 2023-11-28T12:59:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.735+0000 7fefcdb47700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefc806ac00 con 0x7fefc80ca430 2023-11-28T12:59:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.735+0000 7fefcdb47700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc8159b10 con 0x7fefc806a7b0 2023-11-28T12:59:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.735+0000 7fefc77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fefc8142e80 0x7fefc806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.735+0000 7fefc77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fefc8142e80 0x7fefc806a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32812/0 (socket says 172.21.15.111:32812) 2023-11-28T12:59:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.735+0000 7fefc77fe700 1 -- 172.21.15.111:0/4014254830 learned_addr learned my addr 172.21.15.111:0/4014254830 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.735+0000 7fefc6ffd700 1 --2- 172.21.15.111:0/4014254830 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fefc806a7b0 0x7fefc80c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.735+0000 7fefc7fff700 1 --2- 172.21.15.111:0/4014254830 >> v2:172.21.15.149:3300/0 conn(0x7fefc80ca430 0x7fefc806d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefc77fe700 1 -- 172.21.15.111:0/4014254830 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fefc806a7b0 msgr2=0x7fefc80c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefc77fe700 1 --2- 172.21.15.111:0/4014254830 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fefc806a7b0 0x7fefc80c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefc77fe700 1 -- 172.21.15.111:0/4014254830 >> v2:172.21.15.149:3300/0 conn(0x7fefc80ca430 msgr2=0x7fefc806d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:04.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefc77fe700 1 --2- 172.21.15.111:0/4014254830 >> v2:172.21.15.149:3300/0 conn(0x7fefc80ca430 0x7fefc806d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefc77fe700 1 -- 172.21.15.111:0/4014254830 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc80ca890 con 0x7fefc8142e80 2023-11-28T12:59:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefc77fe700 1 --2- 172.21.15.111:0/4014254830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefc8142e80 0x7fefc806a270 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fefc815d120 tx=0x7fefbc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e226a87d276c3e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefc67fc700 1 -- 172.21.15.111:0/4014254830 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc00b870 con 0x7fefc8142e80 2023-11-28T12:59:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefc67fc700 1 -- 172.21.15.111:0/4014254830 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefbc00b9d0 con 0x7fefc8142e80 2023-11-28T12:59:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefc67fc700 1 -- 172.21.15.111:0/4014254830 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc00bd60 con 0x7fefc8142e80 2023-11-28T12:59:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefcdb47700 1 -- 172.21.15.111:0/4014254830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefc8142e80 msgr2=0x7fefc806a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefcdb47700 1 --2- 172.21.15.111:0/4014254830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefc8142e80 0x7fefc806a270 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fefc815d120 tx=0x7fefbc009bd0 comp rx=0 tx=0).stop 2023-11-28T12:59:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefcdb47700 1 -- 172.21.15.111:0/4014254830 shutdown_connections 2023-11-28T12:59:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefcdb47700 1 --2- 172.21.15.111:0/4014254830 >> v2:172.21.15.149:3300/0 conn(0x7fefc80ca430 0x7fefc806d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefcdb47700 1 --2- 172.21.15.111:0/4014254830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefc8142e80 0x7fefc806a270 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefcdb47700 1 --2- 172.21.15.111:0/4014254830 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fefc806a7b0 0x7fefc80c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.736+0000 7fefcdb47700 1 -- 172.21.15.111:0/4014254830 >> 172.21.15.111:0/4014254830 conn(0x7fefc80cb490 msgr2=0x7fefc80cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:04.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.737+0000 7fefcdb47700 1 -- 172.21.15.111:0/4014254830 shutdown_connections 2023-11-28T12:59:04.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.737+0000 7fefcdb47700 1 -- 172.21.15.111:0/4014254830 wait complete. 2023-11-28T12:59:04.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.738+0000 7fefcdb47700 1 Processor -- start 2023-11-28T12:59:04.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.738+0000 7fefcdb47700 1 -- start start 2023-11-28T12:59:04.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefcdb47700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fefc806a7b0 0x7fefc80c1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:04.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefcdb47700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fefc80ca430 0x7fefc80c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:04.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefcdb47700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefc8142e80 0x7fefc81f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:04.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefcdb47700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc80c1db0 con 0x7fefc8142e80 2023-11-28T12:59:04.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefcdb47700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc80c8c80 con 0x7fefc806a7b0 2023-11-28T12:59:04.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefcdb47700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc80c6d50 con 0x7fefc80ca430 2023-11-28T12:59:04.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefc7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefc8142e80 0x7fefc81f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:04.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefc7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefc8142e80 0x7fefc81f2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32822/0 (socket says 172.21.15.111:32822) 2023-11-28T12:59:04.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefc7fff700 1 -- 172.21.15.111:0/2422049038 learned_addr learned my addr 172.21.15.111:0/2422049038 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:04.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefc77fe700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fefc806a7b0 0x7fefc80c1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:04.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefc6ffd700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fefc80ca430 0x7fefc80c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:04.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefc7fff700 1 -- 172.21.15.111:0/2422049038 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fefc80ca430 msgr2=0x7fefc80c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:04.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefc7fff700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fefc80ca430 0x7fefc80c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:04.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefc7fff700 1 -- 172.21.15.111:0/2422049038 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fefc806a7b0 msgr2=0x7fefc80c1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefc7fff700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fefc806a7b0 0x7fefc80c1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefc7fff700 1 -- 172.21.15.111:0/2422049038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefbc009550 con 0x7fefc8142e80 2023-11-28T12:59:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefc77fe700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fefc806a7b0 0x7fefc80c1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefc6ffd700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fefc80ca430 0x7fefc80c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefc7fff700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefc8142e80 0x7fefc81f2390 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fefc0007e70 tx=0x7fefc000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefaffff700 1 -- 172.21.15.111:0/2422049038 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc000ee50 con 0x7fefc8142e80 2023-11-28T12:59:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefaffff700 1 -- 172.21.15.111:0/2422049038 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc0014420 con 0x7fefc8142e80 2023-11-28T12:59:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefcdb47700 1 -- 172.21.15.111:0/2422049038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc80cab00 con 0x7fefc8142e80 2023-11-28T12:59:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefaffff700 1 -- 172.21.15.111:0/2422049038 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc00147d0 con 0x7fefc8142e80 2023-11-28T12:59:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.739+0000 7fefcdb47700 1 -- 172.21.15.111:0/2422049038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc8143250 con 0x7fefc8142e80 2023-11-28T12:59:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.740+0000 7fefcdb47700 1 -- 172.21.15.111:0/2422049038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc80cc5f0 con 0x7fefc8142e80 2023-11-28T12:59:04.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.741+0000 7fefaffff700 1 -- 172.21.15.111:0/2422049038 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fefc0012050 con 0x7fefc8142e80 2023-11-28T12:59:04.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.741+0000 7fefaffff700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fefb4068320 0x7fefb406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:04.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.741+0000 7fefaffff700 1 -- 172.21.15.111:0/2422049038 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fefc0086590 con 0x7fefc8142e80 2023-11-28T12:59:04.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.741+0000 7fefc77fe700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fefb4068320 0x7fefb406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:04.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.742+0000 7fefc77fe700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fefb4068320 0x7fefb406a7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fefbc009130 tx=0x7fefbc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:04.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.745+0000 7fefaffff700 1 -- 172.21.15.111:0/2422049038 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fefc0052b00 con 0x7fefc8142e80 2023-11-28T12:59:04.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.931+0000 7fefcdb47700 1 -- 172.21.15.111:0/2422049038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefc8067880 con 0x7fefc8142e80 2023-11-28T12:59:04.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.932+0000 7fefaffff700 1 -- 172.21.15.111:0/2422049038 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19796 (secure 0 0 0) 0x7fefc0056120 con 0x7fefc8142e80 2023-11-28T12:59:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:04.934 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:00.053943+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:04.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.934+0000 7fefcdb47700 1 -- 172.21.15.111:0/2422049038 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fefb4068320 msgr2=0x7fefb406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:04.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.934+0000 7fefcdb47700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fefb4068320 0x7fefb406a7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fefbc009130 tx=0x7fefbc008030 comp rx=0 tx=0).stop 2023-11-28T12:59:04.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.934+0000 7fefcdb47700 1 -- 172.21.15.111:0/2422049038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefc8142e80 msgr2=0x7fefc81f2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:04.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.934+0000 7fefcdb47700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefc8142e80 0x7fefc81f2390 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fefc0007e70 tx=0x7fefc000e3c0 comp rx=0 tx=0).stop 2023-11-28T12:59:04.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.934+0000 7fefcdb47700 1 -- 172.21.15.111:0/2422049038 shutdown_connections 2023-11-28T12:59:04.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.934+0000 7fefcdb47700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefc8142e80 0x7fefc81f2390 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:04.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.934+0000 7fefcdb47700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fefb4068320 0x7fefb406a7d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:04.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.934+0000 7fefcdb47700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fefc806a7b0 0x7fefc80c1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:04.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.934+0000 7fefcdb47700 1 --2- 172.21.15.111:0/2422049038 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fefc80ca430 0x7fefc80c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:04.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.934+0000 7fefcdb47700 1 -- 172.21.15.111:0/2422049038 >> 172.21.15.111:0/2422049038 conn(0x7fefc80cb490 msgr2=0x7fefc80aeed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:04.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.934+0000 7fefcdb47700 1 -- 172.21.15.111:0/2422049038 shutdown_connections 2023-11-28T12:59:04.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:04.934+0000 7fefcdb47700 1 -- 172.21.15.111:0/2422049038 wait complete. 2023-11-28T12:59:04.947 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T12:58:53.275206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T12:58:54.588499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T12:58:57.666033+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T12:59:04.948 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:59:05.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.070+0000 7fb77e31b700 1 Processor -- start 2023-11-28T12:59:05.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.070+0000 7fb77e31b700 1 -- start start 2023-11-28T12:59:05.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.070+0000 7fb77e31b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb7780cdd90 0x7fb7780b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.070+0000 7fb77e31b700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb7780aecc0 0x7fb7780ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.070+0000 7fb77e31b700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7780ad420 con 0x7fb778159960 2023-11-28T12:59:05.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.070+0000 7fb77e31b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb778159d30 con 0x7fb7780cdd90 2023-11-28T12:59:05.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.070+0000 7fb77e31b700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7780ac810 con 0x7fb7780aecc0 2023-11-28T12:59:05.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.071+0000 7fb7777fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb7780cdd90 0x7fb7780b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.071+0000 7fb7777fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb7780cdd90 0x7fb7780b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32828/0 (socket says 172.21.15.111:32828) 2023-11-28T12:59:05.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.071+0000 7fb7777fe700 1 -- 172.21.15.111:0/2072238593 learned_addr learned my addr 172.21.15.111:0/2072238593 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:05.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.071+0000 7fb77c8b8700 1 --2- 172.21.15.111:0/2072238593 >> v2:172.21.15.149:3300/0 conn(0x7fb7780aecc0 0x7fb7780ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.071+0000 7fb7777fe700 1 -- 172.21.15.111:0/2072238593 >> v2:172.21.15.149:3300/0 conn(0x7fb7780aecc0 msgr2=0x7fb7780ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:05.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.071+0000 7fb7777fe700 1 --2- 172.21.15.111:0/2072238593 >> v2:172.21.15.149:3300/0 conn(0x7fb7780aecc0 0x7fb7780ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.071+0000 7fb7777fe700 1 -- 172.21.15.111:0/2072238593 >> v1:172.21.15.149:6789/0 conn(0x7fb778159960 legacy=0x7fb7780cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.071+0000 7fb7777fe700 1 -- 172.21.15.111:0/2072238593 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7780acaf0 con 0x7fb7780cdd90 2023-11-28T12:59:05.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.071+0000 7fb77c8b8700 1 --2- 172.21.15.111:0/2072238593 >> v2:172.21.15.149:3300/0 conn(0x7fb7780aecc0 0x7fb7780ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:05.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.071+0000 7fb7777fe700 1 --2- 172.21.15.111:0/2072238593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7780cdd90 0x7fb7780b1fd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb7780b14f0 tx=0x7fb76c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3a459b4678c922 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:05.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.071+0000 7fb776ffd700 1 -- 172.21.15.111:0/2072238593 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c00b870 con 0x7fb7780cdd90 2023-11-28T12:59:05.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.071+0000 7fb776ffd700 1 -- 172.21.15.111:0/2072238593 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c00b9d0 con 0x7fb7780cdd90 2023-11-28T12:59:05.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.071+0000 7fb776ffd700 1 -- 172.21.15.111:0/2072238593 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c00bd60 con 0x7fb7780cdd90 2023-11-28T12:59:05.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.072+0000 7fb77e31b700 1 -- 172.21.15.111:0/2072238593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7780cdd90 msgr2=0x7fb7780b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.072+0000 7fb77e31b700 1 --2- 172.21.15.111:0/2072238593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7780cdd90 0x7fb7780b1fd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb7780b14f0 tx=0x7fb76c009bd0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.072+0000 7fb77e31b700 1 -- 172.21.15.111:0/2072238593 shutdown_connections 2023-11-28T12:59:05.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.072+0000 7fb77e31b700 1 --2- 172.21.15.111:0/2072238593 >> v2:172.21.15.149:3300/0 conn(0x7fb7780aecc0 0x7fb7780ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.072+0000 7fb77e31b700 1 --2- 172.21.15.111:0/2072238593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7780cdd90 0x7fb7780b1fd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.072+0000 7fb77e31b700 1 -- 172.21.15.111:0/2072238593 >> 172.21.15.111:0/2072238593 conn(0x7fb778098ee0 msgr2=0x7fb77806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:05.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.072+0000 7fb77e31b700 1 -- 172.21.15.111:0/2072238593 shutdown_connections 2023-11-28T12:59:05.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.072+0000 7fb77e31b700 1 -- 172.21.15.111:0/2072238593 wait complete. 2023-11-28T12:59:05.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.073+0000 7fb77e31b700 1 Processor -- start 2023-11-28T12:59:05.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.074+0000 7fb77e31b700 1 -- start start 2023-11-28T12:59:05.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.074+0000 7fb77e31b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7780aecc0 0x7fb7781f4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.074+0000 7fb77e31b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7780cdd90 0x7fb778143450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.074+0000 7fb77e31b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb778159960 0x7fb77806a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.074+0000 7fb77e31b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780b3c50 con 0x7fb778159960 2023-11-28T12:59:05.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.074+0000 7fb77e31b700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780b2970 con 0x7fb7780aecc0 2023-11-28T12:59:05.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.074+0000 7fb77e31b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7781f4070 con 0x7fb7780cdd90 2023-11-28T12:59:05.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.074+0000 7fb77c8b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb778159960 0x7fb77806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.074+0000 7fb77c8b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb778159960 0x7fb77806a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32836/0 (socket says 172.21.15.111:32836) 2023-11-28T12:59:05.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.074+0000 7fb77c8b8700 1 -- 172.21.15.111:0/3139473702 learned_addr learned my addr 172.21.15.111:0/3139473702 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:05.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.075+0000 7fb777fff700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7780aecc0 0x7fb7781f4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.075+0000 7fb7777fe700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7780cdd90 0x7fb778143450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.075+0000 7fb77c8b8700 1 -- 172.21.15.111:0/3139473702 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7780cdd90 msgr2=0x7fb778143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.075+0000 7fb77c8b8700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7780cdd90 0x7fb778143450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.075+0000 7fb77c8b8700 1 -- 172.21.15.111:0/3139473702 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7780aecc0 msgr2=0x7fb7781f4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.075+0000 7fb77c8b8700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7780aecc0 0x7fb7781f4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.075+0000 7fb77c8b8700 1 -- 172.21.15.111:0/3139473702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c009550 con 0x7fb778159960 2023-11-28T12:59:05.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.075+0000 7fb77c8b8700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb778159960 0x7fb77806a270 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb770007ef0 tx=0x7fb77000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:05.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.075+0000 7fb774ff9700 1 -- 172.21.15.111:0/3139473702 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77000de60 con 0x7fb778159960 2023-11-28T12:59:05.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.075+0000 7fb774ff9700 1 -- 172.21.15.111:0/3139473702 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77000f040 con 0x7fb778159960 2023-11-28T12:59:05.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.075+0000 7fb774ff9700 1 -- 172.21.15.111:0/3139473702 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb770003c60 con 0x7fb778159960 2023-11-28T12:59:05.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.075+0000 7fb77e31b700 1 -- 172.21.15.111:0/3139473702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7780ad0b0 con 0x7fb778159960 2023-11-28T12:59:05.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.075+0000 7fb77e31b700 1 -- 172.21.15.111:0/3139473702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7780acdd0 con 0x7fb778159960 2023-11-28T12:59:05.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.076+0000 7fb77e31b700 1 -- 172.21.15.111:0/3139473702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7780addf0 con 0x7fb778159960 2023-11-28T12:59:05.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.076+0000 7fb774ff9700 1 -- 172.21.15.111:0/3139473702 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb77001c030 con 0x7fb778159960 2023-11-28T12:59:05.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.077+0000 7fb774ff9700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7640682f0 0x7fb76406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.077+0000 7fb774ff9700 1 -- 172.21.15.111:0/3139473702 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb770086160 con 0x7fb778159960 2023-11-28T12:59:05.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.077+0000 7fb777fff700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7640682f0 0x7fb76406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.077+0000 7fb777fff700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7640682f0 0x7fb76406a7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb768000fb0 tx=0x7fb768009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:05.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.081+0000 7fb774ff9700 1 -- 172.21.15.111:0/3139473702 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb7700525e0 con 0x7fb778159960 2023-11-28T12:59:05.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.261+0000 7fb77e31b700 1 -- 172.21.15.111:0/3139473702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb7781f3730 con 0x7fb778159960 2023-11-28T12:59:05.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.261+0000 7fb774ff9700 1 -- 172.21.15.111:0/3139473702 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb770055c00 con 0x7fb778159960 2023-11-28T12:59:05.263 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:59:05.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.263+0000 7fb77e31b700 1 -- 172.21.15.111:0/3139473702 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7640682f0 msgr2=0x7fb76406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.263+0000 7fb77e31b700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7640682f0 0x7fb76406a7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb768000fb0 tx=0x7fb768009040 comp rx=0 tx=0).stop 2023-11-28T12:59:05.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.263+0000 7fb77e31b700 1 -- 172.21.15.111:0/3139473702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb778159960 msgr2=0x7fb77806a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.263+0000 7fb77e31b700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb778159960 0x7fb77806a270 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb770007ef0 tx=0x7fb77000d3b0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.263+0000 7fb77e31b700 1 -- 172.21.15.111:0/3139473702 shutdown_connections 2023-11-28T12:59:05.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.263+0000 7fb77e31b700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb778159960 0x7fb77806a270 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.263+0000 7fb77e31b700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7640682f0 0x7fb76406a7a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.263+0000 7fb77e31b700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7780aecc0 0x7fb7781f4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.263+0000 7fb77e31b700 1 --2- 172.21.15.111:0/3139473702 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7780cdd90 0x7fb778143450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.263+0000 7fb77e31b700 1 -- 172.21.15.111:0/3139473702 >> 172.21.15.111:0/3139473702 conn(0x7fb778098ee0 msgr2=0x7fb778155d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:05.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.263+0000 7fb77e31b700 1 -- 172.21.15.111:0/3139473702 shutdown_connections 2023-11-28T12:59:05.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.263+0000 7fb77e31b700 1 -- 172.21.15.111:0/3139473702 wait complete. 2023-11-28T12:59:05.277 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:59:05.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.398+0000 7f1343973700 1 Processor -- start 2023-11-28T12:59:05.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.398+0000 7f1343973700 1 -- start start 2023-11-28T12:59:05.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1343973700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f133c0cdd90 0x7f133c0b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1343973700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f133c0aecc0 0x7f133c0ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1343973700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c0ad420 con 0x7f133c159960 2023-11-28T12:59:05.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1343973700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f133c159d30 con 0x7f133c0cdd90 2023-11-28T12:59:05.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1343973700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f133c0ac810 con 0x7f133c0aecc0 2023-11-28T12:59:05.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1340f0e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f133c0cdd90 0x7f133c0b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f134170f700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f133c159960 0x7f133c0cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33618/0 (socket says 172.21.15.111:33618) 2023-11-28T12:59:05.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f134170f700 1 -- 172.21.15.111:0/1726495521 learned_addr learned my addr 172.21.15.111:0/1726495521 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:05.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1341f10700 1 --2- 172.21.15.111:0/1726495521 >> v2:172.21.15.149:3300/0 conn(0x7f133c0aecc0 0x7f133c0ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1340f0e700 1 -- 172.21.15.111:0/1726495521 >> v2:172.21.15.149:3300/0 conn(0x7f133c0aecc0 msgr2=0x7f133c0ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1340f0e700 1 --2- 172.21.15.111:0/1726495521 >> v2:172.21.15.149:3300/0 conn(0x7f133c0aecc0 0x7f133c0ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1340f0e700 1 -- 172.21.15.111:0/1726495521 >> v1:172.21.15.111:6789/0 conn(0x7f133c159960 legacy=0x7f133c0cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1333fff700 1 -- 172.21.15.111:0/1726495521 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377452211 0 0) 0x7f133c0ad420 con 0x7f133c159960 2023-11-28T12:59:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1340f0e700 1 -- 172.21.15.111:0/1726495521 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c0acaf0 con 0x7f133c0cdd90 2023-11-28T12:59:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1341f10700 1 --2- 172.21.15.111:0/1726495521 >> v2:172.21.15.149:3300/0 conn(0x7f133c0aecc0 0x7f133c0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.399+0000 7f1340f0e700 1 --2- 172.21.15.111:0/1726495521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f133c0cdd90 0x7f133c0b1fd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f133c0b14f0 tx=0x7f1338009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee913a3346fcce7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.400+0000 7f1333fff700 1 -- 172.21.15.111:0/1726495521 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338011830 con 0x7f133c0cdd90 2023-11-28T12:59:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.400+0000 7f1333fff700 1 -- 172.21.15.111:0/1726495521 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1338011990 con 0x7f133c0cdd90 2023-11-28T12:59:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.400+0000 7f1333fff700 1 -- 172.21.15.111:0/1726495521 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338011d20 con 0x7f133c0cdd90 2023-11-28T12:59:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.400+0000 7f1343973700 1 -- 172.21.15.111:0/1726495521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f133c0cdd90 msgr2=0x7f133c0b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.400+0000 7f1343973700 1 --2- 172.21.15.111:0/1726495521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f133c0cdd90 0x7f133c0b1fd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f133c0b14f0 tx=0x7f1338009b20 comp rx=0 tx=0).stop 2023-11-28T12:59:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.400+0000 7f1343973700 1 -- 172.21.15.111:0/1726495521 shutdown_connections 2023-11-28T12:59:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.400+0000 7f1343973700 1 --2- 172.21.15.111:0/1726495521 >> v2:172.21.15.149:3300/0 conn(0x7f133c0aecc0 0x7f133c0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.400+0000 7f1343973700 1 --2- 172.21.15.111:0/1726495521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f133c0cdd90 0x7f133c0b1fd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.400+0000 7f1343973700 1 -- 172.21.15.111:0/1726495521 >> 172.21.15.111:0/1726495521 conn(0x7f133c098ee0 msgr2=0x7f133c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.400+0000 7f1343973700 1 -- 172.21.15.111:0/1726495521 shutdown_connections 2023-11-28T12:59:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.400+0000 7f1343973700 1 -- 172.21.15.111:0/1726495521 wait complete. 2023-11-28T12:59:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.401+0000 7f1343973700 1 Processor -- start 2023-11-28T12:59:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.401+0000 7f1343973700 1 -- start start 2023-11-28T12:59:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f1343973700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f133c0aecc0 0x7f133c0b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f1343973700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f133c0cdd90 0x7f133c153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f1343973700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f133c159960 0x7f133c0b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f1343973700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c1e4da0 con 0x7f133c0cdd90 2023-11-28T12:59:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f1343973700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c1e5080 con 0x7f133c159960 2023-11-28T12:59:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f1343973700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133c15b060 con 0x7f133c0aecc0 2023-11-28T12:59:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f1340f0e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f133c0cdd90 0x7f133c153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f1340f0e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f133c0cdd90 0x7f133c153640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32848/0 (socket says 172.21.15.111:32848) 2023-11-28T12:59:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f1340f0e700 1 -- 172.21.15.111:0/314973134 learned_addr learned my addr 172.21.15.111:0/314973134 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f134170f700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f133c0aecc0 0x7f133c0b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f1341f10700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f133c159960 0x7f133c0b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f1340f0e700 1 -- 172.21.15.111:0/314973134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f133c0aecc0 msgr2=0x7f133c0b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f1340f0e700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f133c0aecc0 0x7f133c0b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.402+0000 7f1340f0e700 1 -- 172.21.15.111:0/314973134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f133c159960 msgr2=0x7f133c0b2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.403+0000 7f1340f0e700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f133c159960 0x7f133c0b2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.403+0000 7f1340f0e700 1 -- 172.21.15.111:0/314973134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13380094a0 con 0x7f133c0cdd90 2023-11-28T12:59:05.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.403+0000 7f134170f700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f133c0aecc0 0x7f133c0b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:05.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.403+0000 7f1340f0e700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f133c0cdd90 0x7f133c153640 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1338000c00 tx=0x7f1338004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:05.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.403+0000 7f1331ffb700 1 -- 172.21.15.111:0/314973134 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13380038a0 con 0x7f133c0cdd90 2023-11-28T12:59:05.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.403+0000 7f1331ffb700 1 -- 172.21.15.111:0/314973134 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133801f070 con 0x7f133c0cdd90 2023-11-28T12:59:05.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.403+0000 7f1343973700 1 -- 172.21.15.111:0/314973134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133c0ad0b0 con 0x7f133c0cdd90 2023-11-28T12:59:05.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.403+0000 7f1331ffb700 1 -- 172.21.15.111:0/314973134 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338003c40 con 0x7f133c0cdd90 2023-11-28T12:59:05.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.403+0000 7f1343973700 1 -- 172.21.15.111:0/314973134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133c0acdd0 con 0x7f133c0cdd90 2023-11-28T12:59:05.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.404+0000 7f1343973700 1 -- 172.21.15.111:0/314973134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133c0addf0 con 0x7f133c0cdd90 2023-11-28T12:59:05.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.404+0000 7f1331ffb700 1 -- 172.21.15.111:0/314973134 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f133800f050 con 0x7f133c0cdd90 2023-11-28T12:59:05.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.405+0000 7f1331ffb700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13200682f0 0x7f132006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.405+0000 7f1331ffb700 1 -- 172.21.15.111:0/314973134 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f1338021070 con 0x7f133c0cdd90 2023-11-28T12:59:05.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.405+0000 7f134170f700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13200682f0 0x7f132006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.405+0000 7f134170f700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13200682f0 0x7f132006a7a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f132c0017b0 tx=0x7f132c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:05.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.409+0000 7f1331ffb700 1 -- 172.21.15.111:0/314973134 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1338053ed0 con 0x7f133c0cdd90 2023-11-28T12:59:05.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.590+0000 7f1343973700 1 -- 172.21.15.111:0/314973134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f133c15aec0 con 0x7f133c0cdd90 2023-11-28T12:59:05.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.590+0000 7f1331ffb700 1 -- 172.21.15.111:0/314973134 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f133802b020 con 0x7f133c0cdd90 2023-11-28T12:59:05.592 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:59:05.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.592+0000 7f1343973700 1 -- 172.21.15.111:0/314973134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13200682f0 msgr2=0x7f132006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.592+0000 7f1343973700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13200682f0 0x7f132006a7a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f132c0017b0 tx=0x7f132c009040 comp rx=0 tx=0).stop 2023-11-28T12:59:05.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.592+0000 7f1343973700 1 -- 172.21.15.111:0/314973134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f133c0cdd90 msgr2=0x7f133c153640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.592+0000 7f1343973700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f133c0cdd90 0x7f133c153640 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1338000c00 tx=0x7f1338004750 comp rx=0 tx=0).stop 2023-11-28T12:59:05.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.592+0000 7f1343973700 1 -- 172.21.15.111:0/314973134 shutdown_connections 2023-11-28T12:59:05.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.592+0000 7f1343973700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13200682f0 0x7f132006a7a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.592+0000 7f1343973700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f133c159960 0x7f133c0b2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.592+0000 7f1343973700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f133c0aecc0 0x7f133c0b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.592+0000 7f1343973700 1 --2- 172.21.15.111:0/314973134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f133c0cdd90 0x7f133c153640 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.592+0000 7f1343973700 1 -- 172.21.15.111:0/314973134 >> 172.21.15.111:0/314973134 conn(0x7f133c098ee0 msgr2=0x7f133c156590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:05.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.592+0000 7f1343973700 1 -- 172.21.15.111:0/314973134 shutdown_connections 2023-11-28T12:59:05.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.592+0000 7f1343973700 1 -- 172.21.15.111:0/314973134 wait complete. 2023-11-28T12:59:05.605 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T12:59:05.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.727+0000 7f2ca1ead700 1 Processor -- start 2023-11-28T12:59:05.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.727+0000 7f2ca1ead700 1 -- start start 2023-11-28T12:59:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.728+0000 7f2ca1ead700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f2c9c0d0490 0x7f2c9c0cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.728+0000 7f2ca1ead700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c9c06b380 0x7f2c9c06b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.728+0000 7f2ca1ead700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9c0cb810 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.728+0000 7f2ca1ead700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9c06bd40 con 0x7f2c9c0d0490 2023-11-28T12:59:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.728+0000 7f2ca1ead700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c0cc650 con 0x7f2c9c06b380 2023-11-28T12:59:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9affd700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f2c9c0d0490 0x7f2c9c0cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9bfff700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c9c06b380 0x7f2c9c06b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9b7fe700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f2c9c15c7b0 0x7f2c9c0ad8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:42174/0 (socket says 172.21.15.111:42174) 2023-11-28T12:59:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9b7fe700 1 -- 172.21.15.111:0/994552743 learned_addr learned my addr 172.21.15.111:0/994552743 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9a7fc700 1 -- 172.21.15.111:0/994552743 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083067351 0 0) 0x7f2c9c0cb810 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9a7fc700 1 -- 172.21.15.111:0/994552743 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c840035b0 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9affd700 1 -- 172.21.15.111:0/994552743 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c9c06b380 msgr2=0x7f2c9c06b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9affd700 1 --2- 172.21.15.111:0/994552743 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c9c06b380 0x7f2c9c06b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9affd700 1 -- 172.21.15.111:0/994552743 >> v1:172.21.15.149:6789/0 conn(0x7f2c9c15c7b0 legacy=0x7f2c9c0ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9affd700 1 -- 172.21.15.111:0/994552743 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c0cba40 con 0x7f2c9c0d0490 2023-11-28T12:59:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9bfff700 1 --2- 172.21.15.111:0/994552743 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c9c06b380 0x7f2c9c06b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9affd700 1 --2- 172.21.15.111:0/994552743 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c9c0d0490 0x7f2c9c0cb2d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009840 tx=0x7f2c8c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e2f34fe81c43fbd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9a7fc700 1 -- 172.21.15.111:0/994552743 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c011830 con 0x7f2c9c0d0490 2023-11-28T12:59:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9a7fc700 1 -- 172.21.15.111:0/994552743 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c011990 con 0x7f2c9c0d0490 2023-11-28T12:59:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.729+0000 7f2c9a7fc700 1 -- 172.21.15.111:0/994552743 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c011d00 con 0x7f2c9c0d0490 2023-11-28T12:59:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.730+0000 7f2ca1ead700 1 -- 172.21.15.111:0/994552743 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c9c0d0490 msgr2=0x7f2c9c0cb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.730+0000 7f2ca1ead700 1 --2- 172.21.15.111:0/994552743 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c9c0d0490 0x7f2c9c0cb2d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009840 tx=0x7f2c8c009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.730+0000 7f2ca1ead700 1 -- 172.21.15.111:0/994552743 shutdown_connections 2023-11-28T12:59:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.730+0000 7f2ca1ead700 1 --2- 172.21.15.111:0/994552743 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c9c06b380 0x7f2c9c06b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.730+0000 7f2ca1ead700 1 --2- 172.21.15.111:0/994552743 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c9c0d0490 0x7f2c9c0cb2d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.730+0000 7f2ca1ead700 1 -- 172.21.15.111:0/994552743 >> 172.21.15.111:0/994552743 conn(0x7f2c9c09a040 msgr2=0x7f2c9c0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.730+0000 7f2ca1ead700 1 -- 172.21.15.111:0/994552743 shutdown_connections 2023-11-28T12:59:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.730+0000 7f2ca1ead700 1 -- 172.21.15.111:0/994552743 wait complete. 2023-11-28T12:59:05.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.731+0000 7f2ca1ead700 1 Processor -- start 2023-11-28T12:59:05.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.731+0000 7f2ca1ead700 1 -- start start 2023-11-28T12:59:05.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2ca1ead700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c9c06b380 0x7f2c9c158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2ca1ead700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c9c0d0490 0x7f2c9c1f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2ca1ead700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c9c15c7b0 0x7f2c9c0ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2ca1ead700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c0c8d20 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2ca1ead700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c15f880 con 0x7f2c9c0d0490 2023-11-28T12:59:05.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2ca1ead700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c159420 con 0x7f2c9c06b380 2023-11-28T12:59:05.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2c9bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c9c15c7b0 0x7f2c9c0ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2c9bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c9c15c7b0 0x7f2c9c0ca0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32864/0 (socket says 172.21.15.111:32864) 2023-11-28T12:59:05.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2c9bfff700 1 -- 172.21.15.111:0/603093176 learned_addr learned my addr 172.21.15.111:0/603093176 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2c9b7fe700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c9c06b380 0x7f2c9c158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2c9affd700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c9c0d0490 0x7f2c9c1f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2c9bfff700 1 -- 172.21.15.111:0/603093176 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c9c06b380 msgr2=0x7f2c9c158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2c9bfff700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c9c06b380 0x7f2c9c158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2c9bfff700 1 -- 172.21.15.111:0/603093176 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c9c0d0490 msgr2=0x7f2c9c1f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2c9bfff700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c9c0d0490 0x7f2c9c1f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2c9bfff700 1 -- 172.21.15.111:0/603093176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c94009a70 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.732+0000 7f2c9bfff700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c9c15c7b0 0x7f2c9c0ca0f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400efc0 tx=0x7f2c9400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.733+0000 7f2c83fff700 1 -- 172.21.15.111:0/603093176 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9400ca90 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.733+0000 7f2c83fff700 1 -- 172.21.15.111:0/603093176 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9400f040 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.733+0000 7f2ca1ead700 1 -- 172.21.15.111:0/603093176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8c0094a0 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.733+0000 7f2c83fff700 1 -- 172.21.15.111:0/603093176 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9400ce10 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.733+0000 7f2ca1ead700 1 -- 172.21.15.111:0/603093176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9c0cc000 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.734+0000 7f2ca1ead700 1 -- 172.21.15.111:0/603093176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9c0cbd20 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.734+0000 7f2c83fff700 1 -- 172.21.15.111:0/603093176 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2c94014480 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.734+0000 7f2c83fff700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c880682f0 0x7f2c8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:05.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.735+0000 7f2c9b7fe700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c880682f0 0x7f2c8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:05.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.735+0000 7f2c83fff700 1 -- 172.21.15.111:0/603093176 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f2c940866e0 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.735+0000 7f2c9b7fe700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c880682f0 0x7f2c8806a7a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2c90005960 tx=0x7f2c90009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:05.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.739+0000 7f2c83fff700 1 -- 172.21.15.111:0/603093176 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2c94052bf0 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.922+0000 7f2ca1ead700 1 -- 172.21.15.111:0/603093176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2c9c0cc2e0 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.922+0000 7f2c83fff700 1 -- 172.21.15.111:0/603093176 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2c94009d70 con 0x7f2c9c15c7b0 2023-11-28T12:59:05.924 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:05.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.924+0000 7f2ca1ead700 1 -- 172.21.15.111:0/603093176 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c880682f0 msgr2=0x7f2c8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.924+0000 7f2ca1ead700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c880682f0 0x7f2c8806a7a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2c90005960 tx=0x7f2c90009040 comp rx=0 tx=0).stop 2023-11-28T12:59:05.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.924+0000 7f2ca1ead700 1 -- 172.21.15.111:0/603093176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c9c15c7b0 msgr2=0x7f2c9c0ca0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:05.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.924+0000 7f2ca1ead700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c9c15c7b0 0x7f2c9c0ca0f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400efc0 tx=0x7f2c9400c3a0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.924+0000 7f2ca1ead700 1 -- 172.21.15.111:0/603093176 shutdown_connections 2023-11-28T12:59:05.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.924+0000 7f2ca1ead700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c9c15c7b0 0x7f2c9c0ca0f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.924+0000 7f2ca1ead700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c9c06b380 0x7f2c9c158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.924+0000 7f2ca1ead700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c880682f0 0x7f2c8806a7a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.924+0000 7f2ca1ead700 1 --2- 172.21.15.111:0/603093176 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c9c0d0490 0x7f2c9c1f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:05.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.924+0000 7f2ca1ead700 1 -- 172.21.15.111:0/603093176 >> 172.21.15.111:0/603093176 conn(0x7f2c9c09a040 msgr2=0x7f2c9c0d0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:05.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.924+0000 7f2ca1ead700 1 -- 172.21.15.111:0/603093176 shutdown_connections 2023-11-28T12:59:05.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:05.924+0000 7f2ca1ead700 1 -- 172.21.15.111:0/603093176 wait complete. 2023-11-28T12:59:05.937 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T12:59:06.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.058+0000 7efd46189700 1 Processor -- start 2023-11-28T12:59:06.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.058+0000 7efd46189700 1 -- start start 2023-11-28T12:59:06.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd46189700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7efd4015a5d0 0x7efd400ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd46189700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd400b0160 0x7efd400ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd46189700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd400ae8b0 con 0x7efd400cb2d0 2023-11-28T12:59:06.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd46189700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd400cc1b0 con 0x7efd4015a5d0 2023-11-28T12:59:06.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd46189700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd400cc390 con 0x7efd400b0160 2023-11-28T12:59:06.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd3effd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd400b0160 0x7efd400ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd3f7fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7efd4015a5d0 0x7efd400ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd3effd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd400b0160 0x7efd400ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:54618/0 (socket says 172.21.15.111:54618) 2023-11-28T12:59:06.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd3effd700 1 -- 172.21.15.111:0/4044510895 learned_addr learned my addr 172.21.15.111:0/4044510895 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:06.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd3ffff700 1 --1- 172.21.15.111:0/4044510895 >> v1:172.21.15.149:6789/0 conn(0x7efd400cb2d0 0x7efd4015e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:42192/0 (socket says 172.21.15.111:42192) 2023-11-28T12:59:06.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd3f7fe700 1 -- 172.21.15.111:0/4044510895 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd400b0160 msgr2=0x7efd400ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:06.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd3f7fe700 1 --2- 172.21.15.111:0/4044510895 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd400b0160 0x7efd400ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd3f7fe700 1 -- 172.21.15.111:0/4044510895 >> v1:172.21.15.149:6789/0 conn(0x7efd400cb2d0 legacy=0x7efd4015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd3f7fe700 1 -- 172.21.15.111:0/4044510895 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd400cc670 con 0x7efd4015a5d0 2023-11-28T12:59:06.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.059+0000 7efd3effd700 1 --2- 172.21.15.111:0/4044510895 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd400b0160 0x7efd400ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:06.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.060+0000 7efd3f7fe700 1 --2- 172.21.15.111:0/4044510895 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd4015a5d0 0x7efd400ce850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efd34009820 tx=0x7efd34009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ae402f7b36f89fd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:06.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.060+0000 7efd3e7fc700 1 -- 172.21.15.111:0/4044510895 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd34005e30 con 0x7efd4015a5d0 2023-11-28T12:59:06.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.060+0000 7efd3e7fc700 1 -- 172.21.15.111:0/4044510895 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd34006030 con 0x7efd4015a5d0 2023-11-28T12:59:06.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.060+0000 7efd3e7fc700 1 -- 172.21.15.111:0/4044510895 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340063c0 con 0x7efd4015a5d0 2023-11-28T12:59:06.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.060+0000 7efd46189700 1 -- 172.21.15.111:0/4044510895 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd4015a5d0 msgr2=0x7efd400ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.060+0000 7efd46189700 1 --2- 172.21.15.111:0/4044510895 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd4015a5d0 0x7efd400ce850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efd34009820 tx=0x7efd34009b30 comp rx=0 tx=0).stop 2023-11-28T12:59:06.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.060+0000 7efd46189700 1 -- 172.21.15.111:0/4044510895 shutdown_connections 2023-11-28T12:59:06.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.060+0000 7efd46189700 1 --2- 172.21.15.111:0/4044510895 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd4015a5d0 0x7efd400ce850 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.060+0000 7efd46189700 1 --2- 172.21.15.111:0/4044510895 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd400b0160 0x7efd400ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.060+0000 7efd46189700 1 -- 172.21.15.111:0/4044510895 >> 172.21.15.111:0/4044510895 conn(0x7efd4009a040 msgr2=0x7efd400d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:06.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.060+0000 7efd46189700 1 -- 172.21.15.111:0/4044510895 shutdown_connections 2023-11-28T12:59:06.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.060+0000 7efd46189700 1 -- 172.21.15.111:0/4044510895 wait complete. 2023-11-28T12:59:06.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.062+0000 7efd46189700 1 Processor -- start 2023-11-28T12:59:06.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.062+0000 7efd46189700 1 -- start start 2023-11-28T12:59:06.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.063+0000 7efd46189700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd400b0160 0x7efd400b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.063+0000 7efd46189700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd400cb2d0 0x7efd400af530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.063+0000 7efd46189700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd4015a5d0 0x7efd401ef0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.063+0000 7efd46189700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd400cab40 con 0x7efd4015a5d0 2023-11-28T12:59:06.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.063+0000 7efd46189700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd400b1d90 con 0x7efd400cb2d0 2023-11-28T12:59:06.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.063+0000 7efd46189700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd400ca1a0 con 0x7efd400b0160 2023-11-28T12:59:06.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.063+0000 7efd3ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd4015a5d0 0x7efd401ef0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.063+0000 7efd3ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd4015a5d0 0x7efd401ef0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32878/0 (socket says 172.21.15.111:32878) 2023-11-28T12:59:06.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.063+0000 7efd3ffff700 1 -- 172.21.15.111:0/1965754807 learned_addr learned my addr 172.21.15.111:0/1965754807 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:06.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.063+0000 7efd3effd700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd400cb2d0 0x7efd400af530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.063+0000 7efd3f7fe700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd400b0160 0x7efd400b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.064+0000 7efd3ffff700 1 -- 172.21.15.111:0/1965754807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd400b0160 msgr2=0x7efd400b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.064+0000 7efd3ffff700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd400b0160 0x7efd400b1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.064+0000 7efd3ffff700 1 -- 172.21.15.111:0/1965754807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd400cb2d0 msgr2=0x7efd400af530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.064+0000 7efd3ffff700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd400cb2d0 0x7efd400af530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.064+0000 7efd3ffff700 1 -- 172.21.15.111:0/1965754807 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd30009550 con 0x7efd4015a5d0 2023-11-28T12:59:06.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.064+0000 7efd3f7fe700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd400b0160 0x7efd400b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:06.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.064+0000 7efd3ffff700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd4015a5d0 0x7efd401ef0b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efd3800cba0 tx=0x7efd3800ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:06.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.064+0000 7efd27fff700 1 -- 172.21.15.111:0/1965754807 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3800aad0 con 0x7efd4015a5d0 2023-11-28T12:59:06.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.064+0000 7efd27fff700 1 -- 172.21.15.111:0/1965754807 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3800ac30 con 0x7efd4015a5d0 2023-11-28T12:59:06.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.064+0000 7efd46189700 1 -- 172.21.15.111:0/1965754807 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd34009430 con 0x7efd4015a5d0 2023-11-28T12:59:06.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.064+0000 7efd27fff700 1 -- 172.21.15.111:0/1965754807 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd38007070 con 0x7efd4015a5d0 2023-11-28T12:59:06.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.064+0000 7efd46189700 1 -- 172.21.15.111:0/1965754807 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd400ccc30 con 0x7efd4015a5d0 2023-11-28T12:59:06.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.065+0000 7efd46189700 1 -- 172.21.15.111:0/1965754807 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd400cc950 con 0x7efd4015a5d0 2023-11-28T12:59:06.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.065+0000 7efd27fff700 1 -- 172.21.15.111:0/1965754807 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efd38019030 con 0x7efd4015a5d0 2023-11-28T12:59:06.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.066+0000 7efd27fff700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd2c0682f0 0x7efd2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.066+0000 7efd27fff700 1 -- 172.21.15.111:0/1965754807 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7efd38085c80 con 0x7efd4015a5d0 2023-11-28T12:59:06.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.066+0000 7efd3f7fe700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd2c0682f0 0x7efd2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.066+0000 7efd3f7fe700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd2c0682f0 0x7efd2c06a7a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efd34009b70 tx=0x7efd34008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:06.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.070+0000 7efd27fff700 1 -- 172.21.15.111:0/1965754807 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efd3804b5e0 con 0x7efd4015a5d0 2023-11-28T12:59:06.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.252+0000 7efd46189700 1 -- 172.21.15.111:0/1965754807 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efd400ccf10 con 0x7efd4015a5d0 2023-11-28T12:59:06.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.252+0000 7efd27fff700 1 -- 172.21.15.111:0/1965754807 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7efd38017020 con 0x7efd4015a5d0 2023-11-28T12:59:06.254 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T12:59:06.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.254+0000 7efd46189700 1 -- 172.21.15.111:0/1965754807 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd2c0682f0 msgr2=0x7efd2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.254+0000 7efd46189700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd2c0682f0 0x7efd2c06a7a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efd34009b70 tx=0x7efd34008070 comp rx=0 tx=0).stop 2023-11-28T12:59:06.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.254+0000 7efd46189700 1 -- 172.21.15.111:0/1965754807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd4015a5d0 msgr2=0x7efd401ef0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.254+0000 7efd46189700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd4015a5d0 0x7efd401ef0b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efd3800cba0 tx=0x7efd3800ceb0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.254+0000 7efd46189700 1 -- 172.21.15.111:0/1965754807 shutdown_connections 2023-11-28T12:59:06.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.254+0000 7efd46189700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd4015a5d0 0x7efd401ef0b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.254+0000 7efd46189700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd400b0160 0x7efd400b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.254+0000 7efd46189700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd2c0682f0 0x7efd2c06a7a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.254+0000 7efd46189700 1 --2- 172.21.15.111:0/1965754807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd400cb2d0 0x7efd400af530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.254+0000 7efd46189700 1 -- 172.21.15.111:0/1965754807 >> 172.21.15.111:0/1965754807 conn(0x7efd4009a040 msgr2=0x7efd400700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:06.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.255+0000 7efd46189700 1 -- 172.21.15.111:0/1965754807 shutdown_connections 2023-11-28T12:59:06.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.255+0000 7efd46189700 1 -- 172.21.15.111:0/1965754807 wait complete. 2023-11-28T12:59:06.267 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T12:59:06.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.391+0000 7f50bb99e700 1 Processor -- start 2023-11-28T12:59:06.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.391+0000 7f50bb99e700 1 -- start start 2023-11-28T12:59:06.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50bb99e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f50b40b0d70 0x7f50b41604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50bb99e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f50b40b1140 0x7f50b40ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50bb99e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f50b40b15b0 0x7f50b40cc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50bb99e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b415cd90 con 0x7f50b40b0d70 2023-11-28T12:59:06.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50bb99e700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b41613e0 con 0x7f50b40b15b0 2023-11-28T12:59:06.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50bb99e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4160b40 con 0x7f50b40b1140 2023-11-28T12:59:06.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50b973a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f50b40b0d70 0x7f50b41604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50b973a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f50b40b0d70 0x7f50b41604b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32882/0 (socket says 172.21.15.111:32882) 2023-11-28T12:59:06.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50b973a700 1 -- 172.21.15.111:0/1957092901 learned_addr learned my addr 172.21.15.111:0/1957092901 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:06.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50b9f3b700 1 --2- 172.21.15.111:0/1957092901 >> v2:172.21.15.149:3300/0 conn(0x7f50b40b15b0 0x7f50b40cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50b8f39700 1 --2- 172.21.15.111:0/1957092901 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f50b40b1140 0x7f50b40ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50b973a700 1 -- 172.21.15.111:0/1957092901 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f50b40b1140 msgr2=0x7f50b40ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:06.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50b973a700 1 --2- 172.21.15.111:0/1957092901 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f50b40b1140 0x7f50b40ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50b973a700 1 -- 172.21.15.111:0/1957092901 >> v2:172.21.15.149:3300/0 conn(0x7f50b40b15b0 msgr2=0x7f50b40cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:06.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50b973a700 1 --2- 172.21.15.111:0/1957092901 >> v2:172.21.15.149:3300/0 conn(0x7f50b40b15b0 0x7f50b40cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50b973a700 1 -- 172.21.15.111:0/1957092901 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b4160e20 con 0x7f50b40b0d70 2023-11-28T12:59:06.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.392+0000 7f50b973a700 1 --2- 172.21.15.111:0/1957092901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b40b0d70 0x7f50b41604b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f50b40cfb10 tx=0x7f50b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13b60167d59032c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:06.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.393+0000 7f50abfff700 1 -- 172.21.15.111:0/1957092901 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0005e30 con 0x7f50b40b0d70 2023-11-28T12:59:06.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.393+0000 7f50abfff700 1 -- 172.21.15.111:0/1957092901 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b0004030 con 0x7f50b40b0d70 2023-11-28T12:59:06.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.393+0000 7f50abfff700 1 -- 172.21.15.111:0/1957092901 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00043e0 con 0x7f50b40b0d70 2023-11-28T12:59:06.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.393+0000 7f50bb99e700 1 -- 172.21.15.111:0/1957092901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b40b0d70 msgr2=0x7f50b41604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.393+0000 7f50bb99e700 1 --2- 172.21.15.111:0/1957092901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b40b0d70 0x7f50b41604b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f50b40cfb10 tx=0x7f50b0009bd0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.393+0000 7f50bb99e700 1 -- 172.21.15.111:0/1957092901 shutdown_connections 2023-11-28T12:59:06.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.393+0000 7f50bb99e700 1 --2- 172.21.15.111:0/1957092901 >> v2:172.21.15.149:3300/0 conn(0x7f50b40b15b0 0x7f50b40cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.393+0000 7f50bb99e700 1 --2- 172.21.15.111:0/1957092901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b40b0d70 0x7f50b41604b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.393+0000 7f50bb99e700 1 --2- 172.21.15.111:0/1957092901 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f50b40b1140 0x7f50b40ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.393+0000 7f50bb99e700 1 -- 172.21.15.111:0/1957092901 >> 172.21.15.111:0/1957092901 conn(0x7f50b4098ee0 msgr2=0x7f50b406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:06.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.393+0000 7f50bb99e700 1 -- 172.21.15.111:0/1957092901 shutdown_connections 2023-11-28T12:59:06.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.393+0000 7f50bb99e700 1 -- 172.21.15.111:0/1957092901 wait complete. 2023-11-28T12:59:06.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.394+0000 7f50bb99e700 1 Processor -- start 2023-11-28T12:59:06.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.394+0000 7f50bb99e700 1 -- start start 2023-11-28T12:59:06.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.395+0000 7f50bb99e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b40b0d70 0x7f50b4155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.395+0000 7f50bb99e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f50b40b1140 0x7f50b41e62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.395+0000 7f50bb99e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f50b40b15b0 0x7f50b41e7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.395+0000 7f50bb99e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b41e9150 con 0x7f50b40b0d70 2023-11-28T12:59:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.395+0000 7f50bb99e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4156500 con 0x7f50b40b1140 2023-11-28T12:59:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.395+0000 7f50bb99e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b40c63c0 con 0x7f50b40b15b0 2023-11-28T12:59:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.395+0000 7f50b973a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b40b0d70 0x7f50b4155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.395+0000 7f50b973a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b40b0d70 0x7f50b4155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32888/0 (socket says 172.21.15.111:32888) 2023-11-28T12:59:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.395+0000 7f50b973a700 1 -- 172.21.15.111:0/3156079512 learned_addr learned my addr 172.21.15.111:0/3156079512 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.395+0000 7f50b8f39700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f50b40b1140 0x7f50b41e62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.395+0000 7f50b9f3b700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f50b40b15b0 0x7f50b41e7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.396+0000 7f50b973a700 1 -- 172.21.15.111:0/3156079512 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f50b40b15b0 msgr2=0x7f50b41e7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.396+0000 7f50b973a700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f50b40b15b0 0x7f50b41e7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.396+0000 7f50b973a700 1 -- 172.21.15.111:0/3156079512 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f50b40b1140 msgr2=0x7f50b41e62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.396+0000 7f50b973a700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f50b40b1140 0x7f50b41e62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.396+0000 7f50b973a700 1 -- 172.21.15.111:0/3156079512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b0009500 con 0x7f50b40b0d70 2023-11-28T12:59:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.396+0000 7f50b973a700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b40b0d70 0x7f50b4155a20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009130 tx=0x7f50b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.396+0000 7f50a9ffb700 1 -- 172.21.15.111:0/3156079512 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0003cb0 con 0x7f50b40b0d70 2023-11-28T12:59:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.396+0000 7f50a9ffb700 1 -- 172.21.15.111:0/3156079512 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b0003e30 con 0x7f50b40b0d70 2023-11-28T12:59:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.396+0000 7f50bb99e700 1 -- 172.21.15.111:0/3156079512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b40af340 con 0x7f50b40b0d70 2023-11-28T12:59:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.396+0000 7f50a9ffb700 1 -- 172.21.15.111:0/3156079512 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0011660 con 0x7f50b40b0d70 2023-11-28T12:59:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.396+0000 7f50bb99e700 1 -- 172.21.15.111:0/3156079512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b4161100 con 0x7f50b40b0d70 2023-11-28T12:59:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.397+0000 7f50bb99e700 1 -- 172.21.15.111:0/3156079512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b40cec00 con 0x7f50b40b0d70 2023-11-28T12:59:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.398+0000 7f50a9ffb700 1 -- 172.21.15.111:0/3156079512 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f50b002c070 con 0x7f50b40b0d70 2023-11-28T12:59:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.398+0000 7f50a9ffb700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f50980682f0 0x7f509806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.398+0000 7f50a9ffb700 1 -- 172.21.15.111:0/3156079512 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f50b0086f90 con 0x7f50b40b0d70 2023-11-28T12:59:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.398+0000 7f50b8f39700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f50980682f0 0x7f509806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.398+0000 7f50b8f39700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f50980682f0 0x7f509806a7a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f50b40c5370 tx=0x7f50a4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.402+0000 7f50a9ffb700 1 -- 172.21.15.111:0/3156079512 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f50b0053500 con 0x7f50b40b0d70 2023-11-28T12:59:06.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.587+0000 7f50bb99e700 1 -- 172.21.15.111:0/3156079512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f50b4067170 con 0x7f50b40b0d70 2023-11-28T12:59:06.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.588+0000 7f50a9ffb700 1 -- 172.21.15.111:0/3156079512 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f50b0056b20 con 0x7f50b40b0d70 2023-11-28T12:59:06.589 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:06.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.590+0000 7f50bb99e700 1 -- 172.21.15.111:0/3156079512 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f50980682f0 msgr2=0x7f509806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.590+0000 7f50bb99e700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f50980682f0 0x7f509806a7a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f50b40c5370 tx=0x7f50a4008040 comp rx=0 tx=0).stop 2023-11-28T12:59:06.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.590+0000 7f50bb99e700 1 -- 172.21.15.111:0/3156079512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b40b0d70 msgr2=0x7f50b4155a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.590+0000 7f50bb99e700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b40b0d70 0x7f50b4155a20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009130 tx=0x7f50b0004750 comp rx=0 tx=0).stop 2023-11-28T12:59:06.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.590+0000 7f50bb99e700 1 -- 172.21.15.111:0/3156079512 shutdown_connections 2023-11-28T12:59:06.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.590+0000 7f50bb99e700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f50b40b15b0 0x7f50b41e7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.590+0000 7f50bb99e700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b40b0d70 0x7f50b4155a20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.590+0000 7f50bb99e700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f50980682f0 0x7f509806a7a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.590+0000 7f50bb99e700 1 --2- 172.21.15.111:0/3156079512 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f50b40b1140 0x7f50b41e62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.590+0000 7f50bb99e700 1 -- 172.21.15.111:0/3156079512 >> 172.21.15.111:0/3156079512 conn(0x7f50b4098ee0 msgr2=0x7f50b406f380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:06.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.590+0000 7f50bb99e700 1 -- 172.21.15.111:0/3156079512 shutdown_connections 2023-11-28T12:59:06.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.590+0000 7f50bb99e700 1 -- 172.21.15.111:0/3156079512 wait complete. 2023-11-28T12:59:06.603 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T12:59:06.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.723+0000 7f09c0506700 1 Processor -- start 2023-11-28T12:59:06.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.723+0000 7f09c0506700 1 -- start start 2023-11-28T12:59:06.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.724+0000 7f09c0506700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f09b80b0d70 0x7f09b81604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.724+0000 7f09c0506700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09b80b1140 0x7f09b80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.724+0000 7f09c0506700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b815cd90 con 0x7f09b80cc550 2023-11-28T12:59:06.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.724+0000 7f09c0506700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b81613e0 con 0x7f09b80b1140 2023-11-28T12:59:06.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.724+0000 7f09c0506700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b8160b40 con 0x7f09b80b0d70 2023-11-28T12:59:06.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.724+0000 7f09bdaa1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09b80b1140 0x7f09b80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.724+0000 7f09bdaa1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09b80b1140 0x7f09b80ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32898/0 (socket says 172.21.15.111:32898) 2023-11-28T12:59:06.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.724+0000 7f09bdaa1700 1 -- 172.21.15.111:0/2979665371 learned_addr learned my addr 172.21.15.111:0/2979665371 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:06.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.725+0000 7f09be2a2700 1 --2- 172.21.15.111:0/2979665371 >> v2:172.21.15.149:3300/0 conn(0x7f09b80b0d70 0x7f09b81604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.725+0000 7f09bdaa1700 1 -- 172.21.15.111:0/2979665371 >> v2:172.21.15.149:3300/0 conn(0x7f09b80b0d70 msgr2=0x7f09b81604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:06.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.725+0000 7f09bdaa1700 1 --2- 172.21.15.111:0/2979665371 >> v2:172.21.15.149:3300/0 conn(0x7f09b80b0d70 0x7f09b81604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.725+0000 7f09bdaa1700 1 -- 172.21.15.111:0/2979665371 >> v1:172.21.15.149:6789/0 conn(0x7f09b80cc550 legacy=0x7f09b80af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.725+0000 7f09bdaa1700 1 -- 172.21.15.111:0/2979665371 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b8160e20 con 0x7f09b80b1140 2023-11-28T12:59:06.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.725+0000 7f09bdaa1700 1 --2- 172.21.15.111:0/2979665371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09b80b1140 0x7f09b80ce6c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f09b8071960 tx=0x7f09b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b7bbe274419301 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:06.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.725+0000 7f09bd2a0700 1 -- 172.21.15.111:0/2979665371 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b4011830 con 0x7f09b80b1140 2023-11-28T12:59:06.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.725+0000 7f09bd2a0700 1 -- 172.21.15.111:0/2979665371 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b4011990 con 0x7f09b80b1140 2023-11-28T12:59:06.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.725+0000 7f09bd2a0700 1 -- 172.21.15.111:0/2979665371 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b4011d20 con 0x7f09b80b1140 2023-11-28T12:59:06.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.726+0000 7f09c0506700 1 -- 172.21.15.111:0/2979665371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09b80b1140 msgr2=0x7f09b80ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.726+0000 7f09c0506700 1 --2- 172.21.15.111:0/2979665371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09b80b1140 0x7f09b80ce6c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f09b8071960 tx=0x7f09b4009bd0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.726+0000 7f09c0506700 1 -- 172.21.15.111:0/2979665371 shutdown_connections 2023-11-28T12:59:06.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.726+0000 7f09c0506700 1 --2- 172.21.15.111:0/2979665371 >> v2:172.21.15.149:3300/0 conn(0x7f09b80b0d70 0x7f09b81604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.726+0000 7f09c0506700 1 --2- 172.21.15.111:0/2979665371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09b80b1140 0x7f09b80ce6c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.726+0000 7f09c0506700 1 -- 172.21.15.111:0/2979665371 >> 172.21.15.111:0/2979665371 conn(0x7f09b8098ee0 msgr2=0x7f09b806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:06.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.726+0000 7f09c0506700 1 -- 172.21.15.111:0/2979665371 shutdown_connections 2023-11-28T12:59:06.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.726+0000 7f09c0506700 1 -- 172.21.15.111:0/2979665371 wait complete. 2023-11-28T12:59:06.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.727+0000 7f09c0506700 1 Processor -- start 2023-11-28T12:59:06.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.727+0000 7f09c0506700 1 -- start start 2023-11-28T12:59:06.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.727+0000 7f09c0506700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09b80b0d70 0x7f09b814c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.727+0000 7f09c0506700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09b80b1140 0x7f09b81e7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.727+0000 7f09c0506700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09b80cc550 0x7f09b815b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09c0506700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b81e8aa0 con 0x7f09b80cc550 2023-11-28T12:59:06.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09c0506700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b81e7e20 con 0x7f09b80b1140 2023-11-28T12:59:06.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09c0506700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b81e8100 con 0x7f09b80b0d70 2023-11-28T12:59:06.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09beaa3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09b80cc550 0x7f09b815b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09beaa3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09b80cc550 0x7f09b815b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32906/0 (socket says 172.21.15.111:32906) 2023-11-28T12:59:06.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09beaa3700 1 -- 172.21.15.111:0/1702318687 learned_addr learned my addr 172.21.15.111:0/1702318687 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:06.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09be2a2700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09b80b0d70 0x7f09b814c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09bdaa1700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09b80b1140 0x7f09b81e7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09beaa3700 1 -- 172.21.15.111:0/1702318687 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09b80b0d70 msgr2=0x7f09b814c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09beaa3700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09b80b0d70 0x7f09b814c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09beaa3700 1 -- 172.21.15.111:0/1702318687 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09b80b1140 msgr2=0x7f09b81e7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09beaa3700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09b80b1140 0x7f09b81e7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09beaa3700 1 -- 172.21.15.111:0/1702318687 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b4009550 con 0x7f09b80cc550 2023-11-28T12:59:06.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09beaa3700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09b80cc550 0x7f09b815b120 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f09b000cb50 tx=0x7f09b000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:06.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09aeffd700 1 -- 172.21.15.111:0/1702318687 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0003c00 con 0x7f09b80cc550 2023-11-28T12:59:06.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09aeffd700 1 -- 172.21.15.111:0/1702318687 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b0003d60 con 0x7f09b80cc550 2023-11-28T12:59:06.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09c0506700 1 -- 172.21.15.111:0/1702318687 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b80af340 con 0x7f09b80cc550 2023-11-28T12:59:06.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09aeffd700 1 -- 172.21.15.111:0/1702318687 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b000aa20 con 0x7f09b80cc550 2023-11-28T12:59:06.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.728+0000 7f09c0506700 1 -- 172.21.15.111:0/1702318687 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b8161100 con 0x7f09b80cc550 2023-11-28T12:59:06.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.729+0000 7f09c0506700 1 -- 172.21.15.111:0/1702318687 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b80cec00 con 0x7f09b80cc550 2023-11-28T12:59:06.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.729+0000 7f09aeffd700 1 -- 172.21.15.111:0/1702318687 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f09b0010050 con 0x7f09b80cc550 2023-11-28T12:59:06.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.730+0000 7f09aeffd700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f099c0682f0 0x7f099c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:06.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.730+0000 7f09aeffd700 1 -- 172.21.15.111:0/1702318687 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f09b0085c30 con 0x7f09b80cc550 2023-11-28T12:59:06.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.730+0000 7f09be2a2700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f099c0682f0 0x7f099c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:06.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.730+0000 7f09be2a2700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f099c0682f0 0x7f099c06a7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009bc0 tx=0x7f09a80091c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:06.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.734+0000 7f09aeffd700 1 -- 172.21.15.111:0/1702318687 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f09b00521a0 con 0x7f09b80cc550 2023-11-28T12:59:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.933+0000 7f09c0506700 1 -- 172.21.15.111:0/1702318687 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f09b81e8be0 con 0x7f09b80cc550 2023-11-28T12:59:06.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.933+0000 7f09aeffd700 1 -- 172.21.15.111:0/1702318687 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f09b81e8be0 con 0x7f09b80cc550 2023-11-28T12:59:06.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.935+0000 7f09c0506700 1 -- 172.21.15.111:0/1702318687 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f099c0682f0 msgr2=0x7f099c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.935+0000 7f09c0506700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f099c0682f0 0x7f099c06a7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009bc0 tx=0x7f09a80091c0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.935+0000 7f09c0506700 1 -- 172.21.15.111:0/1702318687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09b80cc550 msgr2=0x7f09b815b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:06.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.935+0000 7f09c0506700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09b80cc550 0x7f09b815b120 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f09b000cb50 tx=0x7f09b000ce60 comp rx=0 tx=0).stop 2023-11-28T12:59:06.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.935+0000 7f09c0506700 1 -- 172.21.15.111:0/1702318687 shutdown_connections 2023-11-28T12:59:06.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.935+0000 7f09c0506700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09b80cc550 0x7f09b815b120 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.935+0000 7f09c0506700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09b80b0d70 0x7f09b814c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.935+0000 7f09c0506700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f099c0682f0 0x7f099c06a7a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.935+0000 7f09c0506700 1 --2- 172.21.15.111:0/1702318687 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09b80b1140 0x7f09b81e7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:06.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.935+0000 7f09c0506700 1 -- 172.21.15.111:0/1702318687 >> 172.21.15.111:0/1702318687 conn(0x7f09b8098ee0 msgr2=0x7f09b80af760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:06.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.935+0000 7f09c0506700 1 -- 172.21.15.111:0/1702318687 shutdown_connections 2023-11-28T12:59:06.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:06.935+0000 7f09c0506700 1 -- 172.21.15.111:0/1702318687 wait complete. 2023-11-28T12:59:06.938 INFO:teuthology.orchestra.run.smithi111.stderr:listed 0 entries 2023-11-28T12:59:06.948 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:59:06.948 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:06.965 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:07.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.123+0000 7fe088c0c700 1 Processor -- start 2023-11-28T12:59:07.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.123+0000 7fe088c0c700 1 -- start start 2023-11-28T12:59:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.124+0000 7fe088c0c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe08415c5e0 0x7fe08406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.124+0000 7fe088c0c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe08406b720 0x7fe08406a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.124+0000 7fe088c0c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08415c9b0 con 0x7fe08406acf0 2023-11-28T12:59:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.124+0000 7fe088c0c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0840cce90 con 0x7fe08415c5e0 2023-11-28T12:59:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.124+0000 7fe088c0c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe084160e60 con 0x7fe08406b720 2023-11-28T12:59:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.124+0000 7fe08259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe08415c5e0 0x7fe08406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.124+0000 7fe08259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe08415c5e0 0x7fe08406a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32910/0 (socket says 172.21.15.111:32910) 2023-11-28T12:59:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.124+0000 7fe08259c700 1 -- 172.21.15.111:0/774955093 learned_addr learned my addr 172.21.15.111:0/774955093 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.125+0000 7fe08259c700 1 -- 172.21.15.111:0/774955093 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe08406b720 msgr2=0x7fe08406a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.125+0000 7fe081d9b700 1 --2- 172.21.15.111:0/774955093 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe08406b720 0x7fe08406a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.125+0000 7fe08259c700 1 --2- 172.21.15.111:0/774955093 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe08406b720 0x7fe08406a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.125+0000 7fe08259c700 1 -- 172.21.15.111:0/774955093 >> v1:172.21.15.149:6789/0 conn(0x7fe08406acf0 legacy=0x7fe0840712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:07.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.125+0000 7fe08259c700 1 -- 172.21.15.111:0/774955093 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0840cd0a0 con 0x7fe08415c5e0 2023-11-28T12:59:07.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.125+0000 7fe08259c700 1 --2- 172.21.15.111:0/774955093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe08415c5e0 0x7fe08406a270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe0780098f0 tx=0x7fe078009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b933e80418bde851 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:07.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.125+0000 7fe08159a700 1 -- 172.21.15.111:0/774955093 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe078004230 con 0x7fe08415c5e0 2023-11-28T12:59:07.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.125+0000 7fe08159a700 1 -- 172.21.15.111:0/774955093 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe078004390 con 0x7fe08415c5e0 2023-11-28T12:59:07.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.125+0000 7fe08159a700 1 -- 172.21.15.111:0/774955093 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0780036a0 con 0x7fe08415c5e0 2023-11-28T12:59:07.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.125+0000 7fe088c0c700 1 -- 172.21.15.111:0/774955093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe08415c5e0 msgr2=0x7fe08406a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.125+0000 7fe088c0c700 1 --2- 172.21.15.111:0/774955093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe08415c5e0 0x7fe08406a270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe0780098f0 tx=0x7fe078009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:07.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.126+0000 7fe088c0c700 1 -- 172.21.15.111:0/774955093 shutdown_connections 2023-11-28T12:59:07.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.126+0000 7fe088c0c700 1 --2- 172.21.15.111:0/774955093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe08415c5e0 0x7fe08406a270 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.126+0000 7fe088c0c700 1 --2- 172.21.15.111:0/774955093 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe08406b720 0x7fe08406a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.126+0000 7fe088c0c700 1 -- 172.21.15.111:0/774955093 >> 172.21.15.111:0/774955093 conn(0x7fe0840b0db0 msgr2=0x7fe0840b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:07.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.126+0000 7fe088c0c700 1 -- 172.21.15.111:0/774955093 shutdown_connections 2023-11-28T12:59:07.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.126+0000 7fe088c0c700 1 -- 172.21.15.111:0/774955093 wait complete. 2023-11-28T12:59:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.127+0000 7fe088c0c700 1 Processor -- start 2023-11-28T12:59:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.127+0000 7fe088c0c700 1 -- start start 2023-11-28T12:59:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe088c0c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe08406acf0 0x7fe08415eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe088c0c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe08406b720 0x7fe084145590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe088c0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe08415c5e0 0x7fe0840ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe088c0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08415f0c0 con 0x7fe08415c5e0 2023-11-28T12:59:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe088c0c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840b2190 con 0x7fe08406acf0 2023-11-28T12:59:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe088c0c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08415f2d0 con 0x7fe08406b720 2023-11-28T12:59:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe082d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe08415c5e0 0x7fe0840ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe082d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe08415c5e0 0x7fe0840ad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32920/0 (socket says 172.21.15.111:32920) 2023-11-28T12:59:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe082d9d700 1 -- 172.21.15.111:0/476175079 learned_addr learned my addr 172.21.15.111:0/476175079 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe08259c700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe08406acf0 0x7fe08415eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe081d9b700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe08406b720 0x7fe084145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe082d9d700 1 -- 172.21.15.111:0/476175079 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe08406b720 msgr2=0x7fe084145590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe082d9d700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe08406b720 0x7fe084145590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe082d9d700 1 -- 172.21.15.111:0/476175079 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe08406acf0 msgr2=0x7fe08415eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe082d9d700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe08406acf0 0x7fe08415eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe082d9d700 1 -- 172.21.15.111:0/476175079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe078009550 con 0x7fe08415c5e0 2023-11-28T12:59:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe081d9b700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe08406b720 0x7fe084145590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe08259c700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe08406acf0 0x7fe08415eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe082d9d700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe08415c5e0 0x7fe0840ad2a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00cc40 tx=0x7fe07c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe072ffd700 1 -- 172.21.15.111:0/476175079 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c00aba0 con 0x7fe08415c5e0 2023-11-28T12:59:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe072ffd700 1 -- 172.21.15.111:0/476175079 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07c00ad00 con 0x7fe08415c5e0 2023-11-28T12:59:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe072ffd700 1 -- 172.21.15.111:0/476175079 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c012660 con 0x7fe08415c5e0 2023-11-28T12:59:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe088c0c700 1 -- 172.21.15.111:0/476175079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0840cd660 con 0x7fe08415c5e0 2023-11-28T12:59:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.128+0000 7fe088c0c700 1 -- 172.21.15.111:0/476175079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0840cd380 con 0x7fe08415c5e0 2023-11-28T12:59:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.129+0000 7fe088c0c700 1 -- 172.21.15.111:0/476175079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe084160af0 con 0x7fe08415c5e0 2023-11-28T12:59:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.130+0000 7fe072ffd700 1 -- 172.21.15.111:0/476175079 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe07c022070 con 0x7fe08415c5e0 2023-11-28T12:59:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.130+0000 7fe072ffd700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe0640682f0 0x7fe06406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.130+0000 7fe072ffd700 1 -- 172.21.15.111:0/476175079 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe07c086630 con 0x7fe08415c5e0 2023-11-28T12:59:07.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.130+0000 7fe08259c700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe0640682f0 0x7fe06406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.130+0000 7fe08259c700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe0640682f0 0x7fe06406a7a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe078009130 tx=0x7fe078008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:07.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.134+0000 7fe072ffd700 1 -- 172.21.15.111:0/476175079 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe07c052b10 con 0x7fe08415c5e0 2023-11-28T12:59:07.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.319+0000 7fe088c0c700 1 -- 172.21.15.111:0/476175079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe084067170 con 0x7fe08415c5e0 2023-11-28T12:59:07.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.320+0000 7fe072ffd700 1 -- 172.21.15.111:0/476175079 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19796 (secure 0 0 0) 0x7fe07c016070 con 0x7fe08415c5e0 2023-11-28T12:59:07.322 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:07.322 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:00.053943+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:07.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.322+0000 7fe088c0c700 1 -- 172.21.15.111:0/476175079 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe0640682f0 msgr2=0x7fe06406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.322+0000 7fe088c0c700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe0640682f0 0x7fe06406a7a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe078009130 tx=0x7fe078008030 comp rx=0 tx=0).stop 2023-11-28T12:59:07.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.322+0000 7fe088c0c700 1 -- 172.21.15.111:0/476175079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe08415c5e0 msgr2=0x7fe0840ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.322+0000 7fe088c0c700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe08415c5e0 0x7fe0840ad2a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00cc40 tx=0x7fe07c00a060 comp rx=0 tx=0).stop 2023-11-28T12:59:07.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.322+0000 7fe088c0c700 1 -- 172.21.15.111:0/476175079 shutdown_connections 2023-11-28T12:59:07.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.322+0000 7fe088c0c700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe08415c5e0 0x7fe0840ad2a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.322+0000 7fe088c0c700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe0640682f0 0x7fe06406a7a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.322+0000 7fe088c0c700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe08406acf0 0x7fe08415eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.322+0000 7fe088c0c700 1 --2- 172.21.15.111:0/476175079 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe08406b720 0x7fe084145590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.322+0000 7fe088c0c700 1 -- 172.21.15.111:0/476175079 >> 172.21.15.111:0/476175079 conn(0x7fe0840b0db0 msgr2=0x7fe08406f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:07.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.322+0000 7fe088c0c700 1 -- 172.21.15.111:0/476175079 shutdown_connections 2023-11-28T12:59:07.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.322+0000 7fe088c0c700 1 -- 172.21.15.111:0/476175079 wait complete. 2023-11-28T12:59:07.336 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T12:58:53.275206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T12:58:54.588499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T12:58:57.666033+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T12:59:07.336 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:59:07.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.457+0000 7f1b6b8b9700 1 Processor -- start 2023-11-28T12:59:07.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.457+0000 7f1b6b8b9700 1 -- start start 2023-11-28T12:59:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b6b8b9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1b640cef60 0x7f1b640ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b6b8b9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b640b0160 0x7f1b640adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b6b8b9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b640cc390 con 0x7f1b6415a5d0 2023-11-28T12:59:07.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b6b8b9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b640cc1b0 con 0x7f1b640cef60 2023-11-28T12:59:07.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b6b8b9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b640cc670 con 0x7f1b640b0160 2023-11-28T12:59:07.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b68e54700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1b640cef60 0x7f1b640ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b69655700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1b6415a5d0 0x7f1b640ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33702/0 (socket says 172.21.15.111:33702) 2023-11-28T12:59:07.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b69655700 1 -- 172.21.15.111:0/2513627033 learned_addr learned my addr 172.21.15.111:0/2513627033 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:07.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b68e54700 1 -- 172.21.15.111:0/2513627033 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b640b0160 msgr2=0x7f1b640adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:07.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b69e56700 1 --2- 172.21.15.111:0/2513627033 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b640b0160 0x7f1b640adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b68e54700 1 --2- 172.21.15.111:0/2513627033 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b640b0160 0x7f1b640adde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b68e54700 1 -- 172.21.15.111:0/2513627033 >> v1:172.21.15.111:6789/0 conn(0x7f1b6415a5d0 legacy=0x7f1b640ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b5bfff700 1 -- 172.21.15.111:0/2513627033 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843587531 0 0) 0x7f1b640cc390 con 0x7f1b6415a5d0 2023-11-28T12:59:07.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b68e54700 1 -- 172.21.15.111:0/2513627033 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b640cc950 con 0x7f1b640cef60 2023-11-28T12:59:07.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b68e54700 1 --2- 172.21.15.111:0/2513627033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b640cef60 0x7f1b640ad8a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1b60009840 tx=0x7f1b60009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3d665c812355fe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:07.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b5bfff700 1 -- 172.21.15.111:0/2513627033 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60011830 con 0x7f1b640cef60 2023-11-28T12:59:07.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b5bfff700 1 -- 172.21.15.111:0/2513627033 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b60011990 con 0x7f1b640cef60 2023-11-28T12:59:07.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.458+0000 7f1b5bfff700 1 -- 172.21.15.111:0/2513627033 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60011d20 con 0x7f1b640cef60 2023-11-28T12:59:07.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.459+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/2513627033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b640cef60 msgr2=0x7f1b640ad8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.459+0000 7f1b6b8b9700 1 --2- 172.21.15.111:0/2513627033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b640cef60 0x7f1b640ad8a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1b60009840 tx=0x7f1b60009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:07.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.459+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/2513627033 shutdown_connections 2023-11-28T12:59:07.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.459+0000 7f1b6b8b9700 1 --2- 172.21.15.111:0/2513627033 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b640b0160 0x7f1b640adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.459+0000 7f1b6b8b9700 1 --2- 172.21.15.111:0/2513627033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b640cef60 0x7f1b640ad8a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.459+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/2513627033 >> 172.21.15.111:0/2513627033 conn(0x7f1b6409a040 msgr2=0x7f1b640d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:07.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.459+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/2513627033 shutdown_connections 2023-11-28T12:59:07.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.459+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/2513627033 wait complete. 2023-11-28T12:59:07.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.460+0000 7f1b6b8b9700 1 Processor -- start 2023-11-28T12:59:07.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.460+0000 7f1b6b8b9700 1 -- start start 2023-11-28T12:59:07.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b6b8b9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b640b0160 0x7f1b641f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b6b8b9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b640cef60 0x7f1b640c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b6b8b9700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b6415a5d0 0x7f1b64150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b6b8b9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b640c6370 con 0x7f1b640cef60 2023-11-28T12:59:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b6b8b9700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b640c75c0 con 0x7f1b6415a5d0 2023-11-28T12:59:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b6b8b9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b640c52d0 con 0x7f1b640b0160 2023-11-28T12:59:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b68e54700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b640cef60 0x7f1b640c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b68e54700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b640cef60 0x7f1b640c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32932/0 (socket says 172.21.15.111:32932) 2023-11-28T12:59:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b68e54700 1 -- 172.21.15.111:0/1571274530 learned_addr learned my addr 172.21.15.111:0/1571274530 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b69e56700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b6415a5d0 0x7f1b64150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b69655700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b640b0160 0x7f1b641f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b68e54700 1 -- 172.21.15.111:0/1571274530 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b640b0160 msgr2=0x7f1b641f4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b68e54700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b640b0160 0x7f1b641f4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b68e54700 1 -- 172.21.15.111:0/1571274530 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b6415a5d0 msgr2=0x7f1b64150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b68e54700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b6415a5d0 0x7f1b64150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b68e54700 1 -- 172.21.15.111:0/1571274530 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b600094a0 con 0x7f1b640cef60 2023-11-28T12:59:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b69e56700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b6415a5d0 0x7f1b64150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.461+0000 7f1b68e54700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b640cef60 0x7f1b640c7080 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1b600090b0 tx=0x7f1b60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.462+0000 7f1b59ffb700 1 -- 172.21.15.111:0/1571274530 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b600038a0 con 0x7f1b640cef60 2023-11-28T12:59:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.462+0000 7f1b59ffb700 1 -- 172.21.15.111:0/1571274530 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6001f070 con 0x7f1b640cef60 2023-11-28T12:59:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.462+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/1571274530 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b640ccf10 con 0x7f1b640cef60 2023-11-28T12:59:07.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.462+0000 7f1b59ffb700 1 -- 172.21.15.111:0/1571274530 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60003c60 con 0x7f1b640cef60 2023-11-28T12:59:07.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.462+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/1571274530 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b640ccc30 con 0x7f1b640cef60 2023-11-28T12:59:07.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.463+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/1571274530 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6415a9a0 con 0x7f1b640cef60 2023-11-28T12:59:07.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.463+0000 7f1b59ffb700 1 -- 172.21.15.111:0/1571274530 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1b6000f050 con 0x7f1b640cef60 2023-11-28T12:59:07.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.463+0000 7f1b59ffb700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b480682f0 0x7f1b4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.464+0000 7f1b59ffb700 1 -- 172.21.15.111:0/1571274530 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f1b60021070 con 0x7f1b640cef60 2023-11-28T12:59:07.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.464+0000 7f1b69655700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b480682f0 0x7f1b4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.464+0000 7f1b69655700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b480682f0 0x7f1b4806a7a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1b54007fd0 tx=0x7f1b54009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:07.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.468+0000 7f1b59ffb700 1 -- 172.21.15.111:0/1571274530 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1b60053fc0 con 0x7f1b640cef60 2023-11-28T12:59:07.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.648+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/1571274530 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1b64067170 con 0x7f1b640cef60 2023-11-28T12:59:07.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.648+0000 7f1b59ffb700 1 -- 172.21.15.111:0/1571274530 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1b60015020 con 0x7f1b640cef60 2023-11-28T12:59:07.650 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:59:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.650+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/1571274530 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b480682f0 msgr2=0x7f1b4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.650+0000 7f1b6b8b9700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b480682f0 0x7f1b4806a7a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1b54007fd0 tx=0x7f1b54009040 comp rx=0 tx=0).stop 2023-11-28T12:59:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.650+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/1571274530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b640cef60 msgr2=0x7f1b640c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.650+0000 7f1b6b8b9700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b640cef60 0x7f1b640c7080 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1b600090b0 tx=0x7f1b60004750 comp rx=0 tx=0).stop 2023-11-28T12:59:07.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.650+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/1571274530 shutdown_connections 2023-11-28T12:59:07.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.650+0000 7f1b6b8b9700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b480682f0 0x7f1b4806a7a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.650+0000 7f1b6b8b9700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b6415a5d0 0x7f1b64150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.650+0000 7f1b6b8b9700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b640b0160 0x7f1b641f4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.650+0000 7f1b6b8b9700 1 --2- 172.21.15.111:0/1571274530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b640cef60 0x7f1b640c7080 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.650+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/1571274530 >> 172.21.15.111:0/1571274530 conn(0x7f1b6409a040 msgr2=0x7f1b64157c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:07.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.650+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/1571274530 shutdown_connections 2023-11-28T12:59:07.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.650+0000 7f1b6b8b9700 1 -- 172.21.15.111:0/1571274530 wait complete. 2023-11-28T12:59:07.663 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:59:07.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.786+0000 7f6efa001700 1 Processor -- start 2023-11-28T12:59:07.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.786+0000 7f6efa001700 1 -- start start 2023-11-28T12:59:07.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.787+0000 7f6efa001700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ef40b0d70 0x7f6ef41604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.787+0000 7f6efa001700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef415cd90 con 0x7f6ef40b1140 2023-11-28T12:59:07.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.787+0000 7f6efa001700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef41613e0 con 0x7f6ef40b15b0 2023-11-28T12:59:07.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.787+0000 7f6efa001700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef40cedd0 con 0x7f6ef40b0d70 2023-11-28T12:59:07.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.787+0000 7f6ef2ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6ef40b1140 0x7f6ef40ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33716/0 (socket says 172.21.15.111:33716) 2023-11-28T12:59:07.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.787+0000 7f6ef2ffd700 1 -- 172.21.15.111:0/4237751857 learned_addr learned my addr 172.21.15.111:0/4237751857 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:07.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474064530 0 0) 0x7f6ee4001110 con 0x7f6ef40b1140 2023-11-28T12:59:07.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef37fe700 1 --2- 172.21.15.111:0/4237751857 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ef40b0d70 0x7f6ef41604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6edc0035b0 con 0x7f6ef40b1140 2023-11-28T12:59:07.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 389353071 0 0) 0x7f6edc0035b0 con 0x7f6ef40b1140 2023-11-28T12:59:07.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ef40b0d70 msgr2=0x7f6ef41604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:07.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 --2- 172.21.15.111:0/4237751857 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ef40b0d70 0x7f6ef41604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 >> v1:172.21.15.149:6789/0 conn(0x7f6ef40b15b0 legacy=0x7f6ef40cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef4160b40 con 0x7f6ef40b1140 2023-11-28T12:59:07.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f6ee4001800 con 0x7f6ef40b1140 2023-11-28T12:59:07.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 >> v1:172.21.15.111:6789/0 conn(0x7f6ef40b1140 legacy=0x7f6ef40ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 --2- 172.21.15.111:0/4237751857 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6edc003a30 0x7f6edc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 --2- 172.21.15.111:0/4237751857 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ef40b0d70 0x7f6edc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 --2- 172.21.15.111:0/4237751857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6edc004d90 0x7f6edc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef40cedd0 con 0x7f6edc004d90 2023-11-28T12:59:07.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc00bac0 con 0x7f6edc003a30 2023-11-28T12:59:07.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc00bc50 con 0x7f6ef40b0d70 2023-11-28T12:59:07.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef3fff700 1 --2- 172.21.15.111:0/4237751857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6edc004d90 0x7f6edc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef2ffd700 1 --2- 172.21.15.111:0/4237751857 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ef40b0d70 0x7f6edc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef37fe700 1 --2- 172.21.15.111:0/4237751857 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6edc003a30 0x7f6edc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef3fff700 1 -- 172.21.15.111:0/4237751857 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ef40b0d70 msgr2=0x7f6edc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:07.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef3fff700 1 --2- 172.21.15.111:0/4237751857 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ef40b0d70 0x7f6edc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef3fff700 1 -- 172.21.15.111:0/4237751857 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6edc003a30 msgr2=0x7f6edc003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:07.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef3fff700 1 --2- 172.21.15.111:0/4237751857 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6edc003a30 0x7f6edc003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef3fff700 1 -- 172.21.15.111:0/4237751857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc0035b0 con 0x7f6edc004d90 2023-11-28T12:59:07.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.788+0000 7f6ef3fff700 1 --2- 172.21.15.111:0/4237751857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6edc004d90 0x7f6edc005200 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4071fd0 tx=0x7f6edc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac96210eb66c1b2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:07.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.789+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec001db0 con 0x7f6edc004d90 2023-11-28T12:59:07.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.789+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eec00cc70 con 0x7f6edc004d90 2023-11-28T12:59:07.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.789+0000 7f6ef27fc700 1 -- 172.21.15.111:0/4237751857 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec003a00 con 0x7f6edc004d90 2023-11-28T12:59:07.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.789+0000 7f6efa001700 1 -- 172.21.15.111:0/4237751857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6edc004d90 msgr2=0x7f6edc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.789+0000 7f6efa001700 1 --2- 172.21.15.111:0/4237751857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6edc004d90 0x7f6edc005200 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4071fd0 tx=0x7f6edc005880 comp rx=0 tx=0).stop 2023-11-28T12:59:07.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.789+0000 7f6ef3fff700 1 -- 172.21.15.111:0/4237751857 reap_dead start 2023-11-28T12:59:07.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.789+0000 7f6efa001700 1 -- 172.21.15.111:0/4237751857 shutdown_connections 2023-11-28T12:59:07.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.789+0000 7f6efa001700 1 -- 172.21.15.111:0/4237751857 >> 172.21.15.111:0/4237751857 conn(0x7f6ef4098ee0 msgr2=0x7f6ef406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:07.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.789+0000 7f6efa001700 1 -- 172.21.15.111:0/4237751857 shutdown_connections 2023-11-28T12:59:07.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.789+0000 7f6efa001700 1 -- 172.21.15.111:0/4237751857 wait complete. 2023-11-28T12:59:07.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.791+0000 7f6efa001700 1 Processor -- start 2023-11-28T12:59:07.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.791+0000 7f6efa001700 1 -- start start 2023-11-28T12:59:07.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.791+0000 7f6efa001700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ef415bc10 0x7f6ef40b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.791+0000 7f6efa001700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ef415bfe0 0x7f6ef415dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6efa001700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ef40c6b80 0x7f6ef40c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6efa001700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef415e2d0 con 0x7f6ef40c6b80 2023-11-28T12:59:07.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6efa001700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef415e500 con 0x7f6ef415bfe0 2023-11-28T12:59:07.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6efa001700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef415e7e0 con 0x7f6ef415bc10 2023-11-28T12:59:07.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6ef3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ef40c6b80 0x7f6ef40c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6ef3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ef40c6b80 0x7f6ef40c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32954/0 (socket says 172.21.15.111:32954) 2023-11-28T12:59:07.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6ef3fff700 1 -- 172.21.15.111:0/2508055164 learned_addr learned my addr 172.21.15.111:0/2508055164 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:07.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6ef37fe700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ef415bc10 0x7f6ef40b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6ef2ffd700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ef415bfe0 0x7f6ef415dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6ef3fff700 1 -- 172.21.15.111:0/2508055164 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ef415bc10 msgr2=0x7f6ef40b55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6ef3fff700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ef415bc10 0x7f6ef40b55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6ef3fff700 1 -- 172.21.15.111:0/2508055164 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ef415bfe0 msgr2=0x7f6ef415dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6ef3fff700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ef415bfe0 0x7f6ef415dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6ef3fff700 1 -- 172.21.15.111:0/2508055164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef4161100 con 0x7f6ef40c6b80 2023-11-28T12:59:07.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6ef2ffd700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ef415bfe0 0x7f6ef415dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:07.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6ef3fff700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ef40c6b80 0x7f6ef40c7000 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6eec005040 tx=0x7f6eec0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:07.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6edbfff700 1 -- 172.21.15.111:0/2508055164 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec003e10 con 0x7f6ef40c6b80 2023-11-28T12:59:07.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6edbfff700 1 -- 172.21.15.111:0/2508055164 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eec0050d0 con 0x7f6ef40c6b80 2023-11-28T12:59:07.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6efa001700 1 -- 172.21.15.111:0/2508055164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef4160e20 con 0x7f6ef40c6b80 2023-11-28T12:59:07.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6edbfff700 1 -- 172.21.15.111:0/2508055164 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec0054a0 con 0x7f6ef40c6b80 2023-11-28T12:59:07.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.792+0000 7f6efa001700 1 -- 172.21.15.111:0/2508055164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef40af340 con 0x7f6ef40c6b80 2023-11-28T12:59:07.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.793+0000 7f6efa001700 1 -- 172.21.15.111:0/2508055164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef4067170 con 0x7f6ef40c6b80 2023-11-28T12:59:07.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.794+0000 7f6edbfff700 1 -- 172.21.15.111:0/2508055164 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6eec010050 con 0x7f6ef40c6b80 2023-11-28T12:59:07.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.794+0000 7f6edbfff700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ee00682f0 0x7f6ee006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:07.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.794+0000 7f6edbfff700 1 -- 172.21.15.111:0/2508055164 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6eec086f20 con 0x7f6ef40c6b80 2023-11-28T12:59:07.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.794+0000 7f6ef37fe700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ee00682f0 0x7f6ee006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:07.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.795+0000 7f6ef37fe700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ee00682f0 0x7f6ee006a7a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6ef41f5e70 tx=0x7f6ee8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:07.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.798+0000 7f6edbfff700 1 -- 172.21.15.111:0/2508055164 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6eec053490 con 0x7f6ef40c6b80 2023-11-28T12:59:07.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.979+0000 7f6efa001700 1 -- 172.21.15.111:0/2508055164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6ef406d9a0 con 0x7f6ef40c6b80 2023-11-28T12:59:07.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.980+0000 7f6edbfff700 1 -- 172.21.15.111:0/2508055164 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6eec017020 con 0x7f6ef40c6b80 2023-11-28T12:59:07.982 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:59:07.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.982+0000 7f6efa001700 1 -- 172.21.15.111:0/2508055164 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ee00682f0 msgr2=0x7f6ee006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.982+0000 7f6efa001700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ee00682f0 0x7f6ee006a7a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6ef41f5e70 tx=0x7f6ee8008040 comp rx=0 tx=0).stop 2023-11-28T12:59:07.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.982+0000 7f6efa001700 1 -- 172.21.15.111:0/2508055164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ef40c6b80 msgr2=0x7f6ef40c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:07.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.982+0000 7f6efa001700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ef40c6b80 0x7f6ef40c7000 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6eec005040 tx=0x7f6eec0014a0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.982+0000 7f6efa001700 1 -- 172.21.15.111:0/2508055164 shutdown_connections 2023-11-28T12:59:07.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.982+0000 7f6efa001700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ef40c6b80 0x7f6ef40c7000 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.982+0000 7f6efa001700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ef415bc10 0x7f6ef40b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.982+0000 7f6efa001700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ee00682f0 0x7f6ee006a7a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.982+0000 7f6efa001700 1 --2- 172.21.15.111:0/2508055164 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ef415bfe0 0x7f6ef415dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:07.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.982+0000 7f6efa001700 1 -- 172.21.15.111:0/2508055164 >> 172.21.15.111:0/2508055164 conn(0x7f6ef4098ee0 msgr2=0x7f6ef40ac670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:07.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.982+0000 7f6efa001700 1 -- 172.21.15.111:0/2508055164 shutdown_connections 2023-11-28T12:59:07.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:07.982+0000 7f6efa001700 1 -- 172.21.15.111:0/2508055164 wait complete. 2023-11-28T12:59:07.995 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T12:59:08.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.116+0000 7f16688fd700 1 Processor -- start 2023-11-28T12:59:08.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.116+0000 7f16688fd700 1 -- start start 2023-11-28T12:59:08.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f16688fd700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f166406ba90 0x7f16640cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f16688fd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640cb810 con 0x7f16640d0490 2023-11-28T12:59:08.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f16688fd700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640cc650 con 0x7f166415c7b0 2023-11-28T12:59:08.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f16688fd700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16640cba40 con 0x7f166406ba90 2023-11-28T12:59:08.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f1661d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f16640d0490 0x7f166406b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33738/0 (socket says 172.21.15.111:33738) 2023-11-28T12:59:08.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f1661d9b700 1 -- 172.21.15.111:0/2406508148 learned_addr learned my addr 172.21.15.111:0/2406508148 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:08.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f1662d9d700 1 --2- 172.21.15.111:0/2406508148 >> v2:172.21.15.149:3300/0 conn(0x7f166406ba90 0x7f16640cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817571977 0 0) 0x7f16640cb810 con 0x7f16640d0490 2023-11-28T12:59:08.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f164c0035b0 con 0x7f16640d0490 2023-11-28T12:59:08.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935538488 0 0) 0x7f16640cc650 con 0x7f166415c7b0 2023-11-28T12:59:08.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16640cb810 con 0x7f166415c7b0 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3157179762 0 0) 0x7f164c0035b0 con 0x7f16640d0490 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 >> v2:172.21.15.149:3300/0 conn(0x7f166406ba90 msgr2=0x7f16640cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f166159a700 1 --2- 172.21.15.111:0/2406508148 >> v2:172.21.15.149:3300/0 conn(0x7f166406ba90 0x7f16640cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 >> v1:172.21.15.149:6789/0 conn(0x7f166415c7b0 legacy=0x7f16640ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16640cbd20 con 0x7f16640d0490 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f1658001670 con 0x7f16640d0490 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 >> v1:172.21.15.111:6789/0 conn(0x7f16640d0490 legacy=0x7f166406b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f166159a700 1 --2- 172.21.15.111:0/2406508148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f164c003be0 0x7f164c003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.117+0000 7f166159a700 1 --2- 172.21.15.111:0/2406508148 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f164c004850 0x7f164c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166159a700 1 --2- 172.21.15.111:0/2406508148 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f164c005250 0x7f164c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640cba40 con 0x7f164c003be0 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c00c160 con 0x7f164c004850 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164c00c320 con 0x7f164c005250 2023-11-28T12:59:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166259c700 1 --2- 172.21.15.111:0/2406508148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f164c003be0 0x7f164c003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f1661d9b700 1 --2- 172.21.15.111:0/2406508148 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f164c004850 0x7f164c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f1662d9d700 1 --2- 172.21.15.111:0/2406508148 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f164c005250 0x7f164c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166259c700 1 -- 172.21.15.111:0/2406508148 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f164c005250 msgr2=0x7f164c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166259c700 1 --2- 172.21.15.111:0/2406508148 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f164c005250 0x7f164c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166259c700 1 -- 172.21.15.111:0/2406508148 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f164c004850 msgr2=0x7f164c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166259c700 1 --2- 172.21.15.111:0/2406508148 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f164c004850 0x7f164c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166259c700 1 -- 172.21.15.111:0/2406508148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c0035b0 con 0x7f164c003be0 2023-11-28T12:59:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f1662d9d700 1 -- 172.21.15.111:0/2406508148 reap_dead start 2023-11-28T12:59:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166259c700 1 --2- 172.21.15.111:0/2406508148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f164c003be0 0x7f164c003fd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1654001970 tx=0x7f164c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb553c2cc496aa1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654001460 con 0x7f164c003be0 2023-11-28T12:59:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16540052d0 con 0x7f164c003be0 2023-11-28T12:59:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.118+0000 7f166159a700 1 -- 172.21.15.111:0/2406508148 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654001da0 con 0x7f164c003be0 2023-11-28T12:59:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.119+0000 7f16688fd700 1 -- 172.21.15.111:0/2406508148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f164c003be0 msgr2=0x7f164c003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.119+0000 7f16688fd700 1 --2- 172.21.15.111:0/2406508148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f164c003be0 0x7f164c003fd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1654001970 tx=0x7f164c005880 comp rx=0 tx=0).stop 2023-11-28T12:59:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.119+0000 7f16688fd700 1 -- 172.21.15.111:0/2406508148 shutdown_connections 2023-11-28T12:59:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.119+0000 7f16688fd700 1 --2- 172.21.15.111:0/2406508148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f164c003be0 0x7f164c003fd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.119+0000 7f16688fd700 1 -- 172.21.15.111:0/2406508148 >> 172.21.15.111:0/2406508148 conn(0x7f166409a040 msgr2=0x7f16640d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.119+0000 7f16688fd700 1 -- 172.21.15.111:0/2406508148 shutdown_connections 2023-11-28T12:59:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.119+0000 7f16688fd700 1 -- 172.21.15.111:0/2406508148 wait complete. 2023-11-28T12:59:08.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.120+0000 7f16688fd700 1 Processor -- start 2023-11-28T12:59:08.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.120+0000 7f16688fd700 1 -- start start 2023-11-28T12:59:08.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f16688fd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f164c003be0 0x7f16640b2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f16688fd700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f166415ce90 0x7f16640b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f16688fd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f166415e4a0 0x7f166415e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f16688fd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16641576d0 con 0x7f164c003be0 2023-11-28T12:59:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f16688fd700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166415e010 con 0x7f166415ce90 2023-11-28T12:59:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f16688fd700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166415d400 con 0x7f166415e4a0 2023-11-28T12:59:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f166259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f164c003be0 0x7f16640b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f166259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f164c003be0 0x7f16640b2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32970/0 (socket says 172.21.15.111:32970) 2023-11-28T12:59:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f166259c700 1 -- 172.21.15.111:0/4248709401 learned_addr learned my addr 172.21.15.111:0/4248709401 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f1661d9b700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f166415ce90 0x7f16640b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f1662d9d700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f166415e4a0 0x7f166415e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f166259c700 1 -- 172.21.15.111:0/4248709401 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f166415e4a0 msgr2=0x7f166415e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f166259c700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f166415e4a0 0x7f166415e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f166259c700 1 -- 172.21.15.111:0/4248709401 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f166415ce90 msgr2=0x7f16640b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f166259c700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f166415ce90 0x7f16640b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f166259c700 1 -- 172.21.15.111:0/4248709401 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16640cc2e0 con 0x7f164c003be0 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.121+0000 7f166259c700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f164c003be0 0x7f16640b2630 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f164c00c130 tx=0x7f1654001320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.122+0000 7f1652ffd700 1 -- 172.21.15.111:0/4248709401 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654001da0 con 0x7f164c003be0 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.122+0000 7f1652ffd700 1 -- 172.21.15.111:0/4248709401 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1654003d10 con 0x7f164c003be0 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.122+0000 7f1652ffd700 1 -- 172.21.15.111:0/4248709401 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16540068b0 con 0x7f164c003be0 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.122+0000 7f16688fd700 1 -- 172.21.15.111:0/4248709401 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16640cc000 con 0x7f164c003be0 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.122+0000 7f16688fd700 1 -- 172.21.15.111:0/4248709401 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166415cb80 con 0x7f164c003be0 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.122+0000 7f16688fd700 1 -- 172.21.15.111:0/4248709401 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166415e2f0 con 0x7f164c003be0 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.123+0000 7f1652ffd700 1 -- 172.21.15.111:0/4248709401 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f165402d070 con 0x7f164c003be0 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.123+0000 7f1652ffd700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f16440682f0 0x7f164406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.124+0000 7f1652ffd700 1 -- 172.21.15.111:0/4248709401 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f1654087170 con 0x7f164c003be0 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.124+0000 7f1661d9b700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f16440682f0 0x7f164406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.124+0000 7f1661d9b700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f16440682f0 0x7f164406a7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1658000c00 tx=0x7f1658001470 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:08.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.127+0000 7f1652ffd700 1 -- 172.21.15.111:0/4248709401 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1654057160 con 0x7f164c003be0 2023-11-28T12:59:08.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.307+0000 7f16688fd700 1 -- 172.21.15.111:0/4248709401 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1664067170 con 0x7f164c003be0 2023-11-28T12:59:08.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.307+0000 7f1652ffd700 1 -- 172.21.15.111:0/4248709401 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1654056d50 con 0x7f164c003be0 2023-11-28T12:59:08.309 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:08.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.309+0000 7f16688fd700 1 -- 172.21.15.111:0/4248709401 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f16440682f0 msgr2=0x7f164406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.309+0000 7f16688fd700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f16440682f0 0x7f164406a7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1658000c00 tx=0x7f1658001470 comp rx=0 tx=0).stop 2023-11-28T12:59:08.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.309+0000 7f16688fd700 1 -- 172.21.15.111:0/4248709401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f164c003be0 msgr2=0x7f16640b2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.309+0000 7f16688fd700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f164c003be0 0x7f16640b2630 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f164c00c130 tx=0x7f1654001320 comp rx=0 tx=0).stop 2023-11-28T12:59:08.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.309+0000 7f16688fd700 1 -- 172.21.15.111:0/4248709401 shutdown_connections 2023-11-28T12:59:08.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.309+0000 7f16688fd700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f166415e4a0 0x7f166415e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.309+0000 7f16688fd700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f164c003be0 0x7f16640b2630 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.309+0000 7f16688fd700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f16440682f0 0x7f164406a7a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.309+0000 7f16688fd700 1 --2- 172.21.15.111:0/4248709401 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f166415ce90 0x7f16640b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.309+0000 7f16688fd700 1 -- 172.21.15.111:0/4248709401 >> 172.21.15.111:0/4248709401 conn(0x7f166409a040 msgr2=0x7f16640d09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:08.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.309+0000 7f16688fd700 1 -- 172.21.15.111:0/4248709401 shutdown_connections 2023-11-28T12:59:08.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.309+0000 7f16688fd700 1 -- 172.21.15.111:0/4248709401 wait complete. 2023-11-28T12:59:08.322 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T12:59:08.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.444+0000 7f11d898e700 1 Processor -- start 2023-11-28T12:59:08.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.444+0000 7f11d898e700 1 -- start start 2023-11-28T12:59:08.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.445+0000 7f11d898e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f11d40cdd90 0x7f11d40b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.445+0000 7f11d898e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d40ad420 con 0x7f11d4159960 2023-11-28T12:59:08.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.445+0000 7f11d898e700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d40ac810 con 0x7f11d40aecc0 2023-11-28T12:59:08.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.445+0000 7f11d898e700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d4159d30 con 0x7f11d40cdd90 2023-11-28T12:59:08.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.445+0000 7f11d259c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f11d4159960 0x7f11d40cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33756/0 (socket says 172.21.15.111:33756) 2023-11-28T12:59:08.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.445+0000 7f11d259c700 1 -- 172.21.15.111:0/420642545 learned_addr learned my addr 172.21.15.111:0/420642545 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:08.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.445+0000 7f11d1d9b700 1 --2- 172.21.15.111:0/420642545 >> v2:172.21.15.149:3300/0 conn(0x7f11d40cdd90 0x7f11d40b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.445+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371146650 0 0) 0x7f11d40ad420 con 0x7f11d4159960 2023-11-28T12:59:08.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.445+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c00035b0 con 0x7f11d4159960 2023-11-28T12:59:08.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.445+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3415391607 0 0) 0x7f11c00035b0 con 0x7f11d4159960 2023-11-28T12:59:08.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.445+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 >> v2:172.21.15.149:3300/0 conn(0x7f11d40cdd90 msgr2=0x7f11d40b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:08.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.445+0000 7f11d159a700 1 --2- 172.21.15.111:0/420642545 >> v2:172.21.15.149:3300/0 conn(0x7f11d40cdd90 0x7f11d40b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.445+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 >> v1:172.21.15.149:6789/0 conn(0x7f11d40aecc0 legacy=0x7f11d40ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d40acaf0 con 0x7f11d4159960 2023-11-28T12:59:08.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f11c40017f0 con 0x7f11d4159960 2023-11-28T12:59:08.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 >> v1:172.21.15.111:6789/0 conn(0x7f11d4159960 legacy=0x7f11d40cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d159a700 1 --2- 172.21.15.111:0/420642545 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11c0003a00 0x7f11c0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d159a700 1 --2- 172.21.15.111:0/420642545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c00043f0 0x7f11c00048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d159a700 1 --2- 172.21.15.111:0/420642545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f11c00050b0 0x7f11c000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d4159d30 con 0x7f11c00043f0 2023-11-28T12:59:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c000c000 con 0x7f11c0003a00 2023-11-28T12:59:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c000c1c0 con 0x7f11c00050b0 2023-11-28T12:59:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d1d9b700 1 --2- 172.21.15.111:0/420642545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c00043f0 0x7f11c00048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d259c700 1 --2- 172.21.15.111:0/420642545 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11c0003a00 0x7f11c0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d2d9d700 1 --2- 172.21.15.111:0/420642545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f11c00050b0 0x7f11c000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d1d9b700 1 -- 172.21.15.111:0/420642545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f11c00050b0 msgr2=0x7f11c000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d1d9b700 1 --2- 172.21.15.111:0/420642545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f11c00050b0 0x7f11c000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d1d9b700 1 -- 172.21.15.111:0/420642545 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11c0003a00 msgr2=0x7f11c0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d1d9b700 1 --2- 172.21.15.111:0/420642545 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11c0003a00 0x7f11c0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d1d9b700 1 -- 172.21.15.111:0/420642545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c00035b0 con 0x7f11c00043f0 2023-11-28T12:59:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d2d9d700 1 -- 172.21.15.111:0/420642545 reap_dead start 2023-11-28T12:59:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d1d9b700 1 --2- 172.21.15.111:0/420642545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c00043f0 0x7f11c00048b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009b30 tx=0x7f11c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779209304aec3c6a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:08.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc00e6c0 con 0x7f11c00043f0 2023-11-28T12:59:08.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11bc00e820 con 0x7f11c00043f0 2023-11-28T12:59:08.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.446+0000 7f11d159a700 1 -- 172.21.15.111:0/420642545 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc00eb90 con 0x7f11c00043f0 2023-11-28T12:59:08.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.447+0000 7f11d898e700 1 -- 172.21.15.111:0/420642545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c00043f0 msgr2=0x7f11c00048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.447+0000 7f11d898e700 1 --2- 172.21.15.111:0/420642545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c00043f0 0x7f11c00048b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009b30 tx=0x7f11c0005880 comp rx=0 tx=0).stop 2023-11-28T12:59:08.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.447+0000 7f11d898e700 1 -- 172.21.15.111:0/420642545 shutdown_connections 2023-11-28T12:59:08.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.447+0000 7f11d898e700 1 --2- 172.21.15.111:0/420642545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c00043f0 0x7f11c00048b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.447+0000 7f11d898e700 1 -- 172.21.15.111:0/420642545 >> 172.21.15.111:0/420642545 conn(0x7f11d4098ee0 msgr2=0x7f11d406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:08.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.447+0000 7f11d898e700 1 -- 172.21.15.111:0/420642545 shutdown_connections 2023-11-28T12:59:08.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.447+0000 7f11d898e700 1 -- 172.21.15.111:0/420642545 wait complete. 2023-11-28T12:59:08.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.448+0000 7f11d898e700 1 Processor -- start 2023-11-28T12:59:08.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.448+0000 7f11d898e700 1 -- start start 2023-11-28T12:59:08.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d898e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c00043f0 0x7f11d41f1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d898e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11d40c5420 0x7f11d41575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d898e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f11d40ad970 0x7f11d4150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d898e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d4157b30 con 0x7f11c00043f0 2023-11-28T12:59:08.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d898e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41f2790 con 0x7f11d40c5420 2023-11-28T12:59:08.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d898e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41f1ef0 con 0x7f11d40ad970 2023-11-28T12:59:08.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c00043f0 0x7f11d41f1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c00043f0 0x7f11d41f1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32990/0 (socket says 172.21.15.111:32990) 2023-11-28T12:59:08.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d259c700 1 -- 172.21.15.111:0/2025540656 learned_addr learned my addr 172.21.15.111:0/2025540656 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:08.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d1d9b700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11d40c5420 0x7f11d41575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d2d9d700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f11d40ad970 0x7f11d4150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d259c700 1 -- 172.21.15.111:0/2025540656 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f11d40ad970 msgr2=0x7f11d4150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d259c700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f11d40ad970 0x7f11d4150080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d259c700 1 -- 172.21.15.111:0/2025540656 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11d40c5420 msgr2=0x7f11d41575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d259c700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11d40c5420 0x7f11d41575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.449+0000 7f11d259c700 1 -- 172.21.15.111:0/2025540656 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d40ad0b0 con 0x7f11c00043f0 2023-11-28T12:59:08.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.450+0000 7f11d259c700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c00043f0 0x7f11d41f1830 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11c400fae0 tx=0x7f11c400fdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:08.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.450+0000 7f11caffd700 1 -- 172.21.15.111:0/2025540656 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c40017a0 con 0x7f11c00043f0 2023-11-28T12:59:08.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.450+0000 7f11caffd700 1 -- 172.21.15.111:0/2025540656 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c40072e0 con 0x7f11c00043f0 2023-11-28T12:59:08.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.450+0000 7f11d898e700 1 -- 172.21.15.111:0/2025540656 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d40acdd0 con 0x7f11c00043f0 2023-11-28T12:59:08.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.450+0000 7f11caffd700 1 -- 172.21.15.111:0/2025540656 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c4007690 con 0x7f11c00043f0 2023-11-28T12:59:08.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.450+0000 7f11d898e700 1 -- 172.21.15.111:0/2025540656 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d40b1fd0 con 0x7f11c00043f0 2023-11-28T12:59:08.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.451+0000 7f11d898e700 1 -- 172.21.15.111:0/2025540656 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d406c4f0 con 0x7f11c00043f0 2023-11-28T12:59:08.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.451+0000 7f11caffd700 1 -- 172.21.15.111:0/2025540656 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f11c4005830 con 0x7f11c00043f0 2023-11-28T12:59:08.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.451+0000 7f11caffd700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f11b40682f0 0x7f11b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.452+0000 7f11d1d9b700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f11b40682f0 0x7f11b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.452+0000 7f11caffd700 1 -- 172.21.15.111:0/2025540656 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f11c4086010 con 0x7f11c00043f0 2023-11-28T12:59:08.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.452+0000 7f11d1d9b700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f11b40682f0 0x7f11b406a7a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0090b0 tx=0x7f11bc01e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:08.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.456+0000 7f11caffd700 1 -- 172.21.15.111:0/2025540656 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f11c4052580 con 0x7f11c00043f0 2023-11-28T12:59:08.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.639+0000 7f11d898e700 1 -- 172.21.15.111:0/2025540656 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f11d4067170 con 0x7f11c00043f0 2023-11-28T12:59:08.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.639+0000 7f11caffd700 1 -- 172.21.15.111:0/2025540656 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f11c4001060 con 0x7f11c00043f0 2023-11-28T12:59:08.641 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T12:59:08.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.642+0000 7f11d898e700 1 -- 172.21.15.111:0/2025540656 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f11b40682f0 msgr2=0x7f11b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.642+0000 7f11d898e700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f11b40682f0 0x7f11b406a7a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0090b0 tx=0x7f11bc01e040 comp rx=0 tx=0).stop 2023-11-28T12:59:08.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.642+0000 7f11d898e700 1 -- 172.21.15.111:0/2025540656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c00043f0 msgr2=0x7f11d41f1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.642+0000 7f11d898e700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c00043f0 0x7f11d41f1830 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11c400fae0 tx=0x7f11c400fdf0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.642+0000 7f11d898e700 1 -- 172.21.15.111:0/2025540656 shutdown_connections 2023-11-28T12:59:08.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.642+0000 7f11d898e700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f11d40ad970 0x7f11d4150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.642+0000 7f11d898e700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c00043f0 0x7f11d41f1830 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.642+0000 7f11d898e700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f11b40682f0 0x7f11b406a7a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.642+0000 7f11d898e700 1 --2- 172.21.15.111:0/2025540656 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11d40c5420 0x7f11d41575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.642+0000 7f11d898e700 1 -- 172.21.15.111:0/2025540656 >> 172.21.15.111:0/2025540656 conn(0x7f11d4098ee0 msgr2=0x7f11d406f220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:08.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.642+0000 7f11d898e700 1 -- 172.21.15.111:0/2025540656 shutdown_connections 2023-11-28T12:59:08.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.642+0000 7f11d898e700 1 -- 172.21.15.111:0/2025540656 wait complete. 2023-11-28T12:59:08.655 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T12:59:08.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.776+0000 7fe4c363d700 1 Processor -- start 2023-11-28T12:59:08.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.776+0000 7fe4c363d700 1 -- start start 2023-11-28T12:59:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.777+0000 7fe4c363d700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe4bc1604b0 0x7fe4bc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.777+0000 7fe4c363d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4bc15cd90 con 0x7fe4bc0b0d70 2023-11-28T12:59:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.777+0000 7fe4c363d700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4bc1613e0 con 0x7fe4bc0cc550 2023-11-28T12:59:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.777+0000 7fe4c363d700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4bc160950 con 0x7fe4bc1604b0 2023-11-28T12:59:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.777+0000 7fe4c13d9700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe4bc0b0d70 0x7fe4bc0b1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33776/0 (socket says 172.21.15.111:33776) 2023-11-28T12:59:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.777+0000 7fe4c13d9700 1 -- 172.21.15.111:0/2814272967 learned_addr learned my addr 172.21.15.111:0/2814272967 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.777+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715372608 0 0) 0x7fe4ac001290 con 0x7fe4bc0b0d70 2023-11-28T12:59:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.777+0000 7fe4c0bd8700 1 --2- 172.21.15.111:0/2814272967 >> v2:172.21.15.149:3300/0 conn(0x7fe4bc1604b0 0x7fe4bc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a80035b0 con 0x7fe4bc0b0d70 2023-11-28T12:59:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 278256801 0 0) 0x7fe4a80035b0 con 0x7fe4bc0b0d70 2023-11-28T12:59:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 >> v2:172.21.15.149:3300/0 conn(0x7fe4bc1604b0 msgr2=0x7fe4bc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 --2- 172.21.15.111:0/2814272967 >> v2:172.21.15.149:3300/0 conn(0x7fe4bc1604b0 0x7fe4bc0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 >> v1:172.21.15.149:6789/0 conn(0x7fe4bc0cc550 legacy=0x7fe4bc0af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4bc160b40 con 0x7fe4bc0b0d70 2023-11-28T12:59:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fe4ac001980 con 0x7fe4bc0b0d70 2023-11-28T12:59:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 >> v1:172.21.15.111:6789/0 conn(0x7fe4bc0b0d70 legacy=0x7fe4bc0b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 --2- 172.21.15.111:0/2814272967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4a8003a00 0x7fe4a8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 --2- 172.21.15.111:0/2814272967 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4a8004850 0x7fe4a8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 --2- 172.21.15.111:0/2814272967 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe4a8005200 0x7fe4a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a800c050 con 0x7fe4a8003a00 2023-11-28T12:59:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a800c1c0 con 0x7fe4a8004850 2023-11-28T12:59:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a800c380 con 0x7fe4a8005200 2023-11-28T12:59:08.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4c13d9700 1 --2- 172.21.15.111:0/2814272967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4a8003a00 0x7fe4a8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4c0bd8700 1 --2- 172.21.15.111:0/2814272967 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4a8004850 0x7fe4a8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4c1bda700 1 --2- 172.21.15.111:0/2814272967 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe4a8005200 0x7fe4a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4c13d9700 1 -- 172.21.15.111:0/2814272967 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe4a8005200 msgr2=0x7fe4a800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:08.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4c13d9700 1 --2- 172.21.15.111:0/2814272967 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe4a8005200 0x7fe4a800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4c13d9700 1 -- 172.21.15.111:0/2814272967 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4a8004850 msgr2=0x7fe4a8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:08.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4c13d9700 1 --2- 172.21.15.111:0/2814272967 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4a8004850 0x7fe4a8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4c1bda700 1 -- 172.21.15.111:0/2814272967 reap_dead start 2023-11-28T12:59:08.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4c13d9700 1 -- 172.21.15.111:0/2814272967 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a80035b0 con 0x7fe4a8003a00 2023-11-28T12:59:08.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4c13d9700 1 --2- 172.21.15.111:0/2814272967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4a8003a00 0x7fe4a8003e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00c210 tx=0x7fe4a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a72150e05c09cd3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:08.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.778+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac00cd70 con 0x7fe4a8003a00 2023-11-28T12:59:08.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.779+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ac007d40 con 0x7fe4a8003a00 2023-11-28T12:59:08.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.779+0000 7fe4b3fff700 1 -- 172.21.15.111:0/2814272967 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac007310 con 0x7fe4a8003a00 2023-11-28T12:59:08.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.779+0000 7fe4c363d700 1 -- 172.21.15.111:0/2814272967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4a8003a00 msgr2=0x7fe4a8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.779+0000 7fe4c363d700 1 --2- 172.21.15.111:0/2814272967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4a8003a00 0x7fe4a8003e10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00c210 tx=0x7fe4a8005880 comp rx=0 tx=0).stop 2023-11-28T12:59:08.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.779+0000 7fe4c363d700 1 -- 172.21.15.111:0/2814272967 shutdown_connections 2023-11-28T12:59:08.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.779+0000 7fe4c363d700 1 --2- 172.21.15.111:0/2814272967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4a8003a00 0x7fe4a8003e10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.779+0000 7fe4c363d700 1 -- 172.21.15.111:0/2814272967 >> 172.21.15.111:0/2814272967 conn(0x7fe4bc098ee0 msgr2=0x7fe4bc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:08.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.779+0000 7fe4c363d700 1 -- 172.21.15.111:0/2814272967 shutdown_connections 2023-11-28T12:59:08.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.779+0000 7fe4c363d700 1 -- 172.21.15.111:0/2814272967 wait complete. 2023-11-28T12:59:08.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.780+0000 7fe4c363d700 1 Processor -- start 2023-11-28T12:59:08.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.780+0000 7fe4c363d700 1 -- start start 2023-11-28T12:59:08.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c363d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4a8003a00 0x7fe4bc15df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c363d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe4bc0b54e0 0x7fe4bc0b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c363d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc14aae0 0x7fe4bc14af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c363d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc14b490 con 0x7fe4bc14aae0 2023-11-28T12:59:08.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c363d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc15e610 con 0x7fe4a8003a00 2023-11-28T12:59:08.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c363d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc15e8f0 con 0x7fe4bc0b54e0 2023-11-28T12:59:08.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c1bda700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc14aae0 0x7fe4bc14af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c1bda700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc14aae0 0x7fe4bc14af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33014/0 (socket says 172.21.15.111:33014) 2023-11-28T12:59:08.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c1bda700 1 -- 172.21.15.111:0/1520399423 learned_addr learned my addr 172.21.15.111:0/1520399423 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:08.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c13d9700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4a8003a00 0x7fe4bc15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c0bd8700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe4bc0b54e0 0x7fe4bc0b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c1bda700 1 -- 172.21.15.111:0/1520399423 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe4bc0b54e0 msgr2=0x7fe4bc0b2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c1bda700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe4bc0b54e0 0x7fe4bc0b2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c1bda700 1 -- 172.21.15.111:0/1520399423 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4a8003a00 msgr2=0x7fe4bc15df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c1bda700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4a8003a00 0x7fe4bc15df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.781+0000 7fe4c1bda700 1 -- 172.21.15.111:0/1520399423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4bc161100 con 0x7fe4bc14aae0 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.782+0000 7fe4c1bda700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc14aae0 0x7fe4bc14af50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc0b1ee0 tx=0x7fe4b400d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.782+0000 7fe4b1ffb700 1 -- 172.21.15.111:0/1520399423 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4003c10 con 0x7fe4bc14aae0 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.782+0000 7fe4b1ffb700 1 -- 172.21.15.111:0/1520399423 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b4003d90 con 0x7fe4bc14aae0 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.782+0000 7fe4c363d700 1 -- 172.21.15.111:0/1520399423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4bc160e20 con 0x7fe4bc14aae0 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.782+0000 7fe4b1ffb700 1 -- 172.21.15.111:0/1520399423 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b400d950 con 0x7fe4bc14aae0 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.782+0000 7fe4c363d700 1 -- 172.21.15.111:0/1520399423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4bc0af340 con 0x7fe4bc14aae0 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.783+0000 7fe4c363d700 1 -- 172.21.15.111:0/1520399423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4bc067170 con 0x7fe4bc14aae0 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.783+0000 7fe4b1ffb700 1 -- 172.21.15.111:0/1520399423 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe4b400dab0 con 0x7fe4bc14aae0 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.784+0000 7fe4b1ffb700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe4a00682f0 0x7fe4a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.784+0000 7fe4b1ffb700 1 -- 172.21.15.111:0/1520399423 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe4b40861c0 con 0x7fe4bc14aae0 2023-11-28T12:59:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.784+0000 7fe4c13d9700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe4a00682f0 0x7fe4a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:08.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.784+0000 7fe4c13d9700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe4a00682f0 0x7fe4a006a7a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac000c00 tx=0x7fe4ac007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:08.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.788+0000 7fe4b1ffb700 1 -- 172.21.15.111:0/1520399423 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe4b4052730 con 0x7fe4bc14aae0 2023-11-28T12:59:08.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.971+0000 7fe4c363d700 1 -- 172.21.15.111:0/1520399423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe4bc15c820 con 0x7fe4bc14aae0 2023-11-28T12:59:08.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.972+0000 7fe4b1ffb700 1 -- 172.21.15.111:0/1520399423 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe4b4055d50 con 0x7fe4bc14aae0 2023-11-28T12:59:08.973 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:08.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.974+0000 7fe4c363d700 1 -- 172.21.15.111:0/1520399423 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe4a00682f0 msgr2=0x7fe4a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.974+0000 7fe4c363d700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe4a00682f0 0x7fe4a006a7a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac000c00 tx=0x7fe4ac007000 comp rx=0 tx=0).stop 2023-11-28T12:59:08.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.974+0000 7fe4c363d700 1 -- 172.21.15.111:0/1520399423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc14aae0 msgr2=0x7fe4bc14af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:08.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.974+0000 7fe4c363d700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc14aae0 0x7fe4bc14af50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc0b1ee0 tx=0x7fe4b400d150 comp rx=0 tx=0).stop 2023-11-28T12:59:08.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.974+0000 7fe4c363d700 1 -- 172.21.15.111:0/1520399423 shutdown_connections 2023-11-28T12:59:08.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.974+0000 7fe4c363d700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc14aae0 0x7fe4bc14af50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.974+0000 7fe4c363d700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe4a00682f0 0x7fe4a006a7a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.974+0000 7fe4c363d700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe4a8003a00 0x7fe4bc15df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.974+0000 7fe4c363d700 1 --2- 172.21.15.111:0/1520399423 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe4bc0b54e0 0x7fe4bc0b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:08.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.974+0000 7fe4c363d700 1 -- 172.21.15.111:0/1520399423 >> 172.21.15.111:0/1520399423 conn(0x7fe4bc098ee0 msgr2=0x7fe4bc0ae680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:08.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.974+0000 7fe4c363d700 1 -- 172.21.15.111:0/1520399423 shutdown_connections 2023-11-28T12:59:08.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:08.974+0000 7fe4c363d700 1 -- 172.21.15.111:0/1520399423 wait complete. 2023-11-28T12:59:08.987 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T12:59:09.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.106+0000 7fc4cdeba700 1 Processor -- start 2023-11-28T12:59:09.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.106+0000 7fc4cdeba700 1 -- start start 2023-11-28T12:59:09.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.107+0000 7fc4cdeba700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc4c81604b0 0x7fc4c80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:09.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.107+0000 7fc4cdeba700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4c80cc550 0x7fc4c80cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:09.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.107+0000 7fc4cdeba700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c815cd90 con 0x7fc4c80b0d70 2023-11-28T12:59:09.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.107+0000 7fc4cdeba700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c8160950 con 0x7fc4c81604b0 2023-11-28T12:59:09.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.107+0000 7fc4cdeba700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80ccf10 con 0x7fc4c80cc550 2023-11-28T12:59:09.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.107+0000 7fc4c77fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc4c80b0d70 0x7fc4c80b1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33796/0 (socket says 172.21.15.111:33796) 2023-11-28T12:59:09.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.107+0000 7fc4c77fe700 1 -- 172.21.15.111:0/1522686725 learned_addr learned my addr 172.21.15.111:0/1522686725 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:09.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.107+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700776176 0 0) 0x7fc4bc001290 con 0x7fc4c80b0d70 2023-11-28T12:59:09.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.107+0000 7fc4c6ffd700 1 --2- 172.21.15.111:0/1522686725 >> v2:172.21.15.149:3300/0 conn(0x7fc4c81604b0 0x7fc4c80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:09.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.107+0000 7fc4c7fff700 1 --2- 172.21.15.111:0/1522686725 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4c80cc550 0x7fc4c80cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:09.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b00035b0 con 0x7fc4c80b0d70 2023-11-28T12:59:09.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3674557181 0 0) 0x7fc4b00035b0 con 0x7fc4c80b0d70 2023-11-28T12:59:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4c80cc550 msgr2=0x7fc4c80cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 --2- 172.21.15.111:0/1522686725 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4c80cc550 0x7fc4c80cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 >> v2:172.21.15.149:3300/0 conn(0x7fc4c81604b0 msgr2=0x7fc4c80ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 --2- 172.21.15.111:0/1522686725 >> v2:172.21.15.149:3300/0 conn(0x7fc4c81604b0 0x7fc4c80ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c81613e0 con 0x7fc4c80b0d70 2023-11-28T12:59:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fc4bc001980 con 0x7fc4c80b0d70 2023-11-28T12:59:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 >> v1:172.21.15.111:6789/0 conn(0x7fc4c80b0d70 legacy=0x7fc4c80b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 --2- 172.21.15.111:0/1522686725 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4b0003a00 0x7fc4b0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 --2- 172.21.15.111:0/1522686725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0004430 0x7fc4b00048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 --2- 172.21.15.111:0/1522686725 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4c80cc550 0x7fc4b0004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80ccf10 con 0x7fc4b0004430 2023-11-28T12:59:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b0005700 con 0x7fc4b0003a00 2023-11-28T12:59:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b000ba90 con 0x7fc4c80cc550 2023-11-28T12:59:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c6ffd700 1 --2- 172.21.15.111:0/1522686725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0004430 0x7fc4b00048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c77fe700 1 --2- 172.21.15.111:0/1522686725 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4b0003a00 0x7fc4b0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c7fff700 1 --2- 172.21.15.111:0/1522686725 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4c80cc550 0x7fc4b0004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c6ffd700 1 -- 172.21.15.111:0/1522686725 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4c80cc550 msgr2=0x7fc4b0004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c6ffd700 1 --2- 172.21.15.111:0/1522686725 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4c80cc550 0x7fc4b0004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c6ffd700 1 -- 172.21.15.111:0/1522686725 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4b0003a00 msgr2=0x7fc4b0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c6ffd700 1 --2- 172.21.15.111:0/1522686725 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4b0003a00 0x7fc4b0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c6ffd700 1 -- 172.21.15.111:0/1522686725 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b00035b0 con 0x7fc4b0004430 2023-11-28T12:59:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c6ffd700 1 --2- 172.21.15.111:0/1522686725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0004430 0x7fc4b00048f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8009b60 tx=0x7fc4b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3731c80671803f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b800e6c0 con 0x7fc4b0004430 2023-11-28T12:59:09.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b800e820 con 0x7fc4b0004430 2023-11-28T12:59:09.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.108+0000 7fc4c67fc700 1 -- 172.21.15.111:0/1522686725 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b800eb90 con 0x7fc4b0004430 2023-11-28T12:59:09.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.109+0000 7fc4cdeba700 1 -- 172.21.15.111:0/1522686725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0004430 msgr2=0x7fc4b00048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:09.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.109+0000 7fc4cdeba700 1 --2- 172.21.15.111:0/1522686725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0004430 0x7fc4b00048f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8009b60 tx=0x7fc4b0005880 comp rx=0 tx=0).stop 2023-11-28T12:59:09.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.109+0000 7fc4c7fff700 1 -- 172.21.15.111:0/1522686725 reap_dead start 2023-11-28T12:59:09.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.109+0000 7fc4cdeba700 1 -- 172.21.15.111:0/1522686725 shutdown_connections 2023-11-28T12:59:09.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.109+0000 7fc4cdeba700 1 -- 172.21.15.111:0/1522686725 >> 172.21.15.111:0/1522686725 conn(0x7fc4c8098ee0 msgr2=0x7fc4c806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:09.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.109+0000 7fc4cdeba700 1 -- 172.21.15.111:0/1522686725 shutdown_connections 2023-11-28T12:59:09.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.109+0000 7fc4cdeba700 1 -- 172.21.15.111:0/1522686725 wait complete. 2023-11-28T12:59:09.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.110+0000 7fc4cdeba700 1 Processor -- start 2023-11-28T12:59:09.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.110+0000 7fc4cdeba700 1 -- start start 2023-11-28T12:59:09.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4cdeba700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0004430 0x7fc4c81f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:09.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4cdeba700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4c81f5c10 0x7fc4c815ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:09.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4cdeba700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4c815ef40 0x7fc4c814a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:09.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4cdeba700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c8148360 con 0x7fc4b0004430 2023-11-28T12:59:09.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4cdeba700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c81f6080 con 0x7fc4c81f5c10 2023-11-28T12:59:09.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4cdeba700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c80c3920 con 0x7fc4c815ef40 2023-11-28T12:59:09.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4c77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0004430 0x7fc4c81f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:09.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4c77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0004430 0x7fc4c81f56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33030/0 (socket says 172.21.15.111:33030) 2023-11-28T12:59:09.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4c77fe700 1 -- 172.21.15.111:0/3424723286 learned_addr learned my addr 172.21.15.111:0/3424723286 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:09.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4c6ffd700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4c81f5c10 0x7fc4c815ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:09.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4c7fff700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4c815ef40 0x7fc4c814a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4c77fe700 1 -- 172.21.15.111:0/3424723286 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4c815ef40 msgr2=0x7fc4c814a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4c77fe700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4c815ef40 0x7fc4c814a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4c77fe700 1 -- 172.21.15.111:0/3424723286 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4c81f5c10 msgr2=0x7fc4c815ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4c77fe700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4c81f5c10 0x7fc4c815ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.111+0000 7fc4c77fe700 1 -- 172.21.15.111:0/3424723286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c8160e20 con 0x7fc4b0004430 2023-11-28T12:59:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.112+0000 7fc4c6ffd700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4c81f5c10 0x7fc4c815ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.112+0000 7fc4c77fe700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0004430 0x7fc4c81f56d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc007320 tx=0x7fc4bc0076e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.112+0000 7fc4affff700 1 -- 172.21.15.111:0/3424723286 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc007800 con 0x7fc4b0004430 2023-11-28T12:59:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.112+0000 7fc4affff700 1 -- 172.21.15.111:0/3424723286 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4bc00f870 con 0x7fc4b0004430 2023-11-28T12:59:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.112+0000 7fc4cdeba700 1 -- 172.21.15.111:0/3424723286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c8160b40 con 0x7fc4b0004430 2023-11-28T12:59:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.112+0000 7fc4affff700 1 -- 172.21.15.111:0/3424723286 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc00fc20 con 0x7fc4b0004430 2023-11-28T12:59:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.112+0000 7fc4cdeba700 1 -- 172.21.15.111:0/3424723286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c8161100 con 0x7fc4b0004430 2023-11-28T12:59:09.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.113+0000 7fc4cdeba700 1 -- 172.21.15.111:0/3424723286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c80c68b0 con 0x7fc4b0004430 2023-11-28T12:59:09.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.113+0000 7fc4affff700 1 -- 172.21.15.111:0/3424723286 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc4bc023070 con 0x7fc4b0004430 2023-11-28T12:59:09.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.113+0000 7fc4affff700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4b40682f0 0x7fc4b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:09.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.114+0000 7fc4c6ffd700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4b40682f0 0x7fc4b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:09.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.114+0000 7fc4affff700 1 -- 172.21.15.111:0/3424723286 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc4bc085f50 con 0x7fc4b0004430 2023-11-28T12:59:09.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.114+0000 7fc4c6ffd700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4b40682f0 0x7fc4b406a7a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80090b0 tx=0x7fc4b8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:09.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.118+0000 7fc4affff700 1 -- 172.21.15.111:0/3424723286 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc4bc0524c0 con 0x7fc4b0004430 2023-11-28T12:59:09.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.316+0000 7fc4cdeba700 1 -- 172.21.15.111:0/3424723286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc4c815cd90 con 0x7fc4b0004430 2023-11-28T12:59:09.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.316+0000 7fc4affff700 1 -- 172.21.15.111:0/3424723286 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc4c815cd90 con 0x7fc4b0004430 2023-11-28T12:59:09.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.318+0000 7fc4cdeba700 1 -- 172.21.15.111:0/3424723286 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4b40682f0 msgr2=0x7fc4b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:09.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.318+0000 7fc4cdeba700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4b40682f0 0x7fc4b406a7a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80090b0 tx=0x7fc4b8008030 comp rx=0 tx=0).stop 2023-11-28T12:59:09.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.318+0000 7fc4cdeba700 1 -- 172.21.15.111:0/3424723286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0004430 msgr2=0x7fc4c81f56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:09.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.318+0000 7fc4cdeba700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0004430 0x7fc4c81f56d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc007320 tx=0x7fc4bc0076e0 comp rx=0 tx=0).stop 2023-11-28T12:59:09.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.318+0000 7fc4cdeba700 1 -- 172.21.15.111:0/3424723286 shutdown_connections 2023-11-28T12:59:09.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.318+0000 7fc4cdeba700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4c815ef40 0x7fc4c814a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:09.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.318+0000 7fc4cdeba700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4b0004430 0x7fc4c81f56d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:09.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.318+0000 7fc4cdeba700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4b40682f0 0x7fc4b406a7a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:09.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.318+0000 7fc4cdeba700 1 --2- 172.21.15.111:0/3424723286 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4c81f5c10 0x7fc4c815ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:09.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.318+0000 7fc4cdeba700 1 -- 172.21.15.111:0/3424723286 >> 172.21.15.111:0/3424723286 conn(0x7fc4c8098ee0 msgr2=0x7fc4c80ae680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:09.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.318+0000 7fc4cdeba700 1 -- 172.21.15.111:0/3424723286 shutdown_connections 2023-11-28T12:59:09.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:09.318+0000 7fc4cdeba700 1 -- 172.21.15.111:0/3424723286 wait complete. 2023-11-28T12:59:09.321 INFO:teuthology.orchestra.run.smithi111.stderr:listed 0 entries 2023-11-28T12:59:09.331 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-11-28T12:59:09.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:09.332 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-11-28T12:59:09.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi111.stderr:ceph-fuse[111241]: fuse finished with error 0 and tester_r 0 2023-11-28T12:59:09.366 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T12:59:15.467 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T12:59:15.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:15.468 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-11-28T12:59:15.498 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:15.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.668+0000 7fe5e7311700 1 Processor -- start 2023-11-28T12:59:15.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.669+0000 7fe5e7311700 1 -- start start 2023-11-28T12:59:15.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.669+0000 7fe5e7311700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e006a980 0x7fe5e00c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:15.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.669+0000 7fe5e7311700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe5e00ca430 0x7fe5e00cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:15.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.669+0000 7fe5e7311700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e0159b10 con 0x7fe5e0142e80 2023-11-28T12:59:15.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5e7311700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e00c9380 con 0x7fe5e00ca430 2023-11-28T12:59:15.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5e7311700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e0143250 con 0x7fe5e006a980 2023-11-28T12:59:15.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5e58ae700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe5e00ca430 0x7fe5e00cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:15.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5e50ad700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe5e0142e80 0x7fe5e006a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33816/0 (socket says 172.21.15.111:33816) 2023-11-28T12:59:15.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5e50ad700 1 -- 172.21.15.111:0/2724114587 learned_addr learned my addr 172.21.15.111:0/2724114587 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:15.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5e48ac700 1 --2- 172.21.15.111:0/2724114587 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e006a980 0x7fe5e00c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:15.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5e58ae700 1 -- 172.21.15.111:0/2724114587 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e006a980 msgr2=0x7fe5e00c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:15.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5e58ae700 1 --2- 172.21.15.111:0/2724114587 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e006a980 0x7fe5e00c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:15.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5e58ae700 1 -- 172.21.15.111:0/2724114587 >> v1:172.21.15.111:6789/0 conn(0x7fe5e0142e80 legacy=0x7fe5e006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:15.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5e58ae700 1 -- 172.21.15.111:0/2724114587 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e00ca870 con 0x7fe5e00ca430 2023-11-28T12:59:15.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5d7fff700 1 -- 172.21.15.111:0/2724114587 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447127186 0 0) 0x7fe5e0159b10 con 0x7fe5e0142e80 2023-11-28T12:59:15.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5e58ae700 1 --2- 172.21.15.111:0/2724114587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e00ca430 0x7fe5e00cd0d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8009d90 tx=0x7fe5d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea5b1130584465b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:15.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5d7fff700 1 -- 172.21.15.111:0/2724114587 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d800e070 con 0x7fe5e00ca430 2023-11-28T12:59:15.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5d7fff700 1 -- 172.21.15.111:0/2724114587 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d800bb20 con 0x7fe5e00ca430 2023-11-28T12:59:15.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.670+0000 7fe5d7fff700 1 -- 172.21.15.111:0/2724114587 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8012430 con 0x7fe5e00ca430 2023-11-28T12:59:15.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.671+0000 7fe5e7311700 1 -- 172.21.15.111:0/2724114587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e00ca430 msgr2=0x7fe5e00cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:15.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.671+0000 7fe5e7311700 1 --2- 172.21.15.111:0/2724114587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e00ca430 0x7fe5e00cd0d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8009d90 tx=0x7fe5d800b330 comp rx=0 tx=0).stop 2023-11-28T12:59:15.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.671+0000 7fe5e7311700 1 -- 172.21.15.111:0/2724114587 shutdown_connections 2023-11-28T12:59:15.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.671+0000 7fe5e7311700 1 --2- 172.21.15.111:0/2724114587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e00ca430 0x7fe5e00cd0d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:15.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.671+0000 7fe5e7311700 1 --2- 172.21.15.111:0/2724114587 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e006a980 0x7fe5e00c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:15.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.671+0000 7fe5e7311700 1 -- 172.21.15.111:0/2724114587 >> 172.21.15.111:0/2724114587 conn(0x7fe5e00cb490 msgr2=0x7fe5e00cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:15.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.671+0000 7fe5e7311700 1 -- 172.21.15.111:0/2724114587 shutdown_connections 2023-11-28T12:59:15.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.671+0000 7fe5e7311700 1 -- 172.21.15.111:0/2724114587 wait complete. 2023-11-28T12:59:15.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.672+0000 7fe5e7311700 1 Processor -- start 2023-11-28T12:59:15.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.672+0000 7fe5e7311700 1 -- start start 2023-11-28T12:59:15.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e7311700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e006a980 0x7fe5e014c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e7311700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5e00ca430 0x7fe5e015a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e7311700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e0142e80 0x7fe5e015ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e7311700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00c6aa0 con 0x7fe5e0142e80 2023-11-28T12:59:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e7311700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00ce3d0 con 0x7fe5e00ca430 2023-11-28T12:59:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e7311700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e015add0 con 0x7fe5e006a980 2023-11-28T12:59:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e58ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e0142e80 0x7fe5e015ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e58ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e0142e80 0x7fe5e015ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33048/0 (socket says 172.21.15.111:33048) 2023-11-28T12:59:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e58ae700 1 -- 172.21.15.111:0/3543931400 learned_addr learned my addr 172.21.15.111:0/3543931400 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e50ad700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e006a980 0x7fe5e014c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e48ac700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5e00ca430 0x7fe5e015a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e58ae700 1 -- 172.21.15.111:0/3543931400 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e006a980 msgr2=0x7fe5e014c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:15.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e58ae700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e006a980 0x7fe5e014c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e58ae700 1 -- 172.21.15.111:0/3543931400 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5e00ca430 msgr2=0x7fe5e015a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e58ae700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5e00ca430 0x7fe5e015a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.673+0000 7fe5e58ae700 1 -- 172.21.15.111:0/3543931400 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d8009a70 con 0x7fe5e0142e80 2023-11-28T12:59:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.674+0000 7fe5e58ae700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e0142e80 0x7fe5e015ba30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800b680 tx=0x7fe5d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.674+0000 7fe5d5ffb700 1 -- 172.21.15.111:0/3543931400 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8012430 con 0x7fe5e0142e80 2023-11-28T12:59:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.674+0000 7fe5d5ffb700 1 -- 172.21.15.111:0/3543931400 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d8020070 con 0x7fe5e0142e80 2023-11-28T12:59:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.674+0000 7fe5e7311700 1 -- 172.21.15.111:0/3543931400 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e00cc5f0 con 0x7fe5e0142e80 2023-11-28T12:59:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.674+0000 7fe5d5ffb700 1 -- 172.21.15.111:0/3543931400 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8004280 con 0x7fe5e0142e80 2023-11-28T12:59:15.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.675+0000 7fe5e7311700 1 -- 172.21.15.111:0/3543931400 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e00cab00 con 0x7fe5e0142e80 2023-11-28T12:59:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.676+0000 7fe5d5ffb700 1 -- 172.21.15.111:0/3543931400 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe5d802e070 con 0x7fe5e0142e80 2023-11-28T12:59:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.676+0000 7fe5e7311700 1 -- 172.21.15.111:0/3543931400 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e00cc8d0 con 0x7fe5e0142e80 2023-11-28T12:59:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.677+0000 7fe5d5ffb700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5c40682f0 0x7fe5c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.677+0000 7fe5e50ad700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5c40682f0 0x7fe5c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.677+0000 7fe5d5ffb700 1 -- 172.21.15.111:0/3543931400 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe5d8022070 con 0x7fe5e0142e80 2023-11-28T12:59:15.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.677+0000 7fe5e50ad700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5c40682f0 0x7fe5c406a7a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc000f60 tx=0x7fe5dc007e70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:15.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.682+0000 7fe5d5ffb700 1 -- 172.21.15.111:0/3543931400 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe5d80541c0 con 0x7fe5e0142e80 2023-11-28T12:59:15.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.864+0000 7fe5e7311700 1 -- 172.21.15.111:0/3543931400 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5e015b2c0 con 0x7fe5e0142e80 2023-11-28T12:59:15.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.865+0000 7fe5d5ffb700 1 -- 172.21.15.111:0/3543931400 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19796 (secure 0 0 0) 0x7fe5d80577e0 con 0x7fe5e0142e80 2023-11-28T12:59:15.867 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:15.867 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:00.053943+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:15.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.867+0000 7fe5e7311700 1 -- 172.21.15.111:0/3543931400 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5c40682f0 msgr2=0x7fe5c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:15.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.867+0000 7fe5e7311700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5c40682f0 0x7fe5c406a7a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc000f60 tx=0x7fe5dc007e70 comp rx=0 tx=0).stop 2023-11-28T12:59:15.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.867+0000 7fe5e7311700 1 -- 172.21.15.111:0/3543931400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e0142e80 msgr2=0x7fe5e015ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:15.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.867+0000 7fe5e7311700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e0142e80 0x7fe5e015ba30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800b680 tx=0x7fe5d8004ad0 comp rx=0 tx=0).stop 2023-11-28T12:59:15.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.867+0000 7fe5e7311700 1 -- 172.21.15.111:0/3543931400 shutdown_connections 2023-11-28T12:59:15.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.867+0000 7fe5e7311700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e0142e80 0x7fe5e015ba30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:15.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.867+0000 7fe5e7311700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e006a980 0x7fe5e014c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:15.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.867+0000 7fe5e7311700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5c40682f0 0x7fe5c406a7a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:15.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.867+0000 7fe5e7311700 1 --2- 172.21.15.111:0/3543931400 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5e00ca430 0x7fe5e015a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:15.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.867+0000 7fe5e7311700 1 -- 172.21.15.111:0/3543931400 >> 172.21.15.111:0/3543931400 conn(0x7fe5e00cb490 msgr2=0x7fe5e00c2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:15.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.867+0000 7fe5e7311700 1 -- 172.21.15.111:0/3543931400 shutdown_connections 2023-11-28T12:59:15.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:15.867+0000 7fe5e7311700 1 -- 172.21.15.111:0/3543931400 wait complete. 2023-11-28T12:59:15.882 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:16.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.000+0000 7fd9f7263700 1 Processor -- start 2023-11-28T12:59:16.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.000+0000 7fd9f7263700 1 -- start start 2023-11-28T12:59:16.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.001+0000 7fd9f7263700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd9f015a5d0 0x7fd9f00ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.001+0000 7fd9f7263700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd9f00cb2d0 0x7fd9f00ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.001+0000 7fd9f7263700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f00ae8b0 con 0x7fd9f00b0160 2023-11-28T12:59:16.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.001+0000 7fd9f7263700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f00cc1b0 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.001+0000 7fd9f7263700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f00cc390 con 0x7fd9f015a5d0 2023-11-28T12:59:16.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.001+0000 7fd9f5800700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd9f00cb2d0 0x7fd9f00ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.001+0000 7fd9f5800700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd9f00cb2d0 0x7fd9f00ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33056/0 (socket says 172.21.15.111:33056) 2023-11-28T12:59:16.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.001+0000 7fd9f5800700 1 -- 172.21.15.111:0/919403080 learned_addr learned my addr 172.21.15.111:0/919403080 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:16.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.001+0000 7fd9f4fff700 1 --2- 172.21.15.111:0/919403080 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd9f015a5d0 0x7fd9f00ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.001+0000 7fd9f5800700 1 -- 172.21.15.111:0/919403080 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd9f015a5d0 msgr2=0x7fd9f00ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:16.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.001+0000 7fd9f5800700 1 --2- 172.21.15.111:0/919403080 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd9f015a5d0 0x7fd9f00ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.001+0000 7fd9f5800700 1 -- 172.21.15.111:0/919403080 >> v1:172.21.15.149:6789/0 conn(0x7fd9f00b0160 legacy=0x7fd9f015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.002+0000 7fd9f5800700 1 -- 172.21.15.111:0/919403080 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f00cc670 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.002+0000 7fd9f5800700 1 --2- 172.21.15.111:0/919403080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9f00cb2d0 0x7fd9f00ced90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8009e50 tx=0x7fd9e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19698aba5a7fe521 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:16.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.002+0000 7fd9ef7fe700 1 -- 172.21.15.111:0/919403080 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e800e070 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.002+0000 7fd9ef7fe700 1 -- 172.21.15.111:0/919403080 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e800b9c0 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.002+0000 7fd9ef7fe700 1 -- 172.21.15.111:0/919403080 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e800bd70 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.002+0000 7fd9f7263700 1 -- 172.21.15.111:0/919403080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9f00cb2d0 msgr2=0x7fd9f00ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.002+0000 7fd9f7263700 1 --2- 172.21.15.111:0/919403080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9f00cb2d0 0x7fd9f00ced90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8009e50 tx=0x7fd9e800b330 comp rx=0 tx=0).stop 2023-11-28T12:59:16.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.002+0000 7fd9f7263700 1 -- 172.21.15.111:0/919403080 shutdown_connections 2023-11-28T12:59:16.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.002+0000 7fd9f7263700 1 --2- 172.21.15.111:0/919403080 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd9f015a5d0 0x7fd9f00ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.002+0000 7fd9f7263700 1 --2- 172.21.15.111:0/919403080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9f00cb2d0 0x7fd9f00ced90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.002+0000 7fd9f7263700 1 -- 172.21.15.111:0/919403080 >> 172.21.15.111:0/919403080 conn(0x7fd9f009a040 msgr2=0x7fd9f00d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:16.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.003+0000 7fd9f7263700 1 -- 172.21.15.111:0/919403080 shutdown_connections 2023-11-28T12:59:16.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.003+0000 7fd9f7263700 1 -- 172.21.15.111:0/919403080 wait complete. 2023-11-28T12:59:16.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.004+0000 7fd9f7263700 1 Processor -- start 2023-11-28T12:59:16.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.004+0000 7fd9f7263700 1 -- start start 2023-11-28T12:59:16.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9f7263700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd9f00b0160 0x7fd9f01e5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9f7263700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9f00cb2d0 0x7fd9f0155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9f7263700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd9f015a5d0 0x7fd9f00b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9f7263700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f01efbc0 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9f7263700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f01f0120 con 0x7fd9f015a5d0 2023-11-28T12:59:16.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9f7263700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f00b6bf0 con 0x7fd9f00b0160 2023-11-28T12:59:16.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9f00cb2d0 0x7fd9f0155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9f00cb2d0 0x7fd9f0155610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33060/0 (socket says 172.21.15.111:33060) 2023-11-28T12:59:16.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9effff700 1 -- 172.21.15.111:0/3629371945 learned_addr learned my addr 172.21.15.111:0/3629371945 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:16.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9f5800700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd9f015a5d0 0x7fd9f00b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9f4fff700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd9f00b0160 0x7fd9f01e5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9effff700 1 -- 172.21.15.111:0/3629371945 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd9f00b0160 msgr2=0x7fd9f01e5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9effff700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd9f00b0160 0x7fd9f01e5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9effff700 1 -- 172.21.15.111:0/3629371945 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd9f015a5d0 msgr2=0x7fd9f00b5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9effff700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd9f015a5d0 0x7fd9f00b5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9effff700 1 -- 172.21.15.111:0/3629371945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e8009b30 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9f5800700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd9f015a5d0 0x7fd9f00b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:16.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9f4fff700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd9f00b0160 0x7fd9f01e5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:16.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9effff700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9f00cb2d0 0x7fd9f0155610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00c4e0 tx=0x7fd9dc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:16.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9ed7fa700 1 -- 172.21.15.111:0/3629371945 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc0052d0 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9ed7fa700 1 -- 172.21.15.111:0/3629371945 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc005430 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9f7263700 1 -- 172.21.15.111:0/3629371945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f00ccc30 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9ed7fa700 1 -- 172.21.15.111:0/3629371945 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc0038c0 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.005+0000 7fd9f7263700 1 -- 172.21.15.111:0/3629371945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f00cc950 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.006+0000 7fd9f7263700 1 -- 172.21.15.111:0/3629371945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f00ccf10 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.007+0000 7fd9ed7fa700 1 -- 172.21.15.111:0/3629371945 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd9dc019030 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.007+0000 7fd9ed7fa700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9d40682f0 0x7fd9d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.007+0000 7fd9ed7fa700 1 -- 172.21.15.111:0/3629371945 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd9dc085330 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.007+0000 7fd9f4fff700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9d40682f0 0x7fd9d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.007+0000 7fd9f4fff700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9d40682f0 0x7fd9d406a7a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009820 tx=0x7fd9e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:16.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.011+0000 7fd9ed7fa700 1 -- 172.21.15.111:0/3629371945 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd9dc0518a0 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.198+0000 7fd9f7263700 1 -- 172.21.15.111:0/3629371945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9f0067170 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.199+0000 7fd9ed7fa700 1 -- 172.21.15.111:0/3629371945 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19796 (secure 0 0 0) 0x7fd9dc054ec0 con 0x7fd9f00cb2d0 2023-11-28T12:59:16.201 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:16.201 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:00.053943+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:16.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.201+0000 7fd9f7263700 1 -- 172.21.15.111:0/3629371945 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9d40682f0 msgr2=0x7fd9d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.201+0000 7fd9f7263700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9d40682f0 0x7fd9d406a7a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009820 tx=0x7fd9e4008040 comp rx=0 tx=0).stop 2023-11-28T12:59:16.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.201+0000 7fd9f7263700 1 -- 172.21.15.111:0/3629371945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9f00cb2d0 msgr2=0x7fd9f0155610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.201+0000 7fd9f7263700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9f00cb2d0 0x7fd9f0155610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00c4e0 tx=0x7fd9dc00c8a0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.201+0000 7fd9f7263700 1 -- 172.21.15.111:0/3629371945 shutdown_connections 2023-11-28T12:59:16.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.201+0000 7fd9f7263700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9d40682f0 0x7fd9d406a7a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.201+0000 7fd9f7263700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd9f015a5d0 0x7fd9f00b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.201+0000 7fd9f7263700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd9f00b0160 0x7fd9f01e5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.201+0000 7fd9f7263700 1 --2- 172.21.15.111:0/3629371945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9f00cb2d0 0x7fd9f0155610 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.201+0000 7fd9f7263700 1 -- 172.21.15.111:0/3629371945 >> 172.21.15.111:0/3629371945 conn(0x7fd9f009a040 msgr2=0x7fd9f006ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:16.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.201+0000 7fd9f7263700 1 -- 172.21.15.111:0/3629371945 shutdown_connections 2023-11-28T12:59:16.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.201+0000 7fd9f7263700 1 -- 172.21.15.111:0/3629371945 wait complete. 2023-11-28T12:59:16.215 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T12:59:16.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.337+0000 7fcb275b9700 1 Processor -- start 2023-11-28T12:59:16.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.337+0000 7fcb275b9700 1 -- start start 2023-11-28T12:59:16.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.338+0000 7fcb275b9700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fcb200b2220 0x7fcb200cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.338+0000 7fcb275b9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcb200cf010 0x7fcb200b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.338+0000 7fcb275b9700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb200cc7b0 con 0x7fcb200cb380 2023-11-28T12:59:16.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.338+0000 7fcb275b9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb2008ae90 con 0x7fcb200cf010 2023-11-28T12:59:16.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.338+0000 7fcb275b9700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb200b25f0 con 0x7fcb200b2220 2023-11-28T12:59:16.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.338+0000 7fcb25b56700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcb200cf010 0x7fcb200b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.338+0000 7fcb25b56700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcb200cf010 0x7fcb200b07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33070/0 (socket says 172.21.15.111:33070) 2023-11-28T12:59:16.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.338+0000 7fcb25b56700 1 -- 172.21.15.111:0/3888230482 learned_addr learned my addr 172.21.15.111:0/3888230482 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:16.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.338+0000 7fcb25355700 1 --2- 172.21.15.111:0/3888230482 >> v2:172.21.15.149:3300/0 conn(0x7fcb200b2220 0x7fcb200cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.338+0000 7fcb25b56700 1 -- 172.21.15.111:0/3888230482 >> v2:172.21.15.149:3300/0 conn(0x7fcb200b2220 msgr2=0x7fcb200cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:16.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.338+0000 7fcb25b56700 1 --2- 172.21.15.111:0/3888230482 >> v2:172.21.15.149:3300/0 conn(0x7fcb200b2220 0x7fcb200cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.338+0000 7fcb25b56700 1 -- 172.21.15.111:0/3888230482 >> v1:172.21.15.149:6789/0 conn(0x7fcb200cb380 legacy=0x7fcb200ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:16.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.338+0000 7fcb25b56700 1 -- 172.21.15.111:0/3888230482 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200cca10 con 0x7fcb200cf010 2023-11-28T12:59:16.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.339+0000 7fcb25b56700 1 --2- 172.21.15.111:0/3888230482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200cf010 0x7fcb200b07b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcb18009e50 tx=0x7fcb1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7d7309ec2526af server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:16.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.339+0000 7fcb17fff700 1 -- 172.21.15.111:0/3888230482 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1800e070 con 0x7fcb200cf010 2023-11-28T12:59:16.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.339+0000 7fcb17fff700 1 -- 172.21.15.111:0/3888230482 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb18003a20 con 0x7fcb200cf010 2023-11-28T12:59:16.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.339+0000 7fcb17fff700 1 -- 172.21.15.111:0/3888230482 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb18003dd0 con 0x7fcb200cf010 2023-11-28T12:59:16.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.339+0000 7fcb275b9700 1 -- 172.21.15.111:0/3888230482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200cf010 msgr2=0x7fcb200b07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.339+0000 7fcb275b9700 1 --2- 172.21.15.111:0/3888230482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200cf010 0x7fcb200b07b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcb18009e50 tx=0x7fcb1800b330 comp rx=0 tx=0).stop 2023-11-28T12:59:16.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.339+0000 7fcb275b9700 1 -- 172.21.15.111:0/3888230482 shutdown_connections 2023-11-28T12:59:16.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.339+0000 7fcb275b9700 1 --2- 172.21.15.111:0/3888230482 >> v2:172.21.15.149:3300/0 conn(0x7fcb200b2220 0x7fcb200cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.339+0000 7fcb275b9700 1 --2- 172.21.15.111:0/3888230482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200cf010 0x7fcb200b07b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.339+0000 7fcb275b9700 1 -- 172.21.15.111:0/3888230482 >> 172.21.15.111:0/3888230482 conn(0x7fcb2009a040 msgr2=0x7fcb200d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:16.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.339+0000 7fcb275b9700 1 -- 172.21.15.111:0/3888230482 shutdown_connections 2023-11-28T12:59:16.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.339+0000 7fcb275b9700 1 -- 172.21.15.111:0/3888230482 wait complete. 2023-11-28T12:59:16.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.341+0000 7fcb275b9700 1 Processor -- start 2023-11-28T12:59:16.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.341+0000 7fcb275b9700 1 -- start start 2023-11-28T12:59:16.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb275b9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b2220 0x7fcb201f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb275b9700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb200cb380 0x7fcb200b65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb275b9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb200cf010 0x7fcb200b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb275b9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb201f4410 con 0x7fcb200b2220 2023-11-28T12:59:16.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb275b9700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200ca1d0 con 0x7fcb200cb380 2023-11-28T12:59:16.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb275b9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200b6af0 con 0x7fcb200cf010 2023-11-28T12:59:16.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb25355700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b2220 0x7fcb201f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb25355700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b2220 0x7fcb201f3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33072/0 (socket says 172.21.15.111:33072) 2023-11-28T12:59:16.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb25355700 1 -- 172.21.15.111:0/113592177 learned_addr learned my addr 172.21.15.111:0/113592177 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:16.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb24b54700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb200cb380 0x7fcb200b65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb25b56700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb200cf010 0x7fcb200b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb25355700 1 -- 172.21.15.111:0/113592177 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb200cf010 msgr2=0x7fcb200b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb25355700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb200cf010 0x7fcb200b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb25355700 1 -- 172.21.15.111:0/113592177 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb200cb380 msgr2=0x7fcb200b65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb25355700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb200cb380 0x7fcb200b65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb25355700 1 -- 172.21.15.111:0/113592177 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb18009b30 con 0x7fcb200b2220 2023-11-28T12:59:16.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.342+0000 7fcb25355700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b2220 0x7fcb201f3ed0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00e810 tx=0x7fcb1c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:16.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.343+0000 7fcb15ffb700 1 -- 172.21.15.111:0/113592177 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c00c070 con 0x7fcb200b2220 2023-11-28T12:59:16.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.343+0000 7fcb15ffb700 1 -- 172.21.15.111:0/113592177 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1c013610 con 0x7fcb200b2220 2023-11-28T12:59:16.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.343+0000 7fcb15ffb700 1 -- 172.21.15.111:0/113592177 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c0139c0 con 0x7fcb200b2220 2023-11-28T12:59:16.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.343+0000 7fcb275b9700 1 -- 172.21.15.111:0/113592177 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb200ccfd0 con 0x7fcb200b2220 2023-11-28T12:59:16.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.343+0000 7fcb275b9700 1 -- 172.21.15.111:0/113592177 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb200cccf0 con 0x7fcb200b2220 2023-11-28T12:59:16.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.344+0000 7fcb275b9700 1 -- 172.21.15.111:0/113592177 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb200cd2b0 con 0x7fcb200b2220 2023-11-28T12:59:16.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.344+0000 7fcb15ffb700 1 -- 172.21.15.111:0/113592177 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcb1c023070 con 0x7fcb200b2220 2023-11-28T12:59:16.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.345+0000 7fcb15ffb700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb040682f0 0x7fcb0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.345+0000 7fcb15ffb700 1 -- 172.21.15.111:0/113592177 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fcb1c085e40 con 0x7fcb200b2220 2023-11-28T12:59:16.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.345+0000 7fcb24b54700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb040682f0 0x7fcb0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.345+0000 7fcb24b54700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb040682f0 0x7fcb0406a7a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcb2008a5e0 tx=0x7fcb10007ec0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:16.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.349+0000 7fcb15ffb700 1 -- 172.21.15.111:0/113592177 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcb1c052240 con 0x7fcb200b2220 2023-11-28T12:59:16.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.577+0000 7fcb275b9700 1 -- 172.21.15.111:0/113592177 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fcb200c4f30 con 0x7fcb200b2220 2023-11-28T12:59:16.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.702+0000 7fcb15ffb700 1 -- 172.21.15.111:0/113592177 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7fcb1c059e60 con 0x7fcb200b2220 2023-11-28T12:59:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.704+0000 7fcb275b9700 1 -- 172.21.15.111:0/113592177 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb040682f0 msgr2=0x7fcb0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.704+0000 7fcb275b9700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb040682f0 0x7fcb0406a7a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcb2008a5e0 tx=0x7fcb10007ec0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.704+0000 7fcb275b9700 1 -- 172.21.15.111:0/113592177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b2220 msgr2=0x7fcb201f3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.704+0000 7fcb275b9700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b2220 0x7fcb201f3ed0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00e810 tx=0x7fcb1c00eb20 comp rx=0 tx=0).stop 2023-11-28T12:59:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.704+0000 7fcb275b9700 1 -- 172.21.15.111:0/113592177 shutdown_connections 2023-11-28T12:59:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.704+0000 7fcb275b9700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb200cf010 0x7fcb200b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.704+0000 7fcb275b9700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b2220 0x7fcb201f3ed0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.704+0000 7fcb275b9700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb040682f0 0x7fcb0406a7a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.704+0000 7fcb275b9700 1 --2- 172.21.15.111:0/113592177 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb200cb380 0x7fcb200b65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.704+0000 7fcb275b9700 1 -- 172.21.15.111:0/113592177 >> 172.21.15.111:0/113592177 conn(0x7fcb2009a040 msgr2=0x7fcb200b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.704+0000 7fcb275b9700 1 -- 172.21.15.111:0/113592177 shutdown_connections 2023-11-28T12:59:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.704+0000 7fcb275b9700 1 -- 172.21.15.111:0/113592177 wait complete. 2023-11-28T12:59:16.717 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2023-11-28T12:59:16.717 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T12:59:16.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.839+0000 7ff7d3b4a700 1 Processor -- start 2023-11-28T12:59:16.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.839+0000 7ff7d3b4a700 1 -- start start 2023-11-28T12:59:16.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d3b4a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff7cc1604b0 0x7ff7cc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d3b4a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7cc15cd90 con 0x7ff7cc0b0d70 2023-11-28T12:59:16.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d3b4a700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7cc1613e0 con 0x7ff7cc0cc550 2023-11-28T12:59:16.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d3b4a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7cc160950 con 0x7ff7cc1604b0 2023-11-28T12:59:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d10e5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff7cc1604b0 0x7ff7cc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d18e6700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff7cc0b0d70 0x7ff7cc0b1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33858/0 (socket says 172.21.15.111:33858) 2023-11-28T12:59:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d18e6700 1 -- 172.21.15.111:0/1642957748 learned_addr learned my addr 172.21.15.111:0/1642957748 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d10e5700 1 -- 172.21.15.111:0/1642957748 >> v1:172.21.15.149:6789/0 conn(0x7ff7cc0cc550 legacy=0x7ff7cc0af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d10e5700 1 -- 172.21.15.111:0/1642957748 >> v1:172.21.15.111:6789/0 conn(0x7ff7cc0b0d70 legacy=0x7ff7cc0b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d08e4700 1 -- 172.21.15.111:0/1642957748 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479425054 0 0) 0x7ff7c8001290 con 0x7ff7cc0b0d70 2023-11-28T12:59:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d10e5700 1 -- 172.21.15.111:0/1642957748 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc160b40 con 0x7ff7cc1604b0 2023-11-28T12:59:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d10e5700 1 --2- 172.21.15.111:0/1642957748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7cc1604b0 0x7ff7cc0ce6c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009840 tx=0x7ff7bc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a67d592c44e523b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d08e4700 1 -- 172.21.15.111:0/1642957748 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc011830 con 0x7ff7cc1604b0 2023-11-28T12:59:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d08e4700 1 -- 172.21.15.111:0/1642957748 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7bc011990 con 0x7ff7cc1604b0 2023-11-28T12:59:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.840+0000 7ff7d08e4700 1 -- 172.21.15.111:0/1642957748 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc011d20 con 0x7ff7cc1604b0 2023-11-28T12:59:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.841+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/1642957748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7cc1604b0 msgr2=0x7ff7cc0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.841+0000 7ff7d3b4a700 1 --2- 172.21.15.111:0/1642957748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7cc1604b0 0x7ff7cc0ce6c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009840 tx=0x7ff7bc009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:16.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.841+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/1642957748 shutdown_connections 2023-11-28T12:59:16.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.841+0000 7ff7d3b4a700 1 --2- 172.21.15.111:0/1642957748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7cc1604b0 0x7ff7cc0ce6c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.841+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/1642957748 >> 172.21.15.111:0/1642957748 conn(0x7ff7cc098ee0 msgr2=0x7ff7cc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:16.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.841+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/1642957748 shutdown_connections 2023-11-28T12:59:16.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.841+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/1642957748 wait complete. 2023-11-28T12:59:16.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.843+0000 7ff7d3b4a700 1 Processor -- start 2023-11-28T12:59:16.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.843+0000 7ff7d3b4a700 1 -- start start 2023-11-28T12:59:16.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d3b4a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7cc0b0d70 0x7ff7cc158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d3b4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7cc0cc550 0x7ff7cc0b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d3b4a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7cc1604b0 0x7ff7cc158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d3b4a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc0c43c0 con 0x7ff7cc0cc550 2023-11-28T12:59:16.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d3b4a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc0c63f0 con 0x7ff7cc0b0d70 2023-11-28T12:59:16.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d3b4a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc0c5bd0 con 0x7ff7cc1604b0 2023-11-28T12:59:16.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d10e5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7cc0cc550 0x7ff7cc0b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d10e5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7cc0cc550 0x7ff7cc0b2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33088/0 (socket says 172.21.15.111:33088) 2023-11-28T12:59:16.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d10e5700 1 -- 172.21.15.111:0/4239804858 learned_addr learned my addr 172.21.15.111:0/4239804858 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:16.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d18e6700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7cc0b0d70 0x7ff7cc158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d20e7700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7cc1604b0 0x7ff7cc158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d10e5700 1 -- 172.21.15.111:0/4239804858 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7cc1604b0 msgr2=0x7ff7cc158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d10e5700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7cc1604b0 0x7ff7cc158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d10e5700 1 -- 172.21.15.111:0/4239804858 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7cc0b0d70 msgr2=0x7ff7cc158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:16.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d10e5700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7cc0b0d70 0x7ff7cc158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:16.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d10e5700 1 -- 172.21.15.111:0/4239804858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc0094a0 con 0x7ff7cc0cc550 2023-11-28T12:59:16.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d20e7700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7cc1604b0 0x7ff7cc158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:16.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d10e5700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7cc0cc550 0x7ff7cc0b2970 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc067c90 tx=0x7ff7bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:16.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7c27fc700 1 -- 172.21.15.111:0/4239804858 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc0036a0 con 0x7ff7cc0cc550 2023-11-28T12:59:16.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7c27fc700 1 -- 172.21.15.111:0/4239804858 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7bc003800 con 0x7ff7cc0cc550 2023-11-28T12:59:16.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/4239804858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7cc161100 con 0x7ff7cc0cc550 2023-11-28T12:59:16.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7c27fc700 1 -- 172.21.15.111:0/4239804858 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc003bb0 con 0x7ff7cc0cc550 2023-11-28T12:59:16.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.844+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/4239804858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7cc160e20 con 0x7ff7cc0cc550 2023-11-28T12:59:16.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.845+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/4239804858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7cc0af340 con 0x7ff7cc0cc550 2023-11-28T12:59:16.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.846+0000 7ff7c27fc700 1 -- 172.21.15.111:0/4239804858 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff7bc00f050 con 0x7ff7cc0cc550 2023-11-28T12:59:16.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.846+0000 7ff7c27fc700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7b00682f0 0x7ff7b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:16.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.846+0000 7ff7c27fc700 1 -- 172.21.15.111:0/4239804858 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff7bc0873a0 con 0x7ff7cc0cc550 2023-11-28T12:59:16.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.846+0000 7ff7d18e6700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7b00682f0 0x7ff7b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:16.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.846+0000 7ff7d18e6700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7b00682f0 0x7ff7b006a7a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8007ae0 tx=0x7ff7c8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:16.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:16.850+0000 7ff7c27fc700 1 -- 172.21.15.111:0/4239804858 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff7bc053910 con 0x7ff7cc0cc550 2023-11-28T12:59:17.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.074+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/4239804858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff7cc0c4500 con 0x7ff7cc0cc550 2023-11-28T12:59:17.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.704+0000 7ff7c27fc700 1 -- 172.21.15.111:0/4239804858 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff7cc0c4500 con 0x7ff7cc0cc550 2023-11-28T12:59:17.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.707+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/4239804858 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7b00682f0 msgr2=0x7ff7b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:17.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.707+0000 7ff7d3b4a700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7b00682f0 0x7ff7b006a7a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8007ae0 tx=0x7ff7c8009040 comp rx=0 tx=0).stop 2023-11-28T12:59:17.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.707+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/4239804858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7cc0cc550 msgr2=0x7ff7cc0b2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:17.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.707+0000 7ff7d3b4a700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7cc0cc550 0x7ff7cc0b2970 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc067c90 tx=0x7ff7bc004750 comp rx=0 tx=0).stop 2023-11-28T12:59:17.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.707+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/4239804858 shutdown_connections 2023-11-28T12:59:17.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.707+0000 7ff7d3b4a700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7cc1604b0 0x7ff7cc158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:17.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.707+0000 7ff7d3b4a700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7b00682f0 0x7ff7b006a7a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:17.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.707+0000 7ff7d3b4a700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7cc0b0d70 0x7ff7cc158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:17.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.707+0000 7ff7d3b4a700 1 --2- 172.21.15.111:0/4239804858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7cc0cc550 0x7ff7cc0b2970 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:17.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.707+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/4239804858 >> 172.21.15.111:0/4239804858 conn(0x7ff7cc098ee0 msgr2=0x7ff7cc0af760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:17.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.707+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/4239804858 shutdown_connections 2023-11-28T12:59:17.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.707+0000 7ff7d3b4a700 1 -- 172.21.15.111:0/4239804858 wait complete. 2023-11-28T12:59:17.723 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T12:59:17.724 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T12:59:17.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.848+0000 7faee71dd700 1 Processor -- start 2023-11-28T12:59:17.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.848+0000 7faee71dd700 1 -- start start 2023-11-28T12:59:17.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faee71dd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee00b0d70 0x7faee01604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:17.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faee71dd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faee00b1140 0x7faee00ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:17.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faee71dd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee015cd90 con 0x7faee00b15b0 2023-11-28T12:59:17.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faee71dd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee01613e0 con 0x7faee00b1140 2023-11-28T12:59:17.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faee71dd700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee0160b40 con 0x7faee00b0d70 2023-11-28T12:59:17.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faedffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faee00b1140 0x7faee00ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:17.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faee577a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7faee00b15b0 0x7faee00cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33876/0 (socket says 172.21.15.111:33876) 2023-11-28T12:59:17.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faee577a700 1 -- 172.21.15.111:0/2176918063 learned_addr learned my addr 172.21.15.111:0/2176918063 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:17.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faee4f79700 1 --2- 172.21.15.111:0/2176918063 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee00b0d70 0x7faee01604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:17.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faedffff700 1 -- 172.21.15.111:0/2176918063 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee00b0d70 msgr2=0x7faee01604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faedffff700 1 --2- 172.21.15.111:0/2176918063 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee00b0d70 0x7faee01604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faedffff700 1 -- 172.21.15.111:0/2176918063 >> v1:172.21.15.111:6789/0 conn(0x7faee00b15b0 legacy=0x7faee00cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faedffff700 1 -- 172.21.15.111:0/2176918063 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee0160e20 con 0x7faee00b1140 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.849+0000 7faedffff700 1 --2- 172.21.15.111:0/2176918063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee00b1140 0x7faee00ce6c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7faecc009840 tx=0x7faecc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3c32ba2d5f493 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.850+0000 7faedf7fe700 1 -- 172.21.15.111:0/2176918063 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc011830 con 0x7faee00b1140 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.850+0000 7faedf7fe700 1 -- 172.21.15.111:0/2176918063 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faecc011990 con 0x7faee00b1140 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.850+0000 7faedf7fe700 1 -- 172.21.15.111:0/2176918063 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc011d20 con 0x7faee00b1140 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.850+0000 7faee71dd700 1 -- 172.21.15.111:0/2176918063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee00b1140 msgr2=0x7faee00ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.850+0000 7faee71dd700 1 --2- 172.21.15.111:0/2176918063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee00b1140 0x7faee00ce6c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7faecc009840 tx=0x7faecc009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.850+0000 7faee71dd700 1 -- 172.21.15.111:0/2176918063 shutdown_connections 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.850+0000 7faee71dd700 1 --2- 172.21.15.111:0/2176918063 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee00b0d70 0x7faee01604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.850+0000 7faee71dd700 1 --2- 172.21.15.111:0/2176918063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee00b1140 0x7faee00ce6c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.850+0000 7faee71dd700 1 -- 172.21.15.111:0/2176918063 >> 172.21.15.111:0/2176918063 conn(0x7faee0098ee0 msgr2=0x7faee006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:17.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.850+0000 7faee71dd700 1 -- 172.21.15.111:0/2176918063 shutdown_connections 2023-11-28T12:59:17.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.850+0000 7faee71dd700 1 -- 172.21.15.111:0/2176918063 wait complete. 2023-11-28T12:59:17.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.851+0000 7faee71dd700 1 Processor -- start 2023-11-28T12:59:17.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.851+0000 7faee71dd700 1 -- start start 2023-11-28T12:59:17.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.852+0000 7faee71dd700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee00b0d70 0x7faee015dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:17.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.852+0000 7faee71dd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee00b1140 0x7faee00ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:17.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.852+0000 7faee71dd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee00b15b0 0x7faee00adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:17.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.852+0000 7faee71dd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee015f150 con 0x7faee00b15b0 2023-11-28T12:59:17.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.852+0000 7faee71dd700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee015d0d0 con 0x7faee00b0d70 2023-11-28T12:59:17.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.852+0000 7faee71dd700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee015d3b0 con 0x7faee00b1140 2023-11-28T12:59:17.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.852+0000 7faee577a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee00b15b0 0x7faee00adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:17.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.852+0000 7faee577a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee00b15b0 0x7faee00adc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33104/0 (socket says 172.21.15.111:33104) 2023-11-28T12:59:17.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.852+0000 7faee577a700 1 -- 172.21.15.111:0/158149504 learned_addr learned my addr 172.21.15.111:0/158149504 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:17.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.852+0000 7faee4f79700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee00b0d70 0x7faee015dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:17.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.852+0000 7faedffff700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee00b1140 0x7faee00ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:17.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.853+0000 7faee577a700 1 -- 172.21.15.111:0/158149504 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee00b1140 msgr2=0x7faee00ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:17.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.853+0000 7faee577a700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee00b1140 0x7faee00ad6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:17.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.853+0000 7faee577a700 1 -- 172.21.15.111:0/158149504 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee00b0d70 msgr2=0x7faee015dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:17.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.853+0000 7faee577a700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee00b0d70 0x7faee015dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:17.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.853+0000 7faee577a700 1 -- 172.21.15.111:0/158149504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc0094a0 con 0x7faee00b15b0 2023-11-28T12:59:17.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.853+0000 7faee577a700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee00b15b0 0x7faee00adc00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faed800ca20 tx=0x7faed800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:17.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.853+0000 7faedd7fa700 1 -- 172.21.15.111:0/158149504 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed800a9e0 con 0x7faee00b15b0 2023-11-28T12:59:17.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.853+0000 7faedd7fa700 1 -- 172.21.15.111:0/158149504 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed800ab40 con 0x7faee00b15b0 2023-11-28T12:59:17.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.853+0000 7faee71dd700 1 -- 172.21.15.111:0/158149504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee00af340 con 0x7faee00b15b0 2023-11-28T12:59:17.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.853+0000 7faedd7fa700 1 -- 172.21.15.111:0/158149504 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8012400 con 0x7faee00b15b0 2023-11-28T12:59:17.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.853+0000 7faee71dd700 1 -- 172.21.15.111:0/158149504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee0161100 con 0x7faee00b15b0 2023-11-28T12:59:17.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.854+0000 7faee71dd700 1 -- 172.21.15.111:0/158149504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee00cec00 con 0x7faee00b15b0 2023-11-28T12:59:17.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.854+0000 7faedd7fa700 1 -- 172.21.15.111:0/158149504 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faed8019030 con 0x7faee00b15b0 2023-11-28T12:59:17.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.854+0000 7faedd7fa700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faec40682f0 0x7faec406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:17.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.855+0000 7faee4f79700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faec40682f0 0x7faec406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:17.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.855+0000 7faedd7fa700 1 -- 172.21.15.111:0/158149504 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7faed8085620 con 0x7faee00b15b0 2023-11-28T12:59:17.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.855+0000 7faee4f79700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faec40682f0 0x7faec406a7a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faed4006fd0 tx=0x7faed4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:17.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:17.859+0000 7faedd7fa700 1 -- 172.21.15.111:0/158149504 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faed804b5e0 con 0x7faee00b15b0 2023-11-28T12:59:18.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.081+0000 7faee71dd700 1 -- 172.21.15.111:0/158149504 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faee0067170 con 0x7faee00b15b0 2023-11-28T12:59:18.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.082+0000 7faedd7fa700 1 -- 172.21.15.111:0/158149504 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4656 (secure 0 0 0) 0x7faed8051c40 con 0x7faee00b15b0 2023-11-28T12:59:18.083 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:18.083 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2513950989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2513950989},{"type":"v1","addr":"172.21.15.149:6834","nonce":2513950989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1258891714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1258891714},{"type":"v1","addr":"172.21.15.149:6835","nonce":1258891714}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/3923909288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3923909288},{"type":"v1","addr":"172.21.15.111:6836","nonce":3923909288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T12:58:55.633507+0000","modified":"2023-11-28T12:59:17.702148+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/3180164366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":3180164366},{"type":"v1","addr":"172.21.15.111:6837","nonce":3180164366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T12:59:18.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.084+0000 7faee71dd700 1 -- 172.21.15.111:0/158149504 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faec40682f0 msgr2=0x7faec406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.084+0000 7faee71dd700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faec40682f0 0x7faec406a7a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faed4006fd0 tx=0x7faed4008040 comp rx=0 tx=0).stop 2023-11-28T12:59:18.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.084+0000 7faee71dd700 1 -- 172.21.15.111:0/158149504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee00b15b0 msgr2=0x7faee00adc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.084+0000 7faee71dd700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee00b15b0 0x7faee00adc00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faed800ca20 tx=0x7faed800cd30 comp rx=0 tx=0).stop 2023-11-28T12:59:18.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.084+0000 7faee71dd700 1 -- 172.21.15.111:0/158149504 shutdown_connections 2023-11-28T12:59:18.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.084+0000 7faee71dd700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faee00b15b0 0x7faee00adc00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.084+0000 7faee71dd700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faec40682f0 0x7faec406a7a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.084+0000 7faee71dd700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faee00b0d70 0x7faee015dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.084+0000 7faee71dd700 1 --2- 172.21.15.111:0/158149504 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faee00b1140 0x7faee00ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.084+0000 7faee71dd700 1 -- 172.21.15.111:0/158149504 >> 172.21.15.111:0/158149504 conn(0x7faee0098ee0 msgr2=0x7faee015c230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:18.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.084+0000 7faee71dd700 1 -- 172.21.15.111:0/158149504 shutdown_connections 2023-11-28T12:59:18.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.084+0000 7faee71dd700 1 -- 172.21.15.111:0/158149504 wait complete. 2023-11-28T12:59:18.087 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 9 2023-11-28T12:59:18.098 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:18.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.215+0000 7f6ce762e700 1 Processor -- start 2023-11-28T12:59:18.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.216+0000 7f6ce762e700 1 -- start start 2023-11-28T12:59:18.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.216+0000 7f6ce762e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f6ce01596e0 0x7f6ce015cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.216+0000 7f6ce762e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce015de70 con 0x7f6ce0159310 2023-11-28T12:59:18.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.216+0000 7f6ce762e700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce00ced10 con 0x7f6ce015d510 2023-11-28T12:59:18.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.216+0000 7f6ce762e700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce015d980 con 0x7f6ce01596e0 2023-11-28T12:59:18.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6ce53ca700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6ce0159310 0x7f6ce00ce1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33886/0 (socket says 172.21.15.111:33886) 2023-11-28T12:59:18.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6ce53ca700 1 -- 172.21.15.111:0/2285153865 learned_addr learned my addr 172.21.15.111:0/2285153865 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:18.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580957971 0 0) 0x7f6ce015de70 con 0x7f6ce0159310 2023-11-28T12:59:18.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ccc0035b0 con 0x7f6ce0159310 2023-11-28T12:59:18.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6ce4bc9700 1 --2- 172.21.15.111:0/2285153865 >> v2:172.21.15.149:3300/0 conn(0x7f6ce01596e0 0x7f6ce015cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2777520652 0 0) 0x7f6ccc0035b0 con 0x7f6ce0159310 2023-11-28T12:59:18.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 >> v2:172.21.15.149:3300/0 conn(0x7f6ce01596e0 msgr2=0x7f6ce015cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:18.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 --2- 172.21.15.111:0/2285153865 >> v2:172.21.15.149:3300/0 conn(0x7f6ce01596e0 0x7f6ce015cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 >> v1:172.21.15.149:6789/0 conn(0x7f6ce015d510 legacy=0x7f6ce00cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce00ceff0 con 0x7f6ce0159310 2023-11-28T12:59:18.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f6cd0001670 con 0x7f6ce0159310 2023-11-28T12:59:18.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 >> v1:172.21.15.111:6789/0 conn(0x7f6ce0159310 legacy=0x7f6ce00ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 --2- 172.21.15.111:0/2285153865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ccc003a50 0x7f6ccc003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 --2- 172.21.15.111:0/2285153865 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ccc004850 0x7f6ccc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 --2- 172.21.15.111:0/2285153865 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ccc005250 0x7f6ccc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce015d980 con 0x7f6ccc003a50 2023-11-28T12:59:18.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc00c190 con 0x7f6ccc005250 2023-11-28T12:59:18.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc00c350 con 0x7f6ccc004850 2023-11-28T12:59:18.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6ce53ca700 1 --2- 172.21.15.111:0/2285153865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ccc003a50 0x7f6ccc003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6ce4bc9700 1 --2- 172.21.15.111:0/2285153865 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ccc004850 0x7f6ccc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.217+0000 7f6ce5bcb700 1 --2- 172.21.15.111:0/2285153865 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ccc005250 0x7f6ccc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.218+0000 7f6ce53ca700 1 -- 172.21.15.111:0/2285153865 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ccc004850 msgr2=0x7f6ccc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:18.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.218+0000 7f6ce53ca700 1 --2- 172.21.15.111:0/2285153865 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ccc004850 0x7f6ccc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.218+0000 7f6ce53ca700 1 -- 172.21.15.111:0/2285153865 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ccc005250 msgr2=0x7f6ccc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:18.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.218+0000 7f6ce53ca700 1 --2- 172.21.15.111:0/2285153865 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ccc005250 0x7f6ccc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.218+0000 7f6ce5bcb700 1 -- 172.21.15.111:0/2285153865 reap_dead start 2023-11-28T12:59:18.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.218+0000 7f6ce53ca700 1 -- 172.21.15.111:0/2285153865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ccc0035b0 con 0x7f6ccc003a50 2023-11-28T12:59:18.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.218+0000 7f6ce53ca700 1 --2- 172.21.15.111:0/2285153865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ccc003a50 0x7f6ccc003e60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0047d0 tx=0x7f6ccc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514b3630fc7ca7b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:18.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.218+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0007b70 con 0x7f6ccc003a50 2023-11-28T12:59:18.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.218+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd000ccc0 con 0x7f6ccc003a50 2023-11-28T12:59:18.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.218+0000 7f6cd7fff700 1 -- 172.21.15.111:0/2285153865 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0005ba0 con 0x7f6ccc003a50 2023-11-28T12:59:18.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.218+0000 7f6ce762e700 1 -- 172.21.15.111:0/2285153865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ccc003a50 msgr2=0x7f6ccc003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.218+0000 7f6ce762e700 1 --2- 172.21.15.111:0/2285153865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ccc003a50 0x7f6ccc003e60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0047d0 tx=0x7f6ccc005880 comp rx=0 tx=0).stop 2023-11-28T12:59:18.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.219+0000 7f6ce762e700 1 -- 172.21.15.111:0/2285153865 shutdown_connections 2023-11-28T12:59:18.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.219+0000 7f6ce762e700 1 --2- 172.21.15.111:0/2285153865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ccc003a50 0x7f6ccc003e60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.219+0000 7f6ce762e700 1 -- 172.21.15.111:0/2285153865 >> 172.21.15.111:0/2285153865 conn(0x7f6ce006db10 msgr2=0x7f6ce006df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:18.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.219+0000 7f6ce762e700 1 -- 172.21.15.111:0/2285153865 shutdown_connections 2023-11-28T12:59:18.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.219+0000 7f6ce762e700 1 -- 172.21.15.111:0/2285153865 wait complete. 2023-11-28T12:59:18.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.220+0000 7f6ce762e700 1 Processor -- start 2023-11-28T12:59:18.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.220+0000 7f6ce762e700 1 -- start start 2023-11-28T12:59:18.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce762e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ccc003a50 0x7f6ce0154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce762e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ce01e3e70 0x7f6ce01e42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce762e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ce00c73d0 0x7f6ce00c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce762e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce01e5570 con 0x7f6ccc003a50 2023-11-28T12:59:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce762e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce01e5990 con 0x7f6ce01e3e70 2023-11-28T12:59:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce762e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00c7d80 con 0x7f6ce00c73d0 2023-11-28T12:59:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce53ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ccc003a50 0x7f6ce0154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce53ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ccc003a50 0x7f6ce0154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33120/0 (socket says 172.21.15.111:33120) 2023-11-28T12:59:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce53ca700 1 -- 172.21.15.111:0/190474901 learned_addr learned my addr 172.21.15.111:0/190474901 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce4bc9700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ce01e3e70 0x7f6ce01e42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce5bcb700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ce00c73d0 0x7f6ce00c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce53ca700 1 -- 172.21.15.111:0/190474901 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ce00c73d0 msgr2=0x7f6ce00c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce53ca700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ce00c73d0 0x7f6ce00c7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce53ca700 1 -- 172.21.15.111:0/190474901 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ce01e3e70 msgr2=0x7f6ce01e42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce53ca700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ce01e3e70 0x7f6ce01e42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.221+0000 7f6ce53ca700 1 -- 172.21.15.111:0/190474901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce00cb0f0 con 0x7f6ccc003a50 2023-11-28T12:59:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.222+0000 7f6ce53ca700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ccc003a50 0x7f6ce0154230 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0005f60 tx=0x7f6cd0001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.222+0000 7f6cd5ffb700 1 -- 172.21.15.111:0/190474901 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd000cb60 con 0x7f6ccc003a50 2023-11-28T12:59:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.222+0000 7f6cd5ffb700 1 -- 172.21.15.111:0/190474901 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd0001650 con 0x7f6ccc003a50 2023-11-28T12:59:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.222+0000 7f6ce762e700 1 -- 172.21.15.111:0/190474901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce00cf2d0 con 0x7f6ccc003a50 2023-11-28T12:59:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.222+0000 7f6cd5ffb700 1 -- 172.21.15.111:0/190474901 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0007310 con 0x7f6ccc003a50 2023-11-28T12:59:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.222+0000 7f6ce762e700 1 -- 172.21.15.111:0/190474901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce00cb3d0 con 0x7f6ccc003a50 2023-11-28T12:59:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.223+0000 7f6ce762e700 1 -- 172.21.15.111:0/190474901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce006c4d0 con 0x7f6ccc003a50 2023-11-28T12:59:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.223+0000 7f6cd5ffb700 1 -- 172.21.15.111:0/190474901 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6cd0024030 con 0x7f6ccc003a50 2023-11-28T12:59:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.224+0000 7f6cd5ffb700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6cc40682f0 0x7f6cc406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.224+0000 7f6cd5ffb700 1 -- 172.21.15.111:0/190474901 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6cd00873c0 con 0x7f6ccc003a50 2023-11-28T12:59:18.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.224+0000 7f6ce4bc9700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6cc40682f0 0x7f6cc406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.224+0000 7f6ce4bc9700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6cc40682f0 0x7f6cc406a7a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00c34c0 tx=0x7f6cdc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:18.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.228+0000 7f6cd5ffb700 1 -- 172.21.15.111:0/190474901 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6cd00538b0 con 0x7f6ccc003a50 2023-11-28T12:59:18.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.414+0000 7f6ce762e700 1 -- 172.21.15.111:0/190474901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ce00b5b40 con 0x7f6ccc003a50 2023-11-28T12:59:18.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.415+0000 7f6cd5ffb700 1 -- 172.21.15.111:0/190474901 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19796 (secure 0 0 0) 0x7f6cd0056ed0 con 0x7f6ccc003a50 2023-11-28T12:59:18.416 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:18.417 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:00.053943+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:18.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.417+0000 7f6ce762e700 1 -- 172.21.15.111:0/190474901 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6cc40682f0 msgr2=0x7f6cc406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.417+0000 7f6ce762e700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6cc40682f0 0x7f6cc406a7a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00c34c0 tx=0x7f6cdc008040 comp rx=0 tx=0).stop 2023-11-28T12:59:18.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.417+0000 7f6ce762e700 1 -- 172.21.15.111:0/190474901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ccc003a50 msgr2=0x7f6ce0154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.417+0000 7f6ce762e700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ccc003a50 0x7f6ce0154230 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0005f60 tx=0x7f6cd0001930 comp rx=0 tx=0).stop 2023-11-28T12:59:18.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.417+0000 7f6ce762e700 1 -- 172.21.15.111:0/190474901 shutdown_connections 2023-11-28T12:59:18.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.417+0000 7f6ce762e700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ce00c73d0 0x7f6ce00c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.417+0000 7f6ce762e700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ccc003a50 0x7f6ce0154230 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.417+0000 7f6ce762e700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6cc40682f0 0x7f6cc406a7a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.417+0000 7f6ce762e700 1 --2- 172.21.15.111:0/190474901 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ce01e3e70 0x7f6ce01e42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.417+0000 7f6ce762e700 1 -- 172.21.15.111:0/190474901 >> 172.21.15.111:0/190474901 conn(0x7f6ce006db10 msgr2=0x7f6ce00b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:18.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.417+0000 7f6ce762e700 1 -- 172.21.15.111:0/190474901 shutdown_connections 2023-11-28T12:59:18.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.417+0000 7f6ce762e700 1 -- 172.21.15.111:0/190474901 wait complete. 2023-11-28T12:59:18.430 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T12:59:18.431 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T12:59:18.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.549+0000 7f630cbdf700 1 Processor -- start 2023-11-28T12:59:18.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.549+0000 7f630cbdf700 1 -- start start 2023-11-28T12:59:18.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.550+0000 7f630cbdf700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f630815a5d0 0x7f63080ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.550+0000 7f630cbdf700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63080b0160 0x7f63080ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.550+0000 7f630cbdf700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63080ae8b0 con 0x7f63080cb2d0 2023-11-28T12:59:18.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.550+0000 7f630cbdf700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63080cc1b0 con 0x7f630815a5d0 2023-11-28T12:59:18.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.550+0000 7f630cbdf700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63080cc390 con 0x7f63080b0160 2023-11-28T12:59:18.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.550+0000 7f630659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f630815a5d0 0x7f63080ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.550+0000 7f6306d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f63080cb2d0 0x7f630815e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33908/0 (socket says 172.21.15.111:33908) 2023-11-28T12:59:18.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.550+0000 7f6306d9d700 1 -- 172.21.15.111:0/3090629258 learned_addr learned my addr 172.21.15.111:0/3090629258 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:18.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.551+0000 7f6305d9b700 1 --2- 172.21.15.111:0/3090629258 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63080b0160 0x7f63080ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.551+0000 7f630659c700 1 -- 172.21.15.111:0/3090629258 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63080b0160 msgr2=0x7f63080ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:18.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.551+0000 7f630659c700 1 --2- 172.21.15.111:0/3090629258 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63080b0160 0x7f63080ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.551+0000 7f630659c700 1 -- 172.21.15.111:0/3090629258 >> v1:172.21.15.111:6789/0 conn(0x7f63080cb2d0 legacy=0x7f630815e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.551+0000 7f630659c700 1 -- 172.21.15.111:0/3090629258 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63080cc670 con 0x7f630815a5d0 2023-11-28T12:59:18.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.551+0000 7f630559a700 1 -- 172.21.15.111:0/3090629258 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124563926 0 0) 0x7f63080ae8b0 con 0x7f63080cb2d0 2023-11-28T12:59:18.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.551+0000 7f630659c700 1 --2- 172.21.15.111:0/3090629258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630815a5d0 0x7f63080ce850 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009820 tx=0x7f62fc009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7ffedea06bd43c0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:18.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.551+0000 7f630559a700 1 -- 172.21.15.111:0/3090629258 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc005e30 con 0x7f630815a5d0 2023-11-28T12:59:18.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.551+0000 7f630559a700 1 -- 172.21.15.111:0/3090629258 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc006030 con 0x7f630815a5d0 2023-11-28T12:59:18.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.551+0000 7f630559a700 1 -- 172.21.15.111:0/3090629258 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0063e0 con 0x7f630815a5d0 2023-11-28T12:59:18.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.552+0000 7f630cbdf700 1 -- 172.21.15.111:0/3090629258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630815a5d0 msgr2=0x7f63080ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.552+0000 7f630cbdf700 1 --2- 172.21.15.111:0/3090629258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630815a5d0 0x7f63080ce850 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009820 tx=0x7f62fc009b30 comp rx=0 tx=0).stop 2023-11-28T12:59:18.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.552+0000 7f630cbdf700 1 -- 172.21.15.111:0/3090629258 shutdown_connections 2023-11-28T12:59:18.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.552+0000 7f630cbdf700 1 --2- 172.21.15.111:0/3090629258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f630815a5d0 0x7f63080ce850 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.552+0000 7f630cbdf700 1 --2- 172.21.15.111:0/3090629258 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63080b0160 0x7f63080ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.552+0000 7f630cbdf700 1 -- 172.21.15.111:0/3090629258 >> 172.21.15.111:0/3090629258 conn(0x7f630809a040 msgr2=0x7f63080d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:18.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.552+0000 7f630cbdf700 1 -- 172.21.15.111:0/3090629258 shutdown_connections 2023-11-28T12:59:18.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.552+0000 7f630cbdf700 1 -- 172.21.15.111:0/3090629258 wait complete. 2023-11-28T12:59:18.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.553+0000 7f630cbdf700 1 Processor -- start 2023-11-28T12:59:18.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.553+0000 7f630cbdf700 1 -- start start 2023-11-28T12:59:18.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f630cbdf700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63080b0160 0x7f6308155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f630cbdf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63080cb2d0 0x7f63080b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f630cbdf700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f630815a5d0 0x7f63080b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f630cbdf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63080b4c70 con 0x7f63080cb2d0 2023-11-28T12:59:18.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f630cbdf700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63080b5ac0 con 0x7f63080b0160 2023-11-28T12:59:18.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f630cbdf700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63080c7e80 con 0x7f630815a5d0 2023-11-28T12:59:18.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f6305d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63080cb2d0 0x7f63080b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f6305d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63080cb2d0 0x7f63080b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33134/0 (socket says 172.21.15.111:33134) 2023-11-28T12:59:18.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f6305d9b700 1 -- 172.21.15.111:0/3742751638 learned_addr learned my addr 172.21.15.111:0/3742751638 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:18.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f6306d9d700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f630815a5d0 0x7f63080b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f6305d9b700 1 -- 172.21.15.111:0/3742751638 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f630815a5d0 msgr2=0x7f63080b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f6305d9b700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f630815a5d0 0x7f63080b4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f6305d9b700 1 -- 172.21.15.111:0/3742751638 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63080b0160 msgr2=0x7f6308155260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f630659c700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63080b0160 0x7f6308155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f6305d9b700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63080b0160 0x7f6308155260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f6305d9b700 1 -- 172.21.15.111:0/3742751638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc009430 con 0x7f63080cb2d0 2023-11-28T12:59:18.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.554+0000 7f6305d9b700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63080cb2d0 0x7f63080b3b40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f62f800e810 tx=0x7f62f800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:18.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.555+0000 7f62f6ffd700 1 -- 172.21.15.111:0/3742751638 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8013610 con 0x7f63080cb2d0 2023-11-28T12:59:18.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.555+0000 7f62f6ffd700 1 -- 172.21.15.111:0/3742751638 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f8013770 con 0x7f63080cb2d0 2023-11-28T12:59:18.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.555+0000 7f630cbdf700 1 -- 172.21.15.111:0/3742751638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63080ccc30 con 0x7f63080cb2d0 2023-11-28T12:59:18.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.555+0000 7f62f6ffd700 1 -- 172.21.15.111:0/3742751638 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f8013b20 con 0x7f63080cb2d0 2023-11-28T12:59:18.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.555+0000 7f630cbdf700 1 -- 172.21.15.111:0/3742751638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63080cc950 con 0x7f63080cb2d0 2023-11-28T12:59:18.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.555+0000 7f630cbdf700 1 -- 172.21.15.111:0/3742751638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63080ccf10 con 0x7f63080cb2d0 2023-11-28T12:59:18.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.556+0000 7f62f6ffd700 1 -- 172.21.15.111:0/3742751638 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f62f8023070 con 0x7f63080cb2d0 2023-11-28T12:59:18.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.556+0000 7f62f6ffd700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f62e80682f0 0x7f62e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.556+0000 7f630659c700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f62e80682f0 0x7f62e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.556+0000 7f62f6ffd700 1 -- 172.21.15.111:0/3742751638 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f62f8085be0 con 0x7f63080cb2d0 2023-11-28T12:59:18.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.557+0000 7f630659c700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f62e80682f0 0x7f62e806a7a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f63080702d0 tx=0x7f62fc008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:18.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.561+0000 7f62f6ffd700 1 -- 172.21.15.111:0/3742751638 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f62f80520d0 con 0x7f63080cb2d0 2023-11-28T12:59:18.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.740+0000 7f630cbdf700 1 -- 172.21.15.111:0/3742751638 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6308067170 con 0x7f62e80682f0 2023-11-28T12:59:18.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.750+0000 7f62f6ffd700 1 -- 172.21.15.111:0/3742751638 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367250 (secure 0 0 0) 0x7f6308067170 con 0x7f62e80682f0 2023-11-28T12:59:18.756 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:18.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.754+0000 7f630cbdf700 1 -- 172.21.15.111:0/3742751638 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f62e80682f0 msgr2=0x7f62e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.754+0000 7f630cbdf700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f62e80682f0 0x7f62e806a7a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f63080702d0 tx=0x7f62fc008070 comp rx=0 tx=0).stop 2023-11-28T12:59:18.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.754+0000 7f630cbdf700 1 -- 172.21.15.111:0/3742751638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63080cb2d0 msgr2=0x7f63080b3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.754+0000 7f630cbdf700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63080cb2d0 0x7f63080b3b40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f62f800e810 tx=0x7f62f800eb20 comp rx=0 tx=0).stop 2023-11-28T12:59:18.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.754+0000 7f630cbdf700 1 -- 172.21.15.111:0/3742751638 shutdown_connections 2023-11-28T12:59:18.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.754+0000 7f630cbdf700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f630815a5d0 0x7f63080b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.754+0000 7f630cbdf700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f62e80682f0 0x7f62e806a7a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.754+0000 7f630cbdf700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63080b0160 0x7f6308155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.754+0000 7f630cbdf700 1 --2- 172.21.15.111:0/3742751638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63080cb2d0 0x7f63080b3b40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.754+0000 7f630cbdf700 1 -- 172.21.15.111:0/3742751638 >> 172.21.15.111:0/3742751638 conn(0x7f630809a040 msgr2=0x7f63080af750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:18.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.754+0000 7f630cbdf700 1 -- 172.21.15.111:0/3742751638 shutdown_connections 2023-11-28T12:59:18.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.754+0000 7f630cbdf700 1 -- 172.21.15.111:0/3742751638 wait complete. 2023-11-28T12:59:18.761 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T12:59:18.768 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-11-28T12:59:18.337950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594896,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":67400,"kb_used_data":3400,"kb_used_omap":0,"kb_used_meta":63936,"kb_avail":754907320,"statfs":{"total":773094113280,"available":773025095680,"internally_reserved":0,"allocated":3481600,"data_stored":2108544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65470464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2198,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672369+0000","last_change":"2023-11-28T12:58:55.661175+0000","last_active":"2023-11-28T12:58:59.672369+0000","last_peered":"2023-11-28T12:58:59.672369+0000","last_clean":"2023-11-28T12:58:59.672369+0000","last_became_active":"2023-11-28T12:58:55.660919+0000","last_became_peered":"2023-11-28T12:58:55.660919+0000","last_unstale":"2023-11-28T12:58:59.672369+0000","last_undegraded":"2023-11-28T12:58:59.672369+0000","last_fullsized":"2023-11-28T12:58:59.672369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:12.137203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666089+0000","last_change":"2023-11-28T12:58:55.652749+0000","last_active":"2023-11-28T12:58:59.666089+0000","last_peered":"2023-11-28T12:58:59.666089+0000","last_clean":"2023-11-28T12:58:59.666089+0000","last_became_active":"2023-11-28T12:58:55.652544+0000","last_became_peered":"2023-11-28T12:58:55.652544+0000","last_unstale":"2023-11-28T12:58:59.666089+0000","last_undegraded":"2023-11-28T12:58:59.666089+0000","last_fullsized":"2023-11-28T12:58:59.666089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660310+0000","last_change":"2023-11-28T12:58:55.669208+0000","last_active":"2023-11-28T12:58:59.660310+0000","last_peered":"2023-11-28T12:58:59.660310+0000","last_clean":"2023-11-28T12:58:59.660310+0000","last_became_active":"2023-11-28T12:58:55.668930+0000","last_became_peered":"2023-11-28T12:58:55.668930+0000","last_unstale":"2023-11-28T12:58:59.660310+0000","last_undegraded":"2023-11-28T12:58:59.660310+0000","last_fullsized":"2023-11-28T12:58:59.660310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:18:05.065329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657123+0000","last_change":"2023-11-28T12:58:55.653156+0000","last_active":"2023-11-28T12:58:59.657123+0000","last_peered":"2023-11-28T12:58:59.657123+0000","last_clean":"2023-11-28T12:58:59.657123+0000","last_became_active":"2023-11-28T12:58:55.652820+0000","last_became_peered":"2023-11-28T12:58:55.652820+0000","last_unstale":"2023-11-28T12:58:59.657123+0000","last_undegraded":"2023-11-28T12:58:59.657123+0000","last_fullsized":"2023-11-28T12:58:59.657123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:19.178525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669305+0000","last_change":"2023-11-28T12:58:55.664471+0000","last_active":"2023-11-28T12:58:59.669305+0000","last_peered":"2023-11-28T12:58:59.669305+0000","last_clean":"2023-11-28T12:58:59.669305+0000","last_became_active":"2023-11-28T12:58:55.664221+0000","last_became_peered":"2023-11-28T12:58:55.664221+0000","last_unstale":"2023-11-28T12:58:59.669305+0000","last_undegraded":"2023-11-28T12:58:59.669305+0000","last_fullsized":"2023-11-28T12:58:59.669305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:34:15.223824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659490+0000","last_change":"2023-11-28T12:58:55.669109+0000","last_active":"2023-11-28T12:58:59.659490+0000","last_peered":"2023-11-28T12:58:59.659490+0000","last_clean":"2023-11-28T12:58:59.659490+0000","last_became_active":"2023-11-28T12:58:55.668857+0000","last_became_peered":"2023-11-28T12:58:55.668857+0000","last_unstale":"2023-11-28T12:58:59.659490+0000","last_undegraded":"2023-11-28T12:58:59.659490+0000","last_fullsized":"2023-11-28T12:58:59.659490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:14:02.504692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.663977+0000","last_change":"2023-11-28T12:58:55.662854+0000","last_active":"2023-11-28T12:58:59.663977+0000","last_peered":"2023-11-28T12:58:59.663977+0000","last_clean":"2023-11-28T12:58:59.663977+0000","last_became_active":"2023-11-28T12:58:55.662539+0000","last_became_peered":"2023-11-28T12:58:55.662539+0000","last_unstale":"2023-11-28T12:58:59.663977+0000","last_undegraded":"2023-11-28T12:58:59.663977+0000","last_fullsized":"2023-11-28T12:58:59.663977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661304+0000","last_change":"2023-11-28T12:58:55.657958+0000","last_active":"2023-11-28T12:58:59.661304+0000","last_peered":"2023-11-28T12:58:59.661304+0000","last_clean":"2023-11-28T12:58:59.661304+0000","last_became_active":"2023-11-28T12:58:55.657681+0000","last_became_peered":"2023-11-28T12:58:55.657681+0000","last_unstale":"2023-11-28T12:58:59.661304+0000","last_undegraded":"2023-11-28T12:58:59.661304+0000","last_fullsized":"2023-11-28T12:58:59.661304+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674142+0000","last_change":"2023-11-28T12:58:55.666485+0000","last_active":"2023-11-28T12:58:59.674142+0000","last_peered":"2023-11-28T12:58:59.674142+0000","last_clean":"2023-11-28T12:58:59.674142+0000","last_became_active":"2023-11-28T12:58:55.665670+0000","last_became_peered":"2023-11-28T12:58:55.665670+0000","last_unstale":"2023-11-28T12:58:59.674142+0000","last_undegraded":"2023-11-28T12:58:59.674142+0000","last_fullsized":"2023-11-28T12:58:59.674142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668534+0000","last_change":"2023-11-28T12:58:55.655129+0000","last_active":"2023-11-28T12:58:59.668534+0000","last_peered":"2023-11-28T12:58:59.668534+0000","last_clean":"2023-11-28T12:58:59.668534+0000","last_became_active":"2023-11-28T12:58:55.654839+0000","last_became_peered":"2023-11-28T12:58:55.654839+0000","last_unstale":"2023-11-28T12:58:59.668534+0000","last_undegraded":"2023-11-28T12:58:59.668534+0000","last_fullsized":"2023-11-28T12:58:59.668534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660394+0000","last_change":"2023-11-28T12:58:55.663796+0000","last_active":"2023-11-28T12:58:59.660394+0000","last_peered":"2023-11-28T12:58:59.660394+0000","last_clean":"2023-11-28T12:58:59.660394+0000","last_became_active":"2023-11-28T12:58:55.663553+0000","last_became_peered":"2023-11-28T12:58:55.663553+0000","last_unstale":"2023-11-28T12:58:59.660394+0000","last_undegraded":"2023-11-28T12:58:59.660394+0000","last_fullsized":"2023-11-28T12:58:59.660394+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:31:46.923203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672559+0000","last_change":"2023-11-28T12:58:55.666858+0000","last_active":"2023-11-28T12:58:59.672559+0000","last_peered":"2023-11-28T12:58:59.672559+0000","last_clean":"2023-11-28T12:58:59.672559+0000","last_became_active":"2023-11-28T12:58:55.666597+0000","last_became_peered":"2023-11-28T12:58:55.666597+0000","last_unstale":"2023-11-28T12:58:59.672559+0000","last_undegraded":"2023-11-28T12:58:59.672559+0000","last_fullsized":"2023-11-28T12:58:59.672559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:50.239136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.673902+0000","last_change":"2023-11-28T12:58:55.658497+0000","last_active":"2023-11-28T12:58:59.673902+0000","last_peered":"2023-11-28T12:58:59.673902+0000","last_clean":"2023-11-28T12:58:59.673902+0000","last_became_active":"2023-11-28T12:58:55.658258+0000","last_became_peered":"2023-11-28T12:58:55.658258+0000","last_unstale":"2023-11-28T12:58:59.673902+0000","last_undegraded":"2023-11-28T12:58:59.673902+0000","last_fullsized":"2023-11-28T12:58:59.673902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:38:46.935297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664436+0000","last_change":"2023-11-28T12:58:55.648629+0000","last_active":"2023-11-28T12:58:59.664436+0000","last_peered":"2023-11-28T12:58:59.664436+0000","last_clean":"2023-11-28T12:58:59.664436+0000","last_became_active":"2023-11-28T12:58:55.648143+0000","last_became_peered":"2023-11-28T12:58:55.648143+0000","last_unstale":"2023-11-28T12:58:59.664436+0000","last_undegraded":"2023-11-28T12:58:59.664436+0000","last_fullsized":"2023-11-28T12:58:59.664436+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:06:39.770893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.663795+0000","last_change":"2023-11-28T12:58:55.665560+0000","last_active":"2023-11-28T12:58:59.663795+0000","last_peered":"2023-11-28T12:58:59.663795+0000","last_clean":"2023-11-28T12:58:59.663795+0000","last_became_active":"2023-11-28T12:58:55.665304+0000","last_became_peered":"2023-11-28T12:58:55.665304+0000","last_unstale":"2023-11-28T12:58:59.663795+0000","last_undegraded":"2023-11-28T12:58:59.663795+0000","last_fullsized":"2023-11-28T12:58:59.663795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.658903+0000","last_change":"2023-11-28T12:58:55.669222+0000","last_active":"2023-11-28T12:58:59.658903+0000","last_peered":"2023-11-28T12:58:59.658903+0000","last_clean":"2023-11-28T12:58:59.658903+0000","last_became_active":"2023-11-28T12:58:55.668931+0000","last_became_peered":"2023-11-28T12:58:55.668931+0000","last_unstale":"2023-11-28T12:58:59.658903+0000","last_undegraded":"2023-11-28T12:58:59.658903+0000","last_fullsized":"2023-11-28T12:58:59.658903+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:12:28.351912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674186+0000","last_change":"2023-11-28T12:58:55.666643+0000","last_active":"2023-11-28T12:58:59.674186+0000","last_peered":"2023-11-28T12:58:59.674186+0000","last_clean":"2023-11-28T12:58:59.674186+0000","last_became_active":"2023-11-28T12:58:55.666224+0000","last_became_peered":"2023-11-28T12:58:55.666224+0000","last_unstale":"2023-11-28T12:58:59.674186+0000","last_undegraded":"2023-11-28T12:58:59.674186+0000","last_fullsized":"2023-11-28T12:58:59.674186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660392+0000","last_change":"2023-11-28T12:58:55.660396+0000","last_active":"2023-11-28T12:58:59.660392+0000","last_peered":"2023-11-28T12:58:59.660392+0000","last_clean":"2023-11-28T12:58:59.660392+0000","last_became_active":"2023-11-28T12:58:55.659518+0000","last_became_peered":"2023-11-28T12:58:55.659518+0000","last_unstale":"2023-11-28T12:58:59.660392+0000","last_undegraded":"2023-11-28T12:58:59.660392+0000","last_fullsized":"2023-11-28T12:58:59.660392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:49:50.356082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672413+0000","last_change":"2023-11-28T12:58:55.650573+0000","last_active":"2023-11-28T12:58:59.672413+0000","last_peered":"2023-11-28T12:58:59.672413+0000","last_clean":"2023-11-28T12:58:59.672413+0000","last_became_active":"2023-11-28T12:58:55.650382+0000","last_became_peered":"2023-11-28T12:58:55.650382+0000","last_unstale":"2023-11-28T12:58:59.672413+0000","last_undegraded":"2023-11-28T12:58:59.672413+0000","last_fullsized":"2023-11-28T12:58:59.672413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:54:07.154200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659749+0000","last_change":"2023-11-28T12:58:55.663920+0000","last_active":"2023-11-28T12:58:59.659749+0000","last_peered":"2023-11-28T12:58:59.659749+0000","last_clean":"2023-11-28T12:58:59.659749+0000","last_became_active":"2023-11-28T12:58:55.663347+0000","last_became_peered":"2023-11-28T12:58:55.663347+0000","last_unstale":"2023-11-28T12:58:59.659749+0000","last_undegraded":"2023-11-28T12:58:59.659749+0000","last_fullsized":"2023-11-28T12:58:59.659749+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664143+0000","last_change":"2023-11-28T12:58:55.661755+0000","last_active":"2023-11-28T12:58:59.664143+0000","last_peered":"2023-11-28T12:58:59.664143+0000","last_clean":"2023-11-28T12:58:59.664143+0000","last_became_active":"2023-11-28T12:58:55.661496+0000","last_became_peered":"2023-11-28T12:58:55.661496+0000","last_unstale":"2023-11-28T12:58:59.664143+0000","last_undegraded":"2023-11-28T12:58:59.664143+0000","last_fullsized":"2023-11-28T12:58:59.664143+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669008+0000","last_change":"2023-11-28T12:58:55.666002+0000","last_active":"2023-11-28T12:58:59.669008+0000","last_peered":"2023-11-28T12:58:59.669008+0000","last_clean":"2023-11-28T12:58:59.669008+0000","last_became_active":"2023-11-28T12:58:55.657322+0000","last_became_peered":"2023-11-28T12:58:55.657322+0000","last_unstale":"2023-11-28T12:58:59.669008+0000","last_undegraded":"2023-11-28T12:58:59.669008+0000","last_fullsized":"2023-11-28T12:58:59.669008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655373+0000","last_change":"2023-11-28T12:58:55.669042+0000","last_active":"2023-11-28T12:58:59.655373+0000","last_peered":"2023-11-28T12:58:59.655373+0000","last_clean":"2023-11-28T12:58:59.655373+0000","last_became_active":"2023-11-28T12:58:55.668810+0000","last_became_peered":"2023-11-28T12:58:55.668810+0000","last_unstale":"2023-11-28T12:58:59.655373+0000","last_undegraded":"2023-11-28T12:58:59.655373+0000","last_fullsized":"2023-11-28T12:58:59.655373+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:49:50.356082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666243+0000","last_change":"2023-11-28T12:58:55.658006+0000","last_active":"2023-11-28T12:58:59.666243+0000","last_peered":"2023-11-28T12:58:59.666243+0000","last_clean":"2023-11-28T12:58:59.666243+0000","last_became_active":"2023-11-28T12:58:55.657791+0000","last_became_peered":"2023-11-28T12:58:55.657791+0000","last_unstale":"2023-11-28T12:58:59.666243+0000","last_undegraded":"2023-11-28T12:58:59.666243+0000","last_fullsized":"2023-11-28T12:58:59.666243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667574+0000","last_change":"2023-11-28T12:58:55.658052+0000","last_active":"2023-11-28T12:58:59.667574+0000","last_peered":"2023-11-28T12:58:59.667574+0000","last_clean":"2023-11-28T12:58:59.667574+0000","last_became_active":"2023-11-28T12:58:55.657752+0000","last_became_peered":"2023-11-28T12:58:55.657752+0000","last_unstale":"2023-11-28T12:58:59.667574+0000","last_undegraded":"2023-11-28T12:58:59.667574+0000","last_fullsized":"2023-11-28T12:58:59.667574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665870+0000","last_change":"2023-11-28T12:58:55.657589+0000","last_active":"2023-11-28T12:58:59.665870+0000","last_peered":"2023-11-28T12:58:59.665870+0000","last_clean":"2023-11-28T12:58:59.665870+0000","last_became_active":"2023-11-28T12:58:55.654770+0000","last_became_peered":"2023-11-28T12:58:55.654770+0000","last_unstale":"2023-11-28T12:58:59.665870+0000","last_undegraded":"2023-11-28T12:58:59.665870+0000","last_fullsized":"2023-11-28T12:58:59.665870+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:06:39.770893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665748+0000","last_change":"2023-11-28T12:58:55.657244+0000","last_active":"2023-11-28T12:58:59.665748+0000","last_peered":"2023-11-28T12:58:59.665748+0000","last_clean":"2023-11-28T12:58:59.665748+0000","last_became_active":"2023-11-28T12:58:55.656984+0000","last_became_peered":"2023-11-28T12:58:55.656984+0000","last_unstale":"2023-11-28T12:58:59.665748+0000","last_undegraded":"2023-11-28T12:58:59.665748+0000","last_fullsized":"2023-11-28T12:58:59.665748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059204+0000","last_change":"2023-11-28T12:58:59.681861+0000","last_active":"2023-11-28T12:59:00.059204+0000","last_peered":"2023-11-28T12:59:00.059204+0000","last_clean":"2023-11-28T12:59:00.059204+0000","last_became_active":"2023-11-28T12:58:59.681503+0000","last_became_peered":"2023-11-28T12:58:59.681503+0000","last_unstale":"2023-11-28T12:59:00.059204+0000","last_undegraded":"2023-11-28T12:59:00.059204+0000","last_fullsized":"2023-11-28T12:59:00.059204+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:27.673942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659368+0000","last_change":"2023-11-28T12:58:55.248299+0000","last_active":"2023-11-28T12:58:59.659368+0000","last_peered":"2023-11-28T12:58:59.659368+0000","last_clean":"2023-11-28T12:58:59.659368+0000","last_became_active":"2023-11-28T12:58:55.247956+0000","last_became_peered":"2023-11-28T12:58:55.247956+0000","last_unstale":"2023-11-28T12:58:59.659368+0000","last_undegraded":"2023-11-28T12:58:59.659368+0000","last_fullsized":"2023-11-28T12:58:59.659368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:25:52.356939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655726+0000","last_change":"2023-11-28T12:58:55.660586+0000","last_active":"2023-11-28T12:58:59.655726+0000","last_peered":"2023-11-28T12:58:59.655726+0000","last_clean":"2023-11-28T12:58:59.655726+0000","last_became_active":"2023-11-28T12:58:55.658146+0000","last_became_peered":"2023-11-28T12:58:55.658146+0000","last_unstale":"2023-11-28T12:58:59.655726+0000","last_undegraded":"2023-11-28T12:58:59.655726+0000","last_fullsized":"2023-11-28T12:58:59.655726+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058649+0000","last_change":"2023-11-28T12:58:59.672932+0000","last_active":"2023-11-28T12:59:00.058649+0000","last_peered":"2023-11-28T12:59:00.058649+0000","last_clean":"2023-11-28T12:59:00.058649+0000","last_became_active":"2023-11-28T12:58:59.672540+0000","last_became_peered":"2023-11-28T12:58:59.672540+0000","last_unstale":"2023-11-28T12:59:00.058649+0000","last_undegraded":"2023-11-28T12:59:00.058649+0000","last_fullsized":"2023-11-28T12:59:00.058649+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:12:31.778491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661524+0000","last_change":"2023-11-28T12:58:55.247493+0000","last_active":"2023-11-28T12:58:59.661524+0000","last_peered":"2023-11-28T12:58:59.661524+0000","last_clean":"2023-11-28T12:58:59.661524+0000","last_became_active":"2023-11-28T12:58:55.247218+0000","last_became_peered":"2023-11-28T12:58:55.247218+0000","last_unstale":"2023-11-28T12:58:59.661524+0000","last_undegraded":"2023-11-28T12:58:59.661524+0000","last_fullsized":"2023-11-28T12:58:59.661524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:38:45.929395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674242+0000","last_change":"2023-11-28T12:58:55.666790+0000","last_active":"2023-11-28T12:58:59.674242+0000","last_peered":"2023-11-28T12:58:59.674242+0000","last_clean":"2023-11-28T12:58:59.674242+0000","last_became_active":"2023-11-28T12:58:55.666541+0000","last_became_peered":"2023-11-28T12:58:55.666541+0000","last_unstale":"2023-11-28T12:58:59.674242+0000","last_undegraded":"2023-11-28T12:58:59.674242+0000","last_fullsized":"2023-11-28T12:58:59.674242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313692+0000","last_change":"2023-11-28T12:58:59.685314+0000","last_active":"2023-11-28T12:59:00.313692+0000","last_peered":"2023-11-28T12:59:00.313692+0000","last_clean":"2023-11-28T12:59:00.313692+0000","last_became_active":"2023-11-28T12:58:59.684940+0000","last_became_peered":"2023-11-28T12:58:59.684940+0000","last_unstale":"2023-11-28T12:59:00.313692+0000","last_undegraded":"2023-11-28T12:59:00.313692+0000","last_fullsized":"2023-11-28T12:59:00.313692+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:53.458898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657621+0000","last_change":"2023-11-28T12:58:55.256308+0000","last_active":"2023-11-28T12:58:59.657621+0000","last_peered":"2023-11-28T12:58:59.657621+0000","last_clean":"2023-11-28T12:58:59.657621+0000","last_became_active":"2023-11-28T12:58:55.256020+0000","last_became_peered":"2023-11-28T12:58:55.256020+0000","last_unstale":"2023-11-28T12:58:59.657621+0000","last_undegraded":"2023-11-28T12:58:59.657621+0000","last_fullsized":"2023-11-28T12:58:59.657621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:49.233234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659104+0000","last_change":"2023-11-28T12:58:55.658284+0000","last_active":"2023-11-28T12:58:59.659104+0000","last_peered":"2023-11-28T12:58:59.659104+0000","last_clean":"2023-11-28T12:58:59.659104+0000","last_became_active":"2023-11-28T12:58:55.657995+0000","last_became_peered":"2023-11-28T12:58:55.657995+0000","last_unstale":"2023-11-28T12:58:59.659104+0000","last_undegraded":"2023-11-28T12:58:59.659104+0000","last_fullsized":"2023-11-28T12:58:59.659104+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:11:49.546022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058116+0000","last_change":"2023-11-28T12:58:59.669835+0000","last_active":"2023-11-28T12:59:00.058116+0000","last_peered":"2023-11-28T12:59:00.058116+0000","last_clean":"2023-11-28T12:59:00.058116+0000","last_became_active":"2023-11-28T12:58:59.669428+0000","last_became_peered":"2023-11-28T12:58:59.669428+0000","last_unstale":"2023-11-28T12:59:00.058116+0000","last_undegraded":"2023-11-28T12:59:00.058116+0000","last_fullsized":"2023-11-28T12:59:00.058116+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:18:08.491909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674258+0000","last_change":"2023-11-28T12:58:55.242328+0000","last_active":"2023-11-28T12:58:59.674258+0000","last_peered":"2023-11-28T12:58:59.674258+0000","last_clean":"2023-11-28T12:58:59.674258+0000","last_became_active":"2023-11-28T12:58:55.242028+0000","last_became_peered":"2023-11-28T12:58:55.242028+0000","last_unstale":"2023-11-28T12:58:59.674258+0000","last_undegraded":"2023-11-28T12:58:59.674258+0000","last_fullsized":"2023-11-28T12:58:59.674258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:42.300078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655600+0000","last_change":"2023-11-28T12:58:55.655732+0000","last_active":"2023-11-28T12:58:59.655600+0000","last_peered":"2023-11-28T12:58:59.655600+0000","last_clean":"2023-11-28T12:58:59.655600+0000","last_became_active":"2023-11-28T12:58:55.655490+0000","last_became_peered":"2023-11-28T12:58:55.655490+0000","last_unstale":"2023-11-28T12:58:59.655600+0000","last_undegraded":"2023-11-28T12:58:59.655600+0000","last_fullsized":"2023-11-28T12:58:59.655600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829559+0000","last_change":"2023-11-28T12:58:59.676365+0000","last_active":"2023-11-28T12:59:00.829559+0000","last_peered":"2023-11-28T12:59:00.829559+0000","last_clean":"2023-11-28T12:59:00.829559+0000","last_became_active":"2023-11-28T12:58:59.675934+0000","last_became_peered":"2023-11-28T12:58:59.675934+0000","last_unstale":"2023-11-28T12:59:00.829559+0000","last_undegraded":"2023-11-28T12:59:00.829559+0000","last_fullsized":"2023-11-28T12:59:00.829559+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:04:35.226907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668457+0000","last_change":"2023-11-28T12:58:55.244542+0000","last_active":"2023-11-28T12:58:59.668457+0000","last_peered":"2023-11-28T12:58:59.668457+0000","last_clean":"2023-11-28T12:58:59.668457+0000","last_became_active":"2023-11-28T12:58:55.244251+0000","last_became_peered":"2023-11-28T12:58:55.244251+0000","last_unstale":"2023-11-28T12:58:59.668457+0000","last_undegraded":"2023-11-28T12:58:59.668457+0000","last_fullsized":"2023-11-28T12:58:59.668457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:06.134459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655451+0000","last_change":"2023-11-28T12:58:55.661735+0000","last_active":"2023-11-28T12:58:59.655451+0000","last_peered":"2023-11-28T12:58:59.655451+0000","last_clean":"2023-11-28T12:58:59.655451+0000","last_became_active":"2023-11-28T12:58:55.661468+0000","last_became_peered":"2023-11-28T12:58:55.661468+0000","last_unstale":"2023-11-28T12:58:59.655451+0000","last_undegraded":"2023-11-28T12:58:59.655451+0000","last_fullsized":"2023-11-28T12:58:59.655451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:34.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059839+0000","last_change":"2023-11-28T12:58:59.677031+0000","last_active":"2023-11-28T12:59:00.059839+0000","last_peered":"2023-11-28T12:59:00.059839+0000","last_clean":"2023-11-28T12:59:00.059839+0000","last_became_active":"2023-11-28T12:58:59.676510+0000","last_became_peered":"2023-11-28T12:58:59.676510+0000","last_unstale":"2023-11-28T12:59:00.059839+0000","last_undegraded":"2023-11-28T12:59:00.059839+0000","last_fullsized":"2023-11-28T12:59:00.059839+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:03:33.042937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"21'3","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664803+0000","last_change":"2023-11-28T12:58:55.254041+0000","last_active":"2023-11-28T12:58:59.664803+0000","last_peered":"2023-11-28T12:58:59.664803+0000","last_clean":"2023-11-28T12:58:59.664803+0000","last_became_active":"2023-11-28T12:58:55.253797+0000","last_became_peered":"2023-11-28T12:58:55.253797+0000","last_unstale":"2023-11-28T12:58:59.664803+0000","last_undegraded":"2023-11-28T12:58:59.664803+0000","last_fullsized":"2023-11-28T12:58:59.664803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:42.442743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664769+0000","last_change":"2023-11-28T12:58:55.648632+0000","last_active":"2023-11-28T12:58:59.664769+0000","last_peered":"2023-11-28T12:58:59.664769+0000","last_clean":"2023-11-28T12:58:59.664769+0000","last_became_active":"2023-11-28T12:58:55.648367+0000","last_became_peered":"2023-11-28T12:58:55.648367+0000","last_unstale":"2023-11-28T12:58:59.664769+0000","last_undegraded":"2023-11-28T12:58:59.664769+0000","last_fullsized":"2023-11-28T12:58:59.664769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.315401+0000","last_change":"2023-11-28T12:58:59.675361+0000","last_active":"2023-11-28T12:59:00.315401+0000","last_peered":"2023-11-28T12:59:00.315401+0000","last_clean":"2023-11-28T12:59:00.315401+0000","last_became_active":"2023-11-28T12:58:59.674991+0000","last_became_peered":"2023-11-28T12:58:59.674991+0000","last_unstale":"2023-11-28T12:59:00.315401+0000","last_undegraded":"2023-11-28T12:59:00.315401+0000","last_fullsized":"2023-11-28T12:59:00.315401+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:21:52.978291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"26'9","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:04.555380+0000","last_change":"2023-11-28T12:58:55.245189+0000","last_active":"2023-11-28T12:59:04.555380+0000","last_peered":"2023-11-28T12:59:04.555380+0000","last_clean":"2023-11-28T12:59:04.555380+0000","last_became_active":"2023-11-28T12:58:55.244896+0000","last_became_peered":"2023-11-28T12:58:55.244896+0000","last_unstale":"2023-11-28T12:59:04.555380+0000","last_undegraded":"2023-11-28T12:59:04.555380+0000","last_fullsized":"2023-11-28T12:59:04.555380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3746,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659437+0000","last_change":"2023-11-28T12:58:55.658816+0000","last_active":"2023-11-28T12:58:59.659437+0000","last_peered":"2023-11-28T12:58:59.659437+0000","last_clean":"2023-11-28T12:58:59.659437+0000","last_became_active":"2023-11-28T12:58:55.657238+0000","last_became_peered":"2023-11-28T12:58:55.657238+0000","last_unstale":"2023-11-28T12:58:59.659437+0000","last_undegraded":"2023-11-28T12:58:59.659437+0000","last_fullsized":"2023-11-28T12:58:59.659437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:14:28.821164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830316+0000","last_change":"2023-11-28T12:58:59.685273+0000","last_active":"2023-11-28T12:59:00.830316+0000","last_peered":"2023-11-28T12:59:00.830316+0000","last_clean":"2023-11-28T12:59:00.830316+0000","last_became_active":"2023-11-28T12:58:59.684917+0000","last_became_peered":"2023-11-28T12:58:59.684917+0000","last_unstale":"2023-11-28T12:59:00.830316+0000","last_undegraded":"2023-11-28T12:59:00.830316+0000","last_fullsized":"2023-11-28T12:59:00.830316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:31.632801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664060+0000","last_change":"2023-11-28T12:58:55.249201+0000","last_active":"2023-11-28T12:58:59.664060+0000","last_peered":"2023-11-28T12:58:59.664060+0000","last_clean":"2023-11-28T12:58:59.664060+0000","last_became_active":"2023-11-28T12:58:55.248828+0000","last_became_peered":"2023-11-28T12:58:55.248828+0000","last_unstale":"2023-11-28T12:58:59.664060+0000","last_undegraded":"2023-11-28T12:58:59.664060+0000","last_fullsized":"2023-11-28T12:58:59.664060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:11:48.540120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655687+0000","last_change":"2023-11-28T12:58:55.660397+0000","last_active":"2023-11-28T12:58:59.655687+0000","last_peered":"2023-11-28T12:58:59.655687+0000","last_clean":"2023-11-28T12:58:59.655687+0000","last_became_active":"2023-11-28T12:58:55.657972+0000","last_became_peered":"2023-11-28T12:58:55.657972+0000","last_unstale":"2023-11-28T12:58:59.655687+0000","last_undegraded":"2023-11-28T12:58:59.655687+0000","last_fullsized":"2023-11-28T12:58:59.655687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059788+0000","last_change":"2023-11-28T12:58:59.681226+0000","last_active":"2023-11-28T12:59:00.059788+0000","last_peered":"2023-11-28T12:59:00.059788+0000","last_clean":"2023-11-28T12:59:00.059788+0000","last_became_active":"2023-11-28T12:58:59.680846+0000","last_became_peered":"2023-11-28T12:58:59.680846+0000","last_unstale":"2023-11-28T12:59:00.059788+0000","last_undegraded":"2023-11-28T12:59:00.059788+0000","last_fullsized":"2023-11-28T12:59:00.059788+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:59.373707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668909+0000","last_change":"2023-11-28T12:58:55.246343+0000","last_active":"2023-11-28T12:58:59.668909+0000","last_peered":"2023-11-28T12:58:59.668909+0000","last_clean":"2023-11-28T12:58:59.668909+0000","last_became_active":"2023-11-28T12:58:55.244840+0000","last_became_peered":"2023-11-28T12:58:55.244840+0000","last_unstale":"2023-11-28T12:58:59.668909+0000","last_undegraded":"2023-11-28T12:58:59.668909+0000","last_fullsized":"2023-11-28T12:58:59.668909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665665+0000","last_change":"2023-11-28T12:58:55.656948+0000","last_active":"2023-11-28T12:58:59.665665+0000","last_peered":"2023-11-28T12:58:59.665665+0000","last_clean":"2023-11-28T12:58:59.665665+0000","last_became_active":"2023-11-28T12:58:55.656678+0000","last_became_peered":"2023-11-28T12:58:55.656678+0000","last_unstale":"2023-11-28T12:58:59.665665+0000","last_undegraded":"2023-11-28T12:58:59.665665+0000","last_fullsized":"2023-11-28T12:58:59.665665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.060115+0000","last_change":"2023-11-28T12:58:59.677384+0000","last_active":"2023-11-28T12:59:00.060115+0000","last_peered":"2023-11-28T12:59:00.060115+0000","last_clean":"2023-11-28T12:59:00.060115+0000","last_became_active":"2023-11-28T12:58:59.676997+0000","last_became_peered":"2023-11-28T12:58:59.676997+0000","last_unstale":"2023-11-28T12:59:00.060115+0000","last_undegraded":"2023-11-28T12:59:00.060115+0000","last_fullsized":"2023-11-28T12:59:00.060115+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:31.632801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.663889+0000","last_change":"2023-11-28T12:58:55.249928+0000","last_active":"2023-11-28T12:58:59.663889+0000","last_peered":"2023-11-28T12:58:59.663889+0000","last_clean":"2023-11-28T12:58:59.663889+0000","last_became_active":"2023-11-28T12:58:55.249500+0000","last_became_peered":"2023-11-28T12:58:55.249500+0000","last_unstale":"2023-11-28T12:58:59.663889+0000","last_undegraded":"2023-11-28T12:58:59.663889+0000","last_fullsized":"2023-11-28T12:58:59.663889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:18.172623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674319+0000","last_change":"2023-11-28T12:58:55.663464+0000","last_active":"2023-11-28T12:58:59.674319+0000","last_peered":"2023-11-28T12:58:59.674319+0000","last_clean":"2023-11-28T12:58:59.674319+0000","last_became_active":"2023-11-28T12:58:55.663205+0000","last_became_peered":"2023-11-28T12:58:55.663205+0000","last_unstale":"2023-11-28T12:58:59.674319+0000","last_undegraded":"2023-11-28T12:58:59.674319+0000","last_fullsized":"2023-11-28T12:58:59.674319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.831228+0000","last_change":"2023-11-28T12:58:59.675282+0000","last_active":"2023-11-28T12:59:00.831228+0000","last_peered":"2023-11-28T12:59:00.831228+0000","last_clean":"2023-11-28T12:59:00.831228+0000","last_became_active":"2023-11-28T12:58:59.674859+0000","last_became_peered":"2023-11-28T12:58:59.674859+0000","last_unstale":"2023-11-28T12:59:00.831228+0000","last_undegraded":"2023-11-28T12:59:00.831228+0000","last_fullsized":"2023-11-28T12:59:00.831228+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:59.373707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"21'8","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.656905+0000","last_change":"2023-11-28T12:58:55.236728+0000","last_active":"2023-11-28T12:58:59.656905+0000","last_peered":"2023-11-28T12:58:59.656905+0000","last_clean":"2023-11-28T12:58:59.656905+0000","last_became_active":"2023-11-28T12:58:55.236452+0000","last_became_peered":"2023-11-28T12:58:55.236452+0000","last_unstale":"2023-11-28T12:58:59.656905+0000","last_undegraded":"2023-11-28T12:58:59.656905+0000","last_fullsized":"2023-11-28T12:58:59.656905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655960+0000","last_change":"2023-11-28T12:58:55.662005+0000","last_active":"2023-11-28T12:58:59.655960+0000","last_peered":"2023-11-28T12:58:59.655960+0000","last_clean":"2023-11-28T12:58:59.655960+0000","last_became_active":"2023-11-28T12:58:55.661531+0000","last_became_peered":"2023-11-28T12:58:55.661531+0000","last_unstale":"2023-11-28T12:58:59.655960+0000","last_undegraded":"2023-11-28T12:58:59.655960+0000","last_fullsized":"2023-11-28T12:58:59.655960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:34:15.223824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830169+0000","last_change":"2023-11-28T12:58:59.675000+0000","last_active":"2023-11-28T12:59:00.830169+0000","last_peered":"2023-11-28T12:59:00.830169+0000","last_clean":"2023-11-28T12:59:00.830169+0000","last_became_active":"2023-11-28T12:58:59.674646+0000","last_became_peered":"2023-11-28T12:58:59.674646+0000","last_unstale":"2023-11-28T12:59:00.830169+0000","last_undegraded":"2023-11-28T12:59:00.830169+0000","last_fullsized":"2023-11-28T12:59:00.830169+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:46:17.417359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667637+0000","last_change":"2023-11-28T12:58:55.249124+0000","last_active":"2023-11-28T12:58:59.667637+0000","last_peered":"2023-11-28T12:58:59.667637+0000","last_clean":"2023-11-28T12:58:59.667637+0000","last_became_active":"2023-11-28T12:58:55.248501+0000","last_became_peered":"2023-11-28T12:58:55.248501+0000","last_unstale":"2023-11-28T12:58:59.667637+0000","last_undegraded":"2023-11-28T12:58:59.667637+0000","last_fullsized":"2023-11-28T12:58:59.667637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:25.569310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661198+0000","last_change":"2023-11-28T12:58:55.659587+0000","last_active":"2023-11-28T12:58:59.661198+0000","last_peered":"2023-11-28T12:58:59.661198+0000","last_clean":"2023-11-28T12:58:59.661198+0000","last_became_active":"2023-11-28T12:58:55.659319+0000","last_became_peered":"2023-11-28T12:58:55.659319+0000","last_unstale":"2023-11-28T12:58:59.661198+0000","last_undegraded":"2023-11-28T12:58:59.661198+0000","last_fullsized":"2023-11-28T12:58:59.661198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.828936+0000","last_change":"2023-11-28T12:58:59.686412+0000","last_active":"2023-11-28T12:59:00.828936+0000","last_peered":"2023-11-28T12:59:00.828936+0000","last_clean":"2023-11-28T12:59:00.828936+0000","last_became_active":"2023-11-28T12:58:59.686079+0000","last_became_peered":"2023-11-28T12:58:59.686079+0000","last_unstale":"2023-11-28T12:59:00.828936+0000","last_undegraded":"2023-11-28T12:59:00.828936+0000","last_fullsized":"2023-11-28T12:59:00.828936+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:58:48.492561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664149+0000","last_change":"2023-11-28T12:58:55.248876+0000","last_active":"2023-11-28T12:58:59.664149+0000","last_peered":"2023-11-28T12:58:59.664149+0000","last_clean":"2023-11-28T12:58:59.664149+0000","last_became_active":"2023-11-28T12:58:55.248584+0000","last_became_peered":"2023-11-28T12:58:55.248584+0000","last_unstale":"2023-11-28T12:58:59.664149+0000","last_undegraded":"2023-11-28T12:58:59.664149+0000","last_fullsized":"2023-11-28T12:58:59.664149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:47:12.446132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659149+0000","last_change":"2023-11-28T12:58:55.663768+0000","last_active":"2023-11-28T12:58:59.659149+0000","last_peered":"2023-11-28T12:58:59.659149+0000","last_clean":"2023-11-28T12:58:59.659149+0000","last_became_active":"2023-11-28T12:58:55.663476+0000","last_became_peered":"2023-11-28T12:58:55.663476+0000","last_unstale":"2023-11-28T12:58:59.659149+0000","last_undegraded":"2023-11-28T12:58:59.659149+0000","last_fullsized":"2023-11-28T12:58:59.659149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:07:49.345226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829075+0000","last_change":"2023-11-28T12:58:59.685644+0000","last_active":"2023-11-28T12:59:00.829075+0000","last_peered":"2023-11-28T12:59:00.829075+0000","last_clean":"2023-11-28T12:59:00.829075+0000","last_became_active":"2023-11-28T12:58:59.685259+0000","last_became_peered":"2023-11-28T12:58:59.685259+0000","last_unstale":"2023-11-28T12:59:00.829075+0000","last_undegraded":"2023-11-28T12:59:00.829075+0000","last_fullsized":"2023-11-28T12:59:00.829075+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:07:52.771805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664976+0000","last_change":"2023-11-28T12:58:55.247686+0000","last_active":"2023-11-28T12:58:59.664976+0000","last_peered":"2023-11-28T12:58:59.664976+0000","last_clean":"2023-11-28T12:58:59.664976+0000","last_became_active":"2023-11-28T12:58:55.247328+0000","last_became_peered":"2023-11-28T12:58:55.247328+0000","last_unstale":"2023-11-28T12:58:59.664976+0000","last_undegraded":"2023-11-28T12:58:59.664976+0000","last_fullsized":"2023-11-28T12:58:59.664976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:22:00.609577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.656940+0000","last_change":"2023-11-28T12:58:55.649490+0000","last_active":"2023-11-28T12:58:59.656940+0000","last_peered":"2023-11-28T12:58:59.656940+0000","last_clean":"2023-11-28T12:58:59.656940+0000","last_became_active":"2023-11-28T12:58:55.649143+0000","last_became_peered":"2023-11-28T12:58:55.649143+0000","last_unstale":"2023-11-28T12:58:59.656940+0000","last_undegraded":"2023-11-28T12:58:59.656940+0000","last_fullsized":"2023-11-28T12:58:59.656940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:26.575212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829146+0000","last_change":"2023-11-28T12:58:59.679465+0000","last_active":"2023-11-28T12:59:00.829146+0000","last_peered":"2023-11-28T12:59:00.829146+0000","last_clean":"2023-11-28T12:59:00.829146+0000","last_became_active":"2023-11-28T12:58:59.679026+0000","last_became_peered":"2023-11-28T12:58:59.679026+0000","last_unstale":"2023-11-28T12:59:00.829146+0000","last_undegraded":"2023-11-28T12:59:00.829146+0000","last_fullsized":"2023-11-28T12:59:00.829146+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:52.339645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665301+0000","last_change":"2023-11-28T12:58:55.249645+0000","last_active":"2023-11-28T12:58:59.665301+0000","last_peered":"2023-11-28T12:58:59.665301+0000","last_clean":"2023-11-28T12:58:59.665301+0000","last_became_active":"2023-11-28T12:58:55.249284+0000","last_became_peered":"2023-11-28T12:58:55.249284+0000","last_unstale":"2023-11-28T12:58:59.665301+0000","last_undegraded":"2023-11-28T12:58:59.665301+0000","last_fullsized":"2023-11-28T12:58:59.665301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:49.233234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669390+0000","last_change":"2023-11-28T12:58:55.663994+0000","last_active":"2023-11-28T12:58:59.669390+0000","last_peered":"2023-11-28T12:58:59.669390+0000","last_clean":"2023-11-28T12:58:59.669390+0000","last_became_active":"2023-11-28T12:58:55.657280+0000","last_became_peered":"2023-11-28T12:58:55.657280+0000","last_unstale":"2023-11-28T12:58:59.669390+0000","last_undegraded":"2023-11-28T12:58:59.669390+0000","last_fullsized":"2023-11-28T12:58:59.669390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:34.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829349+0000","last_change":"2023-11-28T12:58:59.680307+0000","last_active":"2023-11-28T12:59:00.829349+0000","last_peered":"2023-11-28T12:59:00.829349+0000","last_clean":"2023-11-28T12:59:00.829349+0000","last_became_active":"2023-11-28T12:58:59.679837+0000","last_became_peered":"2023-11-28T12:58:59.679837+0000","last_unstale":"2023-11-28T12:59:00.829349+0000","last_undegraded":"2023-11-28T12:59:00.829349+0000","last_fullsized":"2023-11-28T12:59:00.829349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:18:08.491909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655480+0000","last_change":"2023-11-28T12:58:55.256347+0000","last_active":"2023-11-28T12:58:59.655480+0000","last_peered":"2023-11-28T12:58:59.655480+0000","last_clean":"2023-11-28T12:58:59.655480+0000","last_became_active":"2023-11-28T12:58:55.256027+0000","last_became_peered":"2023-11-28T12:58:55.256027+0000","last_unstale":"2023-11-28T12:58:59.655480+0000","last_undegraded":"2023-11-28T12:58:59.655480+0000","last_fullsized":"2023-11-28T12:58:59.655480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:42.300078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659538+0000","last_change":"2023-11-28T12:58:55.664332+0000","last_active":"2023-11-28T12:58:59.659538+0000","last_peered":"2023-11-28T12:58:59.659538+0000","last_clean":"2023-11-28T12:58:59.659538+0000","last_became_active":"2023-11-28T12:58:55.663913+0000","last_became_peered":"2023-11-28T12:58:55.663913+0000","last_unstale":"2023-11-28T12:58:59.659538+0000","last_undegraded":"2023-11-28T12:58:59.659538+0000","last_fullsized":"2023-11-28T12:58:59.659538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:50.590629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.315498+0000","last_change":"2023-11-28T12:58:59.678114+0000","last_active":"2023-11-28T12:59:00.315498+0000","last_peered":"2023-11-28T12:59:00.315498+0000","last_clean":"2023-11-28T12:59:00.315498+0000","last_became_active":"2023-11-28T12:58:59.677704+0000","last_became_peered":"2023-11-28T12:58:59.677704+0000","last_unstale":"2023-11-28T12:59:00.315498+0000","last_undegraded":"2023-11-28T12:59:00.315498+0000","last_fullsized":"2023-11-28T12:59:00.315498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:53.458898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657011+0000","last_change":"2023-11-28T12:58:55.241207+0000","last_active":"2023-11-28T12:58:59.657011+0000","last_peered":"2023-11-28T12:58:59.657011+0000","last_clean":"2023-11-28T12:58:59.657011+0000","last_became_active":"2023-11-28T12:58:55.240917+0000","last_became_peered":"2023-11-28T12:58:55.240917+0000","last_unstale":"2023-11-28T12:58:59.657011+0000","last_undegraded":"2023-11-28T12:58:59.657011+0000","last_fullsized":"2023-11-28T12:58:59.657011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659347+0000","last_change":"2023-11-28T12:58:55.660463+0000","last_active":"2023-11-28T12:58:59.659347+0000","last_peered":"2023-11-28T12:58:59.659347+0000","last_clean":"2023-11-28T12:58:59.659347+0000","last_became_active":"2023-11-28T12:58:55.660208+0000","last_became_peered":"2023-11-28T12:58:55.660208+0000","last_unstale":"2023-11-28T12:58:59.659347+0000","last_undegraded":"2023-11-28T12:58:59.659347+0000","last_fullsized":"2023-11-28T12:58:59.659347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830829+0000","last_change":"2023-11-28T12:58:59.688100+0000","last_active":"2023-11-28T12:59:00.830829+0000","last_peered":"2023-11-28T12:59:00.830829+0000","last_clean":"2023-11-28T12:59:00.830829+0000","last_became_active":"2023-11-28T12:58:59.687752+0000","last_became_peered":"2023-11-28T12:58:59.687752+0000","last_unstale":"2023-11-28T12:59:00.830829+0000","last_undegraded":"2023-11-28T12:59:00.830829+0000","last_fullsized":"2023-11-28T12:59:00.830829+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:38:05.937526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659014+0000","last_change":"2023-11-28T12:58:55.246754+0000","last_active":"2023-11-28T12:58:59.659014+0000","last_peered":"2023-11-28T12:58:59.659014+0000","last_clean":"2023-11-28T12:58:59.659014+0000","last_became_active":"2023-11-28T12:58:55.246480+0000","last_became_peered":"2023-11-28T12:58:55.246480+0000","last_unstale":"2023-11-28T12:58:59.659014+0000","last_undegraded":"2023-11-28T12:58:59.659014+0000","last_fullsized":"2023-11-28T12:58:59.659014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:08.923409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660472+0000","last_change":"2023-11-28T12:58:55.660027+0000","last_active":"2023-11-28T12:58:59.660472+0000","last_peered":"2023-11-28T12:58:59.660472+0000","last_clean":"2023-11-28T12:58:59.660472+0000","last_became_active":"2023-11-28T12:58:55.659775+0000","last_became_peered":"2023-11-28T12:58:55.659775+0000","last_unstale":"2023-11-28T12:58:59.660472+0000","last_undegraded":"2023-11-28T12:58:59.660472+0000","last_fullsized":"2023-11-28T12:58:59.660472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058758+0000","last_change":"2023-11-28T12:58:59.681700+0000","last_active":"2023-11-28T12:59:00.058758+0000","last_peered":"2023-11-28T12:59:00.058758+0000","last_clean":"2023-11-28T12:59:00.058758+0000","last_became_active":"2023-11-28T12:58:59.681305+0000","last_became_peered":"2023-11-28T12:58:59.681305+0000","last_unstale":"2023-11-28T12:59:00.058758+0000","last_undegraded":"2023-11-28T12:59:00.058758+0000","last_fullsized":"2023-11-28T12:59:00.058758+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:46.532522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655511+0000","last_change":"2023-11-28T12:58:55.249674+0000","last_active":"2023-11-28T12:58:59.655511+0000","last_peered":"2023-11-28T12:58:59.655511+0000","last_clean":"2023-11-28T12:58:59.655511+0000","last_became_active":"2023-11-28T12:58:55.249391+0000","last_became_peered":"2023-11-28T12:58:55.249391+0000","last_unstale":"2023-11-28T12:58:59.655511+0000","last_undegraded":"2023-11-28T12:58:59.655511+0000","last_fullsized":"2023-11-28T12:58:59.655511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667665+0000","last_change":"2023-11-28T12:58:55.661741+0000","last_active":"2023-11-28T12:58:59.667665+0000","last_peered":"2023-11-28T12:58:59.667665+0000","last_clean":"2023-11-28T12:58:59.667665+0000","last_became_active":"2023-11-28T12:58:55.661473+0000","last_became_peered":"2023-11-28T12:58:55.661473+0000","last_unstale":"2023-11-28T12:58:59.667665+0000","last_undegraded":"2023-11-28T12:58:59.667665+0000","last_fullsized":"2023-11-28T12:58:59.667665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:41:27.479117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313283+0000","last_change":"2023-11-28T12:58:59.675425+0000","last_active":"2023-11-28T12:59:00.313283+0000","last_peered":"2023-11-28T12:59:00.313283+0000","last_clean":"2023-11-28T12:59:00.313283+0000","last_became_active":"2023-11-28T12:58:59.675044+0000","last_became_peered":"2023-11-28T12:58:59.675044+0000","last_unstale":"2023-11-28T12:59:00.313283+0000","last_undegraded":"2023-11-28T12:59:00.313283+0000","last_fullsized":"2023-11-28T12:59:00.313283+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:46:17.417359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664599+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T12:58:59.664599+0000","last_peered":"2023-11-28T12:58:59.664599+0000","last_clean":"2023-11-28T12:58:59.664599+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T12:58:59.664599+0000","last_undegraded":"2023-11-28T12:58:59.664599+0000","last_fullsized":"2023-11-28T12:58:59.664599+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665575+0000","last_change":"2023-11-28T12:58:55.247919+0000","last_active":"2023-11-28T12:58:59.665575+0000","last_peered":"2023-11-28T12:58:59.665575+0000","last_clean":"2023-11-28T12:58:59.665575+0000","last_became_active":"2023-11-28T12:58:55.247653+0000","last_became_peered":"2023-11-28T12:58:55.247653+0000","last_unstale":"2023-11-28T12:58:59.665575+0000","last_undegraded":"2023-11-28T12:58:59.665575+0000","last_fullsized":"2023-11-28T12:58:59.665575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:54:06.148298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664140+0000","last_change":"2023-11-28T12:58:55.662430+0000","last_active":"2023-11-28T12:58:59.664140+0000","last_peered":"2023-11-28T12:58:59.664140+0000","last_clean":"2023-11-28T12:58:59.664140+0000","last_became_active":"2023-11-28T12:58:55.657899+0000","last_became_peered":"2023-11-28T12:58:55.657899+0000","last_unstale":"2023-11-28T12:58:59.664140+0000","last_undegraded":"2023-11-28T12:58:59.664140+0000","last_fullsized":"2023-11-28T12:58:59.664140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:22:01.615479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313377+0000","last_change":"2023-11-28T12:58:59.675356+0000","last_active":"2023-11-28T12:59:00.313377+0000","last_peered":"2023-11-28T12:59:00.313377+0000","last_clean":"2023-11-28T12:59:00.313377+0000","last_became_active":"2023-11-28T12:58:59.674917+0000","last_became_peered":"2023-11-28T12:58:59.674917+0000","last_unstale":"2023-11-28T12:59:00.313377+0000","last_undegraded":"2023-11-28T12:59:00.313377+0000","last_fullsized":"2023-11-28T12:59:00.313377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:51:00.263485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661311+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T12:58:59.661311+0000","last_peered":"2023-11-28T12:58:59.661311+0000","last_clean":"2023-11-28T12:58:59.661311+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T12:58:59.661311+0000","last_undegraded":"2023-11-28T12:58:59.661311+0000","last_fullsized":"2023-11-28T12:58:59.661311+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659552+0000","last_change":"2023-11-28T12:58:55.240294+0000","last_active":"2023-11-28T12:58:59.659552+0000","last_peered":"2023-11-28T12:58:59.659552+0000","last_clean":"2023-11-28T12:58:59.659552+0000","last_became_active":"2023-11-28T12:58:55.239947+0000","last_became_peered":"2023-11-28T12:58:55.239947+0000","last_unstale":"2023-11-28T12:58:59.659552+0000","last_undegraded":"2023-11-28T12:58:59.659552+0000","last_fullsized":"2023-11-28T12:58:59.659552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664509+0000","last_change":"2023-11-28T12:58:55.657537+0000","last_active":"2023-11-28T12:58:59.664509+0000","last_peered":"2023-11-28T12:58:59.664509+0000","last_clean":"2023-11-28T12:58:59.664509+0000","last_became_active":"2023-11-28T12:58:55.655666+0000","last_became_peered":"2023-11-28T12:58:55.655666+0000","last_unstale":"2023-11-28T12:58:59.664509+0000","last_undegraded":"2023-11-28T12:58:59.664509+0000","last_fullsized":"2023-11-28T12:58:59.664509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829428+0000","last_change":"2023-11-28T12:58:59.680479+0000","last_active":"2023-11-28T12:59:00.829428+0000","last_peered":"2023-11-28T12:59:00.829428+0000","last_clean":"2023-11-28T12:59:00.829428+0000","last_became_active":"2023-11-28T12:58:59.680089+0000","last_became_peered":"2023-11-28T12:58:59.680089+0000","last_unstale":"2023-11-28T12:59:00.829428+0000","last_undegraded":"2023-11-28T12:59:00.829428+0000","last_fullsized":"2023-11-28T12:59:00.829428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:12:31.778491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.673610+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T12:58:59.673610+0000","last_peered":"2023-11-28T12:58:59.673610+0000","last_clean":"2023-11-28T12:58:59.673610+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T12:58:59.673610+0000","last_undegraded":"2023-11-28T12:58:59.673610+0000","last_fullsized":"2023-11-28T12:58:59.673610+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660849+0000","last_change":"2023-11-28T12:58:55.238872+0000","last_active":"2023-11-28T12:58:59.660849+0000","last_peered":"2023-11-28T12:58:59.660849+0000","last_clean":"2023-11-28T12:58:59.660849+0000","last_became_active":"2023-11-28T12:58:55.238457+0000","last_became_peered":"2023-11-28T12:58:55.238457+0000","last_unstale":"2023-11-28T12:58:59.660849+0000","last_undegraded":"2023-11-28T12:58:59.660849+0000","last_fullsized":"2023-11-28T12:58:59.660849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667641+0000","last_change":"2023-11-28T12:58:55.659175+0000","last_active":"2023-11-28T12:58:59.667641+0000","last_peered":"2023-11-28T12:58:59.667641+0000","last_clean":"2023-11-28T12:58:59.667641+0000","last_became_active":"2023-11-28T12:58:55.658043+0000","last_became_peered":"2023-11-28T12:58:55.658043+0000","last_unstale":"2023-11-28T12:58:59.667641+0000","last_undegraded":"2023-11-28T12:58:59.667641+0000","last_fullsized":"2023-11-28T12:58:59.667641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:43.448645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830642+0000","last_change":"2023-11-28T12:58:59.682758+0000","last_active":"2023-11-28T12:59:00.830642+0000","last_peered":"2023-11-28T12:59:00.830642+0000","last_clean":"2023-11-28T12:59:00.830642+0000","last_became_active":"2023-11-28T12:58:59.682429+0000","last_became_peered":"2023-11-28T12:58:59.682429+0000","last_unstale":"2023-11-28T12:59:00.830642+0000","last_undegraded":"2023-11-28T12:59:00.830642+0000","last_fullsized":"2023-11-28T12:59:00.830642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:59.373707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668370+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T12:58:59.668370+0000","last_peered":"2023-11-28T12:58:59.668370+0000","last_clean":"2023-11-28T12:58:59.668370+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T12:58:59.668370+0000","last_undegraded":"2023-11-28T12:58:59.668370+0000","last_fullsized":"2023-11-28T12:58:59.668370+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666175+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T12:58:59.666175+0000","last_peered":"2023-11-28T12:58:59.666175+0000","last_clean":"2023-11-28T12:58:59.666175+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T12:58:59.666175+0000","last_undegraded":"2023-11-28T12:58:59.666175+0000","last_fullsized":"2023-11-28T12:58:59.666175+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655438+0000","last_change":"2023-11-28T12:58:55.248780+0000","last_active":"2023-11-28T12:58:59.655438+0000","last_peered":"2023-11-28T12:58:59.655438+0000","last_clean":"2023-11-28T12:58:59.655438+0000","last_became_active":"2023-11-28T12:58:55.248456+0000","last_became_peered":"2023-11-28T12:58:55.248456+0000","last_unstale":"2023-11-28T12:58:59.655438+0000","last_undegraded":"2023-11-28T12:58:59.655438+0000","last_fullsized":"2023-11-28T12:58:59.655438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:25:52.356939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657383+0000","last_change":"2023-11-28T12:58:55.661508+0000","last_active":"2023-11-28T12:58:59.657383+0000","last_peered":"2023-11-28T12:58:59.657383+0000","last_clean":"2023-11-28T12:58:59.657383+0000","last_became_active":"2023-11-28T12:58:55.661252+0000","last_became_peered":"2023-11-28T12:58:55.661252+0000","last_unstale":"2023-11-28T12:58:59.657383+0000","last_undegraded":"2023-11-28T12:58:59.657383+0000","last_fullsized":"2023-11-28T12:58:59.657383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:22:01.615479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830736+0000","last_change":"2023-11-28T12:58:59.675557+0000","last_active":"2023-11-28T12:59:00.830736+0000","last_peered":"2023-11-28T12:59:00.830736+0000","last_clean":"2023-11-28T12:59:00.830736+0000","last_became_active":"2023-11-28T12:58:59.675183+0000","last_became_peered":"2023-11-28T12:58:59.675183+0000","last_unstale":"2023-11-28T12:59:00.830736+0000","last_undegraded":"2023-11-28T12:59:00.830736+0000","last_fullsized":"2023-11-28T12:59:00.830736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:17.338033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666288+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T12:58:59.666288+0000","last_peered":"2023-11-28T12:58:59.666288+0000","last_clean":"2023-11-28T12:58:59.666288+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T12:58:59.666288+0000","last_undegraded":"2023-11-28T12:58:59.666288+0000","last_fullsized":"2023-11-28T12:58:59.666288+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668355+0000","last_change":"2023-11-28T12:58:55.246924+0000","last_active":"2023-11-28T12:58:59.668355+0000","last_peered":"2023-11-28T12:58:59.668355+0000","last_clean":"2023-11-28T12:58:59.668355+0000","last_became_active":"2023-11-28T12:58:55.244984+0000","last_became_peered":"2023-11-28T12:58:55.244984+0000","last_unstale":"2023-11-28T12:58:59.668355+0000","last_undegraded":"2023-11-28T12:58:59.668355+0000","last_fullsized":"2023-11-28T12:58:59.668355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674088+0000","last_change":"2023-11-28T12:58:55.666303+0000","last_active":"2023-11-28T12:58:59.674088+0000","last_peered":"2023-11-28T12:58:59.674088+0000","last_clean":"2023-11-28T12:58:59.674088+0000","last_became_active":"2023-11-28T12:58:55.666058+0000","last_became_peered":"2023-11-28T12:58:55.666058+0000","last_unstale":"2023-11-28T12:58:59.674088+0000","last_undegraded":"2023-11-28T12:58:59.674088+0000","last_fullsized":"2023-11-28T12:58:59.674088+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058107+0000","last_change":"2023-11-28T12:58:59.681611+0000","last_active":"2023-11-28T12:59:00.058107+0000","last_peered":"2023-11-28T12:59:00.058107+0000","last_clean":"2023-11-28T12:59:00.058107+0000","last_became_active":"2023-11-28T12:58:59.681195+0000","last_became_peered":"2023-11-28T12:58:59.681195+0000","last_unstale":"2023-11-28T12:59:00.058107+0000","last_undegraded":"2023-11-28T12:59:00.058107+0000","last_fullsized":"2023-11-28T12:59:00.058107+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:58:48.492561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664884+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T12:58:59.664884+0000","last_peered":"2023-11-28T12:58:59.664884+0000","last_clean":"2023-11-28T12:58:59.664884+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T12:58:59.664884+0000","last_undegraded":"2023-11-28T12:58:59.664884+0000","last_fullsized":"2023-11-28T12:58:59.664884+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664779+0000","last_change":"2023-11-28T12:58:55.240088+0000","last_active":"2023-11-28T12:58:59.664779+0000","last_peered":"2023-11-28T12:58:59.664779+0000","last_clean":"2023-11-28T12:58:59.664779+0000","last_became_active":"2023-11-28T12:58:55.237949+0000","last_became_peered":"2023-11-28T12:58:55.237949+0000","last_unstale":"2023-11-28T12:58:59.664779+0000","last_undegraded":"2023-11-28T12:58:59.664779+0000","last_fullsized":"2023-11-28T12:58:59.664779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669188+0000","last_change":"2023-11-28T12:58:55.657049+0000","last_active":"2023-11-28T12:58:59.669188+0000","last_peered":"2023-11-28T12:58:59.669188+0000","last_clean":"2023-11-28T12:58:59.669188+0000","last_became_active":"2023-11-28T12:58:55.656787+0000","last_became_peered":"2023-11-28T12:58:55.656787+0000","last_unstale":"2023-11-28T12:58:59.669188+0000","last_undegraded":"2023-11-28T12:58:59.669188+0000","last_fullsized":"2023-11-28T12:58:59.669188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:06:39.770893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058517+0000","last_change":"2023-11-28T12:58:59.676382+0000","last_active":"2023-11-28T12:59:00.058517+0000","last_peered":"2023-11-28T12:59:00.058517+0000","last_clean":"2023-11-28T12:59:00.058517+0000","last_became_active":"2023-11-28T12:58:59.675939+0000","last_became_peered":"2023-11-28T12:58:59.675939+0000","last_unstale":"2023-11-28T12:59:00.058517+0000","last_undegraded":"2023-11-28T12:59:00.058517+0000","last_fullsized":"2023-11-28T12:59:00.058517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:50:50.704627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.656336+0000","last_change":"2023-11-28T12:58:55.250390+0000","last_active":"2023-11-28T12:58:59.656336+0000","last_peered":"2023-11-28T12:58:59.656336+0000","last_clean":"2023-11-28T12:58:59.656336+0000","last_became_active":"2023-11-28T12:58:55.249919+0000","last_became_peered":"2023-11-28T12:58:55.249919+0000","last_unstale":"2023-11-28T12:58:59.656336+0000","last_undegraded":"2023-11-28T12:58:59.656336+0000","last_fullsized":"2023-11-28T12:58:59.656336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:11.131301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665787+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T12:58:59.665787+0000","last_peered":"2023-11-28T12:58:59.665787+0000","last_clean":"2023-11-28T12:58:59.665787+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T12:58:59.665787+0000","last_undegraded":"2023-11-28T12:58:59.665787+0000","last_fullsized":"2023-11-28T12:58:59.665787+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655406+0000","last_change":"2023-11-28T12:58:55.661761+0000","last_active":"2023-11-28T12:58:59.655406+0000","last_peered":"2023-11-28T12:58:59.655406+0000","last_clean":"2023-11-28T12:58:59.655406+0000","last_became_active":"2023-11-28T12:58:55.661488+0000","last_became_peered":"2023-11-28T12:58:55.661488+0000","last_unstale":"2023-11-28T12:58:59.655406+0000","last_undegraded":"2023-11-28T12:58:59.655406+0000","last_fullsized":"2023-11-28T12:58:59.655406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:14:28.821164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059438+0000","last_change":"2023-11-28T12:58:59.678960+0000","last_active":"2023-11-28T12:59:00.059438+0000","last_peered":"2023-11-28T12:59:00.059438+0000","last_clean":"2023-11-28T12:59:00.059438+0000","last_became_active":"2023-11-28T12:58:59.678568+0000","last_became_peered":"2023-11-28T12:58:59.678568+0000","last_unstale":"2023-11-28T12:59:00.059438+0000","last_undegraded":"2023-11-28T12:59:00.059438+0000","last_fullsized":"2023-11-28T12:59:00.059438+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:17.338033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659020+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T12:58:59.659020+0000","last_peered":"2023-11-28T12:58:59.659020+0000","last_clean":"2023-11-28T12:58:59.659020+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T12:58:59.659020+0000","last_undegraded":"2023-11-28T12:58:59.659020+0000","last_fullsized":"2023-11-28T12:58:59.659020+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"26'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313562+0000","last_change":"2023-11-28T12:58:55.251038+0000","last_active":"2023-11-28T12:59:00.313562+0000","last_peered":"2023-11-28T12:59:00.313562+0000","last_clean":"2023-11-28T12:59:00.313562+0000","last_became_active":"2023-11-28T12:58:55.250328+0000","last_became_peered":"2023-11-28T12:58:55.250328+0000","last_unstale":"2023-11-28T12:59:00.313562+0000","last_undegraded":"2023-11-28T12:59:00.313562+0000","last_fullsized":"2023-11-28T12:59:00.313562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:11.131301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659036+0000","last_change":"2023-11-28T12:58:55.658965+0000","last_active":"2023-11-28T12:58:59.659036+0000","last_peered":"2023-11-28T12:58:59.659036+0000","last_clean":"2023-11-28T12:58:59.659036+0000","last_became_active":"2023-11-28T12:58:55.658709+0000","last_became_peered":"2023-11-28T12:58:55.658709+0000","last_unstale":"2023-11-28T12:58:59.659036+0000","last_undegraded":"2023-11-28T12:58:59.659036+0000","last_fullsized":"2023-11-28T12:58:59.659036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:58:45.065981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.316209+0000","last_change":"2023-11-28T12:58:59.679267+0000","last_active":"2023-11-28T12:59:00.316209+0000","last_peered":"2023-11-28T12:59:00.316209+0000","last_clean":"2023-11-28T12:59:00.316209+0000","last_became_active":"2023-11-28T12:58:59.678822+0000","last_became_peered":"2023-11-28T12:58:59.678822+0000","last_unstale":"2023-11-28T12:59:00.316209+0000","last_undegraded":"2023-11-28T12:59:00.316209+0000","last_fullsized":"2023-11-28T12:59:00.316209+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:38:05.937526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669102+0000","last_change":"2023-11-28T12:58:55.250742+0000","last_active":"2023-11-28T12:58:59.669102+0000","last_peered":"2023-11-28T12:58:59.669102+0000","last_clean":"2023-11-28T12:58:59.669102+0000","last_became_active":"2023-11-28T12:58:55.245578+0000","last_became_peered":"2023-11-28T12:58:55.245578+0000","last_unstale":"2023-11-28T12:58:59.669102+0000","last_undegraded":"2023-11-28T12:58:59.669102+0000","last_fullsized":"2023-11-28T12:58:59.669102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:25:52.356939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672422+0000","last_change":"2023-11-28T12:58:55.647087+0000","last_active":"2023-11-28T12:58:59.672422+0000","last_peered":"2023-11-28T12:58:59.672422+0000","last_clean":"2023-11-28T12:58:59.672422+0000","last_became_active":"2023-11-28T12:58:55.646827+0000","last_became_peered":"2023-11-28T12:58:55.646827+0000","last_unstale":"2023-11-28T12:58:59.672422+0000","last_undegraded":"2023-11-28T12:58:59.672422+0000","last_fullsized":"2023-11-28T12:58:59.672422+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:43.305980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830493+0000","last_change":"2023-11-28T12:58:59.673358+0000","last_active":"2023-11-28T12:59:00.830493+0000","last_peered":"2023-11-28T12:59:00.830493+0000","last_clean":"2023-11-28T12:59:00.830493+0000","last_became_active":"2023-11-28T12:58:59.673010+0000","last_became_peered":"2023-11-28T12:58:59.673010+0000","last_unstale":"2023-11-28T12:59:00.830493+0000","last_undegraded":"2023-11-28T12:59:00.830493+0000","last_fullsized":"2023-11-28T12:59:00.830493+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:53.458898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665394+0000","last_change":"2023-11-28T12:58:55.242678+0000","last_active":"2023-11-28T12:58:59.665394+0000","last_peered":"2023-11-28T12:58:59.665394+0000","last_clean":"2023-11-28T12:58:59.665394+0000","last_became_active":"2023-11-28T12:58:55.242467+0000","last_became_peered":"2023-11-28T12:58:55.242467+0000","last_unstale":"2023-11-28T12:58:59.665394+0000","last_undegraded":"2023-11-28T12:58:59.665394+0000","last_fullsized":"2023-11-28T12:58:59.665394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:11.131301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659707+0000","last_change":"2023-11-28T12:58:55.664310+0000","last_active":"2023-11-28T12:58:59.659707+0000","last_peered":"2023-11-28T12:58:59.659707+0000","last_clean":"2023-11-28T12:58:59.659707+0000","last_became_active":"2023-11-28T12:58:55.664036+0000","last_became_peered":"2023-11-28T12:58:55.664036+0000","last_unstale":"2023-11-28T12:58:59.659707+0000","last_undegraded":"2023-11-28T12:58:59.659707+0000","last_fullsized":"2023-11-28T12:58:59.659707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:38:36.781235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830051+0000","last_change":"2023-11-28T12:58:59.685443+0000","last_active":"2023-11-28T12:59:00.830051+0000","last_peered":"2023-11-28T12:59:00.830051+0000","last_clean":"2023-11-28T12:59:00.830051+0000","last_became_active":"2023-11-28T12:58:59.685042+0000","last_became_peered":"2023-11-28T12:58:59.685042+0000","last_unstale":"2023-11-28T12:59:00.830051+0000","last_undegraded":"2023-11-28T12:59:00.830051+0000","last_fullsized":"2023-11-28T12:59:00.830051+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:03:33.042937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665495+0000","last_change":"2023-11-28T12:58:55.241170+0000","last_active":"2023-11-28T12:58:59.665495+0000","last_peered":"2023-11-28T12:58:59.665495+0000","last_clean":"2023-11-28T12:58:59.665495+0000","last_became_active":"2023-11-28T12:58:55.240880+0000","last_became_peered":"2023-11-28T12:58:55.240880+0000","last_unstale":"2023-11-28T12:58:59.665495+0000","last_undegraded":"2023-11-28T12:58:59.665495+0000","last_fullsized":"2023-11-28T12:58:59.665495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:42.300078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659284+0000","last_change":"2023-11-28T12:58:55.664483+0000","last_active":"2023-11-28T12:58:59.659284+0000","last_peered":"2023-11-28T12:58:59.659284+0000","last_clean":"2023-11-28T12:58:59.659284+0000","last_became_active":"2023-11-28T12:58:55.662151+0000","last_became_peered":"2023-11-28T12:58:55.662151+0000","last_unstale":"2023-11-28T12:58:59.659284+0000","last_undegraded":"2023-11-28T12:58:59.659284+0000","last_fullsized":"2023-11-28T12:58:59.659284+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:00:06.262366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058024+0000","last_change":"2023-11-28T12:58:59.679533+0000","last_active":"2023-11-28T12:59:00.058024+0000","last_peered":"2023-11-28T12:59:00.058024+0000","last_clean":"2023-11-28T12:59:00.058024+0000","last_became_active":"2023-11-28T12:58:59.679174+0000","last_became_peered":"2023-11-28T12:58:59.679174+0000","last_unstale":"2023-11-28T12:59:00.058024+0000","last_undegraded":"2023-11-28T12:59:00.058024+0000","last_fullsized":"2023-11-28T12:59:00.058024+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:34:44.322352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655854+0000","last_change":"2023-11-28T12:58:55.246420+0000","last_active":"2023-11-28T12:58:59.655854+0000","last_peered":"2023-11-28T12:58:59.655854+0000","last_clean":"2023-11-28T12:58:59.655854+0000","last_became_active":"2023-11-28T12:58:55.246075+0000","last_became_peered":"2023-11-28T12:58:55.246075+0000","last_unstale":"2023-11-28T12:58:59.655854+0000","last_undegraded":"2023-11-28T12:58:59.655854+0000","last_fullsized":"2023-11-28T12:58:59.655854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:38:45.929395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655474+0000","last_change":"2023-11-28T12:58:55.660837+0000","last_active":"2023-11-28T12:58:59.655474+0000","last_peered":"2023-11-28T12:58:59.655474+0000","last_clean":"2023-11-28T12:58:59.655474+0000","last_became_active":"2023-11-28T12:58:55.657932+0000","last_became_peered":"2023-11-28T12:58:55.657932+0000","last_unstale":"2023-11-28T12:58:59.655474+0000","last_undegraded":"2023-11-28T12:58:59.655474+0000","last_fullsized":"2023-11-28T12:58:59.655474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.315551+0000","last_change":"2023-11-28T12:58:59.683087+0000","last_active":"2023-11-28T12:59:00.315551+0000","last_peered":"2023-11-28T12:59:00.315551+0000","last_clean":"2023-11-28T12:59:00.315551+0000","last_became_active":"2023-11-28T12:58:59.682678+0000","last_became_peered":"2023-11-28T12:58:59.682678+0000","last_unstale":"2023-11-28T12:59:00.315551+0000","last_undegraded":"2023-11-28T12:59:00.315551+0000","last_fullsized":"2023-11-28T12:59:00.315551+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:07:52.771805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"21'4","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674325+0000","last_change":"2023-11-28T12:58:55.246094+0000","last_active":"2023-11-28T12:58:59.674325+0000","last_peered":"2023-11-28T12:58:59.674325+0000","last_clean":"2023-11-28T12:58:59.674325+0000","last_became_active":"2023-11-28T12:58:55.245800+0000","last_became_peered":"2023-11-28T12:58:55.245800+0000","last_unstale":"2023-11-28T12:58:59.674325+0000","last_undegraded":"2023-11-28T12:58:59.674325+0000","last_fullsized":"2023-11-28T12:58:59.674325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:49.233234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.658861+0000","last_change":"2023-11-28T12:58:55.660995+0000","last_active":"2023-11-28T12:58:59.658861+0000","last_peered":"2023-11-28T12:58:59.658861+0000","last_clean":"2023-11-28T12:58:59.658861+0000","last_became_active":"2023-11-28T12:58:55.660761+0000","last_became_peered":"2023-11-28T12:58:55.660761+0000","last_unstale":"2023-11-28T12:58:59.658861+0000","last_undegraded":"2023-11-28T12:58:59.658861+0000","last_fullsized":"2023-11-28T12:58:59.658861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657014+0000","last_change":"2023-11-28T12:58:55.650780+0000","last_active":"2023-11-28T12:58:59.657014+0000","last_peered":"2023-11-28T12:58:59.657014+0000","last_clean":"2023-11-28T12:58:59.657014+0000","last_became_active":"2023-11-28T12:58:55.650511+0000","last_became_peered":"2023-11-28T12:58:55.650511+0000","last_unstale":"2023-11-28T12:58:59.657014+0000","last_undegraded":"2023-11-28T12:58:59.657014+0000","last_fullsized":"2023-11-28T12:58:59.657014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:43.448645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655863+0000","last_change":"2023-11-28T12:58:55.663313+0000","last_active":"2023-11-28T12:58:59.655863+0000","last_peered":"2023-11-28T12:58:59.655863+0000","last_clean":"2023-11-28T12:58:59.655863+0000","last_became_active":"2023-11-28T12:58:55.663036+0000","last_became_peered":"2023-11-28T12:58:55.663036+0000","last_unstale":"2023-11-28T12:58:59.655863+0000","last_undegraded":"2023-11-28T12:58:59.655863+0000","last_fullsized":"2023-11-28T12:58:59.655863+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659437+0000","last_change":"2023-11-28T12:58:55.661206+0000","last_active":"2023-11-28T12:58:59.659437+0000","last_peered":"2023-11-28T12:58:59.659437+0000","last_clean":"2023-11-28T12:58:59.659437+0000","last_became_active":"2023-11-28T12:58:55.660952+0000","last_became_peered":"2023-11-28T12:58:55.660952+0000","last_unstale":"2023-11-28T12:58:59.659437+0000","last_undegraded":"2023-11-28T12:58:59.659437+0000","last_fullsized":"2023-11-28T12:58:59.659437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:34.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655629+0000","last_change":"2023-11-28T12:58:55.661675+0000","last_active":"2023-11-28T12:58:59.655629+0000","last_peered":"2023-11-28T12:58:59.655629+0000","last_clean":"2023-11-28T12:58:59.655629+0000","last_became_active":"2023-11-28T12:58:55.661418+0000","last_became_peered":"2023-11-28T12:58:55.661418+0000","last_unstale":"2023-11-28T12:58:59.655629+0000","last_undegraded":"2023-11-28T12:58:59.655629+0000","last_fullsized":"2023-11-28T12:58:59.655629+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664699+0000","last_change":"2023-11-28T12:58:55.651538+0000","last_active":"2023-11-28T12:58:59.664699+0000","last_peered":"2023-11-28T12:58:59.664699+0000","last_clean":"2023-11-28T12:58:59.664699+0000","last_became_active":"2023-11-28T12:58:55.651297+0000","last_became_peered":"2023-11-28T12:58:55.651297+0000","last_unstale":"2023-11-28T12:58:59.664699+0000","last_undegraded":"2023-11-28T12:58:59.664699+0000","last_fullsized":"2023-11-28T12:58:59.664699+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:56:55.140717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4528,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9272,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362568,"statfs":{"total":96636764160,"available":96627269632,"internally_reserved":0,"allocated":901120,"data_stored":709174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8088,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363752,"statfs":{"total":96636764160,"available":96628482048,"internally_reserved":0,"allocated":278528,"data_stored":114844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8152,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363688,"statfs":{"total":96636764160,"available":96628416512,"internally_reserved":0,"allocated":278528,"data_stored":114844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9248,"kb_used_data":856,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362592,"statfs":{"total":96636764160,"available":96627294208,"internally_reserved":0,"allocated":876544,"data_stored":705778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8088,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363752,"statfs":{"total":96636764160,"available":96628482048,"internally_reserved":0,"allocated":278528,"data_stored":114844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8156,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363684,"statfs":{"total":96636764160,"available":96628412416,"internally_reserved":0,"allocated":282624,"data_stored":114934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8236,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363604,"statfs":{"total":96636764160,"available":96628330496,"internally_reserved":0,"allocated":299008,"data_stored":118646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8160,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363680,"statfs":{"total":96636764160,"available":96628408320,"internally_reserved":0,"allocated":286720,"data_stored":115480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T12:59:18.776 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T12:59:18.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.927+0000 7faa0d8ea700 1 Processor -- start 2023-11-28T12:59:18.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.927+0000 7faa0d8ea700 1 -- start start 2023-11-28T12:59:18.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa0d8ea700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7faa080b8510 0x7faa0806b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa0d8ea700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faa080d0d60 0x7faa080d1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa0d8ea700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa080b88e0 con 0x7faa080cd460 2023-11-28T12:59:18.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa0d8ea700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa080cdbd0 con 0x7faa080b8510 2023-11-28T12:59:18.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa0d8ea700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa080d02d0 con 0x7faa080d0d60 2023-11-28T12:59:18.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa067fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7faa080cd460 0x7faa0806beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33918/0 (socket says 172.21.15.111:33918) 2023-11-28T12:59:18.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa067fc700 1 -- 172.21.15.111:0/2030325466 learned_addr learned my addr 172.21.15.111:0/2030325466 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:18.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa077fe700 1 --2- 172.21.15.111:0/2030325466 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faa080d0d60 0x7faa080d1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa06ffd700 1 --2- 172.21.15.111:0/2030325466 >> v2:172.21.15.149:3300/0 conn(0x7faa080b8510 0x7faa0806b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa05ffb700 1 -- 172.21.15.111:0/2030325466 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534703243 0 0) 0x7faa080b88e0 con 0x7faa080cd460 2023-11-28T12:59:18.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa05ffb700 1 -- 172.21.15.111:0/2030325466 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9f00035b0 con 0x7faa080cd460 2023-11-28T12:59:18.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa05ffb700 1 -- 172.21.15.111:0/2030325466 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 155042341 0 0) 0x7fa9f00035b0 con 0x7faa080cd460 2023-11-28T12:59:18.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa077fe700 1 -- 172.21.15.111:0/2030325466 >> v2:172.21.15.149:3300/0 conn(0x7faa080b8510 msgr2=0x7faa0806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:18.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa077fe700 1 --2- 172.21.15.111:0/2030325466 >> v2:172.21.15.149:3300/0 conn(0x7faa080b8510 0x7faa0806b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa077fe700 1 -- 172.21.15.111:0/2030325466 >> v1:172.21.15.111:6789/0 conn(0x7faa080cd460 legacy=0x7faa0806beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa077fe700 1 -- 172.21.15.111:0/2030325466 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa080cd8f0 con 0x7faa080d0d60 2023-11-28T12:59:18.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.928+0000 7faa06ffd700 1 --2- 172.21.15.111:0/2030325466 >> v2:172.21.15.149:3300/0 conn(0x7faa080b8510 0x7faa0806b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:18.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.929+0000 7faa077fe700 1 --2- 172.21.15.111:0/2030325466 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faa080d0d60 0x7faa080d1200 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faa00009d90 tx=0x7faa0000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=216959f1117af83c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:18.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.929+0000 7faa05ffb700 1 -- 172.21.15.111:0/2030325466 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0000e070 con 0x7faa080d0d60 2023-11-28T12:59:18.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.929+0000 7faa05ffb700 1 -- 172.21.15.111:0/2030325466 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa00003a20 con 0x7faa080d0d60 2023-11-28T12:59:18.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.929+0000 7faa05ffb700 1 -- 172.21.15.111:0/2030325466 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8001670 con 0x7faa080d0d60 2023-11-28T12:59:18.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.929+0000 7faa0d8ea700 1 -- 172.21.15.111:0/2030325466 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faa080d0d60 msgr2=0x7faa080d1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.929+0000 7faa0d8ea700 1 --2- 172.21.15.111:0/2030325466 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faa080d0d60 0x7faa080d1200 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faa00009d90 tx=0x7faa0000b330 comp rx=0 tx=0).stop 2023-11-28T12:59:18.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.929+0000 7faa0d8ea700 1 -- 172.21.15.111:0/2030325466 shutdown_connections 2023-11-28T12:59:18.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.929+0000 7faa0d8ea700 1 --2- 172.21.15.111:0/2030325466 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faa080d0d60 0x7faa080d1200 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.929+0000 7faa0d8ea700 1 --2- 172.21.15.111:0/2030325466 >> v2:172.21.15.149:3300/0 conn(0x7faa080b8510 0x7faa0806b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.929+0000 7faa0d8ea700 1 -- 172.21.15.111:0/2030325466 >> 172.21.15.111:0/2030325466 conn(0x7faa080719a0 msgr2=0x7faa08071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:18.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.929+0000 7faa0d8ea700 1 -- 172.21.15.111:0/2030325466 shutdown_connections 2023-11-28T12:59:18.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.929+0000 7faa0d8ea700 1 -- 172.21.15.111:0/2030325466 wait complete. 2023-11-28T12:59:18.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.931+0000 7faa0d8ea700 1 Processor -- start 2023-11-28T12:59:18.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.931+0000 7faa0d8ea700 1 -- start start 2023-11-28T12:59:18.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.931+0000 7faa0d8ea700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faa080b8510 0x7faa080cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.931+0000 7faa0d8ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa080cd460 0x7faa08158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.931+0000 7faa0d8ea700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faa080d0d60 0x7faa08159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.931+0000 7faa0d8ea700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa081e4300 con 0x7faa080cd460 2023-11-28T12:59:18.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa0d8ea700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa081e3d10 con 0x7faa080b8510 2023-11-28T12:59:18.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa0d8ea700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa080c9ee0 con 0x7faa080d0d60 2023-11-28T12:59:18.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa067fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa080cd460 0x7faa08158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa067fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa080cd460 0x7faa08158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33146/0 (socket says 172.21.15.111:33146) 2023-11-28T12:59:18.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa067fc700 1 -- 172.21.15.111:0/3357872330 learned_addr learned my addr 172.21.15.111:0/3357872330 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:18.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa077fe700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faa080d0d60 0x7faa08159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa06ffd700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faa080b8510 0x7faa080cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa067fc700 1 -- 172.21.15.111:0/3357872330 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faa080d0d60 msgr2=0x7faa08159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa067fc700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faa080d0d60 0x7faa08159e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa067fc700 1 -- 172.21.15.111:0/3357872330 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faa080b8510 msgr2=0x7faa080cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:18.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa067fc700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faa080b8510 0x7faa080cced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:18.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa067fc700 1 -- 172.21.15.111:0/3357872330 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc0094a0 con 0x7faa080cd460 2023-11-28T12:59:18.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa067fc700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa080cd460 0x7faa08158420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa9f800c750 tx=0x7fa9f800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:18.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7fa9effff700 1 -- 172.21.15.111:0/3357872330 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8007560 con 0x7faa080cd460 2023-11-28T12:59:18.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7fa9effff700 1 -- 172.21.15.111:0/3357872330 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f80076c0 con 0x7faa080cd460 2023-11-28T12:59:18.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa0d8ea700 1 -- 172.21.15.111:0/3357872330 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa00009a70 con 0x7faa080cd460 2023-11-28T12:59:18.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7fa9effff700 1 -- 172.21.15.111:0/3357872330 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8004ab0 con 0x7faa080cd460 2023-11-28T12:59:18.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.932+0000 7faa0d8ea700 1 -- 172.21.15.111:0/3357872330 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa080cfff0 con 0x7faa080cd460 2023-11-28T12:59:18.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.933+0000 7faa0d8ea700 1 -- 172.21.15.111:0/3357872330 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa080cfd10 con 0x7faa080cd460 2023-11-28T12:59:18.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.934+0000 7fa9effff700 1 -- 172.21.15.111:0/3357872330 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa9f801a030 con 0x7faa080cd460 2023-11-28T12:59:18.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.934+0000 7fa9effff700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9f40682f0 0x7fa9f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:18.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.934+0000 7fa9effff700 1 -- 172.21.15.111:0/3357872330 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa9f80854e0 con 0x7faa080cd460 2023-11-28T12:59:18.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.934+0000 7faa06ffd700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9f40682f0 0x7fa9f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:18.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.934+0000 7faa06ffd700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9f40682f0 0x7fa9f406a7a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009d00 tx=0x7fa9fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:18.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:18.938+0000 7fa9effff700 1 -- 172.21.15.111:0/3357872330 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa9f8051a50 con 0x7faa080cd460 2023-11-28T12:59:19.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.117+0000 7faa0d8ea700 1 -- 172.21.15.111:0/3357872330 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faa081e4440 con 0x7fa9f40682f0 2023-11-28T12:59:19.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.126+0000 7fa9effff700 1 -- 172.21.15.111:0/3357872330 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367250 (secure 0 0 0) 0x7faa081e4440 con 0x7fa9f40682f0 2023-11-28T12:59:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:19.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.129+0000 7faa0d8ea700 1 -- 172.21.15.111:0/3357872330 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9f40682f0 msgr2=0x7fa9f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.129+0000 7faa0d8ea700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9f40682f0 0x7fa9f406a7a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009d00 tx=0x7fa9fc008040 comp rx=0 tx=0).stop 2023-11-28T12:59:19.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.129+0000 7faa0d8ea700 1 -- 172.21.15.111:0/3357872330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa080cd460 msgr2=0x7faa08158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.129+0000 7faa0d8ea700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa080cd460 0x7faa08158420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa9f800c750 tx=0x7fa9f800cb10 comp rx=0 tx=0).stop 2023-11-28T12:59:19.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.130+0000 7faa0d8ea700 1 -- 172.21.15.111:0/3357872330 shutdown_connections 2023-11-28T12:59:19.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.130+0000 7faa0d8ea700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faa080d0d60 0x7faa08159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.130+0000 7faa0d8ea700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9f40682f0 0x7fa9f406a7a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.130+0000 7faa0d8ea700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faa080b8510 0x7faa080cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.130+0000 7faa0d8ea700 1 --2- 172.21.15.111:0/3357872330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa080cd460 0x7faa08158420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.130+0000 7faa0d8ea700 1 -- 172.21.15.111:0/3357872330 >> 172.21.15.111:0/3357872330 conn(0x7faa080719a0 msgr2=0x7faa080b6670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:19.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.130+0000 7faa0d8ea700 1 -- 172.21.15.111:0/3357872330 shutdown_connections 2023-11-28T12:59:19.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.130+0000 7faa0d8ea700 1 -- 172.21.15.111:0/3357872330 wait complete. 2023-11-28T12:59:19.143 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T12:59:19.144 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-11-28T12:59:18.337950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594896,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":67400,"kb_used_data":3400,"kb_used_omap":0,"kb_used_meta":63936,"kb_avail":754907320,"statfs":{"total":773094113280,"available":773025095680,"internally_reserved":0,"allocated":3481600,"data_stored":2108544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65470464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2198,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672369+0000","last_change":"2023-11-28T12:58:55.661175+0000","last_active":"2023-11-28T12:58:59.672369+0000","last_peered":"2023-11-28T12:58:59.672369+0000","last_clean":"2023-11-28T12:58:59.672369+0000","last_became_active":"2023-11-28T12:58:55.660919+0000","last_became_peered":"2023-11-28T12:58:55.660919+0000","last_unstale":"2023-11-28T12:58:59.672369+0000","last_undegraded":"2023-11-28T12:58:59.672369+0000","last_fullsized":"2023-11-28T12:58:59.672369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:12.137203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666089+0000","last_change":"2023-11-28T12:58:55.652749+0000","last_active":"2023-11-28T12:58:59.666089+0000","last_peered":"2023-11-28T12:58:59.666089+0000","last_clean":"2023-11-28T12:58:59.666089+0000","last_became_active":"2023-11-28T12:58:55.652544+0000","last_became_peered":"2023-11-28T12:58:55.652544+0000","last_unstale":"2023-11-28T12:58:59.666089+0000","last_undegraded":"2023-11-28T12:58:59.666089+0000","last_fullsized":"2023-11-28T12:58:59.666089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660310+0000","last_change":"2023-11-28T12:58:55.669208+0000","last_active":"2023-11-28T12:58:59.660310+0000","last_peered":"2023-11-28T12:58:59.660310+0000","last_clean":"2023-11-28T12:58:59.660310+0000","last_became_active":"2023-11-28T12:58:55.668930+0000","last_became_peered":"2023-11-28T12:58:55.668930+0000","last_unstale":"2023-11-28T12:58:59.660310+0000","last_undegraded":"2023-11-28T12:58:59.660310+0000","last_fullsized":"2023-11-28T12:58:59.660310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:18:05.065329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657123+0000","last_change":"2023-11-28T12:58:55.653156+0000","last_active":"2023-11-28T12:58:59.657123+0000","last_peered":"2023-11-28T12:58:59.657123+0000","last_clean":"2023-11-28T12:58:59.657123+0000","last_became_active":"2023-11-28T12:58:55.652820+0000","last_became_peered":"2023-11-28T12:58:55.652820+0000","last_unstale":"2023-11-28T12:58:59.657123+0000","last_undegraded":"2023-11-28T12:58:59.657123+0000","last_fullsized":"2023-11-28T12:58:59.657123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:19.178525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669305+0000","last_change":"2023-11-28T12:58:55.664471+0000","last_active":"2023-11-28T12:58:59.669305+0000","last_peered":"2023-11-28T12:58:59.669305+0000","last_clean":"2023-11-28T12:58:59.669305+0000","last_became_active":"2023-11-28T12:58:55.664221+0000","last_became_peered":"2023-11-28T12:58:55.664221+0000","last_unstale":"2023-11-28T12:58:59.669305+0000","last_undegraded":"2023-11-28T12:58:59.669305+0000","last_fullsized":"2023-11-28T12:58:59.669305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:34:15.223824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659490+0000","last_change":"2023-11-28T12:58:55.669109+0000","last_active":"2023-11-28T12:58:59.659490+0000","last_peered":"2023-11-28T12:58:59.659490+0000","last_clean":"2023-11-28T12:58:59.659490+0000","last_became_active":"2023-11-28T12:58:55.668857+0000","last_became_peered":"2023-11-28T12:58:55.668857+0000","last_unstale":"2023-11-28T12:58:59.659490+0000","last_undegraded":"2023-11-28T12:58:59.659490+0000","last_fullsized":"2023-11-28T12:58:59.659490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:14:02.504692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.663977+0000","last_change":"2023-11-28T12:58:55.662854+0000","last_active":"2023-11-28T12:58:59.663977+0000","last_peered":"2023-11-28T12:58:59.663977+0000","last_clean":"2023-11-28T12:58:59.663977+0000","last_became_active":"2023-11-28T12:58:55.662539+0000","last_became_peered":"2023-11-28T12:58:55.662539+0000","last_unstale":"2023-11-28T12:58:59.663977+0000","last_undegraded":"2023-11-28T12:58:59.663977+0000","last_fullsized":"2023-11-28T12:58:59.663977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661304+0000","last_change":"2023-11-28T12:58:55.657958+0000","last_active":"2023-11-28T12:58:59.661304+0000","last_peered":"2023-11-28T12:58:59.661304+0000","last_clean":"2023-11-28T12:58:59.661304+0000","last_became_active":"2023-11-28T12:58:55.657681+0000","last_became_peered":"2023-11-28T12:58:55.657681+0000","last_unstale":"2023-11-28T12:58:59.661304+0000","last_undegraded":"2023-11-28T12:58:59.661304+0000","last_fullsized":"2023-11-28T12:58:59.661304+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674142+0000","last_change":"2023-11-28T12:58:55.666485+0000","last_active":"2023-11-28T12:58:59.674142+0000","last_peered":"2023-11-28T12:58:59.674142+0000","last_clean":"2023-11-28T12:58:59.674142+0000","last_became_active":"2023-11-28T12:58:55.665670+0000","last_became_peered":"2023-11-28T12:58:55.665670+0000","last_unstale":"2023-11-28T12:58:59.674142+0000","last_undegraded":"2023-11-28T12:58:59.674142+0000","last_fullsized":"2023-11-28T12:58:59.674142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668534+0000","last_change":"2023-11-28T12:58:55.655129+0000","last_active":"2023-11-28T12:58:59.668534+0000","last_peered":"2023-11-28T12:58:59.668534+0000","last_clean":"2023-11-28T12:58:59.668534+0000","last_became_active":"2023-11-28T12:58:55.654839+0000","last_became_peered":"2023-11-28T12:58:55.654839+0000","last_unstale":"2023-11-28T12:58:59.668534+0000","last_undegraded":"2023-11-28T12:58:59.668534+0000","last_fullsized":"2023-11-28T12:58:59.668534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660394+0000","last_change":"2023-11-28T12:58:55.663796+0000","last_active":"2023-11-28T12:58:59.660394+0000","last_peered":"2023-11-28T12:58:59.660394+0000","last_clean":"2023-11-28T12:58:59.660394+0000","last_became_active":"2023-11-28T12:58:55.663553+0000","last_became_peered":"2023-11-28T12:58:55.663553+0000","last_unstale":"2023-11-28T12:58:59.660394+0000","last_undegraded":"2023-11-28T12:58:59.660394+0000","last_fullsized":"2023-11-28T12:58:59.660394+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:31:46.923203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672559+0000","last_change":"2023-11-28T12:58:55.666858+0000","last_active":"2023-11-28T12:58:59.672559+0000","last_peered":"2023-11-28T12:58:59.672559+0000","last_clean":"2023-11-28T12:58:59.672559+0000","last_became_active":"2023-11-28T12:58:55.666597+0000","last_became_peered":"2023-11-28T12:58:55.666597+0000","last_unstale":"2023-11-28T12:58:59.672559+0000","last_undegraded":"2023-11-28T12:58:59.672559+0000","last_fullsized":"2023-11-28T12:58:59.672559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:50.239136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.673902+0000","last_change":"2023-11-28T12:58:55.658497+0000","last_active":"2023-11-28T12:58:59.673902+0000","last_peered":"2023-11-28T12:58:59.673902+0000","last_clean":"2023-11-28T12:58:59.673902+0000","last_became_active":"2023-11-28T12:58:55.658258+0000","last_became_peered":"2023-11-28T12:58:55.658258+0000","last_unstale":"2023-11-28T12:58:59.673902+0000","last_undegraded":"2023-11-28T12:58:59.673902+0000","last_fullsized":"2023-11-28T12:58:59.673902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:38:46.935297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664436+0000","last_change":"2023-11-28T12:58:55.648629+0000","last_active":"2023-11-28T12:58:59.664436+0000","last_peered":"2023-11-28T12:58:59.664436+0000","last_clean":"2023-11-28T12:58:59.664436+0000","last_became_active":"2023-11-28T12:58:55.648143+0000","last_became_peered":"2023-11-28T12:58:55.648143+0000","last_unstale":"2023-11-28T12:58:59.664436+0000","last_undegraded":"2023-11-28T12:58:59.664436+0000","last_fullsized":"2023-11-28T12:58:59.664436+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:06:39.770893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.663795+0000","last_change":"2023-11-28T12:58:55.665560+0000","last_active":"2023-11-28T12:58:59.663795+0000","last_peered":"2023-11-28T12:58:59.663795+0000","last_clean":"2023-11-28T12:58:59.663795+0000","last_became_active":"2023-11-28T12:58:55.665304+0000","last_became_peered":"2023-11-28T12:58:55.665304+0000","last_unstale":"2023-11-28T12:58:59.663795+0000","last_undegraded":"2023-11-28T12:58:59.663795+0000","last_fullsized":"2023-11-28T12:58:59.663795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.658903+0000","last_change":"2023-11-28T12:58:55.669222+0000","last_active":"2023-11-28T12:58:59.658903+0000","last_peered":"2023-11-28T12:58:59.658903+0000","last_clean":"2023-11-28T12:58:59.658903+0000","last_became_active":"2023-11-28T12:58:55.668931+0000","last_became_peered":"2023-11-28T12:58:55.668931+0000","last_unstale":"2023-11-28T12:58:59.658903+0000","last_undegraded":"2023-11-28T12:58:59.658903+0000","last_fullsized":"2023-11-28T12:58:59.658903+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:12:28.351912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674186+0000","last_change":"2023-11-28T12:58:55.666643+0000","last_active":"2023-11-28T12:58:59.674186+0000","last_peered":"2023-11-28T12:58:59.674186+0000","last_clean":"2023-11-28T12:58:59.674186+0000","last_became_active":"2023-11-28T12:58:55.666224+0000","last_became_peered":"2023-11-28T12:58:55.666224+0000","last_unstale":"2023-11-28T12:58:59.674186+0000","last_undegraded":"2023-11-28T12:58:59.674186+0000","last_fullsized":"2023-11-28T12:58:59.674186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660392+0000","last_change":"2023-11-28T12:58:55.660396+0000","last_active":"2023-11-28T12:58:59.660392+0000","last_peered":"2023-11-28T12:58:59.660392+0000","last_clean":"2023-11-28T12:58:59.660392+0000","last_became_active":"2023-11-28T12:58:55.659518+0000","last_became_peered":"2023-11-28T12:58:55.659518+0000","last_unstale":"2023-11-28T12:58:59.660392+0000","last_undegraded":"2023-11-28T12:58:59.660392+0000","last_fullsized":"2023-11-28T12:58:59.660392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:49:50.356082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672413+0000","last_change":"2023-11-28T12:58:55.650573+0000","last_active":"2023-11-28T12:58:59.672413+0000","last_peered":"2023-11-28T12:58:59.672413+0000","last_clean":"2023-11-28T12:58:59.672413+0000","last_became_active":"2023-11-28T12:58:55.650382+0000","last_became_peered":"2023-11-28T12:58:55.650382+0000","last_unstale":"2023-11-28T12:58:59.672413+0000","last_undegraded":"2023-11-28T12:58:59.672413+0000","last_fullsized":"2023-11-28T12:58:59.672413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:54:07.154200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659749+0000","last_change":"2023-11-28T12:58:55.663920+0000","last_active":"2023-11-28T12:58:59.659749+0000","last_peered":"2023-11-28T12:58:59.659749+0000","last_clean":"2023-11-28T12:58:59.659749+0000","last_became_active":"2023-11-28T12:58:55.663347+0000","last_became_peered":"2023-11-28T12:58:55.663347+0000","last_unstale":"2023-11-28T12:58:59.659749+0000","last_undegraded":"2023-11-28T12:58:59.659749+0000","last_fullsized":"2023-11-28T12:58:59.659749+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664143+0000","last_change":"2023-11-28T12:58:55.661755+0000","last_active":"2023-11-28T12:58:59.664143+0000","last_peered":"2023-11-28T12:58:59.664143+0000","last_clean":"2023-11-28T12:58:59.664143+0000","last_became_active":"2023-11-28T12:58:55.661496+0000","last_became_peered":"2023-11-28T12:58:55.661496+0000","last_unstale":"2023-11-28T12:58:59.664143+0000","last_undegraded":"2023-11-28T12:58:59.664143+0000","last_fullsized":"2023-11-28T12:58:59.664143+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669008+0000","last_change":"2023-11-28T12:58:55.666002+0000","last_active":"2023-11-28T12:58:59.669008+0000","last_peered":"2023-11-28T12:58:59.669008+0000","last_clean":"2023-11-28T12:58:59.669008+0000","last_became_active":"2023-11-28T12:58:55.657322+0000","last_became_peered":"2023-11-28T12:58:55.657322+0000","last_unstale":"2023-11-28T12:58:59.669008+0000","last_undegraded":"2023-11-28T12:58:59.669008+0000","last_fullsized":"2023-11-28T12:58:59.669008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655373+0000","last_change":"2023-11-28T12:58:55.669042+0000","last_active":"2023-11-28T12:58:59.655373+0000","last_peered":"2023-11-28T12:58:59.655373+0000","last_clean":"2023-11-28T12:58:59.655373+0000","last_became_active":"2023-11-28T12:58:55.668810+0000","last_became_peered":"2023-11-28T12:58:55.668810+0000","last_unstale":"2023-11-28T12:58:59.655373+0000","last_undegraded":"2023-11-28T12:58:59.655373+0000","last_fullsized":"2023-11-28T12:58:59.655373+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:49:50.356082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666243+0000","last_change":"2023-11-28T12:58:55.658006+0000","last_active":"2023-11-28T12:58:59.666243+0000","last_peered":"2023-11-28T12:58:59.666243+0000","last_clean":"2023-11-28T12:58:59.666243+0000","last_became_active":"2023-11-28T12:58:55.657791+0000","last_became_peered":"2023-11-28T12:58:55.657791+0000","last_unstale":"2023-11-28T12:58:59.666243+0000","last_undegraded":"2023-11-28T12:58:59.666243+0000","last_fullsized":"2023-11-28T12:58:59.666243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667574+0000","last_change":"2023-11-28T12:58:55.658052+0000","last_active":"2023-11-28T12:58:59.667574+0000","last_peered":"2023-11-28T12:58:59.667574+0000","last_clean":"2023-11-28T12:58:59.667574+0000","last_became_active":"2023-11-28T12:58:55.657752+0000","last_became_peered":"2023-11-28T12:58:55.657752+0000","last_unstale":"2023-11-28T12:58:59.667574+0000","last_undegraded":"2023-11-28T12:58:59.667574+0000","last_fullsized":"2023-11-28T12:58:59.667574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665870+0000","last_change":"2023-11-28T12:58:55.657589+0000","last_active":"2023-11-28T12:58:59.665870+0000","last_peered":"2023-11-28T12:58:59.665870+0000","last_clean":"2023-11-28T12:58:59.665870+0000","last_became_active":"2023-11-28T12:58:55.654770+0000","last_became_peered":"2023-11-28T12:58:55.654770+0000","last_unstale":"2023-11-28T12:58:59.665870+0000","last_undegraded":"2023-11-28T12:58:59.665870+0000","last_fullsized":"2023-11-28T12:58:59.665870+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:06:39.770893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665748+0000","last_change":"2023-11-28T12:58:55.657244+0000","last_active":"2023-11-28T12:58:59.665748+0000","last_peered":"2023-11-28T12:58:59.665748+0000","last_clean":"2023-11-28T12:58:59.665748+0000","last_became_active":"2023-11-28T12:58:55.656984+0000","last_became_peered":"2023-11-28T12:58:55.656984+0000","last_unstale":"2023-11-28T12:58:59.665748+0000","last_undegraded":"2023-11-28T12:58:59.665748+0000","last_fullsized":"2023-11-28T12:58:59.665748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059204+0000","last_change":"2023-11-28T12:58:59.681861+0000","last_active":"2023-11-28T12:59:00.059204+0000","last_peered":"2023-11-28T12:59:00.059204+0000","last_clean":"2023-11-28T12:59:00.059204+0000","last_became_active":"2023-11-28T12:58:59.681503+0000","last_became_peered":"2023-11-28T12:58:59.681503+0000","last_unstale":"2023-11-28T12:59:00.059204+0000","last_undegraded":"2023-11-28T12:59:00.059204+0000","last_fullsized":"2023-11-28T12:59:00.059204+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:27.673942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659368+0000","last_change":"2023-11-28T12:58:55.248299+0000","last_active":"2023-11-28T12:58:59.659368+0000","last_peered":"2023-11-28T12:58:59.659368+0000","last_clean":"2023-11-28T12:58:59.659368+0000","last_became_active":"2023-11-28T12:58:55.247956+0000","last_became_peered":"2023-11-28T12:58:55.247956+0000","last_unstale":"2023-11-28T12:58:59.659368+0000","last_undegraded":"2023-11-28T12:58:59.659368+0000","last_fullsized":"2023-11-28T12:58:59.659368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:25:52.356939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655726+0000","last_change":"2023-11-28T12:58:55.660586+0000","last_active":"2023-11-28T12:58:59.655726+0000","last_peered":"2023-11-28T12:58:59.655726+0000","last_clean":"2023-11-28T12:58:59.655726+0000","last_became_active":"2023-11-28T12:58:55.658146+0000","last_became_peered":"2023-11-28T12:58:55.658146+0000","last_unstale":"2023-11-28T12:58:59.655726+0000","last_undegraded":"2023-11-28T12:58:59.655726+0000","last_fullsized":"2023-11-28T12:58:59.655726+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058649+0000","last_change":"2023-11-28T12:58:59.672932+0000","last_active":"2023-11-28T12:59:00.058649+0000","last_peered":"2023-11-28T12:59:00.058649+0000","last_clean":"2023-11-28T12:59:00.058649+0000","last_became_active":"2023-11-28T12:58:59.672540+0000","last_became_peered":"2023-11-28T12:58:59.672540+0000","last_unstale":"2023-11-28T12:59:00.058649+0000","last_undegraded":"2023-11-28T12:59:00.058649+0000","last_fullsized":"2023-11-28T12:59:00.058649+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:12:31.778491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661524+0000","last_change":"2023-11-28T12:58:55.247493+0000","last_active":"2023-11-28T12:58:59.661524+0000","last_peered":"2023-11-28T12:58:59.661524+0000","last_clean":"2023-11-28T12:58:59.661524+0000","last_became_active":"2023-11-28T12:58:55.247218+0000","last_became_peered":"2023-11-28T12:58:55.247218+0000","last_unstale":"2023-11-28T12:58:59.661524+0000","last_undegraded":"2023-11-28T12:58:59.661524+0000","last_fullsized":"2023-11-28T12:58:59.661524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:38:45.929395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674242+0000","last_change":"2023-11-28T12:58:55.666790+0000","last_active":"2023-11-28T12:58:59.674242+0000","last_peered":"2023-11-28T12:58:59.674242+0000","last_clean":"2023-11-28T12:58:59.674242+0000","last_became_active":"2023-11-28T12:58:55.666541+0000","last_became_peered":"2023-11-28T12:58:55.666541+0000","last_unstale":"2023-11-28T12:58:59.674242+0000","last_undegraded":"2023-11-28T12:58:59.674242+0000","last_fullsized":"2023-11-28T12:58:59.674242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313692+0000","last_change":"2023-11-28T12:58:59.685314+0000","last_active":"2023-11-28T12:59:00.313692+0000","last_peered":"2023-11-28T12:59:00.313692+0000","last_clean":"2023-11-28T12:59:00.313692+0000","last_became_active":"2023-11-28T12:58:59.684940+0000","last_became_peered":"2023-11-28T12:58:59.684940+0000","last_unstale":"2023-11-28T12:59:00.313692+0000","last_undegraded":"2023-11-28T12:59:00.313692+0000","last_fullsized":"2023-11-28T12:59:00.313692+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:53.458898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657621+0000","last_change":"2023-11-28T12:58:55.256308+0000","last_active":"2023-11-28T12:58:59.657621+0000","last_peered":"2023-11-28T12:58:59.657621+0000","last_clean":"2023-11-28T12:58:59.657621+0000","last_became_active":"2023-11-28T12:58:55.256020+0000","last_became_peered":"2023-11-28T12:58:55.256020+0000","last_unstale":"2023-11-28T12:58:59.657621+0000","last_undegraded":"2023-11-28T12:58:59.657621+0000","last_fullsized":"2023-11-28T12:58:59.657621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:49.233234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659104+0000","last_change":"2023-11-28T12:58:55.658284+0000","last_active":"2023-11-28T12:58:59.659104+0000","last_peered":"2023-11-28T12:58:59.659104+0000","last_clean":"2023-11-28T12:58:59.659104+0000","last_became_active":"2023-11-28T12:58:55.657995+0000","last_became_peered":"2023-11-28T12:58:55.657995+0000","last_unstale":"2023-11-28T12:58:59.659104+0000","last_undegraded":"2023-11-28T12:58:59.659104+0000","last_fullsized":"2023-11-28T12:58:59.659104+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:11:49.546022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058116+0000","last_change":"2023-11-28T12:58:59.669835+0000","last_active":"2023-11-28T12:59:00.058116+0000","last_peered":"2023-11-28T12:59:00.058116+0000","last_clean":"2023-11-28T12:59:00.058116+0000","last_became_active":"2023-11-28T12:58:59.669428+0000","last_became_peered":"2023-11-28T12:58:59.669428+0000","last_unstale":"2023-11-28T12:59:00.058116+0000","last_undegraded":"2023-11-28T12:59:00.058116+0000","last_fullsized":"2023-11-28T12:59:00.058116+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:18:08.491909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674258+0000","last_change":"2023-11-28T12:58:55.242328+0000","last_active":"2023-11-28T12:58:59.674258+0000","last_peered":"2023-11-28T12:58:59.674258+0000","last_clean":"2023-11-28T12:58:59.674258+0000","last_became_active":"2023-11-28T12:58:55.242028+0000","last_became_peered":"2023-11-28T12:58:55.242028+0000","last_unstale":"2023-11-28T12:58:59.674258+0000","last_undegraded":"2023-11-28T12:58:59.674258+0000","last_fullsized":"2023-11-28T12:58:59.674258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:42.300078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655600+0000","last_change":"2023-11-28T12:58:55.655732+0000","last_active":"2023-11-28T12:58:59.655600+0000","last_peered":"2023-11-28T12:58:59.655600+0000","last_clean":"2023-11-28T12:58:59.655600+0000","last_became_active":"2023-11-28T12:58:55.655490+0000","last_became_peered":"2023-11-28T12:58:55.655490+0000","last_unstale":"2023-11-28T12:58:59.655600+0000","last_undegraded":"2023-11-28T12:58:59.655600+0000","last_fullsized":"2023-11-28T12:58:59.655600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829559+0000","last_change":"2023-11-28T12:58:59.676365+0000","last_active":"2023-11-28T12:59:00.829559+0000","last_peered":"2023-11-28T12:59:00.829559+0000","last_clean":"2023-11-28T12:59:00.829559+0000","last_became_active":"2023-11-28T12:58:59.675934+0000","last_became_peered":"2023-11-28T12:58:59.675934+0000","last_unstale":"2023-11-28T12:59:00.829559+0000","last_undegraded":"2023-11-28T12:59:00.829559+0000","last_fullsized":"2023-11-28T12:59:00.829559+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:04:35.226907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668457+0000","last_change":"2023-11-28T12:58:55.244542+0000","last_active":"2023-11-28T12:58:59.668457+0000","last_peered":"2023-11-28T12:58:59.668457+0000","last_clean":"2023-11-28T12:58:59.668457+0000","last_became_active":"2023-11-28T12:58:55.244251+0000","last_became_peered":"2023-11-28T12:58:55.244251+0000","last_unstale":"2023-11-28T12:58:59.668457+0000","last_undegraded":"2023-11-28T12:58:59.668457+0000","last_fullsized":"2023-11-28T12:58:59.668457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:06.134459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655451+0000","last_change":"2023-11-28T12:58:55.661735+0000","last_active":"2023-11-28T12:58:59.655451+0000","last_peered":"2023-11-28T12:58:59.655451+0000","last_clean":"2023-11-28T12:58:59.655451+0000","last_became_active":"2023-11-28T12:58:55.661468+0000","last_became_peered":"2023-11-28T12:58:55.661468+0000","last_unstale":"2023-11-28T12:58:59.655451+0000","last_undegraded":"2023-11-28T12:58:59.655451+0000","last_fullsized":"2023-11-28T12:58:59.655451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:34.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059839+0000","last_change":"2023-11-28T12:58:59.677031+0000","last_active":"2023-11-28T12:59:00.059839+0000","last_peered":"2023-11-28T12:59:00.059839+0000","last_clean":"2023-11-28T12:59:00.059839+0000","last_became_active":"2023-11-28T12:58:59.676510+0000","last_became_peered":"2023-11-28T12:58:59.676510+0000","last_unstale":"2023-11-28T12:59:00.059839+0000","last_undegraded":"2023-11-28T12:59:00.059839+0000","last_fullsized":"2023-11-28T12:59:00.059839+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:03:33.042937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"21'3","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664803+0000","last_change":"2023-11-28T12:58:55.254041+0000","last_active":"2023-11-28T12:58:59.664803+0000","last_peered":"2023-11-28T12:58:59.664803+0000","last_clean":"2023-11-28T12:58:59.664803+0000","last_became_active":"2023-11-28T12:58:55.253797+0000","last_became_peered":"2023-11-28T12:58:55.253797+0000","last_unstale":"2023-11-28T12:58:59.664803+0000","last_undegraded":"2023-11-28T12:58:59.664803+0000","last_fullsized":"2023-11-28T12:58:59.664803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:42.442743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664769+0000","last_change":"2023-11-28T12:58:55.648632+0000","last_active":"2023-11-28T12:58:59.664769+0000","last_peered":"2023-11-28T12:58:59.664769+0000","last_clean":"2023-11-28T12:58:59.664769+0000","last_became_active":"2023-11-28T12:58:55.648367+0000","last_became_peered":"2023-11-28T12:58:55.648367+0000","last_unstale":"2023-11-28T12:58:59.664769+0000","last_undegraded":"2023-11-28T12:58:59.664769+0000","last_fullsized":"2023-11-28T12:58:59.664769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.315401+0000","last_change":"2023-11-28T12:58:59.675361+0000","last_active":"2023-11-28T12:59:00.315401+0000","last_peered":"2023-11-28T12:59:00.315401+0000","last_clean":"2023-11-28T12:59:00.315401+0000","last_became_active":"2023-11-28T12:58:59.674991+0000","last_became_peered":"2023-11-28T12:58:59.674991+0000","last_unstale":"2023-11-28T12:59:00.315401+0000","last_undegraded":"2023-11-28T12:59:00.315401+0000","last_fullsized":"2023-11-28T12:59:00.315401+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:21:52.978291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"26'9","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:04.555380+0000","last_change":"2023-11-28T12:58:55.245189+0000","last_active":"2023-11-28T12:59:04.555380+0000","last_peered":"2023-11-28T12:59:04.555380+0000","last_clean":"2023-11-28T12:59:04.555380+0000","last_became_active":"2023-11-28T12:58:55.244896+0000","last_became_peered":"2023-11-28T12:58:55.244896+0000","last_unstale":"2023-11-28T12:59:04.555380+0000","last_undegraded":"2023-11-28T12:59:04.555380+0000","last_fullsized":"2023-11-28T12:59:04.555380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3746,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659437+0000","last_change":"2023-11-28T12:58:55.658816+0000","last_active":"2023-11-28T12:58:59.659437+0000","last_peered":"2023-11-28T12:58:59.659437+0000","last_clean":"2023-11-28T12:58:59.659437+0000","last_became_active":"2023-11-28T12:58:55.657238+0000","last_became_peered":"2023-11-28T12:58:55.657238+0000","last_unstale":"2023-11-28T12:58:59.659437+0000","last_undegraded":"2023-11-28T12:58:59.659437+0000","last_fullsized":"2023-11-28T12:58:59.659437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:14:28.821164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830316+0000","last_change":"2023-11-28T12:58:59.685273+0000","last_active":"2023-11-28T12:59:00.830316+0000","last_peered":"2023-11-28T12:59:00.830316+0000","last_clean":"2023-11-28T12:59:00.830316+0000","last_became_active":"2023-11-28T12:58:59.684917+0000","last_became_peered":"2023-11-28T12:58:59.684917+0000","last_unstale":"2023-11-28T12:59:00.830316+0000","last_undegraded":"2023-11-28T12:59:00.830316+0000","last_fullsized":"2023-11-28T12:59:00.830316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:31.632801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664060+0000","last_change":"2023-11-28T12:58:55.249201+0000","last_active":"2023-11-28T12:58:59.664060+0000","last_peered":"2023-11-28T12:58:59.664060+0000","last_clean":"2023-11-28T12:58:59.664060+0000","last_became_active":"2023-11-28T12:58:55.248828+0000","last_became_peered":"2023-11-28T12:58:55.248828+0000","last_unstale":"2023-11-28T12:58:59.664060+0000","last_undegraded":"2023-11-28T12:58:59.664060+0000","last_fullsized":"2023-11-28T12:58:59.664060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:11:48.540120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655687+0000","last_change":"2023-11-28T12:58:55.660397+0000","last_active":"2023-11-28T12:58:59.655687+0000","last_peered":"2023-11-28T12:58:59.655687+0000","last_clean":"2023-11-28T12:58:59.655687+0000","last_became_active":"2023-11-28T12:58:55.657972+0000","last_became_peered":"2023-11-28T12:58:55.657972+0000","last_unstale":"2023-11-28T12:58:59.655687+0000","last_undegraded":"2023-11-28T12:58:59.655687+0000","last_fullsized":"2023-11-28T12:58:59.655687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059788+0000","last_change":"2023-11-28T12:58:59.681226+0000","last_active":"2023-11-28T12:59:00.059788+0000","last_peered":"2023-11-28T12:59:00.059788+0000","last_clean":"2023-11-28T12:59:00.059788+0000","last_became_active":"2023-11-28T12:58:59.680846+0000","last_became_peered":"2023-11-28T12:58:59.680846+0000","last_unstale":"2023-11-28T12:59:00.059788+0000","last_undegraded":"2023-11-28T12:59:00.059788+0000","last_fullsized":"2023-11-28T12:59:00.059788+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:59.373707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668909+0000","last_change":"2023-11-28T12:58:55.246343+0000","last_active":"2023-11-28T12:58:59.668909+0000","last_peered":"2023-11-28T12:58:59.668909+0000","last_clean":"2023-11-28T12:58:59.668909+0000","last_became_active":"2023-11-28T12:58:55.244840+0000","last_became_peered":"2023-11-28T12:58:55.244840+0000","last_unstale":"2023-11-28T12:58:59.668909+0000","last_undegraded":"2023-11-28T12:58:59.668909+0000","last_fullsized":"2023-11-28T12:58:59.668909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665665+0000","last_change":"2023-11-28T12:58:55.656948+0000","last_active":"2023-11-28T12:58:59.665665+0000","last_peered":"2023-11-28T12:58:59.665665+0000","last_clean":"2023-11-28T12:58:59.665665+0000","last_became_active":"2023-11-28T12:58:55.656678+0000","last_became_peered":"2023-11-28T12:58:55.656678+0000","last_unstale":"2023-11-28T12:58:59.665665+0000","last_undegraded":"2023-11-28T12:58:59.665665+0000","last_fullsized":"2023-11-28T12:58:59.665665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.060115+0000","last_change":"2023-11-28T12:58:59.677384+0000","last_active":"2023-11-28T12:59:00.060115+0000","last_peered":"2023-11-28T12:59:00.060115+0000","last_clean":"2023-11-28T12:59:00.060115+0000","last_became_active":"2023-11-28T12:58:59.676997+0000","last_became_peered":"2023-11-28T12:58:59.676997+0000","last_unstale":"2023-11-28T12:59:00.060115+0000","last_undegraded":"2023-11-28T12:59:00.060115+0000","last_fullsized":"2023-11-28T12:59:00.060115+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:31.632801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.663889+0000","last_change":"2023-11-28T12:58:55.249928+0000","last_active":"2023-11-28T12:58:59.663889+0000","last_peered":"2023-11-28T12:58:59.663889+0000","last_clean":"2023-11-28T12:58:59.663889+0000","last_became_active":"2023-11-28T12:58:55.249500+0000","last_became_peered":"2023-11-28T12:58:55.249500+0000","last_unstale":"2023-11-28T12:58:59.663889+0000","last_undegraded":"2023-11-28T12:58:59.663889+0000","last_fullsized":"2023-11-28T12:58:59.663889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:18.172623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674319+0000","last_change":"2023-11-28T12:58:55.663464+0000","last_active":"2023-11-28T12:58:59.674319+0000","last_peered":"2023-11-28T12:58:59.674319+0000","last_clean":"2023-11-28T12:58:59.674319+0000","last_became_active":"2023-11-28T12:58:55.663205+0000","last_became_peered":"2023-11-28T12:58:55.663205+0000","last_unstale":"2023-11-28T12:58:59.674319+0000","last_undegraded":"2023-11-28T12:58:59.674319+0000","last_fullsized":"2023-11-28T12:58:59.674319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.831228+0000","last_change":"2023-11-28T12:58:59.675282+0000","last_active":"2023-11-28T12:59:00.831228+0000","last_peered":"2023-11-28T12:59:00.831228+0000","last_clean":"2023-11-28T12:59:00.831228+0000","last_became_active":"2023-11-28T12:58:59.674859+0000","last_became_peered":"2023-11-28T12:58:59.674859+0000","last_unstale":"2023-11-28T12:59:00.831228+0000","last_undegraded":"2023-11-28T12:59:00.831228+0000","last_fullsized":"2023-11-28T12:59:00.831228+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:59.373707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"21'8","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.656905+0000","last_change":"2023-11-28T12:58:55.236728+0000","last_active":"2023-11-28T12:58:59.656905+0000","last_peered":"2023-11-28T12:58:59.656905+0000","last_clean":"2023-11-28T12:58:59.656905+0000","last_became_active":"2023-11-28T12:58:55.236452+0000","last_became_peered":"2023-11-28T12:58:55.236452+0000","last_unstale":"2023-11-28T12:58:59.656905+0000","last_undegraded":"2023-11-28T12:58:59.656905+0000","last_fullsized":"2023-11-28T12:58:59.656905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655960+0000","last_change":"2023-11-28T12:58:55.662005+0000","last_active":"2023-11-28T12:58:59.655960+0000","last_peered":"2023-11-28T12:58:59.655960+0000","last_clean":"2023-11-28T12:58:59.655960+0000","last_became_active":"2023-11-28T12:58:55.661531+0000","last_became_peered":"2023-11-28T12:58:55.661531+0000","last_unstale":"2023-11-28T12:58:59.655960+0000","last_undegraded":"2023-11-28T12:58:59.655960+0000","last_fullsized":"2023-11-28T12:58:59.655960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:34:15.223824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830169+0000","last_change":"2023-11-28T12:58:59.675000+0000","last_active":"2023-11-28T12:59:00.830169+0000","last_peered":"2023-11-28T12:59:00.830169+0000","last_clean":"2023-11-28T12:59:00.830169+0000","last_became_active":"2023-11-28T12:58:59.674646+0000","last_became_peered":"2023-11-28T12:58:59.674646+0000","last_unstale":"2023-11-28T12:59:00.830169+0000","last_undegraded":"2023-11-28T12:59:00.830169+0000","last_fullsized":"2023-11-28T12:59:00.830169+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:46:17.417359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667637+0000","last_change":"2023-11-28T12:58:55.249124+0000","last_active":"2023-11-28T12:58:59.667637+0000","last_peered":"2023-11-28T12:58:59.667637+0000","last_clean":"2023-11-28T12:58:59.667637+0000","last_became_active":"2023-11-28T12:58:55.248501+0000","last_became_peered":"2023-11-28T12:58:55.248501+0000","last_unstale":"2023-11-28T12:58:59.667637+0000","last_undegraded":"2023-11-28T12:58:59.667637+0000","last_fullsized":"2023-11-28T12:58:59.667637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:25.569310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661198+0000","last_change":"2023-11-28T12:58:55.659587+0000","last_active":"2023-11-28T12:58:59.661198+0000","last_peered":"2023-11-28T12:58:59.661198+0000","last_clean":"2023-11-28T12:58:59.661198+0000","last_became_active":"2023-11-28T12:58:55.659319+0000","last_became_peered":"2023-11-28T12:58:55.659319+0000","last_unstale":"2023-11-28T12:58:59.661198+0000","last_undegraded":"2023-11-28T12:58:59.661198+0000","last_fullsized":"2023-11-28T12:58:59.661198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.828936+0000","last_change":"2023-11-28T12:58:59.686412+0000","last_active":"2023-11-28T12:59:00.828936+0000","last_peered":"2023-11-28T12:59:00.828936+0000","last_clean":"2023-11-28T12:59:00.828936+0000","last_became_active":"2023-11-28T12:58:59.686079+0000","last_became_peered":"2023-11-28T12:58:59.686079+0000","last_unstale":"2023-11-28T12:59:00.828936+0000","last_undegraded":"2023-11-28T12:59:00.828936+0000","last_fullsized":"2023-11-28T12:59:00.828936+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:58:48.492561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664149+0000","last_change":"2023-11-28T12:58:55.248876+0000","last_active":"2023-11-28T12:58:59.664149+0000","last_peered":"2023-11-28T12:58:59.664149+0000","last_clean":"2023-11-28T12:58:59.664149+0000","last_became_active":"2023-11-28T12:58:55.248584+0000","last_became_peered":"2023-11-28T12:58:55.248584+0000","last_unstale":"2023-11-28T12:58:59.664149+0000","last_undegraded":"2023-11-28T12:58:59.664149+0000","last_fullsized":"2023-11-28T12:58:59.664149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:47:12.446132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659149+0000","last_change":"2023-11-28T12:58:55.663768+0000","last_active":"2023-11-28T12:58:59.659149+0000","last_peered":"2023-11-28T12:58:59.659149+0000","last_clean":"2023-11-28T12:58:59.659149+0000","last_became_active":"2023-11-28T12:58:55.663476+0000","last_became_peered":"2023-11-28T12:58:55.663476+0000","last_unstale":"2023-11-28T12:58:59.659149+0000","last_undegraded":"2023-11-28T12:58:59.659149+0000","last_fullsized":"2023-11-28T12:58:59.659149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:07:49.345226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829075+0000","last_change":"2023-11-28T12:58:59.685644+0000","last_active":"2023-11-28T12:59:00.829075+0000","last_peered":"2023-11-28T12:59:00.829075+0000","last_clean":"2023-11-28T12:59:00.829075+0000","last_became_active":"2023-11-28T12:58:59.685259+0000","last_became_peered":"2023-11-28T12:58:59.685259+0000","last_unstale":"2023-11-28T12:59:00.829075+0000","last_undegraded":"2023-11-28T12:59:00.829075+0000","last_fullsized":"2023-11-28T12:59:00.829075+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:07:52.771805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664976+0000","last_change":"2023-11-28T12:58:55.247686+0000","last_active":"2023-11-28T12:58:59.664976+0000","last_peered":"2023-11-28T12:58:59.664976+0000","last_clean":"2023-11-28T12:58:59.664976+0000","last_became_active":"2023-11-28T12:58:55.247328+0000","last_became_peered":"2023-11-28T12:58:55.247328+0000","last_unstale":"2023-11-28T12:58:59.664976+0000","last_undegraded":"2023-11-28T12:58:59.664976+0000","last_fullsized":"2023-11-28T12:58:59.664976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:22:00.609577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.656940+0000","last_change":"2023-11-28T12:58:55.649490+0000","last_active":"2023-11-28T12:58:59.656940+0000","last_peered":"2023-11-28T12:58:59.656940+0000","last_clean":"2023-11-28T12:58:59.656940+0000","last_became_active":"2023-11-28T12:58:55.649143+0000","last_became_peered":"2023-11-28T12:58:55.649143+0000","last_unstale":"2023-11-28T12:58:59.656940+0000","last_undegraded":"2023-11-28T12:58:59.656940+0000","last_fullsized":"2023-11-28T12:58:59.656940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:26.575212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829146+0000","last_change":"2023-11-28T12:58:59.679465+0000","last_active":"2023-11-28T12:59:00.829146+0000","last_peered":"2023-11-28T12:59:00.829146+0000","last_clean":"2023-11-28T12:59:00.829146+0000","last_became_active":"2023-11-28T12:58:59.679026+0000","last_became_peered":"2023-11-28T12:58:59.679026+0000","last_unstale":"2023-11-28T12:59:00.829146+0000","last_undegraded":"2023-11-28T12:59:00.829146+0000","last_fullsized":"2023-11-28T12:59:00.829146+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:52.339645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665301+0000","last_change":"2023-11-28T12:58:55.249645+0000","last_active":"2023-11-28T12:58:59.665301+0000","last_peered":"2023-11-28T12:58:59.665301+0000","last_clean":"2023-11-28T12:58:59.665301+0000","last_became_active":"2023-11-28T12:58:55.249284+0000","last_became_peered":"2023-11-28T12:58:55.249284+0000","last_unstale":"2023-11-28T12:58:59.665301+0000","last_undegraded":"2023-11-28T12:58:59.665301+0000","last_fullsized":"2023-11-28T12:58:59.665301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:49.233234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669390+0000","last_change":"2023-11-28T12:58:55.663994+0000","last_active":"2023-11-28T12:58:59.669390+0000","last_peered":"2023-11-28T12:58:59.669390+0000","last_clean":"2023-11-28T12:58:59.669390+0000","last_became_active":"2023-11-28T12:58:55.657280+0000","last_became_peered":"2023-11-28T12:58:55.657280+0000","last_unstale":"2023-11-28T12:58:59.669390+0000","last_undegraded":"2023-11-28T12:58:59.669390+0000","last_fullsized":"2023-11-28T12:58:59.669390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:34.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829349+0000","last_change":"2023-11-28T12:58:59.680307+0000","last_active":"2023-11-28T12:59:00.829349+0000","last_peered":"2023-11-28T12:59:00.829349+0000","last_clean":"2023-11-28T12:59:00.829349+0000","last_became_active":"2023-11-28T12:58:59.679837+0000","last_became_peered":"2023-11-28T12:58:59.679837+0000","last_unstale":"2023-11-28T12:59:00.829349+0000","last_undegraded":"2023-11-28T12:59:00.829349+0000","last_fullsized":"2023-11-28T12:59:00.829349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:18:08.491909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655480+0000","last_change":"2023-11-28T12:58:55.256347+0000","last_active":"2023-11-28T12:58:59.655480+0000","last_peered":"2023-11-28T12:58:59.655480+0000","last_clean":"2023-11-28T12:58:59.655480+0000","last_became_active":"2023-11-28T12:58:55.256027+0000","last_became_peered":"2023-11-28T12:58:55.256027+0000","last_unstale":"2023-11-28T12:58:59.655480+0000","last_undegraded":"2023-11-28T12:58:59.655480+0000","last_fullsized":"2023-11-28T12:58:59.655480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:42.300078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659538+0000","last_change":"2023-11-28T12:58:55.664332+0000","last_active":"2023-11-28T12:58:59.659538+0000","last_peered":"2023-11-28T12:58:59.659538+0000","last_clean":"2023-11-28T12:58:59.659538+0000","last_became_active":"2023-11-28T12:58:55.663913+0000","last_became_peered":"2023-11-28T12:58:55.663913+0000","last_unstale":"2023-11-28T12:58:59.659538+0000","last_undegraded":"2023-11-28T12:58:59.659538+0000","last_fullsized":"2023-11-28T12:58:59.659538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:50.590629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.315498+0000","last_change":"2023-11-28T12:58:59.678114+0000","last_active":"2023-11-28T12:59:00.315498+0000","last_peered":"2023-11-28T12:59:00.315498+0000","last_clean":"2023-11-28T12:59:00.315498+0000","last_became_active":"2023-11-28T12:58:59.677704+0000","last_became_peered":"2023-11-28T12:58:59.677704+0000","last_unstale":"2023-11-28T12:59:00.315498+0000","last_undegraded":"2023-11-28T12:59:00.315498+0000","last_fullsized":"2023-11-28T12:59:00.315498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:53.458898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657011+0000","last_change":"2023-11-28T12:58:55.241207+0000","last_active":"2023-11-28T12:58:59.657011+0000","last_peered":"2023-11-28T12:58:59.657011+0000","last_clean":"2023-11-28T12:58:59.657011+0000","last_became_active":"2023-11-28T12:58:55.240917+0000","last_became_peered":"2023-11-28T12:58:55.240917+0000","last_unstale":"2023-11-28T12:58:59.657011+0000","last_undegraded":"2023-11-28T12:58:59.657011+0000","last_fullsized":"2023-11-28T12:58:59.657011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659347+0000","last_change":"2023-11-28T12:58:55.660463+0000","last_active":"2023-11-28T12:58:59.659347+0000","last_peered":"2023-11-28T12:58:59.659347+0000","last_clean":"2023-11-28T12:58:59.659347+0000","last_became_active":"2023-11-28T12:58:55.660208+0000","last_became_peered":"2023-11-28T12:58:55.660208+0000","last_unstale":"2023-11-28T12:58:59.659347+0000","last_undegraded":"2023-11-28T12:58:59.659347+0000","last_fullsized":"2023-11-28T12:58:59.659347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830829+0000","last_change":"2023-11-28T12:58:59.688100+0000","last_active":"2023-11-28T12:59:00.830829+0000","last_peered":"2023-11-28T12:59:00.830829+0000","last_clean":"2023-11-28T12:59:00.830829+0000","last_became_active":"2023-11-28T12:58:59.687752+0000","last_became_peered":"2023-11-28T12:58:59.687752+0000","last_unstale":"2023-11-28T12:59:00.830829+0000","last_undegraded":"2023-11-28T12:59:00.830829+0000","last_fullsized":"2023-11-28T12:59:00.830829+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:38:05.937526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659014+0000","last_change":"2023-11-28T12:58:55.246754+0000","last_active":"2023-11-28T12:58:59.659014+0000","last_peered":"2023-11-28T12:58:59.659014+0000","last_clean":"2023-11-28T12:58:59.659014+0000","last_became_active":"2023-11-28T12:58:55.246480+0000","last_became_peered":"2023-11-28T12:58:55.246480+0000","last_unstale":"2023-11-28T12:58:59.659014+0000","last_undegraded":"2023-11-28T12:58:59.659014+0000","last_fullsized":"2023-11-28T12:58:59.659014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:08.923409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660472+0000","last_change":"2023-11-28T12:58:55.660027+0000","last_active":"2023-11-28T12:58:59.660472+0000","last_peered":"2023-11-28T12:58:59.660472+0000","last_clean":"2023-11-28T12:58:59.660472+0000","last_became_active":"2023-11-28T12:58:55.659775+0000","last_became_peered":"2023-11-28T12:58:55.659775+0000","last_unstale":"2023-11-28T12:58:59.660472+0000","last_undegraded":"2023-11-28T12:58:59.660472+0000","last_fullsized":"2023-11-28T12:58:59.660472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058758+0000","last_change":"2023-11-28T12:58:59.681700+0000","last_active":"2023-11-28T12:59:00.058758+0000","last_peered":"2023-11-28T12:59:00.058758+0000","last_clean":"2023-11-28T12:59:00.058758+0000","last_became_active":"2023-11-28T12:58:59.681305+0000","last_became_peered":"2023-11-28T12:58:59.681305+0000","last_unstale":"2023-11-28T12:59:00.058758+0000","last_undegraded":"2023-11-28T12:59:00.058758+0000","last_fullsized":"2023-11-28T12:59:00.058758+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:46.532522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655511+0000","last_change":"2023-11-28T12:58:55.249674+0000","last_active":"2023-11-28T12:58:59.655511+0000","last_peered":"2023-11-28T12:58:59.655511+0000","last_clean":"2023-11-28T12:58:59.655511+0000","last_became_active":"2023-11-28T12:58:55.249391+0000","last_became_peered":"2023-11-28T12:58:55.249391+0000","last_unstale":"2023-11-28T12:58:59.655511+0000","last_undegraded":"2023-11-28T12:58:59.655511+0000","last_fullsized":"2023-11-28T12:58:59.655511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667665+0000","last_change":"2023-11-28T12:58:55.661741+0000","last_active":"2023-11-28T12:58:59.667665+0000","last_peered":"2023-11-28T12:58:59.667665+0000","last_clean":"2023-11-28T12:58:59.667665+0000","last_became_active":"2023-11-28T12:58:55.661473+0000","last_became_peered":"2023-11-28T12:58:55.661473+0000","last_unstale":"2023-11-28T12:58:59.667665+0000","last_undegraded":"2023-11-28T12:58:59.667665+0000","last_fullsized":"2023-11-28T12:58:59.667665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:41:27.479117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313283+0000","last_change":"2023-11-28T12:58:59.675425+0000","last_active":"2023-11-28T12:59:00.313283+0000","last_peered":"2023-11-28T12:59:00.313283+0000","last_clean":"2023-11-28T12:59:00.313283+0000","last_became_active":"2023-11-28T12:58:59.675044+0000","last_became_peered":"2023-11-28T12:58:59.675044+0000","last_unstale":"2023-11-28T12:59:00.313283+0000","last_undegraded":"2023-11-28T12:59:00.313283+0000","last_fullsized":"2023-11-28T12:59:00.313283+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:46:17.417359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664599+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T12:58:59.664599+0000","last_peered":"2023-11-28T12:58:59.664599+0000","last_clean":"2023-11-28T12:58:59.664599+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T12:58:59.664599+0000","last_undegraded":"2023-11-28T12:58:59.664599+0000","last_fullsized":"2023-11-28T12:58:59.664599+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665575+0000","last_change":"2023-11-28T12:58:55.247919+0000","last_active":"2023-11-28T12:58:59.665575+0000","last_peered":"2023-11-28T12:58:59.665575+0000","last_clean":"2023-11-28T12:58:59.665575+0000","last_became_active":"2023-11-28T12:58:55.247653+0000","last_became_peered":"2023-11-28T12:58:55.247653+0000","last_unstale":"2023-11-28T12:58:59.665575+0000","last_undegraded":"2023-11-28T12:58:59.665575+0000","last_fullsized":"2023-11-28T12:58:59.665575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:54:06.148298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664140+0000","last_change":"2023-11-28T12:58:55.662430+0000","last_active":"2023-11-28T12:58:59.664140+0000","last_peered":"2023-11-28T12:58:59.664140+0000","last_clean":"2023-11-28T12:58:59.664140+0000","last_became_active":"2023-11-28T12:58:55.657899+0000","last_became_peered":"2023-11-28T12:58:55.657899+0000","last_unstale":"2023-11-28T12:58:59.664140+0000","last_undegraded":"2023-11-28T12:58:59.664140+0000","last_fullsized":"2023-11-28T12:58:59.664140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:22:01.615479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313377+0000","last_change":"2023-11-28T12:58:59.675356+0000","last_active":"2023-11-28T12:59:00.313377+0000","last_peered":"2023-11-28T12:59:00.313377+0000","last_clean":"2023-11-28T12:59:00.313377+0000","last_became_active":"2023-11-28T12:58:59.674917+0000","last_became_peered":"2023-11-28T12:58:59.674917+0000","last_unstale":"2023-11-28T12:59:00.313377+0000","last_undegraded":"2023-11-28T12:59:00.313377+0000","last_fullsized":"2023-11-28T12:59:00.313377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:51:00.263485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661311+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T12:58:59.661311+0000","last_peered":"2023-11-28T12:58:59.661311+0000","last_clean":"2023-11-28T12:58:59.661311+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T12:58:59.661311+0000","last_undegraded":"2023-11-28T12:58:59.661311+0000","last_fullsized":"2023-11-28T12:58:59.661311+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659552+0000","last_change":"2023-11-28T12:58:55.240294+0000","last_active":"2023-11-28T12:58:59.659552+0000","last_peered":"2023-11-28T12:58:59.659552+0000","last_clean":"2023-11-28T12:58:59.659552+0000","last_became_active":"2023-11-28T12:58:55.239947+0000","last_became_peered":"2023-11-28T12:58:55.239947+0000","last_unstale":"2023-11-28T12:58:59.659552+0000","last_undegraded":"2023-11-28T12:58:59.659552+0000","last_fullsized":"2023-11-28T12:58:59.659552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664509+0000","last_change":"2023-11-28T12:58:55.657537+0000","last_active":"2023-11-28T12:58:59.664509+0000","last_peered":"2023-11-28T12:58:59.664509+0000","last_clean":"2023-11-28T12:58:59.664509+0000","last_became_active":"2023-11-28T12:58:55.655666+0000","last_became_peered":"2023-11-28T12:58:55.655666+0000","last_unstale":"2023-11-28T12:58:59.664509+0000","last_undegraded":"2023-11-28T12:58:59.664509+0000","last_fullsized":"2023-11-28T12:58:59.664509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829428+0000","last_change":"2023-11-28T12:58:59.680479+0000","last_active":"2023-11-28T12:59:00.829428+0000","last_peered":"2023-11-28T12:59:00.829428+0000","last_clean":"2023-11-28T12:59:00.829428+0000","last_became_active":"2023-11-28T12:58:59.680089+0000","last_became_peered":"2023-11-28T12:58:59.680089+0000","last_unstale":"2023-11-28T12:59:00.829428+0000","last_undegraded":"2023-11-28T12:59:00.829428+0000","last_fullsized":"2023-11-28T12:59:00.829428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:12:31.778491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.673610+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T12:58:59.673610+0000","last_peered":"2023-11-28T12:58:59.673610+0000","last_clean":"2023-11-28T12:58:59.673610+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T12:58:59.673610+0000","last_undegraded":"2023-11-28T12:58:59.673610+0000","last_fullsized":"2023-11-28T12:58:59.673610+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660849+0000","last_change":"2023-11-28T12:58:55.238872+0000","last_active":"2023-11-28T12:58:59.660849+0000","last_peered":"2023-11-28T12:58:59.660849+0000","last_clean":"2023-11-28T12:58:59.660849+0000","last_became_active":"2023-11-28T12:58:55.238457+0000","last_became_peered":"2023-11-28T12:58:55.238457+0000","last_unstale":"2023-11-28T12:58:59.660849+0000","last_undegraded":"2023-11-28T12:58:59.660849+0000","last_fullsized":"2023-11-28T12:58:59.660849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667641+0000","last_change":"2023-11-28T12:58:55.659175+0000","last_active":"2023-11-28T12:58:59.667641+0000","last_peered":"2023-11-28T12:58:59.667641+0000","last_clean":"2023-11-28T12:58:59.667641+0000","last_became_active":"2023-11-28T12:58:55.658043+0000","last_became_peered":"2023-11-28T12:58:55.658043+0000","last_unstale":"2023-11-28T12:58:59.667641+0000","last_undegraded":"2023-11-28T12:58:59.667641+0000","last_fullsized":"2023-11-28T12:58:59.667641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:43.448645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830642+0000","last_change":"2023-11-28T12:58:59.682758+0000","last_active":"2023-11-28T12:59:00.830642+0000","last_peered":"2023-11-28T12:59:00.830642+0000","last_clean":"2023-11-28T12:59:00.830642+0000","last_became_active":"2023-11-28T12:58:59.682429+0000","last_became_peered":"2023-11-28T12:58:59.682429+0000","last_unstale":"2023-11-28T12:59:00.830642+0000","last_undegraded":"2023-11-28T12:59:00.830642+0000","last_fullsized":"2023-11-28T12:59:00.830642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:59.373707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668370+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T12:58:59.668370+0000","last_peered":"2023-11-28T12:58:59.668370+0000","last_clean":"2023-11-28T12:58:59.668370+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T12:58:59.668370+0000","last_undegraded":"2023-11-28T12:58:59.668370+0000","last_fullsized":"2023-11-28T12:58:59.668370+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666175+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T12:58:59.666175+0000","last_peered":"2023-11-28T12:58:59.666175+0000","last_clean":"2023-11-28T12:58:59.666175+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T12:58:59.666175+0000","last_undegraded":"2023-11-28T12:58:59.666175+0000","last_fullsized":"2023-11-28T12:58:59.666175+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655438+0000","last_change":"2023-11-28T12:58:55.248780+0000","last_active":"2023-11-28T12:58:59.655438+0000","last_peered":"2023-11-28T12:58:59.655438+0000","last_clean":"2023-11-28T12:58:59.655438+0000","last_became_active":"2023-11-28T12:58:55.248456+0000","last_became_peered":"2023-11-28T12:58:55.248456+0000","last_unstale":"2023-11-28T12:58:59.655438+0000","last_undegraded":"2023-11-28T12:58:59.655438+0000","last_fullsized":"2023-11-28T12:58:59.655438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:25:52.356939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657383+0000","last_change":"2023-11-28T12:58:55.661508+0000","last_active":"2023-11-28T12:58:59.657383+0000","last_peered":"2023-11-28T12:58:59.657383+0000","last_clean":"2023-11-28T12:58:59.657383+0000","last_became_active":"2023-11-28T12:58:55.661252+0000","last_became_peered":"2023-11-28T12:58:55.661252+0000","last_unstale":"2023-11-28T12:58:59.657383+0000","last_undegraded":"2023-11-28T12:58:59.657383+0000","last_fullsized":"2023-11-28T12:58:59.657383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:22:01.615479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830736+0000","last_change":"2023-11-28T12:58:59.675557+0000","last_active":"2023-11-28T12:59:00.830736+0000","last_peered":"2023-11-28T12:59:00.830736+0000","last_clean":"2023-11-28T12:59:00.830736+0000","last_became_active":"2023-11-28T12:58:59.675183+0000","last_became_peered":"2023-11-28T12:58:59.675183+0000","last_unstale":"2023-11-28T12:59:00.830736+0000","last_undegraded":"2023-11-28T12:59:00.830736+0000","last_fullsized":"2023-11-28T12:59:00.830736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:17.338033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666288+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T12:58:59.666288+0000","last_peered":"2023-11-28T12:58:59.666288+0000","last_clean":"2023-11-28T12:58:59.666288+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T12:58:59.666288+0000","last_undegraded":"2023-11-28T12:58:59.666288+0000","last_fullsized":"2023-11-28T12:58:59.666288+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668355+0000","last_change":"2023-11-28T12:58:55.246924+0000","last_active":"2023-11-28T12:58:59.668355+0000","last_peered":"2023-11-28T12:58:59.668355+0000","last_clean":"2023-11-28T12:58:59.668355+0000","last_became_active":"2023-11-28T12:58:55.244984+0000","last_became_peered":"2023-11-28T12:58:55.244984+0000","last_unstale":"2023-11-28T12:58:59.668355+0000","last_undegraded":"2023-11-28T12:58:59.668355+0000","last_fullsized":"2023-11-28T12:58:59.668355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674088+0000","last_change":"2023-11-28T12:58:55.666303+0000","last_active":"2023-11-28T12:58:59.674088+0000","last_peered":"2023-11-28T12:58:59.674088+0000","last_clean":"2023-11-28T12:58:59.674088+0000","last_became_active":"2023-11-28T12:58:55.666058+0000","last_became_peered":"2023-11-28T12:58:55.666058+0000","last_unstale":"2023-11-28T12:58:59.674088+0000","last_undegraded":"2023-11-28T12:58:59.674088+0000","last_fullsized":"2023-11-28T12:58:59.674088+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058107+0000","last_change":"2023-11-28T12:58:59.681611+0000","last_active":"2023-11-28T12:59:00.058107+0000","last_peered":"2023-11-28T12:59:00.058107+0000","last_clean":"2023-11-28T12:59:00.058107+0000","last_became_active":"2023-11-28T12:58:59.681195+0000","last_became_peered":"2023-11-28T12:58:59.681195+0000","last_unstale":"2023-11-28T12:59:00.058107+0000","last_undegraded":"2023-11-28T12:59:00.058107+0000","last_fullsized":"2023-11-28T12:59:00.058107+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:58:48.492561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664884+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T12:58:59.664884+0000","last_peered":"2023-11-28T12:58:59.664884+0000","last_clean":"2023-11-28T12:58:59.664884+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T12:58:59.664884+0000","last_undegraded":"2023-11-28T12:58:59.664884+0000","last_fullsized":"2023-11-28T12:58:59.664884+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664779+0000","last_change":"2023-11-28T12:58:55.240088+0000","last_active":"2023-11-28T12:58:59.664779+0000","last_peered":"2023-11-28T12:58:59.664779+0000","last_clean":"2023-11-28T12:58:59.664779+0000","last_became_active":"2023-11-28T12:58:55.237949+0000","last_became_peered":"2023-11-28T12:58:55.237949+0000","last_unstale":"2023-11-28T12:58:59.664779+0000","last_undegraded":"2023-11-28T12:58:59.664779+0000","last_fullsized":"2023-11-28T12:58:59.664779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669188+0000","last_change":"2023-11-28T12:58:55.657049+0000","last_active":"2023-11-28T12:58:59.669188+0000","last_peered":"2023-11-28T12:58:59.669188+0000","last_clean":"2023-11-28T12:58:59.669188+0000","last_became_active":"2023-11-28T12:58:55.656787+0000","last_became_peered":"2023-11-28T12:58:55.656787+0000","last_unstale":"2023-11-28T12:58:59.669188+0000","last_undegraded":"2023-11-28T12:58:59.669188+0000","last_fullsized":"2023-11-28T12:58:59.669188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:06:39.770893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058517+0000","last_change":"2023-11-28T12:58:59.676382+0000","last_active":"2023-11-28T12:59:00.058517+0000","last_peered":"2023-11-28T12:59:00.058517+0000","last_clean":"2023-11-28T12:59:00.058517+0000","last_became_active":"2023-11-28T12:58:59.675939+0000","last_became_peered":"2023-11-28T12:58:59.675939+0000","last_unstale":"2023-11-28T12:59:00.058517+0000","last_undegraded":"2023-11-28T12:59:00.058517+0000","last_fullsized":"2023-11-28T12:59:00.058517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:50:50.704627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.656336+0000","last_change":"2023-11-28T12:58:55.250390+0000","last_active":"2023-11-28T12:58:59.656336+0000","last_peered":"2023-11-28T12:58:59.656336+0000","last_clean":"2023-11-28T12:58:59.656336+0000","last_became_active":"2023-11-28T12:58:55.249919+0000","last_became_peered":"2023-11-28T12:58:55.249919+0000","last_unstale":"2023-11-28T12:58:59.656336+0000","last_undegraded":"2023-11-28T12:58:59.656336+0000","last_fullsized":"2023-11-28T12:58:59.656336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:11.131301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665787+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T12:58:59.665787+0000","last_peered":"2023-11-28T12:58:59.665787+0000","last_clean":"2023-11-28T12:58:59.665787+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T12:58:59.665787+0000","last_undegraded":"2023-11-28T12:58:59.665787+0000","last_fullsized":"2023-11-28T12:58:59.665787+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655406+0000","last_change":"2023-11-28T12:58:55.661761+0000","last_active":"2023-11-28T12:58:59.655406+0000","last_peered":"2023-11-28T12:58:59.655406+0000","last_clean":"2023-11-28T12:58:59.655406+0000","last_became_active":"2023-11-28T12:58:55.661488+0000","last_became_peered":"2023-11-28T12:58:55.661488+0000","last_unstale":"2023-11-28T12:58:59.655406+0000","last_undegraded":"2023-11-28T12:58:59.655406+0000","last_fullsized":"2023-11-28T12:58:59.655406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:14:28.821164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059438+0000","last_change":"2023-11-28T12:58:59.678960+0000","last_active":"2023-11-28T12:59:00.059438+0000","last_peered":"2023-11-28T12:59:00.059438+0000","last_clean":"2023-11-28T12:59:00.059438+0000","last_became_active":"2023-11-28T12:58:59.678568+0000","last_became_peered":"2023-11-28T12:58:59.678568+0000","last_unstale":"2023-11-28T12:59:00.059438+0000","last_undegraded":"2023-11-28T12:59:00.059438+0000","last_fullsized":"2023-11-28T12:59:00.059438+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:17.338033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659020+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T12:58:59.659020+0000","last_peered":"2023-11-28T12:58:59.659020+0000","last_clean":"2023-11-28T12:58:59.659020+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T12:58:59.659020+0000","last_undegraded":"2023-11-28T12:58:59.659020+0000","last_fullsized":"2023-11-28T12:58:59.659020+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"26'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313562+0000","last_change":"2023-11-28T12:58:55.251038+0000","last_active":"2023-11-28T12:59:00.313562+0000","last_peered":"2023-11-28T12:59:00.313562+0000","last_clean":"2023-11-28T12:59:00.313562+0000","last_became_active":"2023-11-28T12:58:55.250328+0000","last_became_peered":"2023-11-28T12:58:55.250328+0000","last_unstale":"2023-11-28T12:59:00.313562+0000","last_undegraded":"2023-11-28T12:59:00.313562+0000","last_fullsized":"2023-11-28T12:59:00.313562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:11.131301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659036+0000","last_change":"2023-11-28T12:58:55.658965+0000","last_active":"2023-11-28T12:58:59.659036+0000","last_peered":"2023-11-28T12:58:59.659036+0000","last_clean":"2023-11-28T12:58:59.659036+0000","last_became_active":"2023-11-28T12:58:55.658709+0000","last_became_peered":"2023-11-28T12:58:55.658709+0000","last_unstale":"2023-11-28T12:58:59.659036+0000","last_undegraded":"2023-11-28T12:58:59.659036+0000","last_fullsized":"2023-11-28T12:58:59.659036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:58:45.065981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.316209+0000","last_change":"2023-11-28T12:58:59.679267+0000","last_active":"2023-11-28T12:59:00.316209+0000","last_peered":"2023-11-28T12:59:00.316209+0000","last_clean":"2023-11-28T12:59:00.316209+0000","last_became_active":"2023-11-28T12:58:59.678822+0000","last_became_peered":"2023-11-28T12:58:59.678822+0000","last_unstale":"2023-11-28T12:59:00.316209+0000","last_undegraded":"2023-11-28T12:59:00.316209+0000","last_fullsized":"2023-11-28T12:59:00.316209+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:38:05.937526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669102+0000","last_change":"2023-11-28T12:58:55.250742+0000","last_active":"2023-11-28T12:58:59.669102+0000","last_peered":"2023-11-28T12:58:59.669102+0000","last_clean":"2023-11-28T12:58:59.669102+0000","last_became_active":"2023-11-28T12:58:55.245578+0000","last_became_peered":"2023-11-28T12:58:55.245578+0000","last_unstale":"2023-11-28T12:58:59.669102+0000","last_undegraded":"2023-11-28T12:58:59.669102+0000","last_fullsized":"2023-11-28T12:58:59.669102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:25:52.356939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672422+0000","last_change":"2023-11-28T12:58:55.647087+0000","last_active":"2023-11-28T12:58:59.672422+0000","last_peered":"2023-11-28T12:58:59.672422+0000","last_clean":"2023-11-28T12:58:59.672422+0000","last_became_active":"2023-11-28T12:58:55.646827+0000","last_became_peered":"2023-11-28T12:58:55.646827+0000","last_unstale":"2023-11-28T12:58:59.672422+0000","last_undegraded":"2023-11-28T12:58:59.672422+0000","last_fullsized":"2023-11-28T12:58:59.672422+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:43.305980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830493+0000","last_change":"2023-11-28T12:58:59.673358+0000","last_active":"2023-11-28T12:59:00.830493+0000","last_peered":"2023-11-28T12:59:00.830493+0000","last_clean":"2023-11-28T12:59:00.830493+0000","last_became_active":"2023-11-28T12:58:59.673010+0000","last_became_peered":"2023-11-28T12:58:59.673010+0000","last_unstale":"2023-11-28T12:59:00.830493+0000","last_undegraded":"2023-11-28T12:59:00.830493+0000","last_fullsized":"2023-11-28T12:59:00.830493+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:53.458898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665394+0000","last_change":"2023-11-28T12:58:55.242678+0000","last_active":"2023-11-28T12:58:59.665394+0000","last_peered":"2023-11-28T12:58:59.665394+0000","last_clean":"2023-11-28T12:58:59.665394+0000","last_became_active":"2023-11-28T12:58:55.242467+0000","last_became_peered":"2023-11-28T12:58:55.242467+0000","last_unstale":"2023-11-28T12:58:59.665394+0000","last_undegraded":"2023-11-28T12:58:59.665394+0000","last_fullsized":"2023-11-28T12:58:59.665394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:11.131301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659707+0000","last_change":"2023-11-28T12:58:55.664310+0000","last_active":"2023-11-28T12:58:59.659707+0000","last_peered":"2023-11-28T12:58:59.659707+0000","last_clean":"2023-11-28T12:58:59.659707+0000","last_became_active":"2023-11-28T12:58:55.664036+0000","last_became_peered":"2023-11-28T12:58:55.664036+0000","last_unstale":"2023-11-28T12:58:59.659707+0000","last_undegraded":"2023-11-28T12:58:59.659707+0000","last_fullsized":"2023-11-28T12:58:59.659707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:38:36.781235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830051+0000","last_change":"2023-11-28T12:58:59.685443+0000","last_active":"2023-11-28T12:59:00.830051+0000","last_peered":"2023-11-28T12:59:00.830051+0000","last_clean":"2023-11-28T12:59:00.830051+0000","last_became_active":"2023-11-28T12:58:59.685042+0000","last_became_peered":"2023-11-28T12:58:59.685042+0000","last_unstale":"2023-11-28T12:59:00.830051+0000","last_undegraded":"2023-11-28T12:59:00.830051+0000","last_fullsized":"2023-11-28T12:59:00.830051+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:03:33.042937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665495+0000","last_change":"2023-11-28T12:58:55.241170+0000","last_active":"2023-11-28T12:58:59.665495+0000","last_peered":"2023-11-28T12:58:59.665495+0000","last_clean":"2023-11-28T12:58:59.665495+0000","last_became_active":"2023-11-28T12:58:55.240880+0000","last_became_peered":"2023-11-28T12:58:55.240880+0000","last_unstale":"2023-11-28T12:58:59.665495+0000","last_undegraded":"2023-11-28T12:58:59.665495+0000","last_fullsized":"2023-11-28T12:58:59.665495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:42.300078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659284+0000","last_change":"2023-11-28T12:58:55.664483+0000","last_active":"2023-11-28T12:58:59.659284+0000","last_peered":"2023-11-28T12:58:59.659284+0000","last_clean":"2023-11-28T12:58:59.659284+0000","last_became_active":"2023-11-28T12:58:55.662151+0000","last_became_peered":"2023-11-28T12:58:55.662151+0000","last_unstale":"2023-11-28T12:58:59.659284+0000","last_undegraded":"2023-11-28T12:58:59.659284+0000","last_fullsized":"2023-11-28T12:58:59.659284+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:00:06.262366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058024+0000","last_change":"2023-11-28T12:58:59.679533+0000","last_active":"2023-11-28T12:59:00.058024+0000","last_peered":"2023-11-28T12:59:00.058024+0000","last_clean":"2023-11-28T12:59:00.058024+0000","last_became_active":"2023-11-28T12:58:59.679174+0000","last_became_peered":"2023-11-28T12:58:59.679174+0000","last_unstale":"2023-11-28T12:59:00.058024+0000","last_undegraded":"2023-11-28T12:59:00.058024+0000","last_fullsized":"2023-11-28T12:59:00.058024+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:34:44.322352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655854+0000","last_change":"2023-11-28T12:58:55.246420+0000","last_active":"2023-11-28T12:58:59.655854+0000","last_peered":"2023-11-28T12:58:59.655854+0000","last_clean":"2023-11-28T12:58:59.655854+0000","last_became_active":"2023-11-28T12:58:55.246075+0000","last_became_peered":"2023-11-28T12:58:55.246075+0000","last_unstale":"2023-11-28T12:58:59.655854+0000","last_undegraded":"2023-11-28T12:58:59.655854+0000","last_fullsized":"2023-11-28T12:58:59.655854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:38:45.929395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655474+0000","last_change":"2023-11-28T12:58:55.660837+0000","last_active":"2023-11-28T12:58:59.655474+0000","last_peered":"2023-11-28T12:58:59.655474+0000","last_clean":"2023-11-28T12:58:59.655474+0000","last_became_active":"2023-11-28T12:58:55.657932+0000","last_became_peered":"2023-11-28T12:58:55.657932+0000","last_unstale":"2023-11-28T12:58:59.655474+0000","last_undegraded":"2023-11-28T12:58:59.655474+0000","last_fullsized":"2023-11-28T12:58:59.655474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.315551+0000","last_change":"2023-11-28T12:58:59.683087+0000","last_active":"2023-11-28T12:59:00.315551+0000","last_peered":"2023-11-28T12:59:00.315551+0000","last_clean":"2023-11-28T12:59:00.315551+0000","last_became_active":"2023-11-28T12:58:59.682678+0000","last_became_peered":"2023-11-28T12:58:59.682678+0000","last_unstale":"2023-11-28T12:59:00.315551+0000","last_undegraded":"2023-11-28T12:59:00.315551+0000","last_fullsized":"2023-11-28T12:59:00.315551+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:07:52.771805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"21'4","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674325+0000","last_change":"2023-11-28T12:58:55.246094+0000","last_active":"2023-11-28T12:58:59.674325+0000","last_peered":"2023-11-28T12:58:59.674325+0000","last_clean":"2023-11-28T12:58:59.674325+0000","last_became_active":"2023-11-28T12:58:55.245800+0000","last_became_peered":"2023-11-28T12:58:55.245800+0000","last_unstale":"2023-11-28T12:58:59.674325+0000","last_undegraded":"2023-11-28T12:58:59.674325+0000","last_fullsized":"2023-11-28T12:58:59.674325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:49.233234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.658861+0000","last_change":"2023-11-28T12:58:55.660995+0000","last_active":"2023-11-28T12:58:59.658861+0000","last_peered":"2023-11-28T12:58:59.658861+0000","last_clean":"2023-11-28T12:58:59.658861+0000","last_became_active":"2023-11-28T12:58:55.660761+0000","last_became_peered":"2023-11-28T12:58:55.660761+0000","last_unstale":"2023-11-28T12:58:59.658861+0000","last_undegraded":"2023-11-28T12:58:59.658861+0000","last_fullsized":"2023-11-28T12:58:59.658861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657014+0000","last_change":"2023-11-28T12:58:55.650780+0000","last_active":"2023-11-28T12:58:59.657014+0000","last_peered":"2023-11-28T12:58:59.657014+0000","last_clean":"2023-11-28T12:58:59.657014+0000","last_became_active":"2023-11-28T12:58:55.650511+0000","last_became_peered":"2023-11-28T12:58:55.650511+0000","last_unstale":"2023-11-28T12:58:59.657014+0000","last_undegraded":"2023-11-28T12:58:59.657014+0000","last_fullsized":"2023-11-28T12:58:59.657014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:43.448645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655863+0000","last_change":"2023-11-28T12:58:55.663313+0000","last_active":"2023-11-28T12:58:59.655863+0000","last_peered":"2023-11-28T12:58:59.655863+0000","last_clean":"2023-11-28T12:58:59.655863+0000","last_became_active":"2023-11-28T12:58:55.663036+0000","last_became_peered":"2023-11-28T12:58:55.663036+0000","last_unstale":"2023-11-28T12:58:59.655863+0000","last_undegraded":"2023-11-28T12:58:59.655863+0000","last_fullsized":"2023-11-28T12:58:59.655863+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659437+0000","last_change":"2023-11-28T12:58:55.661206+0000","last_active":"2023-11-28T12:58:59.659437+0000","last_peered":"2023-11-28T12:58:59.659437+0000","last_clean":"2023-11-28T12:58:59.659437+0000","last_became_active":"2023-11-28T12:58:55.660952+0000","last_became_peered":"2023-11-28T12:58:55.660952+0000","last_unstale":"2023-11-28T12:58:59.659437+0000","last_undegraded":"2023-11-28T12:58:59.659437+0000","last_fullsized":"2023-11-28T12:58:59.659437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:34.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655629+0000","last_change":"2023-11-28T12:58:55.661675+0000","last_active":"2023-11-28T12:58:59.655629+0000","last_peered":"2023-11-28T12:58:59.655629+0000","last_clean":"2023-11-28T12:58:59.655629+0000","last_became_active":"2023-11-28T12:58:55.661418+0000","last_became_peered":"2023-11-28T12:58:55.661418+0000","last_unstale":"2023-11-28T12:58:59.655629+0000","last_undegraded":"2023-11-28T12:58:59.655629+0000","last_fullsized":"2023-11-28T12:58:59.655629+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664699+0000","last_change":"2023-11-28T12:58:55.651538+0000","last_active":"2023-11-28T12:58:59.664699+0000","last_peered":"2023-11-28T12:58:59.664699+0000","last_clean":"2023-11-28T12:58:59.664699+0000","last_became_active":"2023-11-28T12:58:55.651297+0000","last_became_peered":"2023-11-28T12:58:55.651297+0000","last_unstale":"2023-11-28T12:58:59.664699+0000","last_undegraded":"2023-11-28T12:58:59.664699+0000","last_fullsized":"2023-11-28T12:58:59.664699+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:56:55.140717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4528,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9272,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362568,"statfs":{"total":96636764160,"available":96627269632,"internally_reserved":0,"allocated":901120,"data_stored":709174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8088,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363752,"statfs":{"total":96636764160,"available":96628482048,"internally_reserved":0,"allocated":278528,"data_stored":114844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8152,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363688,"statfs":{"total":96636764160,"available":96628416512,"internally_reserved":0,"allocated":278528,"data_stored":114844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9248,"kb_used_data":856,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362592,"statfs":{"total":96636764160,"available":96627294208,"internally_reserved":0,"allocated":876544,"data_stored":705778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8088,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363752,"statfs":{"total":96636764160,"available":96628482048,"internally_reserved":0,"allocated":278528,"data_stored":114844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8156,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363684,"statfs":{"total":96636764160,"available":96628412416,"internally_reserved":0,"allocated":282624,"data_stored":114934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8236,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363604,"statfs":{"total":96636764160,"available":96628330496,"internally_reserved":0,"allocated":299008,"data_stored":118646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8160,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363680,"statfs":{"total":96636764160,"available":96628408320,"internally_reserved":0,"allocated":286720,"data_stored":115480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T12:59:19.152 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T12:59:19.153 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-28T12:59:19.153 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-28T12:59:19.153 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-28T12:59:19.154 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-28T12:59:19.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.304+0000 7f06fd31b700 1 Processor -- start 2023-11-28T12:59:19.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.304+0000 7f06fd31b700 1 -- start start 2023-11-28T12:59:19.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06fd31b700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f06f815cbf0 0x7f06f80b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06fd31b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80cfdb0 con 0x7f06f806ba90 2023-11-28T12:59:19.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06fd31b700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80cc740 con 0x7f06f80b40a0 2023-11-28T12:59:19.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06fd31b700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f80b35e0 con 0x7f06f815cbf0 2023-11-28T12:59:19.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f77fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f06f806ba90 0x7f06f80ccbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33934/0 (socket says 172.21.15.111:33934) 2023-11-28T12:59:19.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f77fe700 1 -- 172.21.15.111:0/3870353043 learned_addr learned my addr 172.21.15.111:0/3870353043 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:19.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f6ffd700 1 --2- 172.21.15.111:0/3870353043 >> v2:172.21.15.149:3300/0 conn(0x7f06f815cbf0 0x7f06f80b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936666312 0 0) 0x7f06f80cfdb0 con 0x7f06f806ba90 2023-11-28T12:59:19.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e00035b0 con 0x7f06f806ba90 2023-11-28T12:59:19.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3220313246 0 0) 0x7f06e00035b0 con 0x7f06f806ba90 2023-11-28T12:59:19.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 >> v2:172.21.15.149:3300/0 conn(0x7f06f815cbf0 msgr2=0x7f06f80b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:19.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 --2- 172.21.15.111:0/3870353043 >> v2:172.21.15.149:3300/0 conn(0x7f06f815cbf0 0x7f06f80b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 >> v1:172.21.15.149:6789/0 conn(0x7f06f80b40a0 legacy=0x7f06f806b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80cbe10 con 0x7f06f806ba90 2023-11-28T12:59:19.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f06f00019f0 con 0x7f06f806ba90 2023-11-28T12:59:19.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 >> v1:172.21.15.111:6789/0 conn(0x7f06f806ba90 legacy=0x7f06f80ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 --2- 172.21.15.111:0/3870353043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06e0003a00 0x7f06e0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 --2- 172.21.15.111:0/3870353043 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06e0004850 0x7f06e0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 --2- 172.21.15.111:0/3870353043 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06e0005200 0x7f06e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80b35e0 con 0x7f06e0003a00 2023-11-28T12:59:19.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e000c110 con 0x7f06e0005200 2023-11-28T12:59:19.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e000c2d0 con 0x7f06e0004850 2023-11-28T12:59:19.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f6ffd700 1 --2- 172.21.15.111:0/3870353043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06e0003a00 0x7f06e0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f77fe700 1 --2- 172.21.15.111:0/3870353043 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06e0005200 0x7f06e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.305+0000 7f06f67fc700 1 --2- 172.21.15.111:0/3870353043 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06e0004850 0x7f06e0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06f6ffd700 1 -- 172.21.15.111:0/3870353043 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06e0004850 msgr2=0x7f06e0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:19.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06f6ffd700 1 --2- 172.21.15.111:0/3870353043 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06e0004850 0x7f06e0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06f6ffd700 1 -- 172.21.15.111:0/3870353043 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06e0005200 msgr2=0x7f06e000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:19.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06f6ffd700 1 --2- 172.21.15.111:0/3870353043 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06e0005200 0x7f06e000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06f77fe700 1 -- 172.21.15.111:0/3870353043 reap_dead start 2023-11-28T12:59:19.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06f6ffd700 1 -- 172.21.15.111:0/3870353043 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e00035b0 con 0x7f06e0003a00 2023-11-28T12:59:19.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06f6ffd700 1 --2- 172.21.15.111:0/3870353043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06e0003a00 0x7f06e0003e10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f06e8009b60 tx=0x7f06e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48f82972152d1a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:19.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e800e6c0 con 0x7f06e0003a00 2023-11-28T12:59:19.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e800e820 con 0x7f06e0003a00 2023-11-28T12:59:19.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06f5ffb700 1 -- 172.21.15.111:0/3870353043 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e800eb90 con 0x7f06e0003a00 2023-11-28T12:59:19.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06fd31b700 1 -- 172.21.15.111:0/3870353043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06e0003a00 msgr2=0x7f06e0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06fd31b700 1 --2- 172.21.15.111:0/3870353043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06e0003a00 0x7f06e0003e10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f06e8009b60 tx=0x7f06e0005880 comp rx=0 tx=0).stop 2023-11-28T12:59:19.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06fd31b700 1 -- 172.21.15.111:0/3870353043 shutdown_connections 2023-11-28T12:59:19.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.306+0000 7f06fd31b700 1 --2- 172.21.15.111:0/3870353043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06e0003a00 0x7f06e0003e10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.307+0000 7f06fd31b700 1 -- 172.21.15.111:0/3870353043 >> 172.21.15.111:0/3870353043 conn(0x7f06f809a040 msgr2=0x7f06f80d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:19.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.307+0000 7f06fd31b700 1 -- 172.21.15.111:0/3870353043 shutdown_connections 2023-11-28T12:59:19.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.307+0000 7f06fd31b700 1 -- 172.21.15.111:0/3870353043 wait complete. 2023-11-28T12:59:19.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.308+0000 7f06fd31b700 1 Processor -- start 2023-11-28T12:59:19.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.308+0000 7f06fd31b700 1 -- start start 2023-11-28T12:59:19.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06fd31b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06e0003a00 0x7f06f8154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06fd31b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06f80c58c0 0x7f06f80ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06fd31b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f80cabb0 0x7f06f80cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06fd31b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f81f6620 con 0x7f06f80cabb0 2023-11-28T12:59:19.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06fd31b700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f8155090 con 0x7f06f80c58c0 2023-11-28T12:59:19.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06fd31b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f81f59a0 con 0x7f06e0003a00 2023-11-28T12:59:19.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06f77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f80cabb0 0x7f06f80cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06f77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f80cabb0 0x7f06f80cc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33168/0 (socket says 172.21.15.111:33168) 2023-11-28T12:59:19.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06f77fe700 1 -- 172.21.15.111:0/4240973419 learned_addr learned my addr 172.21.15.111:0/4240973419 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:19.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06f67fc700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06f80c58c0 0x7f06f80ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06f6ffd700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06e0003a00 0x7f06f8154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06f77fe700 1 -- 172.21.15.111:0/4240973419 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06e0003a00 msgr2=0x7f06f8154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06f77fe700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06e0003a00 0x7f06f8154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06f77fe700 1 -- 172.21.15.111:0/4240973419 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06f80c58c0 msgr2=0x7f06f80ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06f77fe700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06f80c58c0 0x7f06f80ca670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06f77fe700 1 -- 172.21.15.111:0/4240973419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80cc3d0 con 0x7f06f80cabb0 2023-11-28T12:59:19.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06f67fc700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06f80c58c0 0x7f06f80ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:19.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06f77fe700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f80cabb0 0x7f06f80cc8f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f06f8071420 tx=0x7f06f000a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:19.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06df7fe700 1 -- 172.21.15.111:0/4240973419 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f000ada0 con 0x7f06f80cabb0 2023-11-28T12:59:19.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06df7fe700 1 -- 172.21.15.111:0/4240973419 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f00133f0 con 0x7f06f80cabb0 2023-11-28T12:59:19.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06df7fe700 1 -- 172.21.15.111:0/4240973419 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f00137a0 con 0x7f06f80cabb0 2023-11-28T12:59:19.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06fd31b700 1 -- 172.21.15.111:0/4240973419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f80cc0f0 con 0x7f06f80cabb0 2023-11-28T12:59:19.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.309+0000 7f06fd31b700 1 -- 172.21.15.111:0/4240973419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f80d06f0 con 0x7f06f80cabb0 2023-11-28T12:59:19.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.310+0000 7f06fd31b700 1 -- 172.21.15.111:0/4240973419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f81f6760 con 0x7f06f80cabb0 2023-11-28T12:59:19.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.311+0000 7f06df7fe700 1 -- 172.21.15.111:0/4240973419 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f06f0013900 con 0x7f06f80cabb0 2023-11-28T12:59:19.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.311+0000 7f06df7fe700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06e40682f0 0x7f06e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.311+0000 7f06df7fe700 1 -- 172.21.15.111:0/4240973419 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f06f0086a70 con 0x7f06f80cabb0 2023-11-28T12:59:19.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.311+0000 7f06f6ffd700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06e40682f0 0x7f06e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.311+0000 7f06f6ffd700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06e40682f0 0x7f06e406a7a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f06e80090b0 tx=0x7f06e801e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:19.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.315+0000 7f06df7fe700 1 -- 172.21.15.111:0/4240973419 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f06f81f6760 con 0x7f06f80cabb0 2023-11-28T12:59:19.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.545+0000 7f06fd31b700 1 -- 172.21.15.111:0/4240973419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f06f81f6760 con 0x7f06f80cabb0 2023-11-28T12:59:19.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.548+0000 7f06df7fe700 1 -- 172.21.15.111:0/4240973419 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+166386 (secure 0 0 0) 0x7f06f81f6760 con 0x7f06f80cabb0 2023-11-28T12:59:19.551 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:19.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.551+0000 7f06fd31b700 1 -- 172.21.15.111:0/4240973419 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06e40682f0 msgr2=0x7f06e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.551+0000 7f06fd31b700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06e40682f0 0x7f06e406a7a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f06e80090b0 tx=0x7f06e801e040 comp rx=0 tx=0).stop 2023-11-28T12:59:19.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.551+0000 7f06fd31b700 1 -- 172.21.15.111:0/4240973419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f80cabb0 msgr2=0x7f06f80cc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.551+0000 7f06fd31b700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f80cabb0 0x7f06f80cc8f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f06f8071420 tx=0x7f06f000a5f0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.551+0000 7f06fd31b700 1 -- 172.21.15.111:0/4240973419 shutdown_connections 2023-11-28T12:59:19.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.551+0000 7f06fd31b700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06f80cabb0 0x7f06f80cc8f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.551+0000 7f06fd31b700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06e0003a00 0x7f06f8154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.551+0000 7f06fd31b700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06e40682f0 0x7f06e406a7a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.551+0000 7f06fd31b700 1 --2- 172.21.15.111:0/4240973419 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06f80c58c0 0x7f06f80ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.551+0000 7f06fd31b700 1 -- 172.21.15.111:0/4240973419 >> 172.21.15.111:0/4240973419 conn(0x7f06f809a040 msgr2=0x7f06f806beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:19.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.551+0000 7f06fd31b700 1 -- 172.21.15.111:0/4240973419 shutdown_connections 2023-11-28T12:59:19.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.551+0000 7f06fd31b700 1 -- 172.21.15.111:0/4240973419 wait complete. 2023-11-28T12:59:19.564 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":5,"active_gid":4110,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":103256},{"type":"v1","addr":"172.21.15.111:6833","nonce":103256}]},"active_addr":"172.21.15.111:6833/103256","active_change":"2023-11-28T12:58:40.326033+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4105,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":1760865100}]},{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":4192805326}]},{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":2383055349}]},{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":1948746027}]}]} 2023-11-28T12:59:19.567 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-28T12:59:19.567 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-28T12:59:19.567 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:19.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.689+0000 7fbcb8762700 1 Processor -- start 2023-11-28T12:59:19.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.689+0000 7fbcb8762700 1 -- start start 2023-11-28T12:59:19.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.690+0000 7fbcb8762700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbcb00b0440 0x7fbcb00b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.690+0000 7fbcb8762700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbcb00ca2f0 0x7fbcb00ca770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.690+0000 7fbcb8762700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb00cc750 con 0x7fbcb015c6c0 2023-11-28T12:59:19.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.690+0000 7fbcb8762700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb00b0260 con 0x7fbcb00ca2f0 2023-11-28T12:59:19.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.690+0000 7fbcb8762700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb00b0df0 con 0x7fbcb00b0440 2023-11-28T12:59:19.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.690+0000 7fbcb6cff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbcb00ca2f0 0x7fbcb00ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.691+0000 7fbcb64fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbcb015c6c0 0x7fbcb00afb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33950/0 (socket says 172.21.15.111:33950) 2023-11-28T12:59:19.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.691+0000 7fbcb64fe700 1 -- 172.21.15.111:0/259302658 learned_addr learned my addr 172.21.15.111:0/259302658 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:19.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.691+0000 7fbcb5cfd700 1 --2- 172.21.15.111:0/259302658 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbcb00b0440 0x7fbcb00b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.691+0000 7fbcb54fc700 1 -- 172.21.15.111:0/259302658 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558426918 0 0) 0x7fbcb00cc750 con 0x7fbcb015c6c0 2023-11-28T12:59:19.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.691+0000 7fbcb54fc700 1 -- 172.21.15.111:0/259302658 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc9c0035b0 con 0x7fbcb015c6c0 2023-11-28T12:59:19.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.691+0000 7fbcb6cff700 1 -- 172.21.15.111:0/259302658 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbcb00b0440 msgr2=0x7fbcb00b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:19.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.691+0000 7fbcb6cff700 1 --2- 172.21.15.111:0/259302658 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbcb00b0440 0x7fbcb00b08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.691+0000 7fbcb6cff700 1 -- 172.21.15.111:0/259302658 >> v1:172.21.15.111:6789/0 conn(0x7fbcb015c6c0 legacy=0x7fbcb00afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.691+0000 7fbcb6cff700 1 -- 172.21.15.111:0/259302658 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb00cca30 con 0x7fbcb00ca2f0 2023-11-28T12:59:19.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.691+0000 7fbcb6cff700 1 --2- 172.21.15.111:0/259302658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcb00ca2f0 0x7fbcb00ca770 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbca8009d90 tx=0x7fbca800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d95694002375fd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:19.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.691+0000 7fbcb54fc700 1 -- 172.21.15.111:0/259302658 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca800e070 con 0x7fbcb00ca2f0 2023-11-28T12:59:19.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.691+0000 7fbcb54fc700 1 -- 172.21.15.111:0/259302658 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca800bb20 con 0x7fbcb00ca2f0 2023-11-28T12:59:19.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.691+0000 7fbcb54fc700 1 -- 172.21.15.111:0/259302658 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8012430 con 0x7fbcb00ca2f0 2023-11-28T12:59:19.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.692+0000 7fbcb8762700 1 -- 172.21.15.111:0/259302658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcb00ca2f0 msgr2=0x7fbcb00ca770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.692+0000 7fbcb8762700 1 --2- 172.21.15.111:0/259302658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcb00ca2f0 0x7fbcb00ca770 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbca8009d90 tx=0x7fbca800b330 comp rx=0 tx=0).stop 2023-11-28T12:59:19.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.692+0000 7fbcb8762700 1 -- 172.21.15.111:0/259302658 shutdown_connections 2023-11-28T12:59:19.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.692+0000 7fbcb8762700 1 --2- 172.21.15.111:0/259302658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcb00ca2f0 0x7fbcb00ca770 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.692+0000 7fbcb8762700 1 --2- 172.21.15.111:0/259302658 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbcb00b0440 0x7fbcb00b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.692+0000 7fbcb8762700 1 -- 172.21.15.111:0/259302658 >> 172.21.15.111:0/259302658 conn(0x7fbcb0098ee0 msgr2=0x7fbcb006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:19.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.692+0000 7fbcb8762700 1 -- 172.21.15.111:0/259302658 shutdown_connections 2023-11-28T12:59:19.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.692+0000 7fbcb8762700 1 -- 172.21.15.111:0/259302658 wait complete. 2023-11-28T12:59:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.693+0000 7fbcb8762700 1 Processor -- start 2023-11-28T12:59:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.693+0000 7fbcb8762700 1 -- start start 2023-11-28T12:59:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb8762700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbcb00b0440 0x7fbcb00b55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb8762700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbcb00ca2f0 0x7fbcb0159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb8762700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcb015c6c0 0x7fbcb0146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb8762700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb01f2420 con 0x7fbcb015c6c0 2023-11-28T12:59:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb8762700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb0146d50 con 0x7fbcb00ca2f0 2023-11-28T12:59:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb8762700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb015a5f0 con 0x7fbcb00b0440 2023-11-28T12:59:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb6cff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcb015c6c0 0x7fbcb0146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb6cff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcb015c6c0 0x7fbcb0146810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33182/0 (socket says 172.21.15.111:33182) 2023-11-28T12:59:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb6cff700 1 -- 172.21.15.111:0/1201923184 learned_addr learned my addr 172.21.15.111:0/1201923184 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb64fe700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbcb00b0440 0x7fbcb00b55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb5cfd700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbcb00ca2f0 0x7fbcb0159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb6cff700 1 -- 172.21.15.111:0/1201923184 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbcb00b0440 msgr2=0x7fbcb00b55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb6cff700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbcb00b0440 0x7fbcb00b55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb6cff700 1 -- 172.21.15.111:0/1201923184 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbcb00ca2f0 msgr2=0x7fbcb0159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb6cff700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbcb00ca2f0 0x7fbcb0159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb6cff700 1 -- 172.21.15.111:0/1201923184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca8009a70 con 0x7fbcb015c6c0 2023-11-28T12:59:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb5cfd700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbcb00ca2f0 0x7fbcb0159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb6cff700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcb015c6c0 0x7fbcb0146810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbca800b370 tx=0x7fbca8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbca6ffd700 1 -- 172.21.15.111:0/1201923184 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8012430 con 0x7fbcb015c6c0 2023-11-28T12:59:19.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbca6ffd700 1 -- 172.21.15.111:0/1201923184 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca8020070 con 0x7fbcb015c6c0 2023-11-28T12:59:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.694+0000 7fbcb8762700 1 -- 172.21.15.111:0/1201923184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb00ccff0 con 0x7fbcb015c6c0 2023-11-28T12:59:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.695+0000 7fbca6ffd700 1 -- 172.21.15.111:0/1201923184 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8004280 con 0x7fbcb015c6c0 2023-11-28T12:59:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.695+0000 7fbcb8762700 1 -- 172.21.15.111:0/1201923184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb00ccd10 con 0x7fbcb015c6c0 2023-11-28T12:59:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.695+0000 7fbcb8762700 1 -- 172.21.15.111:0/1201923184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb00cd2d0 con 0x7fbcb015c6c0 2023-11-28T12:59:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.696+0000 7fbca6ffd700 1 -- 172.21.15.111:0/1201923184 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbca80043e0 con 0x7fbcb015c6c0 2023-11-28T12:59:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.696+0000 7fbca6ffd700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbc940682f0 0x7fbc9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.696+0000 7fbca6ffd700 1 -- 172.21.15.111:0/1201923184 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fbca8022070 con 0x7fbcb015c6c0 2023-11-28T12:59:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.696+0000 7fbcb64fe700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbc940682f0 0x7fbc9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:19.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.697+0000 7fbcb64fe700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbc940682f0 0x7fbc9406a7a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbca0007fd0 tx=0x7fbca0007dc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.700+0000 7fbca6ffd700 1 -- 172.21.15.111:0/1201923184 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbca80548a0 con 0x7fbcb015c6c0 2023-11-28T12:59:19.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.884+0000 7fbcb8762700 1 -- 172.21.15.111:0/1201923184 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcb00cdf80 con 0x7fbcb015c6c0 2023-11-28T12:59:19.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.885+0000 7fbca6ffd700 1 -- 172.21.15.111:0/1201923184 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19796 (secure 0 0 0) 0x7fbca8057ec0 con 0x7fbcb015c6c0 2023-11-28T12:59:19.887 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:19.887 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:00.053943+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:19.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.887+0000 7fbcb8762700 1 -- 172.21.15.111:0/1201923184 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbc940682f0 msgr2=0x7fbc9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.887+0000 7fbcb8762700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbc940682f0 0x7fbc9406a7a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbca0007fd0 tx=0x7fbca0007dc0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.887+0000 7fbcb8762700 1 -- 172.21.15.111:0/1201923184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcb015c6c0 msgr2=0x7fbcb0146810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:19.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.887+0000 7fbcb8762700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcb015c6c0 0x7fbcb0146810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbca800b370 tx=0x7fbca8004ad0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.887+0000 7fbcb8762700 1 -- 172.21.15.111:0/1201923184 shutdown_connections 2023-11-28T12:59:19.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.887+0000 7fbcb8762700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcb015c6c0 0x7fbcb0146810 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.887+0000 7fbcb8762700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbcb00b0440 0x7fbcb00b55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.887+0000 7fbcb8762700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbc940682f0 0x7fbc9406a7a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.887+0000 7fbcb8762700 1 --2- 172.21.15.111:0/1201923184 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbcb00ca2f0 0x7fbcb0159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:19.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.887+0000 7fbcb8762700 1 -- 172.21.15.111:0/1201923184 >> 172.21.15.111:0/1201923184 conn(0x7fbcb0098ee0 msgr2=0x7fbcb00cafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:19.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.887+0000 7fbcb8762700 1 -- 172.21.15.111:0/1201923184 shutdown_connections 2023-11-28T12:59:19.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:19.887+0000 7fbcb8762700 1 -- 172.21.15.111:0/1201923184 wait complete. 2023-11-28T12:59:19.901 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-28T12:59:19.901 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:20.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.019+0000 7f555ee4f700 1 Processor -- start 2023-11-28T12:59:20.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.019+0000 7f555ee4f700 1 -- start start 2023-11-28T12:59:20.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.020+0000 7f555ee4f700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f55580b0d70 0x7f55581604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.020+0000 7f555ee4f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f55580b1140 0x7f55580ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.020+0000 7f555ee4f700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555815cd90 con 0x7f55580b15b0 2023-11-28T12:59:20.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.020+0000 7f555ee4f700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55581613e0 con 0x7f55580b0d70 2023-11-28T12:59:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.020+0000 7f555ee4f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5558160b40 con 0x7f55580b1140 2023-11-28T12:59:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.020+0000 7f5557fff700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f55580b1140 0x7f55580ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.020+0000 7f555cbeb700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f55580b0d70 0x7f55581604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.020+0000 7f555d3ec700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f55580b15b0 0x7f55580cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:42506/0 (socket says 172.21.15.111:42506) 2023-11-28T12:59:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.020+0000 7f555d3ec700 1 -- 172.21.15.111:0/92877533 learned_addr learned my addr 172.21.15.111:0/92877533 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.021+0000 7f5557fff700 1 -- 172.21.15.111:0/92877533 >> v2:172.21.15.149:3300/0 conn(0x7f55580b0d70 msgr2=0x7f55581604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.021+0000 7f5557fff700 1 --2- 172.21.15.111:0/92877533 >> v2:172.21.15.149:3300/0 conn(0x7f55580b0d70 0x7f55581604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.021+0000 7f5557fff700 1 -- 172.21.15.111:0/92877533 >> v1:172.21.15.149:6789/0 conn(0x7f55580b15b0 legacy=0x7f55580cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.021+0000 7f5557fff700 1 -- 172.21.15.111:0/92877533 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5558160e20 con 0x7f55580b1140 2023-11-28T12:59:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.021+0000 7f5557fff700 1 --2- 172.21.15.111:0/92877533 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f55580b1140 0x7f55580ce6c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5544009840 tx=0x7f5544009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e05d42fa9465383 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.021+0000 7f55577fe700 1 -- 172.21.15.111:0/92877533 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5544011830 con 0x7f55580b1140 2023-11-28T12:59:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.021+0000 7f55577fe700 1 -- 172.21.15.111:0/92877533 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5544011990 con 0x7f55580b1140 2023-11-28T12:59:20.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.021+0000 7f55577fe700 1 -- 172.21.15.111:0/92877533 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5544011d20 con 0x7f55580b1140 2023-11-28T12:59:20.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.021+0000 7f555ee4f700 1 -- 172.21.15.111:0/92877533 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f55580b1140 msgr2=0x7f55580ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.021+0000 7f555ee4f700 1 --2- 172.21.15.111:0/92877533 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f55580b1140 0x7f55580ce6c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5544009840 tx=0x7f5544009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:20.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.022+0000 7f555ee4f700 1 -- 172.21.15.111:0/92877533 shutdown_connections 2023-11-28T12:59:20.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.022+0000 7f555ee4f700 1 --2- 172.21.15.111:0/92877533 >> v2:172.21.15.149:3300/0 conn(0x7f55580b0d70 0x7f55581604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.022+0000 7f555ee4f700 1 --2- 172.21.15.111:0/92877533 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f55580b1140 0x7f55580ce6c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.022+0000 7f555ee4f700 1 -- 172.21.15.111:0/92877533 >> 172.21.15.111:0/92877533 conn(0x7f5558098ee0 msgr2=0x7f555806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.022+0000 7f555ee4f700 1 -- 172.21.15.111:0/92877533 shutdown_connections 2023-11-28T12:59:20.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.022+0000 7f555ee4f700 1 -- 172.21.15.111:0/92877533 wait complete. 2023-11-28T12:59:20.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.023+0000 7f555ee4f700 1 Processor -- start 2023-11-28T12:59:20.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.023+0000 7f555ee4f700 1 -- start start 2023-11-28T12:59:20.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555ee4f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f55580b0d70 0x7f555814bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555ee4f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f55580b1140 0x7f55580c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555ee4f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55580b15b0 0x7f55580c9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555ee4f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55580c4f70 con 0x7f55580b15b0 2023-11-28T12:59:20.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555ee4f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55580c9940 con 0x7f55580b0d70 2023-11-28T12:59:20.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555ee4f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555814c130 con 0x7f55580b1140 2023-11-28T12:59:20.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555d3ec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55580b15b0 0x7f55580c9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555d3ec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55580b15b0 0x7f55580c9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33196/0 (socket says 172.21.15.111:33196) 2023-11-28T12:59:20.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555d3ec700 1 -- 172.21.15.111:0/3679740943 learned_addr learned my addr 172.21.15.111:0/3679740943 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f5557fff700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f55580b1140 0x7f55580c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555cbeb700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f55580b0d70 0x7f555814bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555d3ec700 1 -- 172.21.15.111:0/3679740943 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f55580b1140 msgr2=0x7f55580c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555d3ec700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f55580b1140 0x7f55580c8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555d3ec700 1 -- 172.21.15.111:0/3679740943 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f55580b0d70 msgr2=0x7f555814bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555d3ec700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f55580b0d70 0x7f555814bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555d3ec700 1 -- 172.21.15.111:0/3679740943 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55440094a0 con 0x7f55580b15b0 2023-11-28T12:59:20.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555d3ec700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55580b15b0 0x7f55580c9400 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f555000ca20 tx=0x7f555000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f55557fa700 1 -- 172.21.15.111:0/3679740943 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555000aaa0 con 0x7f55580b15b0 2023-11-28T12:59:20.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f55557fa700 1 -- 172.21.15.111:0/3679740943 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555000ac00 con 0x7f55580b15b0 2023-11-28T12:59:20.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f55557fa700 1 -- 172.21.15.111:0/3679740943 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5550012400 con 0x7f55580b15b0 2023-11-28T12:59:20.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555ee4f700 1 -- 172.21.15.111:0/3679740943 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55580af340 con 0x7f55580b15b0 2023-11-28T12:59:20.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.024+0000 7f555ee4f700 1 -- 172.21.15.111:0/3679740943 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5558161100 con 0x7f55580b15b0 2023-11-28T12:59:20.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.025+0000 7f555ee4f700 1 -- 172.21.15.111:0/3679740943 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55580cec00 con 0x7f55580b15b0 2023-11-28T12:59:20.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.026+0000 7f55557fa700 1 -- 172.21.15.111:0/3679740943 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5550019030 con 0x7f55580b15b0 2023-11-28T12:59:20.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.026+0000 7f55557fa700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f553c0682f0 0x7f553c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.026+0000 7f55557fa700 1 -- 172.21.15.111:0/3679740943 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f55500857c0 con 0x7f55580b15b0 2023-11-28T12:59:20.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.026+0000 7f555cbeb700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f553c0682f0 0x7f553c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.026+0000 7f555cbeb700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f553c0682f0 0x7f553c06a7a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f55580ae990 tx=0x7f554c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.030+0000 7f55557fa700 1 -- 172.21.15.111:0/3679740943 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f555004b5e0 con 0x7f55580b15b0 2023-11-28T12:59:20.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.212+0000 7f555ee4f700 1 -- 172.21.15.111:0/3679740943 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f555815cd90 con 0x7f55580b15b0 2023-11-28T12:59:20.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.213+0000 7f55557fa700 1 -- 172.21.15.111:0/3679740943 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19796 (secure 0 0 0) 0x7f555815cd90 con 0x7f55580b15b0 2023-11-28T12:59:20.215 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:20.215 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:00.053943+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:20.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.215+0000 7f555ee4f700 1 -- 172.21.15.111:0/3679740943 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f553c0682f0 msgr2=0x7f553c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.216+0000 7f555ee4f700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f553c0682f0 0x7f553c06a7a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f55580ae990 tx=0x7f554c008040 comp rx=0 tx=0).stop 2023-11-28T12:59:20.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.216+0000 7f555ee4f700 1 -- 172.21.15.111:0/3679740943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55580b15b0 msgr2=0x7f55580c9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.216+0000 7f555ee4f700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55580b15b0 0x7f55580c9400 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f555000ca20 tx=0x7f555000cde0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.216+0000 7f555ee4f700 1 -- 172.21.15.111:0/3679740943 shutdown_connections 2023-11-28T12:59:20.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.216+0000 7f555ee4f700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55580b15b0 0x7f55580c9400 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.216+0000 7f555ee4f700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f553c0682f0 0x7f553c06a7a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.216+0000 7f555ee4f700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f55580b0d70 0x7f555814bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.216+0000 7f555ee4f700 1 --2- 172.21.15.111:0/3679740943 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f55580b1140 0x7f55580c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.216+0000 7f555ee4f700 1 -- 172.21.15.111:0/3679740943 >> 172.21.15.111:0/3679740943 conn(0x7f5558098ee0 msgr2=0x7f555806f200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.216+0000 7f555ee4f700 1 -- 172.21.15.111:0/3679740943 shutdown_connections 2023-11-28T12:59:20.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.216+0000 7f555ee4f700 1 -- 172.21.15.111:0/3679740943 wait complete. 2023-11-28T12:59:20.230 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-28T12:59:20.230 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-28T12:59:20.231 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-28T12:59:20.231 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-28T12:59:20.231 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-28T12:59:20.232 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-28T12:59:20.232 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-11-28T12:59:20.233 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-11-28T12:59:20.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.442+0000 7fce5f0c1700 1 Processor -- start 2023-11-28T12:59:20.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.442+0000 7fce5f0c1700 1 -- start start 2023-11-28T12:59:20.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.443+0000 7fa24c298700 1 Processor -- start 2023-11-28T12:59:20.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.443+0000 7fce5f0c1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce580c9ed0 0x7fce5815bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.443+0000 7fce5f0c1700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fce580cdee0 0x7fce5806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.443+0000 7fce5f0c1700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce580cce40 con 0x7fce580cbf70 2023-11-28T12:59:20.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.443+0000 7fce5f0c1700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce580cd120 con 0x7fce580cdee0 2023-11-28T12:59:20.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.443+0000 7fce5f0c1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce580cd400 con 0x7fce580c9ed0 2023-11-28T12:59:20.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.442+0000 7fd35347c700 1 Processor -- start 2023-11-28T12:59:20.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.443+0000 7fd35347c700 1 -- start start 2023-11-28T12:59:20.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fce5d65e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fce580cdee0 0x7fce5806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fce5d65e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fce580cdee0 0x7fce5806a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:51102/0 (socket says 172.21.15.111:51102) 2023-11-28T12:59:20.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fce5d65e700 1 -- 172.21.15.111:0/882758050 learned_addr learned my addr 172.21.15.111:0/882758050 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fce57fff700 1 --2- 172.21.15.111:0/882758050 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce580c9ed0 0x7fce5815bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fd35347c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd34c0d1d70 0x7fd34c167270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fd35347c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c06c790 con 0x7fd34c0cd460 2023-11-28T12:59:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fd35347c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c0b88e0 con 0x7fd34c0b8510 2023-11-28T12:59:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fd35347c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34c0cdbd0 con 0x7fd34c0d1d70 2023-11-28T12:59:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fce5d65e700 1 -- 172.21.15.111:0/882758050 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce580c9ed0 msgr2=0x7fce5815bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fce5d65e700 1 --2- 172.21.15.111:0/882758050 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce580c9ed0 0x7fce5815bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fce5d65e700 1 -- 172.21.15.111:0/882758050 >> v1:172.21.15.149:6789/0 conn(0x7fce580cbf70 legacy=0x7fce5815ce70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T12:59:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fce5d65e700 1 -- 172.21.15.111:0/882758050 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce580cd6e0 con 0x7fce580cdee0 2023-11-28T12:59:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fd351a19700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd34c0d1d70 0x7fd34c167270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.443+0000 7fa24c298700 1 -- start start 2023-11-28T12:59:20.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fa24c298700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa2440d2df0 0x7fa2440cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fa24c298700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2440d1950 con 0x7fa2440b4fc0 2023-11-28T12:59:20.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fa24c298700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2440b2500 con 0x7fa2440b1c60 2023-11-28T12:59:20.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fa24c298700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2440d3d30 con 0x7fa2440d2df0 2023-11-28T12:59:20.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fce5d65e700 1 --2- 172.21.15.111:0/882758050 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce580cdee0 0x7fce5806a270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fce50009e50 tx=0x7fce5000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5461c1f7ea8556d7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fd351a19700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd34c0d1d70 0x7fd34c167270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33204/0 (socket says 172.21.15.111:33204) 2023-11-28T12:59:20.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.444+0000 7fd351a19700 1 -- 172.21.15.111:0/1506650625 learned_addr learned my addr 172.21.15.111:0/1506650625 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.445+0000 7fa24a835700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7fa2440b1c60 0x7fa2440cf010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:42530/0 (socket says 172.21.15.111:42530) 2023-11-28T12:59:20.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.445+0000 7fa24a835700 1 -- 172.21.15.111:0/3304963790 learned_addr learned my addr 172.21.15.111:0/3304963790 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:20.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.445+0000 7fce577fe700 1 -- 172.21.15.111:0/882758050 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5000e070 con 0x7fce580cdee0 2023-11-28T12:59:20.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.445+0000 7fce577fe700 1 -- 172.21.15.111:0/882758050 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce50003a20 con 0x7fce580cdee0 2023-11-28T12:59:20.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.445+0000 7fce577fe700 1 -- 172.21.15.111:0/882758050 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50003dd0 con 0x7fce580cdee0 2023-11-28T12:59:20.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.445+0000 7fd351a19700 1 -- 172.21.15.111:0/1506650625 >> v1:172.21.15.149:6789/0 conn(0x7fd34c0b8510 legacy=0x7fd34c06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.445+0000 7fd351a19700 1 -- 172.21.15.111:0/1506650625 >> v1:172.21.15.111:6789/0 conn(0x7fd34c0cd460 legacy=0x7fd34c06c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.445+0000 7fd351a19700 1 -- 172.21.15.111:0/1506650625 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c0cd8f0 con 0x7fd34c0d1d70 2023-11-28T12:59:20.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.445+0000 7fd351a19700 1 --2- 172.21.15.111:0/1506650625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd34c0d1d70 0x7fd34c167270 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd344009e50 tx=0x7fd34400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=818379ffb77152bf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196742368 0 0) 0x7fa2440b2500 con 0x7fa2440b1c60 2023-11-28T12:59:20.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2300035b0 con 0x7fa2440b1c60 2023-11-28T12:59:20.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fce5f0c1700 1 -- 172.21.15.111:0/882758050 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce580cdee0 msgr2=0x7fce5806a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fce5f0c1700 1 --2- 172.21.15.111:0/882758050 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce580cdee0 0x7fce5806a270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fce50009e50 tx=0x7fce5000b330 comp rx=0 tx=0).stop 2023-11-28T12:59:20.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7f779c32b700 1 Processor -- start 2023-11-28T12:59:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 <== mon.1 v1:172.21.15.149:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3990544763 0 0) 0x7fa2300035b0 con 0x7fa2440b1c60 2023-11-28T12:59:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa2440d2df0 msgr2=0x7fa2440cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 --2- 172.21.15.111:0/3304963790 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa2440d2df0 0x7fa2440cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 >> v1:172.21.15.111:6789/0 conn(0x7fa2440b4fc0 legacy=0x7fa2440d31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 --> v1:172.21.15.149:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2440d3a50 con 0x7fa2440b1c60 2023-11-28T12:59:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 <== mon.1 v1:172.21.15.149:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fa23c0019f0 con 0x7fa2440b1c60 2023-11-28T12:59:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 >> v1:172.21.15.149:6789/0 conn(0x7fa2440b1c60 legacy=0x7fa2440cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 --2- 172.21.15.111:0/3304963790 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa230003a00 0x7fa230003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 --2- 172.21.15.111:0/3304963790 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa230004850 0x7fa230004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 --2- 172.21.15.111:0/3304963790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa230005250 0x7fa23000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2440d3d30 con 0x7fa230005250 2023-11-28T12:59:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23000c140 con 0x7fa230003a00 2023-11-28T12:59:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23000c300 con 0x7fa230004850 2023-11-28T12:59:20.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.447+0000 7f5290ed5700 1 Processor -- start 2023-11-28T12:59:20.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.447+0000 7fa24a835700 1 --2- 172.21.15.111:0/3304963790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa230005250 0x7fa23000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.447+0000 7f5290ed5700 1 -- start start 2023-11-28T12:59:20.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.447+0000 7f2d1f6e1700 1 Processor -- start 2023-11-28T12:59:20.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.447+0000 7fa24a835700 1 -- 172.21.15.111:0/3304963790 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa230004850 msgr2=0x7fa230004cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T12:59:20.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.447+0000 7fa24a835700 1 --2- 172.21.15.111:0/3304963790 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa230004850 0x7fa230004cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.447+0000 7fa24a835700 1 -- 172.21.15.111:0/3304963790 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa230003a00 msgr2=0x7fa230003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.447+0000 7fa24a835700 1 --2- 172.21.15.111:0/3304963790 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa230003a00 0x7fa230003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.447+0000 7fa24a835700 1 -- 172.21.15.111:0/3304963790 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2440d3a50 con 0x7fa230005250 2023-11-28T12:59:20.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fce5f0c1700 1 -- 172.21.15.111:0/882758050 shutdown_connections 2023-11-28T12:59:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fce5f0c1700 1 --2- 172.21.15.111:0/882758050 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce580cdee0 0x7fce5806a270 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fce5f0c1700 1 --2- 172.21.15.111:0/882758050 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce580c9ed0 0x7fce5815bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7fce5f0c1700 1 -- 172.21.15.111:0/882758050 >> 172.21.15.111:0/882758050 conn(0x7fce580cb490 msgr2=0x7fce580cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.447+0000 7f2d1f6e1700 1 -- start start 2023-11-28T12:59:20.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.447+0000 7fa24a835700 1 --2- 172.21.15.111:0/3304963790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa230005250 0x7fa23000ba10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa23c002230 tx=0x7fa230005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a8df8f593ada0d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.446+0000 7f779c32b700 1 -- start start 2023-11-28T12:59:20.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f779c32b700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f77941607e0 0x7f779406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f779c32b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f779406b870 0x7f779406a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f779c32b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7794165390 con 0x7f779406acf0 2023-11-28T12:59:20.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f779c32b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77940c2910 con 0x7f779406b870 2023-11-28T12:59:20.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f779c32b700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77941645f0 con 0x7f77941607e0 2023-11-28T12:59:20.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7fce5f0c1700 1 -- 172.21.15.111:0/882758050 shutdown_connections 2023-11-28T12:59:20.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f77998c6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f779406b870 0x7f779406a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f779a8c8700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f779406acf0 0x7f77940712e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34000/0 (socket says 172.21.15.111:34000) 2023-11-28T12:59:20.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f779a8c8700 1 -- 172.21.15.111:0/3998801570 learned_addr learned my addr 172.21.15.111:0/3998801570 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:20.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f779a0c7700 1 --2- 172.21.15.111:0/3998801570 >> v2:172.21.15.149:3300/0 conn(0x7f77941607e0 0x7f779406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f5290ed5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f528c0c9300 0x7f528c0ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c012600 con 0x7fa230005250 2023-11-28T12:59:20.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c012760 con 0x7fa230005250 2023-11-28T12:59:20.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7fa249032700 1 -- 172.21.15.111:0/3304963790 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c012ad0 con 0x7fa230005250 2023-11-28T12:59:20.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f5290ed5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f528c06a270 0x7f528c070910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f5290ed5700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c06d870 con 0x7f528c06dd00 2023-11-28T12:59:20.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f5290ed5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528c06e150 con 0x7f528c0c9300 2023-11-28T12:59:20.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f5290ed5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528c06cc60 con 0x7f528c06a270 2023-11-28T12:59:20.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7fd343fff700 1 -- 172.21.15.111:0/1506650625 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34400e070 con 0x7fd34c0d1d70 2023-11-28T12:59:20.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7fd343fff700 1 -- 172.21.15.111:0/1506650625 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd344003a20 con 0x7fd34c0d1d70 2023-11-28T12:59:20.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7fd343fff700 1 -- 172.21.15.111:0/1506650625 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd344003db0 con 0x7fd34c0d1d70 2023-11-28T12:59:20.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f528ad9d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f528c06a270 0x7f528c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f528ad9d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f528c06a270 0x7f528c070910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:54976/0 (socket says 172.21.15.111:54976) 2023-11-28T12:59:20.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f528ad9d700 1 -- 172.21.15.111:0/1937427753 learned_addr learned my addr 172.21.15.111:0/1937427753 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f77998c6700 1 -- 172.21.15.111:0/3998801570 >> v2:172.21.15.149:3300/0 conn(0x7f77941607e0 msgr2=0x7f779406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f77998c6700 1 --2- 172.21.15.111:0/3998801570 >> v2:172.21.15.149:3300/0 conn(0x7f77941607e0 0x7f779406a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f77998c6700 1 -- 172.21.15.111:0/3998801570 >> v1:172.21.15.111:6789/0 conn(0x7f779406acf0 legacy=0x7f77940712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f77998c6700 1 -- 172.21.15.111:0/3998801570 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77941648d0 con 0x7f779406b870 2023-11-28T12:59:20.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f528ad9d700 1 -- 172.21.15.111:0/1937427753 >> v2:172.21.15.111:3300/0 conn(0x7f528c0c9300 msgr2=0x7f528c0ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f528ad9d700 1 --2- 172.21.15.111:0/1937427753 >> v2:172.21.15.111:3300/0 conn(0x7f528c0c9300 0x7f528c0ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f528ad9d700 1 -- 172.21.15.111:0/1937427753 >> v1:172.21.15.149:6789/0 conn(0x7f528c06dd00 legacy=0x7f528c158cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T12:59:20.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f528ad9d700 1 -- 172.21.15.111:0/1937427753 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c06cf40 con 0x7f528c06a270 2023-11-28T12:59:20.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f2d1f6e1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2d180cb240 0x7f2d181590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f2d1f6e1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d1815d270 0x7f2d180cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f2d1f6e1700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d180cd540 con 0x7f2d180cdf10 2023-11-28T12:59:20.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f2d1f6e1700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d18159a20 con 0x7f2d180cb240 2023-11-28T12:59:20.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7f2d1f6e1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d18159d00 con 0x7f2d1815d270 2023-11-28T12:59:20.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f2d1cc7c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d1815d270 0x7f2d180cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f2d1cc7c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d1815d270 0x7f2d180cd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:54980/0 (socket says 172.21.15.111:54980) 2023-11-28T12:59:20.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f2d1cc7c700 1 -- 172.21.15.111:0/2034413891 learned_addr learned my addr 172.21.15.111:0/2034413891 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f77998c6700 1 --2- 172.21.15.111:0/3998801570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f779406b870 0x7f779406a7b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f77940cf220 tx=0x7f7784009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6d83b5f813f51 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7fd35347c700 1 -- 172.21.15.111:0/1506650625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd34c0d1d70 msgr2=0x7fd34c167270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7fd35347c700 1 --2- 172.21.15.111:0/1506650625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd34c0d1d70 0x7fd34c167270 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd344009e50 tx=0x7fd34400b330 comp rx=0 tx=0).stop 2023-11-28T12:59:20.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7fd35347c700 1 -- 172.21.15.111:0/1506650625 shutdown_connections 2023-11-28T12:59:20.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7fd35347c700 1 --2- 172.21.15.111:0/1506650625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd34c0d1d70 0x7fd34c167270 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7fd35347c700 1 -- 172.21.15.111:0/1506650625 >> 172.21.15.111:0/1506650625 conn(0x7fd34c0719a0 msgr2=0x7fd34c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f528ad9d700 1 --2- 172.21.15.111:0/1937427753 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f528c06a270 0x7f528c070910 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f528400d150 tx=0x7f528400d460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=125a7d699c559179 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7fa24c298700 1 -- 172.21.15.111:0/3304963790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa230005250 msgr2=0x7fa23000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7fa24c298700 1 --2- 172.21.15.111:0/3304963790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa230005250 0x7fa23000ba10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa23c002230 tx=0x7fa230005880 comp rx=0 tx=0).stop 2023-11-28T12:59:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7fa24a835700 1 -- 172.21.15.111:0/3304963790 reap_dead start 2023-11-28T12:59:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7fa24c298700 1 -- 172.21.15.111:0/3304963790 shutdown_connections 2023-11-28T12:59:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7fa24c298700 1 -- 172.21.15.111:0/3304963790 >> 172.21.15.111:0/3304963790 conn(0x7fa2440719a0 msgr2=0x7fa244071da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7fa24c298700 1 -- 172.21.15.111:0/3304963790 shutdown_connections 2023-11-28T12:59:20.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7fd35347c700 1 -- 172.21.15.111:0/1506650625 shutdown_connections 2023-11-28T12:59:20.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f8a45bdc700 1 Processor -- start 2023-11-28T12:59:20.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.448+0000 7fce5f0c1700 1 -- 172.21.15.111:0/882758050 wait complete. 2023-11-28T12:59:20.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7fce5f0c1700 1 Processor -- start 2023-11-28T12:59:20.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7f77990c5700 1 -- 172.21.15.111:0/3998801570 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784011830 con 0x7f779406b870 2023-11-28T12:59:20.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7f77990c5700 1 -- 172.21.15.111:0/3998801570 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7784011990 con 0x7f779406b870 2023-11-28T12:59:20.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7f77990c5700 1 -- 172.21.15.111:0/3998801570 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784011d20 con 0x7f779406b870 2023-11-28T12:59:20.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7f528959a700 1 -- 172.21.15.111:0/1937427753 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284010070 con 0x7f528c06a270 2023-11-28T12:59:20.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7fa24c298700 1 -- 172.21.15.111:0/3304963790 wait complete. 2023-11-28T12:59:20.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7f528959a700 1 -- 172.21.15.111:0/1937427753 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528400dd80 con 0x7f528c06a270 2023-11-28T12:59:20.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7f528959a700 1 -- 172.21.15.111:0/1937427753 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528400b310 con 0x7f528c06a270 2023-11-28T12:59:20.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7fa24c298700 1 Processor -- start 2023-11-28T12:59:20.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7f8a45bdc700 1 -- start start 2023-11-28T12:59:20.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7f8a45bdc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a400cdee0 0x7f8a4006c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7f8a45bdc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a400ccb60 con 0x7f8a400c9ed0 2023-11-28T12:59:20.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7f8a45bdc700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a400cce40 con 0x7f8a4006a270 2023-11-28T12:59:20.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7f8a45bdc700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a400cd120 con 0x7f8a400cdee0 2023-11-28T12:59:20.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7f5290ed5700 1 -- 172.21.15.111:0/1937427753 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f528c06a270 msgr2=0x7f528c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7f5290ed5700 1 --2- 172.21.15.111:0/1937427753 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f528c06a270 0x7f528c070910 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f528400d150 tx=0x7f528400d460 comp rx=0 tx=0).stop 2023-11-28T12:59:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7f779c32b700 1 -- 172.21.15.111:0/3998801570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f779406b870 msgr2=0x7f779406a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7f779c32b700 1 --2- 172.21.15.111:0/3998801570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f779406b870 0x7f779406a7b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f77940cf220 tx=0x7f7784009b20 comp rx=0 tx=0).stop 2023-11-28T12:59:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7fd35347c700 1 -- 172.21.15.111:0/1506650625 wait complete. 2023-11-28T12:59:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7fd35347c700 1 Processor -- start 2023-11-28T12:59:20.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7fa24c298700 1 -- start start 2023-11-28T12:59:20.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f2d1d47d700 1 --2- 172.21.15.111:0/2034413891 >> v2:172.21.15.111:3300/0 conn(0x7f2d180cb240 0x7f2d181590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f2d1d47d700 1 -- 172.21.15.111:0/2034413891 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d1815d270 msgr2=0x7f2d180cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f2d1d47d700 1 --2- 172.21.15.111:0/2034413891 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d1815d270 0x7f2d180cd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f2d1d47d700 1 -- 172.21.15.111:0/2034413891 >> v1:172.21.15.149:6789/0 conn(0x7f2d180cdf10 legacy=0x7f2d1806e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.449+0000 7f2d1d47d700 1 -- 172.21.15.111:0/2034413891 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d18159fe0 con 0x7f2d180cb240 2023-11-28T12:59:20.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7f2d1d47d700 1 --2- 172.21.15.111:0/2034413891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d180cb240 0x7f2d181590a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2d14009840 tx=0x7f2d14009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253ef7c1fcff1e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7f2d0ffff700 1 -- 172.21.15.111:0/2034413891 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14004230 con 0x7f2d180cb240 2023-11-28T12:59:20.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f2d0ffff700 1 -- 172.21.15.111:0/2034413891 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d14004390 con 0x7f2d180cb240 2023-11-28T12:59:20.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f2d0ffff700 1 -- 172.21.15.111:0/2034413891 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d140036c0 con 0x7f2d180cb240 2023-11-28T12:59:20.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f8a3f7fe700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f8a4006a270 0x7f8a4015ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:42562/0 (socket says 172.21.15.111:42562) 2023-11-28T12:59:20.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f8a3f7fe700 1 -- 172.21.15.111:0/3344416304 learned_addr learned my addr 172.21.15.111:0/3344416304 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:20.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f8a3ffff700 1 --2- 172.21.15.111:0/3344416304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a400cdee0 0x7f8a4006c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.450+0000 7fce5f0c1700 1 -- start start 2023-11-28T12:59:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7fce5f0c1700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce580c9ed0 0x7fce5815b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7fce5f0c1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce580cbf70 0x7fce58147bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7fce5f0c1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce580cdee0 0x7fce5813cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7fce5f0c1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5815ba10 con 0x7fce580cbf70 2023-11-28T12:59:20.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.451+0000 7fce5f0c1700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce581480f0 con 0x7fce580c9ed0 2023-11-28T12:59:20.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7fce5f0c1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce581493a0 con 0x7fce580cdee0 2023-11-28T12:59:20.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7fce5d65e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce580cdee0 0x7fce5813cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7fce5d65e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce580cdee0 0x7fce5813cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:54994/0 (socket says 172.21.15.111:54994) 2023-11-28T12:59:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7fce5d65e700 1 -- 172.21.15.111:0/74955623 learned_addr learned my addr 172.21.15.111:0/74955623 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7fce57fff700 1 --2- 172.21.15.111:0/74955623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce580cbf70 0x7fce58147bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436874713 0 0) 0x7f8a400cce40 con 0x7f8a4006a270 2023-11-28T12:59:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a280035b0 con 0x7f8a4006a270 2023-11-28T12:59:20.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7fce5d65e700 1 -- 172.21.15.111:0/74955623 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce580c9ed0 msgr2=0x7fce5815b4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T12:59:20.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7fce5d65e700 1 --2- 172.21.15.111:0/74955623 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce580c9ed0 0x7fce5815b4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7fce5d65e700 1 -- 172.21.15.111:0/74955623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce580cbf70 msgr2=0x7fce58147bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7fce5d65e700 1 --2- 172.21.15.111:0/74955623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce580cbf70 0x7fce58147bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7fce5d65e700 1 -- 172.21.15.111:0/74955623 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce50009b30 con 0x7fce580cdee0 2023-11-28T12:59:20.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fce5d65e700 1 --2- 172.21.15.111:0/74955623 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce580cdee0 0x7fce5813cdf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fce5000b680 tx=0x7fce50004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2034413891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d180cb240 msgr2=0x7f2d181590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f2d1f6e1700 1 --2- 172.21.15.111:0/2034413891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d180cb240 0x7f2d181590a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2d14009840 tx=0x7f2d14009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:20.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f5290ed5700 1 -- 172.21.15.111:0/1937427753 shutdown_connections 2023-11-28T12:59:20.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f5290ed5700 1 --2- 172.21.15.111:0/1937427753 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f528c06a270 0x7f528c070910 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f5290ed5700 1 --2- 172.21.15.111:0/1937427753 >> v2:172.21.15.111:3300/0 conn(0x7f528c0c9300 0x7f528c0ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f5290ed5700 1 -- 172.21.15.111:0/1937427753 >> 172.21.15.111:0/1937427753 conn(0x7f528c0cb490 msgr2=0x7f528c0cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f5290ed5700 1 -- 172.21.15.111:0/1937427753 shutdown_connections 2023-11-28T12:59:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7f5290ed5700 1 -- 172.21.15.111:0/1937427753 wait complete. 2023-11-28T12:59:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24c298700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa230005250 0x7fa244159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24c298700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2441598b0 0x7fa24415e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24c298700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa24415bc80 0x7fa2441e5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24c298700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2440cadc0 con 0x7fa2441598b0 2023-11-28T12:59:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24c298700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24415d180 con 0x7fa24415bc80 2023-11-28T12:59:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24c298700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24415d3d0 con 0x7fa230005250 2023-11-28T12:59:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24a034700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa230005250 0x7fa244159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24a034700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa230005250 0x7fa244159240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:54998/0 (socket says 172.21.15.111:54998) 2023-11-28T12:59:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24a034700 1 -- 172.21.15.111:0/159363889 learned_addr learned my addr 172.21.15.111:0/159363889 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 <== mon.1 v1:172.21.15.149:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2563815735 0 0) 0x7f8a280035b0 con 0x7f8a4006a270 2023-11-28T12:59:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a400cdee0 msgr2=0x7f8a4006c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7f8a3e7fc700 1 --2- 172.21.15.111:0/3344416304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a400cdee0 0x7f8a4006c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 >> v1:172.21.15.111:6789/0 conn(0x7f8a400c9ed0 legacy=0x7f8a4006a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 --> v1:172.21.15.149:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a400cd400 con 0x7f8a4006a270 2023-11-28T12:59:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 <== mon.1 v1:172.21.15.149:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f8a38001250 con 0x7f8a4006a270 2023-11-28T12:59:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 >> v1:172.21.15.149:6789/0 conn(0x7f8a4006a270 legacy=0x7f8a4015ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7f8a3e7fc700 1 --2- 172.21.15.111:0/3344416304 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a28003a00 0x7f8a28003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7f8a3e7fc700 1 --2- 172.21.15.111:0/3344416304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a28004430 0x7f8a280048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7f8a3e7fc700 1 --2- 172.21.15.111:0/3344416304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a280050a0 0x7f8a2800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a400cd120 con 0x7f8a280050a0 2023-11-28T12:59:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a28014810 con 0x7f8a28003a00 2023-11-28T12:59:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a28014980 con 0x7f8a28004430 2023-11-28T12:59:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7f8a3effd700 1 --2- 172.21.15.111:0/3344416304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a28004430 0x7f8a280048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7f8a3f7fe700 1 --2- 172.21.15.111:0/3344416304 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a28003a00 0x7f8a28003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7f8a3effd700 1 -- 172.21.15.111:0/3344416304 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a28003a00 msgr2=0x7f8a28003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7f8a3effd700 1 --2- 172.21.15.111:0/3344416304 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a28003a00 0x7f8a28003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7f8a3effd700 1 -- 172.21.15.111:0/3344416304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a280050a0 msgr2=0x7f8a2800ba10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T12:59:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7f8a3effd700 1 --2- 172.21.15.111:0/3344416304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a280050a0 0x7f8a2800ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7f8a3effd700 1 -- 172.21.15.111:0/3344416304 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a400cd400 con 0x7f8a28004430 2023-11-28T12:59:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fce557fa700 1 -- 172.21.15.111:0/74955623 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50004440 con 0x7fce580cdee0 2023-11-28T12:59:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fce5f0c1700 1 -- 172.21.15.111:0/74955623 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce580ce360 con 0x7fce580cdee0 2023-11-28T12:59:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fce557fa700 1 -- 172.21.15.111:0/74955623 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce500045a0 con 0x7fce580cdee0 2023-11-28T12:59:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fce557fa700 1 -- 172.21.15.111:0/74955623 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50004950 con 0x7fce580cdee0 2023-11-28T12:59:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fce5f0c1700 1 -- 172.21.15.111:0/74955623 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce580cd9c0 con 0x7fce580cdee0 2023-11-28T12:59:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.452+0000 7fd35347c700 1 -- start start 2023-11-28T12:59:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fd35347c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd34c0b8510 0x7fd34c06e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fd35347c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd34c0cd460 0x7fd34c1f8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fd35347c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd34c0d1d70 0x7fd34c1fe790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fd35347c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c1ff730 con 0x7fd34c0b8510 2023-11-28T12:59:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fd35347c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c1ff170 con 0x7fd34c0cd460 2023-11-28T12:59:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fd35347c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c1ff450 con 0x7fd34c0d1d70 2023-11-28T12:59:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fd351218700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd34c0b8510 0x7fd34c06e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fd351218700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd34c0b8510 0x7fd34c06e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33252/0 (socket says 172.21.15.111:33252) 2023-11-28T12:59:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fd351218700 1 -- 172.21.15.111:0/2751585426 learned_addr learned my addr 172.21.15.111:0/2751585426 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fd350a17700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd34c0cd460 0x7fd34c1f8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fd351a19700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd34c0d1d70 0x7fd34c1fe790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7fd351a19700 1 -- 172.21.15.111:0/2751585426 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd34c0cd460 msgr2=0x7fd34c1f8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7fd351a19700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd34c0cd460 0x7fd34c1f8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7fd351a19700 1 -- 172.21.15.111:0/2751585426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd34c0b8510 msgr2=0x7fd34c06e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7fd351a19700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd34c0b8510 0x7fd34c06e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7fd351a19700 1 -- 172.21.15.111:0/2751585426 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd344009b30 con 0x7fd34c0d1d70 2023-11-28T12:59:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7fd351a19700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd34c0d1d70 0x7fd34c1fe790 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd344013000 tx=0x7fd344004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24a835700 1 --2- 172.21.15.111:0/159363889 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa24415bc80 0x7fa2441e5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24a034700 1 -- 172.21.15.111:0/159363889 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa24415bc80 msgr2=0x7fa2441e5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24a034700 1 --2- 172.21.15.111:0/159363889 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa24415bc80 0x7fa2441e5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24a034700 1 -- 172.21.15.111:0/159363889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2441598b0 msgr2=0x7fa24415e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24a034700 1 --2- 172.21.15.111:0/159363889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2441598b0 0x7fa24415e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.453+0000 7fa24a034700 1 -- 172.21.15.111:0/159363889 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2440b88f0 con 0x7fa230005250 2023-11-28T12:59:20.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7efd99713700 1 Processor -- start 2023-11-28T12:59:20.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7efd99713700 1 -- start start 2023-11-28T12:59:20.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7fd341ffb700 1 -- 172.21.15.111:0/2751585426 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd344004230 con 0x7fd34c0d1d70 2023-11-28T12:59:20.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7fd341ffb700 1 -- 172.21.15.111:0/2751585426 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd344004390 con 0x7fd34c0d1d70 2023-11-28T12:59:20.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7fd341ffb700 1 -- 172.21.15.111:0/2751585426 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd344004740 con 0x7fd34c0d1d70 2023-11-28T12:59:20.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7fa24a034700 1 --2- 172.21.15.111:0/159363889 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa230005250 0x7fa244159240 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa23400b3f0 tx=0x7fa23400b700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7f8a3effd700 1 --2- 172.21.15.111:0/3344416304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a28004430 0x7f8a280048f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8a300017c0 tx=0x7f8a28005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b492c5c94dc3097 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7f5290ed5700 1 Processor -- start 2023-11-28T12:59:20.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7f779c32b700 1 -- 172.21.15.111:0/3998801570 shutdown_connections 2023-11-28T12:59:20.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7f779c32b700 1 --2- 172.21.15.111:0/3998801570 >> v2:172.21.15.149:3300/0 conn(0x7f77941607e0 0x7f779406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2034413891 shutdown_connections 2023-11-28T12:59:20.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7f2d1f6e1700 1 --2- 172.21.15.111:0/2034413891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d180cb240 0x7f2d181590a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7f2d1f6e1700 1 --2- 172.21.15.111:0/2034413891 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d1815d270 0x7f2d180cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7f779c32b700 1 --2- 172.21.15.111:0/3998801570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f779406b870 0x7f779406a7b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7f779c32b700 1 -- 172.21.15.111:0/3998801570 >> 172.21.15.111:0/3998801570 conn(0x7f77940b0db0 msgr2=0x7f77940b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2034413891 >> 172.21.15.111:0/2034413891 conn(0x7f2d180b0db0 msgr2=0x7f2d180b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a300052d0 con 0x7f8a28004430 2023-11-28T12:59:20.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2034413891 shutdown_connections 2023-11-28T12:59:20.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a30005430 con 0x7f8a28004430 2023-11-28T12:59:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7f8a3e7fc700 1 -- 172.21.15.111:0/3344416304 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a300038c0 con 0x7f8a28004430 2023-11-28T12:59:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7fce5f0c1700 1 -- 172.21.15.111:0/74955623 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fce580ce4f0 con 0x7fce580cdee0 2023-11-28T12:59:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.454+0000 7fd35347c700 1 -- 172.21.15.111:0/2751585426 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34c0cfff0 con 0x7fd34c0d1d70 2023-11-28T12:59:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7fd35347c700 1 -- 172.21.15.111:0/2751585426 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34c0cfd10 con 0x7fd34c0d1d70 2023-11-28T12:59:20.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7fa23affd700 1 -- 172.21.15.111:0/159363889 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa234011830 con 0x7fa230005250 2023-11-28T12:59:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7fa23affd700 1 -- 172.21.15.111:0/159363889 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa234011990 con 0x7fa230005250 2023-11-28T12:59:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7fa23affd700 1 -- 172.21.15.111:0/159363889 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa234011d40 con 0x7fa230005250 2023-11-28T12:59:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7fa24c298700 1 -- 172.21.15.111:0/159363889 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2440d15e0 con 0x7fa230005250 2023-11-28T12:59:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7fa24c298700 1 -- 172.21.15.111:0/159363889 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2440b21b0 con 0x7fa230005250 2023-11-28T12:59:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7efd99713700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd94142e80 0x7efd9406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7efd99713700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7efd9406a7b0 0x7efd940c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7efd99713700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efd940ca430 0x7efd9406d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7efd99713700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd940c9380 con 0x7efd940ca430 2023-11-28T12:59:20.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7efd99713700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd9406ac00 con 0x7efd9406a7b0 2023-11-28T12:59:20.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7efd99713700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd94159b10 con 0x7efd94142e80 2023-11-28T12:59:20.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7efd927fc700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7efd9406a7b0 0x7efd940c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7efd927fc700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7efd9406a7b0 0x7efd940c9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:51168/0 (socket says 172.21.15.111:51168) 2023-11-28T12:59:20.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7efd927fc700 1 -- 172.21.15.111:0/4082627357 learned_addr learned my addr 172.21.15.111:0/4082627357 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7efd92ffd700 1 --2- 172.21.15.111:0/4082627357 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd94142e80 0x7efd9406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3344416304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a28004430 msgr2=0x7f8a280048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7f8a45bdc700 1 --2- 172.21.15.111:0/3344416304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a28004430 0x7f8a280048f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8a300017c0 tx=0x7f8a28005880 comp rx=0 tx=0).stop 2023-11-28T12:59:20.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7f8a3ffff700 1 -- 172.21.15.111:0/3344416304 reap_dead start 2023-11-28T12:59:20.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7efd937fe700 1 --2- 172.21.15.111:0/4082627357 >> v2:172.21.15.111:3300/0 conn(0x7efd940ca430 0x7efd9406d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7efd937fe700 1 -- 172.21.15.111:0/4082627357 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd94142e80 msgr2=0x7efd9406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7efd937fe700 1 --2- 172.21.15.111:0/4082627357 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd94142e80 0x7efd9406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7efd937fe700 1 -- 172.21.15.111:0/4082627357 >> v2:172.21.15.149:3300/0 conn(0x7efd9406a7b0 msgr2=0x7efd940c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7efd937fe700 1 --2- 172.21.15.111:0/4082627357 >> v2:172.21.15.149:3300/0 conn(0x7efd9406a7b0 0x7efd940c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7efd937fe700 1 -- 172.21.15.111:0/4082627357 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd940ca890 con 0x7efd940ca430 2023-11-28T12:59:20.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2034413891 wait complete. 2023-11-28T12:59:20.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7f2d1f6e1700 1 Processor -- start 2023-11-28T12:59:20.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7fce557fa700 1 -- 172.21.15.111:0/74955623 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fce50024030 con 0x7fce580cdee0 2023-11-28T12:59:20.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7fce557fa700 1 --2- 172.21.15.111:0/74955623 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fce3c0682f0 0x7fce3c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7fce557fa700 1 -- 172.21.15.111:0/74955623 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fce500877a0 con 0x7fce580cdee0 2023-11-28T12:59:20.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7fce557fa700 1 --2- 172.21.15.111:0/74955623 >> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] conn(0x7fce3c071ec0 0x7fce3c0742f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7fce557fa700 1 -- 172.21.15.111:0/74955623 --> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce3c0749c0 con 0x7fce3c071ec0 2023-11-28T12:59:20.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7fce557fa700 1 -- 172.21.15.111:0/74955623 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce50053d10 con 0x7fce580cdee0 2023-11-28T12:59:20.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7fd341ffb700 1 -- 172.21.15.111:0/2751585426 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd34401b040 con 0x7fd34c0d1d70 2023-11-28T12:59:20.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7f2d1f6e1700 1 -- start start 2023-11-28T12:59:20.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.455+0000 7f5290ed5700 1 -- start start 2023-11-28T12:59:20.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7f5290ed5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f528c06a270 0x7f528c1f9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7f5290ed5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f528c06dd00 0x7f528c0b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7f5290ed5700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f528c0c9300 0x7f528c157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7f5290ed5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c0b3350 con 0x7f528c06dd00 2023-11-28T12:59:20.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7f5290ed5700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c0b22f0 con 0x7f528c0c9300 2023-11-28T12:59:20.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.456+0000 7f5290ed5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528c0b2590 con 0x7f528c06a270 2023-11-28T12:59:20.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7f528ad9d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f528c0c9300 0x7f528c157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7f528ad9d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f528c0c9300 0x7f528c157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:51172/0 (socket says 172.21.15.111:51172) 2023-11-28T12:59:20.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7f528ad9d700 1 -- 172.21.15.111:0/479974858 learned_addr learned my addr 172.21.15.111:0/479974858 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7f528a59c700 1 --2- 172.21.15.111:0/479974858 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f528c06a270 0x7f528c1f9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7fce5ce5d700 1 --2- 172.21.15.111:0/74955623 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fce3c0682f0 0x7fce3c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7fd341ffb700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd3300683c0 0x7fd33006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7f528ad9d700 1 -- 172.21.15.111:0/479974858 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f528c06a270 msgr2=0x7f528c1f9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7f528ad9d700 1 --2- 172.21.15.111:0/479974858 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f528c06a270 0x7f528c1f9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7f528ad9d700 1 -- 172.21.15.111:0/479974858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f528c06dd00 msgr2=0x7f528c0b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7f528ad9d700 1 --2- 172.21.15.111:0/479974858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f528c06dd00 0x7f528c0b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7f528ad9d700 1 -- 172.21.15.111:0/479974858 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528400bd40 con 0x7f528c0c9300 2023-11-28T12:59:20.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3344416304 shutdown_connections 2023-11-28T12:59:20.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3344416304 >> 172.21.15.111:0/3344416304 conn(0x7f8a400cb490 msgr2=0x7f8a400cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7fd341ffb700 1 -- 172.21.15.111:0/2751585426 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd344087690 con 0x7fd34c0d1d70 2023-11-28T12:59:20.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7fd351218700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd3300683c0 0x7fd33006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7fa23affd700 1 -- 172.21.15.111:0/159363889 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa234021070 con 0x7fa230005250 2023-11-28T12:59:20.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.457+0000 7efd937fe700 1 --2- 172.21.15.111:0/4082627357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd940ca430 0x7efd9406d2d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efd9406eee0 tx=0x7efd8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893e05b8e8ddaacc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f528ad9d700 1 --2- 172.21.15.111:0/479974858 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f528c0c9300 0x7f528c157ba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f528400d860 tx=0x7f5284004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3344416304 shutdown_connections 2023-11-28T12:59:20.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7fce5ce5d700 1 --2- 172.21.15.111:0/74955623 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fce3c0682f0 0x7fce3c06a7a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fce44009710 tx=0x7fce44006c60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7fce57fff700 1 --2- 172.21.15.111:0/74955623 >> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] conn(0x7fce3c071ec0 0x7fce3c0742f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7fd35347c700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] conn(0x7fd34c1fecd0 0x7fd34c0ab390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7fd35347c700 1 -- 172.21.15.111:0/2751585426 --> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd34c067880 con 0x7fd34c1fecd0 2023-11-28T12:59:20.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7fd350a17700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] conn(0x7fd34c1fecd0 0x7fd34c0ab390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7fd351218700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd3300683c0 0x7fd33006a870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd33c000c00 tx=0x7fd33c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f2d1f6e1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d180cb240 0x7f2d18151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f2d1f6e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d180cdf10 0x7f2d180afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f2d1f6e1700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2d1815d270 0x7f2d180b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f2d1f6e1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1815c3f0 con 0x7f2d180cdf10 2023-11-28T12:59:20.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f2d1f6e1700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1815c9b0 con 0x7f2d1815d270 2023-11-28T12:59:20.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f2d1f6e1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d181f4790 con 0x7f2d180cb240 2023-11-28T12:59:20.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f2d1cc7c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d180cdf10 0x7f2d180afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f2d1cc7c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d180cdf10 0x7f2d180afac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33286/0 (socket says 172.21.15.111:33286) 2023-11-28T12:59:20.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f2d1cc7c700 1 -- 172.21.15.111:0/2835301711 learned_addr learned my addr 172.21.15.111:0/2835301711 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7f2d1d47d700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d180cb240 0x7f2d18151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7f2d1dc7e700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2d1815d270 0x7f2d180b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f527affd700 1 -- 172.21.15.111:0/479974858 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284009a10 con 0x7f528c0c9300 2023-11-28T12:59:20.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f527affd700 1 -- 172.21.15.111:0/479974858 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5284009b70 con 0x7f528c0c9300 2023-11-28T12:59:20.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f527affd700 1 -- 172.21.15.111:0/479974858 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284004030 con 0x7f528c0c9300 2023-11-28T12:59:20.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f5290ed5700 1 -- 172.21.15.111:0/479974858 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528c06d500 con 0x7f528c0c9300 2023-11-28T12:59:20.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7f5290ed5700 1 -- 172.21.15.111:0/479974858 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528c06d220 con 0x7f528c0c9300 2023-11-28T12:59:20.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7fa24c298700 1 -- 172.21.15.111:0/159363889 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa244067170 con 0x7fa230005250 2023-11-28T12:59:20.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7efd91ffb700 1 -- 172.21.15.111:0/4082627357 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c00e070 con 0x7efd940ca430 2023-11-28T12:59:20.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7efd91ffb700 1 -- 172.21.15.111:0/4082627357 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8c00bb20 con 0x7efd940ca430 2023-11-28T12:59:20.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.458+0000 7efd91ffb700 1 -- 172.21.15.111:0/4082627357 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c012430 con 0x7efd940ca430 2023-11-28T12:59:20.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7efd99713700 1 -- 172.21.15.111:0/4082627357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd940ca430 msgr2=0x7efd9406d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7efd99713700 1 --2- 172.21.15.111:0/4082627357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd940ca430 0x7efd9406d2d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efd9406eee0 tx=0x7efd8c00b330 comp rx=0 tx=0).stop 2023-11-28T12:59:20.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7efd99713700 1 -- 172.21.15.111:0/4082627357 shutdown_connections 2023-11-28T12:59:20.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7efd99713700 1 --2- 172.21.15.111:0/4082627357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd940ca430 0x7efd9406d2d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7efd99713700 1 --2- 172.21.15.111:0/4082627357 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd94142e80 0x7efd9406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7efd99713700 1 --2- 172.21.15.111:0/4082627357 >> v2:172.21.15.149:3300/0 conn(0x7efd9406a7b0 0x7efd940c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7efd99713700 1 -- 172.21.15.111:0/4082627357 >> 172.21.15.111:0/4082627357 conn(0x7efd940cb490 msgr2=0x7efd940cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7f2d1d47d700 1 -- 172.21.15.111:0/2835301711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2d1815d270 msgr2=0x7f2d180b22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7f2d1d47d700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2d1815d270 0x7f2d180b22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7f2d1d47d700 1 -- 172.21.15.111:0/2835301711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d180cdf10 msgr2=0x7f2d180afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7f2d1d47d700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d180cdf10 0x7f2d180afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7f2d1d47d700 1 -- 172.21.15.111:0/2835301711 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d140094a0 con 0x7f2d180cb240 2023-11-28T12:59:20.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7efd99713700 1 -- 172.21.15.111:0/4082627357 shutdown_connections 2023-11-28T12:59:20.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7f2d1d47d700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d180cb240 0x7f2d18151100 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2d14013000 tx=0x7f2d14004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.460+0000 7f5290ed5700 1 -- 172.21.15.111:0/479974858 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5270000fc0 con 0x7f528c0c9300 2023-11-28T12:59:20.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3344416304 wait complete. 2023-11-28T12:59:20.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7fa23affd700 1 --2- 172.21.15.111:0/159363889 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa2280682f0 0x7fa22806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7fa23affd700 1 -- 172.21.15.111:0/159363889 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa234085d50 con 0x7fa230005250 2023-11-28T12:59:20.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7fa23affd700 1 --2- 172.21.15.111:0/159363889 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x7fa228071fc0 0x7fa2280743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7fa23affd700 1 -- 172.21.15.111:0/159363889 --> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa228074ac0 con 0x7fa228071fc0 2023-11-28T12:59:20.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7fa23affd700 1 -- 172.21.15.111:0/159363889 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa23408b2e0 con 0x7fa230005250 2023-11-28T12:59:20.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7fa24a835700 1 --2- 172.21.15.111:0/159363889 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x7fa228071fc0 0x7fa2280743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7fa249833700 1 --2- 172.21.15.111:0/159363889 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa2280682f0 0x7fa22806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.460+0000 7fa24a835700 1 --2- 172.21.15.111:0/159363889 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x7fa228071fc0 0x7fa2280743f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.460+0000 7fa249833700 1 --2- 172.21.15.111:0/159363889 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa2280682f0 0x7fa22806a7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa240007fd0 tx=0x7fa240007a20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7fd350a17700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] conn(0x7fd34c1fecd0 0x7fd34c0ab390 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.460+0000 7f8a45bdc700 1 Processor -- start 2023-11-28T12:59:20.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.461+0000 7f8a45bdc700 1 -- start start 2023-11-28T12:59:20.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a45bdc700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a28004430 0x7f8a401e5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a45bdc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a400afa30 0x7f8a400b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a45bdc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a400b1040 0x7f8a4015a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a45bdc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a401e56d0 con 0x7f8a400b1040 2023-11-28T12:59:20.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a45bdc700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a400affa0 con 0x7f8a28004430 2023-11-28T12:59:20.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a45bdc700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a400b0280 con 0x7f8a400afa30 2023-11-28T12:59:20.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a3f7fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a28004430 0x7f8a401e5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a3f7fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a28004430 0x7f8a401e5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:51200/0 (socket says 172.21.15.111:51200) 2023-11-28T12:59:20.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a3f7fe700 1 -- 172.21.15.111:0/3541211143 learned_addr learned my addr 172.21.15.111:0/3541211143 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a3effd700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a400afa30 0x7f8a400b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a3ffff700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a400b1040 0x7f8a4015a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a3f7fe700 1 -- 172.21.15.111:0/3541211143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a400afa30 msgr2=0x7f8a400b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a3f7fe700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a400afa30 0x7f8a400b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a3f7fe700 1 -- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a400b1040 msgr2=0x7f8a4015a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a3f7fe700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a400b1040 0x7f8a4015a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f8a3f7fe700 1 -- 172.21.15.111:0/3541211143 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a400cd9c0 con 0x7f8a28004430 2023-11-28T12:59:20.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7f8a3effd700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a400afa30 0x7f8a400b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:20.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7f8a3ffff700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a400b1040 0x7f8a4015a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:20.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7f8a3f7fe700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a28004430 0x7f8a401e5190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8a38006800 tx=0x7f8a38006bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.460+0000 7fce57fff700 1 --2- 172.21.15.111:0/74955623 >> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] conn(0x7fce3c071ec0 0x7fce3c0742f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.461+0000 7fd341ffb700 1 -- 172.21.15.111:0/2751585426 <== osd.2 v2:172.21.15.111:6811/103530 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd34c067880 con 0x7fd34c1fecd0 2023-11-28T12:59:20.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.461+0000 7f779c32b700 1 -- 172.21.15.111:0/3998801570 shutdown_connections 2023-11-28T12:59:20.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.460+0000 7f2d0dffb700 1 -- 172.21.15.111:0/2835301711 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14003ea0 con 0x7f2d180cb240 2023-11-28T12:59:20.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.460+0000 7f2d0dffb700 1 -- 172.21.15.111:0/2835301711 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d14011630 con 0x7f2d180cb240 2023-11-28T12:59:20.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.460+0000 7f2d0dffb700 1 -- 172.21.15.111:0/2835301711 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d140119e0 con 0x7f2d180cb240 2023-11-28T12:59:20.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.460+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2835301711 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d181595e0 con 0x7f2d180cb240 2023-11-28T12:59:20.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.461+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2835301711 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d180cd1d0 con 0x7f2d180cb240 2023-11-28T12:59:20.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2835301711 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2cf8000fc0 con 0x7f2d180cb240 2023-11-28T12:59:20.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.460+0000 7f527affd700 1 -- 172.21.15.111:0/479974858 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5284004190 con 0x7f528c0c9300 2023-11-28T12:59:20.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.461+0000 7f527affd700 1 --2- 172.21.15.111:0/479974858 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f526c0682c0 0x7f526c06a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.461+0000 7f527affd700 1 -- 172.21.15.111:0/479974858 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f52840876e0 con 0x7f528c0c9300 2023-11-28T12:59:20.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.461+0000 7f527affd700 1 --2- 172.21.15.111:0/479974858 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x7f526c071f10 0x7f526c074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.461+0000 7f527affd700 1 -- 172.21.15.111:0/479974858 --> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f526c074a10 con 0x7f526c071f10 2023-11-28T12:59:20.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.461+0000 7f527affd700 1 -- 172.21.15.111:0/479974858 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5284089db0 con 0x7f528c0c9300 2023-11-28T12:59:20.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.461+0000 7f528a59c700 1 --2- 172.21.15.111:0/479974858 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f526c0682c0 0x7f526c06a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f528a59c700 1 --2- 172.21.15.111:0/479974858 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f526c0682c0 0x7f526c06a770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5280006fd0 tx=0x7f5280008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.461+0000 7fa23affd700 1 -- 172.21.15.111:0/159363889 <== osd.6 v2:172.21.15.149:6816/103796 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fa228074ac0 con 0x7fa228071fc0 2023-11-28T12:59:20.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.459+0000 7efd99713700 1 -- 172.21.15.111:0/4082627357 wait complete. 2023-11-28T12:59:20.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.461+0000 7efd99713700 1 Processor -- start 2023-11-28T12:59:20.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.461+0000 7efd99713700 1 -- start start 2023-11-28T12:59:20.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7efd99713700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd9406a7b0 0x7efd941fe740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7efd99713700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd940ca430 0x7efd94158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7efd99713700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd94142e80 0x7efd9415c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7efd99713700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd940c5100 con 0x7efd940ca430 2023-11-28T12:59:20.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7efd99713700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd940c44f0 con 0x7efd94142e80 2023-11-28T12:59:20.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7efd99713700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd940c47d0 con 0x7efd9406a7b0 2023-11-28T12:59:20.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7efd92ffd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd9406a7b0 0x7efd941fe740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7efd92ffd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd9406a7b0 0x7efd941fe740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55054/0 (socket says 172.21.15.111:55054) 2023-11-28T12:59:20.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7efd92ffd700 1 -- 172.21.15.111:0/2478544272 learned_addr learned my addr 172.21.15.111:0/2478544272 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7efd927fc700 1 --2- 172.21.15.111:0/2478544272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd940ca430 0x7efd94158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7efd92ffd700 1 -- 172.21.15.111:0/2478544272 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd94142e80 msgr2=0x7efd9415c080 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T12:59:20.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7efd92ffd700 1 --2- 172.21.15.111:0/2478544272 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd94142e80 0x7efd9415c080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7efd92ffd700 1 -- 172.21.15.111:0/2478544272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd940ca430 msgr2=0x7efd94158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7efd92ffd700 1 --2- 172.21.15.111:0/2478544272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd940ca430 0x7efd94158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7efd92ffd700 1 -- 172.21.15.111:0/2478544272 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c009a70 con 0x7efd9406a7b0 2023-11-28T12:59:20.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7f8a27fff700 1 -- 172.21.15.111:0/3541211143 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a38001a50 con 0x7f8a28004430 2023-11-28T12:59:20.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7f8a27fff700 1 -- 172.21.15.111:0/3541211143 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a38001bb0 con 0x7f8a28004430 2023-11-28T12:59:20.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7f8a27fff700 1 -- 172.21.15.111:0/3541211143 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a38001f20 con 0x7f8a28004430 2023-11-28T12:59:20.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3541211143 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a400cd6e0 con 0x7f8a28004430 2023-11-28T12:59:20.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3541211143 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4015c110 con 0x7f8a28004430 2023-11-28T12:59:20.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.464+0000 7fce557fa700 1 -- 172.21.15.111:0/74955623 <== osd.1 v2:172.21.15.111:6824/103531 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fce3c0749c0 con 0x7fce3c071ec0 2023-11-28T12:59:20.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.464+0000 7efd92ffd700 1 --2- 172.21.15.111:0/2478544272 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd9406a7b0 0x7efd941fe740 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efd8400e870 tx=0x7efd8400eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.464+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3541211143 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8a20000fc0 con 0x7f8a28004430 2023-11-28T12:59:20.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.462+0000 7f779c32b700 1 -- 172.21.15.111:0/3998801570 wait complete. 2023-11-28T12:59:20.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.464+0000 7f779c32b700 1 Processor -- start 2023-11-28T12:59:20.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.463+0000 7f5289d9b700 1 --2- 172.21.15.111:0/479974858 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x7f526c071f10 0x7f526c074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.464+0000 7f5289d9b700 1 --2- 172.21.15.111:0/479974858 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x7f526c071f10 0x7f526c074340 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.465+0000 7f527affd700 1 -- 172.21.15.111:0/479974858 <== osd.5 v2:172.21.15.149:6808/103795 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f526c074a10 con 0x7f526c071f10 2023-11-28T12:59:20.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.465+0000 7f8a27fff700 1 -- 172.21.15.111:0/3541211143 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8a3801c030 con 0x7f8a28004430 2023-11-28T12:59:20.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f8a27fff700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8a2c068340 0x7f8a2c06a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f8a27fff700 1 -- 172.21.15.111:0/3541211143 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f8a38085dc0 con 0x7f8a28004430 2023-11-28T12:59:20.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f8a27fff700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x7f8a2c072010 0x7f8a2c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f8a27fff700 1 -- 172.21.15.111:0/3541211143 --> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a2c074b10 con 0x7f8a2c072010 2023-11-28T12:59:20.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f8a27fff700 1 -- 172.21.15.111:0/3541211143 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a380883a0 con 0x7f8a28004430 2023-11-28T12:59:20.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f8a3ffff700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x7f8a2c072010 0x7f8a2c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.465+0000 7f779c32b700 1 -- start start 2023-11-28T12:59:20.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f779c32b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f779406acf0 0x7f77940c9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f779c32b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f779406b870 0x7f77940c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f779c32b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f77941607e0 0x7f77940c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f779c32b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77940c9570 con 0x7f779406acf0 2023-11-28T12:59:20.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f779c32b700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77940c68d0 con 0x7f77941607e0 2023-11-28T12:59:20.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f779c32b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77941ff040 con 0x7f779406b870 2023-11-28T12:59:20.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.465+0000 7f2d0dffb700 1 -- 172.21.15.111:0/2835301711 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2d1402e070 con 0x7f2d180cb240 2023-11-28T12:59:20.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f2d0dffb700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2cfc0683c0 0x7f2cfc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f2d0dffb700 1 -- 172.21.15.111:0/2835301711 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f2d14086fe0 con 0x7f2d180cb240 2023-11-28T12:59:20.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f2d0dffb700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] conn(0x7f2cfc072010 0x7f2cfc074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f2d0dffb700 1 -- 172.21.15.111:0/2835301711 --> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2cfc074b10 con 0x7f2cfc072010 2023-11-28T12:59:20.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f779a0c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f779406acf0 0x7f77940c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f779a0c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f779406acf0 0x7f77940c9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33318/0 (socket says 172.21.15.111:33318) 2023-11-28T12:59:20.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.466+0000 7f779a0c7700 1 -- 172.21.15.111:0/3589375960 learned_addr learned my addr 172.21.15.111:0/3589375960 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7f779a8c8700 1 --2- 172.21.15.111:0/3589375960 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f77941607e0 0x7f77940c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7f2d1cc7c700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2cfc0683c0 0x7f2cfc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7f2d1dc7e700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] conn(0x7f2cfc072010 0x7f2cfc074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7f8a3effd700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8a2c068340 0x7f8a2c06a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7f779a0c7700 1 -- 172.21.15.111:0/3589375960 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f779406b870 msgr2=0x7f77940c6390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T12:59:20.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7f779a0c7700 1 --2- 172.21.15.111:0/3589375960 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f779406b870 0x7f77940c6390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7f779a0c7700 1 -- 172.21.15.111:0/3589375960 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f77941607e0 msgr2=0x7f77940c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7f779a0c7700 1 --2- 172.21.15.111:0/3589375960 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f77941607e0 0x7f77940c9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7f779a0c7700 1 -- 172.21.15.111:0/3589375960 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77840094a0 con 0x7f779406acf0 2023-11-28T12:59:20.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7f2d0dffb700 1 -- 172.21.15.111:0/2835301711 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d140534e0 con 0x7f2d180cb240 2023-11-28T12:59:20.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7efd7b7fe700 1 -- 172.21.15.111:0/2478544272 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd84013680 con 0x7efd9406a7b0 2023-11-28T12:59:20.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7efd7b7fe700 1 -- 172.21.15.111:0/2478544272 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd840137e0 con 0x7efd9406a7b0 2023-11-28T12:59:20.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7efd7b7fe700 1 -- 172.21.15.111:0/2478544272 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd84013b50 con 0x7efd9406a7b0 2023-11-28T12:59:20.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7f2d1cc7c700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2cfc0683c0 0x7f2cfc06a870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2d180ada00 tx=0x7f2d08008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.467+0000 7f8a3ffff700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x7f8a2c072010 0x7f8a2c074440 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.468+0000 7f8a3effd700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8a2c068340 0x7f8a2c06a7f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8a30000c00 tx=0x7f8a30009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.468+0000 7f779a0c7700 1 --2- 172.21.15.111:0/3589375960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f779406acf0 0x7f77940c9030 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f779000c7f0 tx=0x7f779000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.468+0000 7f778affd700 1 -- 172.21.15.111:0/3589375960 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7790004220 con 0x7f779406acf0 2023-11-28T12:59:20.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.468+0000 7f778affd700 1 -- 172.21.15.111:0/3589375960 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7790004380 con 0x7f779406acf0 2023-11-28T12:59:20.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.468+0000 7f778affd700 1 -- 172.21.15.111:0/3589375960 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77900036a0 con 0x7f779406acf0 2023-11-28T12:59:20.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.468+0000 7f2d1dc7e700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] conn(0x7f2cfc072010 0x7f2cfc074440 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.469+0000 7f8a27fff700 1 -- 172.21.15.111:0/3541211143 <== osd.0 v2:172.21.15.111:6800/103527 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8a2c074b10 con 0x7f8a2c072010 2023-11-28T12:59:20.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.469+0000 7f2d0dffb700 1 -- 172.21.15.111:0/2835301711 <== osd.4 v2:172.21.15.149:6800/103794 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f2cfc074b10 con 0x7f2cfc072010 2023-11-28T12:59:20.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.468+0000 7f779c32b700 1 -- 172.21.15.111:0/3589375960 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7794164e90 con 0x7f779406acf0 2023-11-28T12:59:20.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.468+0000 7f779c32b700 1 -- 172.21.15.111:0/3589375960 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7794164bb0 con 0x7f779406acf0 2023-11-28T12:59:20.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.470+0000 7f778affd700 1 -- 172.21.15.111:0/3589375960 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7790020070 con 0x7f779406acf0 2023-11-28T12:59:20.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.471+0000 7f778affd700 1 --2- 172.21.15.111:0/3589375960 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7778068340 0x7f777806a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.471+0000 7f778affd700 1 -- 172.21.15.111:0/3589375960 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f77900859d0 con 0x7f779406acf0 2023-11-28T12:59:20.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.471+0000 7f77998c6700 1 --2- 172.21.15.111:0/3589375960 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7778068340 0x7f777806a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.472+0000 7f77998c6700 1 --2- 172.21.15.111:0/3589375960 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7778068340 0x7f777806a7f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f77940c1a80 tx=0x7f7784008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.473+0000 7efd99713700 1 -- 172.21.15.111:0/2478544272 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd940cab00 con 0x7efd9406a7b0 2023-11-28T12:59:20.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.473+0000 7efd99713700 1 -- 172.21.15.111:0/2478544272 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd94143250 con 0x7efd9406a7b0 2023-11-28T12:59:20.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.474+0000 7efd7b7fe700 1 -- 172.21.15.111:0/2478544272 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efd84019030 con 0x7efd9406a7b0 2023-11-28T12:59:20.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.474+0000 7efd99713700 1 -- 172.21.15.111:0/2478544272 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efd940cc5f0 con 0x7efd9406a7b0 2023-11-28T12:59:20.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.475+0000 7efd7b7fe700 1 --2- 172.21.15.111:0/2478544272 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd80068340 0x7efd8006a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.475+0000 7efd927fc700 1 --2- 172.21.15.111:0/2478544272 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd80068340 0x7efd8006a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.475+0000 7efd7b7fe700 1 -- 172.21.15.111:0/2478544272 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7efd84085c90 con 0x7efd9406a7b0 2023-11-28T12:59:20.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.475+0000 7efd7b7fe700 1 --2- 172.21.15.111:0/2478544272 >> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] conn(0x7efd80071f90 0x7efd800743c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.475+0000 7efd7b7fe700 1 -- 172.21.15.111:0/2478544272 --> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd80074a90 con 0x7efd80071f90 2023-11-28T12:59:20.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.475+0000 7efd7b7fe700 1 -- 172.21.15.111:0/2478544272 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd84051f60 con 0x7efd9406a7b0 2023-11-28T12:59:20.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.475+0000 7efd927fc700 1 --2- 172.21.15.111:0/2478544272 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd80068340 0x7efd8006a7f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efd940b0c80 tx=0x7efd880092c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.476+0000 7efd937fe700 1 --2- 172.21.15.111:0/2478544272 >> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] conn(0x7efd80071f90 0x7efd800743c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.476+0000 7efd937fe700 1 --2- 172.21.15.111:0/2478544272 >> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] conn(0x7efd80071f90 0x7efd800743c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.477+0000 7efd7b7fe700 1 -- 172.21.15.111:0/2478544272 <== osd.7 v2:172.21.15.149:6824/103827 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7efd80074a90 con 0x7efd80071f90 2023-11-28T12:59:20.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.484+0000 7f779c32b700 1 --2- 172.21.15.111:0/3589375960 >> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] conn(0x7f777c0015e0 0x7f777c003a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.484+0000 7f779c32b700 1 -- 172.21.15.111:0/3589375960 --> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f777c006bc0 con 0x7f777c0015e0 2023-11-28T12:59:20.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.484+0000 7fa24c298700 1 -- 172.21.15.111:0/159363889 --> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa24414c490 con 0x7fa228071fc0 2023-11-28T12:59:20.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.485+0000 7fa23affd700 1 -- 172.21.15.111:0/159363889 <== osd.6 v2:172.21.15.149:6816/103796 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa24414c490 con 0x7fa228071fc0 2023-11-28T12:59:20.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.486+0000 7f779a8c8700 1 --2- 172.21.15.111:0/3589375960 >> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] conn(0x7f777c0015e0 0x7f777c003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.486+0000 7fa24c298700 1 -- 172.21.15.111:0/159363889 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x7fa228071fc0 msgr2=0x7fa2280743f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.486+0000 7fa24c298700 1 --2- 172.21.15.111:0/159363889 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x7fa228071fc0 0x7fa2280743f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.486+0000 7fa24c298700 1 -- 172.21.15.111:0/159363889 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa2280682f0 msgr2=0x7fa22806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.486+0000 7fa24c298700 1 --2- 172.21.15.111:0/159363889 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa2280682f0 0x7fa22806a7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa240007fd0 tx=0x7fa240007a20 comp rx=0 tx=0).stop 2023-11-28T12:59:20.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.486+0000 7fa24c298700 1 -- 172.21.15.111:0/159363889 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa230005250 msgr2=0x7fa244159240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.486+0000 7fa24c298700 1 --2- 172.21.15.111:0/159363889 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa230005250 0x7fa244159240 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa23400b3f0 tx=0x7fa23400b700 comp rx=0 tx=0).stop 2023-11-28T12:59:20.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.486+0000 7fa24a835700 1 -- 172.21.15.111:0/159363889 reap_dead start 2023-11-28T12:59:20.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.486+0000 7f779a8c8700 1 --2- 172.21.15.111:0/3589375960 >> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] conn(0x7f777c0015e0 0x7f777c003a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.486+0000 7fa24c298700 1 -- 172.21.15.111:0/159363889 shutdown_connections 2023-11-28T12:59:20.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.486+0000 7fa24c298700 1 -- 172.21.15.111:0/159363889 >> 172.21.15.111:0/159363889 conn(0x7fa2440719a0 msgr2=0x7fa2440b26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.487+0000 7f778affd700 1 -- 172.21.15.111:0/3589375960 <== osd.3 v2:172.21.15.111:6808/103538 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f777c006bc0 con 0x7f777c0015e0 2023-11-28T12:59:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.488+0000 7fa24c298700 1 -- 172.21.15.111:0/159363889 shutdown_connections 2023-11-28T12:59:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.495+0000 7fce5f0c1700 1 -- 172.21.15.111:0/74955623 --> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fce58067880 con 0x7fce3c071ec0 2023-11-28T12:59:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7fce557fa700 1 -- 172.21.15.111:0/74955623 <== osd.1 v2:172.21.15.111:6824/103531 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fce58067880 con 0x7fce3c071ec0 2023-11-28T12:59:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.498+0000 7fce5f0c1700 1 -- 172.21.15.111:0/74955623 >> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] conn(0x7fce3c071ec0 msgr2=0x7fce3c0742f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.498+0000 7fce5f0c1700 1 --2- 172.21.15.111:0/74955623 >> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] conn(0x7fce3c071ec0 0x7fce3c0742f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.498+0000 7fce5f0c1700 1 -- 172.21.15.111:0/74955623 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fce3c0682f0 msgr2=0x7fce3c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.498+0000 7fce5f0c1700 1 --2- 172.21.15.111:0/74955623 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fce3c0682f0 0x7fce3c06a7a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fce44009710 tx=0x7fce44006c60 comp rx=0 tx=0).stop 2023-11-28T12:59:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.498+0000 7fce5f0c1700 1 -- 172.21.15.111:0/74955623 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce580cdee0 msgr2=0x7fce5813cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.498+0000 7fce5f0c1700 1 --2- 172.21.15.111:0/74955623 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce580cdee0 0x7fce5813cdf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fce5000b680 tx=0x7fce50004ad0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.494+0000 7fd35347c700 1 -- 172.21.15.111:0/2751585426 --> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd34c06a950 con 0x7fd34c1fecd0 2023-11-28T12:59:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.494+0000 7fd341ffb700 1 -- 172.21.15.111:0/2751585426 <== osd.2 v2:172.21.15.111:6811/103530 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd34c06a950 con 0x7fd34c1fecd0 2023-11-28T12:59:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.495+0000 7fd35347c700 1 -- 172.21.15.111:0/2751585426 >> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] conn(0x7fd34c1fecd0 msgr2=0x7fd34c0ab390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.495+0000 7fd35347c700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] conn(0x7fd34c1fecd0 0x7fd34c0ab390 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.495+0000 7fd35347c700 1 -- 172.21.15.111:0/2751585426 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd3300683c0 msgr2=0x7fd33006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.495+0000 7fd35347c700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd3300683c0 0x7fd33006a870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd33c000c00 tx=0x7fd33c009040 comp rx=0 tx=0).stop 2023-11-28T12:59:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.495+0000 7fd35347c700 1 -- 172.21.15.111:0/2751585426 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd34c0d1d70 msgr2=0x7fd34c1fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.495+0000 7fd35347c700 1 --2- 172.21.15.111:0/2751585426 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd34c0d1d70 0x7fd34c1fe790 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd344013000 tx=0x7fd344004ad0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7fd351a19700 1 -- 172.21.15.111:0/2751585426 reap_dead start 2023-11-28T12:59:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7fd35347c700 1 -- 172.21.15.111:0/2751585426 shutdown_connections 2023-11-28T12:59:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7fd35347c700 1 -- 172.21.15.111:0/2751585426 >> 172.21.15.111:0/2751585426 conn(0x7fd34c0719a0 msgr2=0x7fd34c0cbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7fd35347c700 1 -- 172.21.15.111:0/2751585426 shutdown_connections 2023-11-28T12:59:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7fd35347c700 1 -- 172.21.15.111:0/2751585426 wait complete. 2023-11-28T12:59:20.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.495+0000 7f5290ed5700 1 -- 172.21.15.111:0/479974858 --> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5270002d40 con 0x7f526c071f10 2023-11-28T12:59:20.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7f527affd700 1 -- 172.21.15.111:0/479974858 <== osd.5 v2:172.21.15.149:6808/103795 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5270002d40 con 0x7f526c071f10 2023-11-28T12:59:20.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7f5290ed5700 1 -- 172.21.15.111:0/479974858 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x7f526c071f10 msgr2=0x7f526c074340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7f5290ed5700 1 --2- 172.21.15.111:0/479974858 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x7f526c071f10 0x7f526c074340 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7f5290ed5700 1 -- 172.21.15.111:0/479974858 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f526c0682c0 msgr2=0x7f526c06a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7f5290ed5700 1 --2- 172.21.15.111:0/479974858 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f526c0682c0 0x7f526c06a770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5280006fd0 tx=0x7f5280008040 comp rx=0 tx=0).stop 2023-11-28T12:59:20.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7f5290ed5700 1 -- 172.21.15.111:0/479974858 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f528c0c9300 msgr2=0x7f528c157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7f5290ed5700 1 --2- 172.21.15.111:0/479974858 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f528c0c9300 0x7f528c157ba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f528400d860 tx=0x7f5284004ad0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.496+0000 7f528ad9d700 1 -- 172.21.15.111:0/479974858 reap_dead start 2023-11-28T12:59:20.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.488+0000 7fa24c298700 1 -- 172.21.15.111:0/159363889 wait complete. 2023-11-28T12:59:20.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.500+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3541211143 --> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8a20002d40 con 0x7f8a2c072010 2023-11-28T12:59:20.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.503+0000 7f8a27fff700 1 -- 172.21.15.111:0/3541211143 <== osd.0 v2:172.21.15.111:6800/103527 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8a20002d40 con 0x7f8a2c072010 2023-11-28T12:59:20.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.500+0000 7fce5d65e700 1 -- 172.21.15.111:0/74955623 reap_dead start 2023-11-28T12:59:20.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.500+0000 7fce5f0c1700 1 -- 172.21.15.111:0/74955623 shutdown_connections 2023-11-28T12:59:20.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.500+0000 7fce5f0c1700 1 -- 172.21.15.111:0/74955623 >> 172.21.15.111:0/74955623 conn(0x7fce580cb490 msgr2=0x7fce5814d820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.500+0000 7fce5f0c1700 1 -- 172.21.15.111:0/74955623 shutdown_connections 2023-11-28T12:59:20.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.500+0000 7fce5f0c1700 1 -- 172.21.15.111:0/74955623 wait complete. 2023-11-28T12:59:20.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.500+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2835301711 --> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2cf8002d40 con 0x7f2cfc072010 2023-11-28T12:59:20.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.500+0000 7f2d0dffb700 1 -- 172.21.15.111:0/2835301711 <== osd.4 v2:172.21.15.149:6800/103794 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2cf8002d40 con 0x7f2cfc072010 2023-11-28T12:59:20.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.501+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2835301711 >> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] conn(0x7f2cfc072010 msgr2=0x7f2cfc074440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.501+0000 7f2d1f6e1700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] conn(0x7f2cfc072010 0x7f2cfc074440 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.501+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2835301711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2cfc0683c0 msgr2=0x7f2cfc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.501+0000 7f2d1f6e1700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2cfc0683c0 0x7f2cfc06a870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2d180ada00 tx=0x7f2d08008040 comp rx=0 tx=0).stop 2023-11-28T12:59:20.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.501+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2835301711 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d180cb240 msgr2=0x7f2d18151100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.501+0000 7f2d1f6e1700 1 --2- 172.21.15.111:0/2835301711 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d180cb240 0x7f2d18151100 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2d14013000 tx=0x7f2d14004750 comp rx=0 tx=0).stop 2023-11-28T12:59:20.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.501+0000 7f2d1dc7e700 1 -- 172.21.15.111:0/2835301711 reap_dead start 2023-11-28T12:59:20.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.501+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2835301711 shutdown_connections 2023-11-28T12:59:20.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.501+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2835301711 >> 172.21.15.111:0/2835301711 conn(0x7f2d180b0db0 msgr2=0x7f2d180b0590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.501+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2835301711 shutdown_connections 2023-11-28T12:59:20.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.501+0000 7f2d1f6e1700 1 -- 172.21.15.111:0/2835301711 wait complete. 2023-11-28T12:59:20.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.500+0000 7f5290ed5700 1 -- 172.21.15.111:0/479974858 shutdown_connections 2023-11-28T12:59:20.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.500+0000 7f5290ed5700 1 -- 172.21.15.111:0/479974858 >> 172.21.15.111:0/479974858 conn(0x7f528c0cb490 msgr2=0x7f528c0cde80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.503+0000 7efd99713700 1 -- 172.21.15.111:0/2478544272 --> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efd94067880 con 0x7efd80071f90 2023-11-28T12:59:20.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.503+0000 7efd7b7fe700 1 -- 172.21.15.111:0/2478544272 <== osd.7 v2:172.21.15.149:6824/103827 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efd94067880 con 0x7efd80071f90 2023-11-28T12:59:20.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.504+0000 7efd99713700 1 -- 172.21.15.111:0/2478544272 >> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] conn(0x7efd80071f90 msgr2=0x7efd800743c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.504+0000 7efd99713700 1 --2- 172.21.15.111:0/2478544272 >> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] conn(0x7efd80071f90 0x7efd800743c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.504+0000 7efd99713700 1 -- 172.21.15.111:0/2478544272 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd80068340 msgr2=0x7efd8006a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.504+0000 7efd99713700 1 --2- 172.21.15.111:0/2478544272 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd80068340 0x7efd8006a7f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efd940b0c80 tx=0x7efd880092c0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.504+0000 7efd99713700 1 -- 172.21.15.111:0/2478544272 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd9406a7b0 msgr2=0x7efd941fe740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.504+0000 7efd99713700 1 --2- 172.21.15.111:0/2478544272 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd9406a7b0 0x7efd941fe740 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efd8400e870 tx=0x7efd8400eb80 comp rx=0 tx=0).stop 2023-11-28T12:59:20.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.504+0000 7efd937fe700 1 -- 172.21.15.111:0/2478544272 reap_dead start 2023-11-28T12:59:20.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.504+0000 7efd99713700 1 -- 172.21.15.111:0/2478544272 shutdown_connections 2023-11-28T12:59:20.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.504+0000 7efd99713700 1 -- 172.21.15.111:0/2478544272 >> 172.21.15.111:0/2478544272 conn(0x7efd940cb490 msgr2=0x7efd9406de50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.504+0000 7efd99713700 1 -- 172.21.15.111:0/2478544272 shutdown_connections 2023-11-28T12:59:20.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.504+0000 7efd99713700 1 -- 172.21.15.111:0/2478544272 wait complete. 2023-11-28T12:59:20.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.505+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x7f8a2c072010 msgr2=0x7f8a2c074440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.505+0000 7f8a45bdc700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x7f8a2c072010 0x7f8a2c074440 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.510+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8a2c068340 msgr2=0x7f8a2c06a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.510+0000 7f8a45bdc700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8a2c068340 0x7f8a2c06a7f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8a30000c00 tx=0x7f8a30009030 comp rx=0 tx=0).stop 2023-11-28T12:59:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.510+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3541211143 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a28004430 msgr2=0x7f8a401e5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.510+0000 7f8a45bdc700 1 --2- 172.21.15.111:0/3541211143 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a28004430 0x7f8a401e5190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8a38006800 tx=0x7f8a38006bc0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.510+0000 7f8a3ffff700 1 -- 172.21.15.111:0/3541211143 reap_dead start 2023-11-28T12:59:20.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.513+0000 7f5290ed5700 1 -- 172.21.15.111:0/479974858 shutdown_connections 2023-11-28T12:59:20.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.513+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3541211143 shutdown_connections 2023-11-28T12:59:20.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.513+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3541211143 >> 172.21.15.111:0/3541211143 conn(0x7f8a400cb490 msgr2=0x7f8a40157570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.513+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3541211143 shutdown_connections 2023-11-28T12:59:20.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.513+0000 7f8a45bdc700 1 -- 172.21.15.111:0/3541211143 wait complete. 2023-11-28T12:59:20.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.514+0000 7f5290ed5700 1 -- 172.21.15.111:0/479974858 wait complete. 2023-11-28T12:59:20.556 INFO:teuthology.orchestra.run.smithi111.stdout:55834574858 2023-11-28T12:59:20.556 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-28T12:59:20.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.522+0000 7f779c32b700 1 -- 172.21.15.111:0/3589375960 --> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f777c005ca0 con 0x7f777c0015e0 2023-11-28T12:59:20.557 INFO:teuthology.orchestra.run.smithi111.stdout:55834574857 2023-11-28T12:59:20.558 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-28T12:59:20.558 INFO:teuthology.orchestra.run.smithi111.stdout:55834574858 2023-11-28T12:59:20.559 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-28T12:59:20.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.524+0000 7f778affd700 1 -- 172.21.15.111:0/3589375960 <== osd.3 v2:172.21.15.111:6808/103538 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f777c005ca0 con 0x7f777c0015e0 2023-11-28T12:59:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.524+0000 7f779c32b700 1 -- 172.21.15.111:0/3589375960 >> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] conn(0x7f777c0015e0 msgr2=0x7f777c003a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.524+0000 7f779c32b700 1 --2- 172.21.15.111:0/3589375960 >> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] conn(0x7f777c0015e0 0x7f777c003a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.526+0000 7f779c32b700 1 -- 172.21.15.111:0/3589375960 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7778068340 msgr2=0x7f777806a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.526+0000 7f779c32b700 1 --2- 172.21.15.111:0/3589375960 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7778068340 0x7f777806a7f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f77940c1a80 tx=0x7f7784008030 comp rx=0 tx=0).stop 2023-11-28T12:59:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.526+0000 7f779c32b700 1 -- 172.21.15.111:0/3589375960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f779406acf0 msgr2=0x7f77940c9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.526+0000 7f779c32b700 1 --2- 172.21.15.111:0/3589375960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f779406acf0 0x7f77940c9030 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f779000c7f0 tx=0x7f779000cb00 comp rx=0 tx=0).stop 2023-11-28T12:59:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.526+0000 7f779a8c8700 1 -- 172.21.15.111:0/3589375960 reap_dead start 2023-11-28T12:59:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.526+0000 7f779c32b700 1 -- 172.21.15.111:0/3589375960 shutdown_connections 2023-11-28T12:59:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.526+0000 7f779c32b700 1 -- 172.21.15.111:0/3589375960 >> 172.21.15.111:0/3589375960 conn(0x7f77940b0db0 msgr2=0x7f779406e730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.526+0000 7f779c32b700 1 -- 172.21.15.111:0/3589375960 shutdown_connections 2023-11-28T12:59:20.561 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-11-28T12:59:20.561 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-28T12:59:20.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.526+0000 7f779c32b700 1 -- 172.21.15.111:0/3589375960 wait complete. 2023-11-28T12:59:20.562 INFO:teuthology.orchestra.run.smithi111.stdout:55834574859 2023-11-28T12:59:20.563 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-28T12:59:20.563 INFO:teuthology.orchestra.run.smithi111.stdout:55834574858 2023-11-28T12:59:20.563 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-28T12:59:20.564 INFO:teuthology.orchestra.run.smithi111.stdout:55834574858 2023-11-28T12:59:20.565 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-28T12:59:20.565 INFO:teuthology.orchestra.run.smithi111.stdout:55834574858 2023-11-28T12:59:20.566 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-28T12:59:20.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.807+0000 7f4ed2337700 1 Processor -- start 2023-11-28T12:59:20.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.807+0000 7f4ed2337700 1 -- start start 2023-11-28T12:59:20.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.808+0000 7f4ed2337700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ecc0cb240 0x7f4ecc1590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.808+0000 7f4ed2337700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ecc0cd540 con 0x7f4ecc06e730 2023-11-28T12:59:20.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.808+0000 7f4ed2337700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ecc0ce0e0 con 0x7f4ecc15d270 2023-11-28T12:59:20.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.808+0000 7f4ed2337700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc159a20 con 0x7f4ecc0cb240 2023-11-28T12:59:20.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.809+0000 7f4ed08d4700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4ecc06e730 0x7f4ecc0c9ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34114/0 (socket says 172.21.15.111:34114) 2023-11-28T12:59:20.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.809+0000 7f4ed08d4700 1 -- 172.21.15.111:0/974093406 learned_addr learned my addr 172.21.15.111:0/974093406 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:20.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.809+0000 7f4ecbfff700 1 --2- 172.21.15.111:0/974093406 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ecc0cb240 0x7f4ecc1590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.809+0000 7f4ecaffd700 1 -- 172.21.15.111:0/974093406 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795239642 0 0) 0x7f4ecc0ce0e0 con 0x7f4ecc15d270 2023-11-28T12:59:20.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.809+0000 7f4ecaffd700 1 -- 172.21.15.111:0/974093406 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb80035b0 con 0x7f4ecc15d270 2023-11-28T12:59:20.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.809+0000 7f4ecbfff700 1 -- 172.21.15.111:0/974093406 >> v1:172.21.15.149:6789/0 conn(0x7f4ecc15d270 legacy=0x7f4ecc0cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.809+0000 7f4ecbfff700 1 -- 172.21.15.111:0/974093406 >> v1:172.21.15.111:6789/0 conn(0x7f4ecc06e730 legacy=0x7f4ecc0c9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.809+0000 7f4ecbfff700 1 -- 172.21.15.111:0/974093406 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ecc159d00 con 0x7f4ecc0cb240 2023-11-28T12:59:20.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.810+0000 7f4ecbfff700 1 --2- 172.21.15.111:0/974093406 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ecc0cb240 0x7f4ecc1590a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4eb400aa10 tx=0x7f4eb400ad20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=433ed3091627e72d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.810+0000 7f4ecaffd700 1 -- 172.21.15.111:0/974093406 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb400cb70 con 0x7f4ecc0cb240 2023-11-28T12:59:20.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.810+0000 7f4ecaffd700 1 -- 172.21.15.111:0/974093406 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb400ccd0 con 0x7f4ecc0cb240 2023-11-28T12:59:20.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.810+0000 7f4ecaffd700 1 -- 172.21.15.111:0/974093406 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4018600 con 0x7f4ecc0cb240 2023-11-28T12:59:20.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.811+0000 7f4ed2337700 1 -- 172.21.15.111:0/974093406 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ecc0cb240 msgr2=0x7f4ecc1590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.811+0000 7f4ed2337700 1 --2- 172.21.15.111:0/974093406 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ecc0cb240 0x7f4ecc1590a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4eb400aa10 tx=0x7f4eb400ad20 comp rx=0 tx=0).stop 2023-11-28T12:59:20.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.812+0000 7fb54e8c7700 1 Processor -- start 2023-11-28T12:59:20.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.812+0000 7f047cb70700 1 Processor -- start 2023-11-28T12:59:20.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.812+0000 7f4ed2337700 1 -- 172.21.15.111:0/974093406 shutdown_connections 2023-11-28T12:59:20.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.812+0000 7f4ed2337700 1 --2- 172.21.15.111:0/974093406 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ecc0cb240 0x7f4ecc1590a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.812+0000 7f4ed2337700 1 -- 172.21.15.111:0/974093406 >> 172.21.15.111:0/974093406 conn(0x7f4ecc0b0db0 msgr2=0x7f4ecc0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.812+0000 7f4ed2337700 1 -- 172.21.15.111:0/974093406 shutdown_connections 2023-11-28T12:59:20.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.813+0000 7fd623c19700 1 Processor -- start 2023-11-28T12:59:20.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.812+0000 7f047cb70700 1 -- start start 2023-11-28T12:59:20.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.813+0000 7f047cb70700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 0x7f04780c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.813+0000 7f047cb70700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f047815ce70 0x7f047806a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.813+0000 7f047cb70700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f047806ab80 con 0x7f04780cdee0 2023-11-28T12:59:20.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.813+0000 7f047cb70700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04780ccb60 con 0x7f047815ce70 2023-11-28T12:59:20.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.813+0000 7f047cb70700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04780cce40 con 0x7f047806a270 2023-11-28T12:59:20.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7fd623c19700 1 -- start start 2023-11-28T12:59:20.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.813+0000 7f4ed2337700 1 -- 172.21.15.111:0/974093406 wait complete. 2023-11-28T12:59:20.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7f4ed2337700 1 Processor -- start 2023-11-28T12:59:20.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fd623c19700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd61c06cc70 0x7fd61c06dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fd623c19700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd61c070910 0x7fd61c070d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fd623c19700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd61c071270 con 0x7fd61c158b70 2023-11-28T12:59:20.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fd623c19700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd61c0ca620 con 0x7fd61c070910 2023-11-28T12:59:20.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fd623c19700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c06a980 con 0x7fd61c06cc70 2023-11-28T12:59:20.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fd6219b5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd61c06cc70 0x7fd61c06dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fd6219b5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd61c06cc70 0x7fd61c06dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55102/0 (socket says 172.21.15.111:55102) 2023-11-28T12:59:20.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fd6219b5700 1 -- 172.21.15.111:0/1623595913 learned_addr learned my addr 172.21.15.111:0/1623595913 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.813+0000 7f9ce185f700 1 Processor -- start 2023-11-28T12:59:20.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.813+0000 7f9ce185f700 1 -- start start 2023-11-28T12:59:20.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7f9ce185f700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9cdc160e10 0x7f9cdc0ad950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7f9ce185f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cdc06a980 0x7f9cdc0ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7f9ce185f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cdc165740 con 0x7f9cdc0ade90 2023-11-28T12:59:20.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7f9ce185f700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cdc164bc0 con 0x7f9cdc160e10 2023-11-28T12:59:20.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7f9ce185f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc164ea0 con 0x7f9cdc06a980 2023-11-28T12:59:20.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7f9cdb7fe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cdc06a980 0x7f9cdc0ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7f9cdb7fe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cdc06a980 0x7f9cdc0ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55098/0 (socket says 172.21.15.111:55098) 2023-11-28T12:59:20.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7f9cdb7fe700 1 -- 172.21.15.111:0/3958288788 learned_addr learned my addr 172.21.15.111:0/3958288788 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f9cdb7fe700 1 -- 172.21.15.111:0/3958288788 >> v2:172.21.15.149:3300/0 conn(0x7f9cdc160e10 msgr2=0x7f9cdc0ad950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T12:59:20.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f9cdb7fe700 1 --2- 172.21.15.111:0/3958288788 >> v2:172.21.15.149:3300/0 conn(0x7f9cdc160e10 0x7f9cdc0ad950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f9cdb7fe700 1 -- 172.21.15.111:0/3958288788 >> v1:172.21.15.111:6789/0 conn(0x7f9cdc0ade90 legacy=0x7f9cdc06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f9cdb7fe700 1 -- 172.21.15.111:0/3958288788 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc165180 con 0x7f9cdc06a980 2023-11-28T12:59:20.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.812+0000 7fb54e8c7700 1 -- start start 2023-11-28T12:59:20.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7fb54e8c7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb54806a270 0x7fb54806a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7fb54e8c7700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb5480ccf40 0x7fb5480cd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7fb54e8c7700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb548165740 con 0x7fb548160e10 2023-11-28T12:59:20.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7fb54e8c7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54806ac30 con 0x7fb54806a270 2023-11-28T12:59:20.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7fb54e8c7700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5480ae060 con 0x7fb5480ccf40 2023-11-28T12:59:20.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7fb547fff700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7fb548160e10 0x7fb5480ad950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:42664/0 (socket says 172.21.15.111:42664) 2023-11-28T12:59:20.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7fb547fff700 1 -- 172.21.15.111:0/1398455654 learned_addr learned my addr 172.21.15.111:0/1398455654 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:20.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fb54ce64700 1 --2- 172.21.15.111:0/1398455654 >> v2:172.21.15.149:3300/0 conn(0x7fb5480ccf40 0x7fb5480cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fb5477fe700 1 --2- 172.21.15.111:0/1398455654 >> v2:172.21.15.111:3300/0 conn(0x7fb54806a270 0x7fb54806a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fb54ce64700 1 -- 172.21.15.111:0/1398455654 >> v2:172.21.15.111:3300/0 conn(0x7fb54806a270 msgr2=0x7fb54806a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fb54ce64700 1 --2- 172.21.15.111:0/1398455654 >> v2:172.21.15.111:3300/0 conn(0x7fb54806a270 0x7fb54806a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fb54ce64700 1 -- 172.21.15.111:0/1398455654 >> v1:172.21.15.149:6789/0 conn(0x7fb548160e10 legacy=0x7fb5480ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7fb54ce64700 1 -- 172.21.15.111:0/1398455654 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb548164bc0 con 0x7fb5480ccf40 2023-11-28T12:59:20.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7fb54ce64700 1 --2- 172.21.15.111:0/1398455654 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5480ccf40 0x7fb5480cd360 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb5480b4940 tx=0x7fb538009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f565d3de0adda606 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7fd6219b5700 1 -- 172.21.15.111:0/1623595913 >> v2:172.21.15.149:3300/0 conn(0x7fd61c070910 msgr2=0x7fd61c070d30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T12:59:20.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7fd6219b5700 1 --2- 172.21.15.111:0/1623595913 >> v2:172.21.15.149:3300/0 conn(0x7fd61c070910 0x7fd61c070d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7fd6209b3700 1 -- 172.21.15.111:0/1623595913 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783897950 0 0) 0x7fd61c071270 con 0x7fd61c158b70 2023-11-28T12:59:20.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7fd6219b5700 1 -- 172.21.15.111:0/1623595913 >> v1:172.21.15.149:6789/0 conn(0x7fd61c158b70 legacy=0x7fd61c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7fd6219b5700 1 -- 172.21.15.111:0/1623595913 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd61c0c9580 con 0x7fd61c06cc70 2023-11-28T12:59:20.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f41d8d2c700 1 Processor -- start 2023-11-28T12:59:20.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7f4ed2337700 1 -- start start 2023-11-28T12:59:20.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f4ed2337700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ecc06e730 0x7f4ecc0c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f4ed2337700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ecc0cb240 0x7f4ecc0c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f4ed2337700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ecc15d270 0x7f4ecc0af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f4ed2337700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc1589c0 con 0x7f4ecc0cb240 2023-11-28T12:59:20.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f4ed2337700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc1530c0 con 0x7f4ecc15d270 2023-11-28T12:59:20.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f4ed2337700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ecc0c59f0 con 0x7f4ecc06e730 2023-11-28T12:59:20.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f4ed08d4700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ecc15d270 0x7f4ecc0af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7fb546ffd700 1 -- 172.21.15.111:0/1398455654 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb538011830 con 0x7fb5480ccf40 2023-11-28T12:59:20.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7fb546ffd700 1 -- 172.21.15.111:0/1398455654 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb538011990 con 0x7fb5480ccf40 2023-11-28T12:59:20.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7fb546ffd700 1 -- 172.21.15.111:0/1398455654 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb538011d20 con 0x7fb5480ccf40 2023-11-28T12:59:20.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7f047659c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 0x7f04780c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7f047659c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 0x7f04780c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55088/0 (socket says 172.21.15.111:55088) 2023-11-28T12:59:20.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7f047659c700 1 -- 172.21.15.111:0/3208235393 learned_addr learned my addr 172.21.15.111:0/3208235393 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7f047659c700 1 -- 172.21.15.111:0/3208235393 >> v2:172.21.15.149:3300/0 conn(0x7f047815ce70 msgr2=0x7f047806a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7f047659c700 1 --2- 172.21.15.111:0/3208235393 >> v2:172.21.15.149:3300/0 conn(0x7f047815ce70 0x7f047806a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7f047659c700 1 -- 172.21.15.111:0/3208235393 >> v1:172.21.15.149:6789/0 conn(0x7f04780cdee0 legacy=0x7f047806c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.814+0000 7f047659c700 1 -- 172.21.15.111:0/3208235393 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04780cd120 con 0x7f047806a270 2023-11-28T12:59:20.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.815+0000 7f047659c700 1 --2- 172.21.15.111:0/3208235393 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 0x7f04780c9ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0470007a60 tx=0x7f0470010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9deaf8a263297400 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f047559a700 1 -- 172.21.15.111:0/3208235393 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470010270 con 0x7f047806a270 2023-11-28T12:59:20.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f047559a700 1 -- 172.21.15.111:0/3208235393 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0470019a60 con 0x7f047806a270 2023-11-28T12:59:20.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f047559a700 1 -- 172.21.15.111:0/3208235393 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470019df0 con 0x7f047806a270 2023-11-28T12:59:20.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f047cb70700 1 -- 172.21.15.111:0/3208235393 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 msgr2=0x7f04780c9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f047cb70700 1 --2- 172.21.15.111:0/3208235393 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 0x7f04780c9ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0470007a60 tx=0x7f0470010800 comp rx=0 tx=0).stop 2023-11-28T12:59:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f4ed08d4700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ecc15d270 0x7f4ecc0af3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:51260/0 (socket says 172.21.15.111:51260) 2023-11-28T12:59:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f4ed08d4700 1 -- 172.21.15.111:0/189492702 learned_addr learned my addr 172.21.15.111:0/189492702 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f4ecb7fe700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ecc0cb240 0x7f4ecc0c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f4ecb7fe700 1 -- 172.21.15.111:0/189492702 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ecc06e730 msgr2=0x7f4ecc0c54b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-28T12:59:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f4ecb7fe700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ecc06e730 0x7f4ecc0c54b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f4ecb7fe700 1 -- 172.21.15.111:0/189492702 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ecc15d270 msgr2=0x7f4ecc0af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f4ecb7fe700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ecc15d270 0x7f4ecc0af3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f4ecb7fe700 1 -- 172.21.15.111:0/189492702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb400a640 con 0x7f4ecc0cb240 2023-11-28T12:59:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f4ed08d4700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ecc15d270 0x7f4ecc0af3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:20.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f9cdb7fe700 1 --2- 172.21.15.111:0/3958288788 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cdc06a980 0x7f9cdc0ccf40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009e50 tx=0x7f9cd400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67a4a6df3701618 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f9cd9ffb700 1 -- 172.21.15.111:0/3958288788 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd400e070 con 0x7f9cdc06a980 2023-11-28T12:59:20.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f9cd9ffb700 1 -- 172.21.15.111:0/3958288788 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd400b9c0 con 0x7f9cdc06a980 2023-11-28T12:59:20.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f9cd9ffb700 1 -- 172.21.15.111:0/3958288788 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd400bd70 con 0x7f9cdc06a980 2023-11-28T12:59:20.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f9ce185f700 1 -- 172.21.15.111:0/3958288788 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cdc06a980 msgr2=0x7f9cdc0ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f9ce185f700 1 --2- 172.21.15.111:0/3958288788 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cdc06a980 0x7f9cdc0ccf40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009e50 tx=0x7f9cd400b330 comp rx=0 tx=0).stop 2023-11-28T12:59:20.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f4ecb7fe700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ecc0cb240 0x7f4ecc0c6390 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00c3c0 tx=0x7f4ebc00c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1398455654 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5480ccf40 msgr2=0x7fb5480cd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7fb54e8c7700 1 --2- 172.21.15.111:0/1398455654 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5480ccf40 0x7fb5480cd360 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb5480b4940 tx=0x7fb538009b20 comp rx=0 tx=0).stop 2023-11-28T12:59:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.816+0000 7f41d8d2c700 1 -- start start 2023-11-28T12:59:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7fd6219b5700 1 --2- 172.21.15.111:0/1623595913 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd61c06cc70 0x7fd61c06dab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd6180098f0 tx=0x7fd618009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ab16852bfab607f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f9ce185f700 1 -- 172.21.15.111:0/3958288788 shutdown_connections 2023-11-28T12:59:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f9ce185f700 1 --2- 172.21.15.111:0/3958288788 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cdc06a980 0x7f9cdc0ccf40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f9ce185f700 1 --2- 172.21.15.111:0/3958288788 >> v2:172.21.15.149:3300/0 conn(0x7f9cdc160e10 0x7f9cdc0ad950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.817+0000 7f9ce185f700 1 -- 172.21.15.111:0/3958288788 >> 172.21.15.111:0/3958288788 conn(0x7f9cdc0cb490 msgr2=0x7f9cdc0cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f9ce185f700 1 -- 172.21.15.111:0/3958288788 shutdown_connections 2023-11-28T12:59:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f9ce185f700 1 -- 172.21.15.111:0/3958288788 wait complete. 2023-11-28T12:59:20.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f047cb70700 1 -- 172.21.15.111:0/3208235393 shutdown_connections 2023-11-28T12:59:20.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f047cb70700 1 --2- 172.21.15.111:0/3208235393 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 0x7f04780c9ed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f047cb70700 1 --2- 172.21.15.111:0/3208235393 >> v2:172.21.15.149:3300/0 conn(0x7f047815ce70 0x7f047806a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f047cb70700 1 -- 172.21.15.111:0/3208235393 >> 172.21.15.111:0/3208235393 conn(0x7f04780cb490 msgr2=0x7f04780cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f41d8d2c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f41d40c43b0 0x7f41d40ad980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f41d8d2c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d40d1a00 con 0x7f41d40d13e0 2023-11-28T12:59:20.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f41d8d2c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d40cfa50 con 0x7f41d4160e20 2023-11-28T12:59:20.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f41d8d2c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d40cfd30 con 0x7f41d40c43b0 2023-11-28T12:59:20.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7fd6209b3700 1 -- 172.21.15.111:0/1623595913 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61800b8a0 con 0x7fd61c06cc70 2023-11-28T12:59:20.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f41d259c700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f41d4160e20 0x7f41d40c3ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:42690/0 (socket says 172.21.15.111:42690) 2023-11-28T12:59:20.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f41d259c700 1 -- 172.21.15.111:0/2286512237 learned_addr learned my addr 172.21.15.111:0/2286512237 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:20.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f41d1d9b700 1 --2- 172.21.15.111:0/2286512237 >> v2:172.21.15.111:3300/0 conn(0x7f41d40c43b0 0x7f41d40ad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f047cb70700 1 -- 172.21.15.111:0/3208235393 shutdown_connections 2023-11-28T12:59:20.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1398455654 shutdown_connections 2023-11-28T12:59:20.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7fb54e8c7700 1 --2- 172.21.15.111:0/1398455654 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5480ccf40 0x7fb5480cd360 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7fb54e8c7700 1 --2- 172.21.15.111:0/1398455654 >> v2:172.21.15.111:3300/0 conn(0x7fb54806a270 0x7fb54806a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1398455654 >> 172.21.15.111:0/1398455654 conn(0x7fb5480cb490 msgr2=0x7fb5480cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1398455654 shutdown_connections 2023-11-28T12:59:20.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f047cb70700 1 -- 172.21.15.111:0/3208235393 wait complete. 2023-11-28T12:59:20.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.820+0000 7f047cb70700 1 Processor -- start 2023-11-28T12:59:20.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.820+0000 7f047cb70700 1 -- start start 2023-11-28T12:59:20.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f4ec8ff9700 1 -- 172.21.15.111:0/189492702 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc0052d0 con 0x7f4ecc0cb240 2023-11-28T12:59:20.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f4ec8ff9700 1 -- 172.21.15.111:0/189492702 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc005450 con 0x7f4ecc0cb240 2023-11-28T12:59:20.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7f4ec8ff9700 1 -- 172.21.15.111:0/189492702 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc0038c0 con 0x7f4ecc0cb240 2023-11-28T12:59:20.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.819+0000 7f4ed2337700 1 -- 172.21.15.111:0/189492702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ecc0cd1d0 con 0x7f4ecc0cb240 2023-11-28T12:59:20.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.819+0000 7f4ed2337700 1 -- 172.21.15.111:0/189492702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ecc159fe0 con 0x7f4ecc0cb240 2023-11-28T12:59:20.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f4ed2337700 1 -- 172.21.15.111:0/189492702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ecc1595e0 con 0x7f4ecc0cb240 2023-11-28T12:59:20.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.820+0000 7f9ce185f700 1 Processor -- start 2023-11-28T12:59:20.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.819+0000 7f41d159a700 1 -- 172.21.15.111:0/2286512237 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249117204 0 0) 0x7f41d40cfa50 con 0x7f41d4160e20 2023-11-28T12:59:20.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.819+0000 7f41d159a700 1 -- 172.21.15.111:0/2286512237 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41bc0035b0 con 0x7f41d4160e20 2023-11-28T12:59:20.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.820+0000 7f41d1d9b700 1 -- 172.21.15.111:0/2286512237 >> v1:172.21.15.149:6789/0 conn(0x7f41d4160e20 legacy=0x7f41d40c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.820+0000 7f41d1d9b700 1 -- 172.21.15.111:0/2286512237 >> v1:172.21.15.111:6789/0 conn(0x7f41d40d13e0 legacy=0x7f41d40adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.820+0000 7f41d1d9b700 1 -- 172.21.15.111:0/2286512237 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d40d0010 con 0x7f41d40c43b0 2023-11-28T12:59:20.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.820+0000 7f41d1d9b700 1 --2- 172.21.15.111:0/2286512237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d40c43b0 0x7f41d40ad980 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f41c4009840 tx=0x7f41c4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10ba421025e0d84 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7fd6209b3700 1 -- 172.21.15.111:0/1623595913 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61800ba00 con 0x7fd61c06cc70 2023-11-28T12:59:20.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.818+0000 7fd6209b3700 1 -- 172.21.15.111:0/1623595913 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61800bd90 con 0x7fd61c06cc70 2023-11-28T12:59:20.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.819+0000 7fd623c19700 1 -- 172.21.15.111:0/1623595913 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd61c06cc70 msgr2=0x7fd61c06dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.819+0000 7fd623c19700 1 --2- 172.21.15.111:0/1623595913 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd61c06cc70 0x7fd61c06dab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd6180098f0 tx=0x7fd618009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:20.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.819+0000 7fd623c19700 1 -- 172.21.15.111:0/1623595913 shutdown_connections 2023-11-28T12:59:20.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.819+0000 7fd623c19700 1 --2- 172.21.15.111:0/1623595913 >> v2:172.21.15.149:3300/0 conn(0x7fd61c070910 0x7fd61c070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.819+0000 7fd623c19700 1 --2- 172.21.15.111:0/1623595913 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd61c06cc70 0x7fd61c06dab0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.819+0000 7fd623c19700 1 -- 172.21.15.111:0/1623595913 >> 172.21.15.111:0/1623595913 conn(0x7fd61c0cb490 msgr2=0x7fd61c0cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.819+0000 7fd623c19700 1 -- 172.21.15.111:0/1623595913 shutdown_connections 2023-11-28T12:59:20.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.820+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1398455654 wait complete. 2023-11-28T12:59:20.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.819+0000 7fd623c19700 1 -- 172.21.15.111:0/1623595913 wait complete. 2023-11-28T12:59:20.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7fd623c19700 1 Processor -- start 2023-11-28T12:59:20.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7fb54e8c7700 1 Processor -- start 2023-11-28T12:59:20.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.820+0000 7f9ce185f700 1 -- start start 2023-11-28T12:59:20.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f9ce185f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9cdc06a980 0x7f9cdc148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f9ce185f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cdc0ade90 0x7f9cdc150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f9ce185f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cdc160e10 0x7f9cdc152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f9ce185f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc1515b0 con 0x7f9cdc0ade90 2023-11-28T12:59:20.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f9ce185f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc1624d0 con 0x7f9cdc06a980 2023-11-28T12:59:20.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f9ce185f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc150d10 con 0x7f9cdc160e10 2023-11-28T12:59:20.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7fb54e8c7700 1 -- start start 2023-11-28T12:59:20.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f41d159a700 1 -- 172.21.15.111:0/2286512237 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c4004230 con 0x7f41d40c43b0 2023-11-28T12:59:20.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f9cda7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cdc0ade90 0x7f9cdc150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f9cda7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cdc0ade90 0x7f9cdc150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33374/0 (socket says 172.21.15.111:33374) 2023-11-28T12:59:20.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f9cda7fc700 1 -- 172.21.15.111:0/4241937954 learned_addr learned my addr 172.21.15.111:0/4241937954 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f9cdaffd700 1 --2- 172.21.15.111:0/4241937954 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9cdc06a980 0x7f9cdc148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f9cda7fc700 1 -- 172.21.15.111:0/4241937954 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cdc160e10 msgr2=0x7f9cdc152670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T12:59:20.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f9cda7fc700 1 --2- 172.21.15.111:0/4241937954 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cdc160e10 0x7f9cdc152670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f9cda7fc700 1 -- 172.21.15.111:0/4241937954 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9cdc06a980 msgr2=0x7f9cdc148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f9cda7fc700 1 --2- 172.21.15.111:0/4241937954 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9cdc06a980 0x7f9cdc148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f9cda7fc700 1 -- 172.21.15.111:0/4241937954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd4009b30 con 0x7f9cdc0ade90 2023-11-28T12:59:20.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f9cda7fc700 1 --2- 172.21.15.111:0/4241937954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cdc0ade90 0x7f9cdc150370 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00c4e0 tx=0x7f9ccc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7fd623c19700 1 -- start start 2023-11-28T12:59:20.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7fd623c19700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd61c06cc70 0x7fd61c157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7fd623c19700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd61c070910 0x7fd61c0c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7fd623c19700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd61c158b70 0x7fd61c1724e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7fd623c19700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c16c100 con 0x7fd61c070910 2023-11-28T12:59:20.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7fd623c19700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c16c3e0 con 0x7fd61c06cc70 2023-11-28T12:59:20.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7fd623c19700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c16c6c0 con 0x7fd61c158b70 2023-11-28T12:59:20.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fd6219b5700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd61c06cc70 0x7fd61c157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fd6219b5700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd61c06cc70 0x7fd61c157ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:51282/0 (socket says 172.21.15.111:51282) 2023-11-28T12:59:20.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fd6219b5700 1 -- 172.21.15.111:0/2355794779 learned_addr learned my addr 172.21.15.111:0/2355794779 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fb54e8c7700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb54806a270 0x7fb5481611e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fb54e8c7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb5480ccf40 0x7fb548151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fb54e8c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb548160e10 0x7fb548152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fb54e8c7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5481506c0 con 0x7fb548160e10 2023-11-28T12:59:20.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fb54e8c7700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5480aedd0 con 0x7fb54806a270 2023-11-28T12:59:20.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fb54e8c7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5480af0b0 con 0x7fb5480ccf40 2023-11-28T12:59:20.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fd6221b6700 1 --2- 172.21.15.111:0/2355794779 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd61c158b70 0x7fd61c1724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fd6219b5700 1 -- 172.21.15.111:0/2355794779 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd61c158b70 msgr2=0x7fd61c1724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fd6219b5700 1 --2- 172.21.15.111:0/2355794779 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd61c158b70 0x7fd61c1724e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fd6219b5700 1 -- 172.21.15.111:0/2355794779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd61c070910 msgr2=0x7fd61c0c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fd6219b5700 1 --2- 172.21.15.111:0/2355794779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd61c070910 0x7fd61c0c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fd6219b5700 1 -- 172.21.15.111:0/2355794779 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd618009550 con 0x7fd61c06cc70 2023-11-28T12:59:20.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f047cb70700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 0x7f0478152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f047cb70700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04780cdee0 0x7f047806d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f047cb70700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f047815ce70 0x7f0478089400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f047cb70700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0478150340 con 0x7f04780cdee0 2023-11-28T12:59:20.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f047cb70700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047814f6c0 con 0x7f047815ce70 2023-11-28T12:59:20.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.821+0000 7f047cb70700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047814f9a0 con 0x7f047806a270 2023-11-28T12:59:20.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f047659c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 0x7f0478152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f047659c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 0x7f0478152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55124/0 (socket says 172.21.15.111:55124) 2023-11-28T12:59:20.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f047659c700 1 -- 172.21.15.111:0/3135548943 learned_addr learned my addr 172.21.15.111:0/3135548943 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f0476d9d700 1 --2- 172.21.15.111:0/3135548943 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f047815ce70 0x7f0478089400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f047659c700 1 -- 172.21.15.111:0/3135548943 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f047815ce70 msgr2=0x7f0478089400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f047659c700 1 --2- 172.21.15.111:0/3135548943 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f047815ce70 0x7f0478089400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f047659c700 1 -- 172.21.15.111:0/3135548943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04780cdee0 msgr2=0x7f047806d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f047659c700 1 --2- 172.21.15.111:0/3135548943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04780cdee0 0x7f047806d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f047659c700 1 -- 172.21.15.111:0/3135548943 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0470007790 con 0x7f047806a270 2023-11-28T12:59:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.822+0000 7f047659c700 1 --2- 172.21.15.111:0/3135548943 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 0x7f0478152180 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0470007230 tx=0x7f0470004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7f0466ffd700 1 -- 172.21.15.111:0/3135548943 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470016070 con 0x7f047806a270 2023-11-28T12:59:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7f0466ffd700 1 -- 172.21.15.111:0/3135548943 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0470007e80 con 0x7f047806a270 2023-11-28T12:59:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7f0466ffd700 1 -- 172.21.15.111:0/3135548943 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04700194c0 con 0x7f047806a270 2023-11-28T12:59:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7f047cb70700 1 -- 172.21.15.111:0/3135548943 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04780cd6e0 con 0x7f047806a270 2023-11-28T12:59:20.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7f047cb70700 1 -- 172.21.15.111:0/3135548943 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04780cd400 con 0x7f047806a270 2023-11-28T12:59:20.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fd6219b5700 1 --2- 172.21.15.111:0/2355794779 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd61c06cc70 0x7fd61c157ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd618000c00 tx=0x7fd618004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.823+0000 7fd6127fc700 1 -- 172.21.15.111:0/2355794779 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61801a8f0 con 0x7fd61c06cc70 2023-11-28T12:59:20.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7fd6127fc700 1 -- 172.21.15.111:0/2355794779 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61801aa50 con 0x7fd61c06cc70 2023-11-28T12:59:20.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7fd6127fc700 1 -- 172.21.15.111:0/2355794779 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61801ae00 con 0x7fd61c06cc70 2023-11-28T12:59:20.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7f047cb70700 1 -- 172.21.15.111:0/3135548943 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045c0052f0 con 0x7f047806a270 2023-11-28T12:59:20.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7feceebba700 1 Processor -- start 2023-11-28T12:59:20.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7fd623c19700 1 -- 172.21.15.111:0/2355794779 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61c0c9b40 con 0x7fd61c06cc70 2023-11-28T12:59:20.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7fd623c19700 1 -- 172.21.15.111:0/2355794779 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61c0c9860 con 0x7fd61c06cc70 2023-11-28T12:59:20.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.825+0000 7f41d159a700 1 -- 172.21.15.111:0/2286512237 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c4004390 con 0x7f41d40c43b0 2023-11-28T12:59:20.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.825+0000 7f41d159a700 1 -- 172.21.15.111:0/2286512237 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c40036a0 con 0x7f41d40c43b0 2023-11-28T12:59:20.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.825+0000 7fd623c19700 1 -- 172.21.15.111:0/2355794779 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61c0c9e20 con 0x7fd61c06cc70 2023-11-28T12:59:20.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7fb547fff700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb54806a270 0x7fb5481611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7fb5477fe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb5480ccf40 0x7fb548151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7fb5477fe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb5480ccf40 0x7fb548151d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55136/0 (socket says 172.21.15.111:55136) 2023-11-28T12:59:20.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7fb5477fe700 1 -- 172.21.15.111:0/1561002981 learned_addr learned my addr 172.21.15.111:0/1561002981 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7fb5477fe700 1 -- 172.21.15.111:0/1561002981 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb54806a270 msgr2=0x7fb5481611e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7fb5477fe700 1 --2- 172.21.15.111:0/1561002981 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb54806a270 0x7fb5481611e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7fb5477fe700 1 -- 172.21.15.111:0/1561002981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb548160e10 msgr2=0x7fb548152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7fb5477fe700 1 --2- 172.21.15.111:0/1561002981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb548160e10 0x7fb548152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.824+0000 7fb5477fe700 1 -- 172.21.15.111:0/1561002981 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5380094a0 con 0x7fb5480ccf40 2023-11-28T12:59:20.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.825+0000 7fb5477fe700 1 --2- 172.21.15.111:0/1561002981 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb5480ccf40 0x7fb548151d20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00e810 tx=0x7fb53c00eb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7feceebba700 1 -- start start 2023-11-28T12:59:20.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7f41d8d2c700 1 -- 172.21.15.111:0/2286512237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d40c43b0 msgr2=0x7f41d40ad980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7f41d8d2c700 1 --2- 172.21.15.111:0/2286512237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d40c43b0 0x7f41d40ad980 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f41c4009840 tx=0x7f41c4009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:20.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7f0466ffd700 1 -- 172.21.15.111:0/3135548943 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f047001f020 con 0x7f047806a270 2023-11-28T12:59:20.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7f41d8d2c700 1 -- 172.21.15.111:0/2286512237 shutdown_connections 2023-11-28T12:59:20.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7f41d8d2c700 1 --2- 172.21.15.111:0/2286512237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d40c43b0 0x7f41d40ad980 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7f41d8d2c700 1 -- 172.21.15.111:0/2286512237 >> 172.21.15.111:0/2286512237 conn(0x7f41d409a020 msgr2=0x7f41d40d0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7fc4f5981700 1 Processor -- start 2023-11-28T12:59:20.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7f4ec8ff9700 1 -- 172.21.15.111:0/189492702 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ebc019030 con 0x7f4ecc0cb240 2023-11-28T12:59:20.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7f4ec8ff9700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4eac068390 0x7f4eac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7f4ec8ff9700 1 -- 172.21.15.111:0/189492702 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f4ebc085740 con 0x7f4ecc0cb240 2023-11-28T12:59:20.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7fc4f5981700 1 -- start start 2023-11-28T12:59:20.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7f41d8d2c700 1 -- 172.21.15.111:0/2286512237 shutdown_connections 2023-11-28T12:59:20.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7f0466ffd700 1 --2- 172.21.15.111:0/3135548943 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f04580683c0 0x7f045806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7f0466ffd700 1 -- 172.21.15.111:0/3135548943 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0470086c10 con 0x7f047806a270 2023-11-28T12:59:20.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7f4ecbfff700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4eac068390 0x7f4eac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7f0475d9b700 1 --2- 172.21.15.111:0/3135548943 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f04580683c0 0x7f045806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.826+0000 7fd6127fc700 1 -- 172.21.15.111:0/2355794779 <== mon.1 v2:172.21.15.149:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd61800f050 con 0x7fd61c06cc70 2023-11-28T12:59:20.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7fd6127fc700 1 --2- 172.21.15.111:0/2355794779 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6000682f0 0x7fd60006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7f41d8d2c700 1 -- 172.21.15.111:0/2286512237 wait complete. 2023-11-28T12:59:20.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7feceebba700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fece80cd460 0x7fece806c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7feceebba700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fece80d0d60 0x7fece80d11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7feceebba700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece80b88e0 con 0x7fece80b8510 2023-11-28T12:59:20.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7feceebba700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece80cdbd0 con 0x7fece80d0d60 2023-11-28T12:59:20.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7feceebba700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece80d02d0 con 0x7fece80cd460 2023-11-28T12:59:20.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7fd6127fc700 1 -- 172.21.15.111:0/2355794779 <== mon.1 v2:172.21.15.149:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd618087d00 con 0x7fd61c06cc70 2023-11-28T12:59:20.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7fecec956700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fece80b8510 0x7fece806b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34176/0 (socket says 172.21.15.111:34176) 2023-11-28T12:59:20.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7fecec956700 1 -- 172.21.15.111:0/3618994894 learned_addr learned my addr 172.21.15.111:0/3618994894 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:20.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7fece7fff700 1 --2- 172.21.15.111:0/3618994894 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fece80cd460 0x7fece806c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7feced157700 1 --2- 172.21.15.111:0/3618994894 >> v2:172.21.15.149:3300/0 conn(0x7fece80d0d60 0x7fece80d11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7fc4f5981700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc4f00ce850 0x7fc4f00cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7fc4f5981700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f00ad8a0 con 0x7fc4f00afba0 2023-11-28T12:59:20.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fc4f5981700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f00cf160 con 0x7fc4f015a920 2023-11-28T12:59:20.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fc4f5981700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f00adaf0 con 0x7fc4f00ce850 2023-11-28T12:59:20.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.827+0000 7fd6211b4700 1 --2- 172.21.15.111:0/2355794779 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6000682f0 0x7fd60006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fece7fff700 1 -- 172.21.15.111:0/3618994894 >> v2:172.21.15.149:3300/0 conn(0x7fece80d0d60 msgr2=0x7fece80d11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:20.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fece7fff700 1 --2- 172.21.15.111:0/3618994894 >> v2:172.21.15.149:3300/0 conn(0x7fece80d0d60 0x7fece80d11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fece7fff700 1 -- 172.21.15.111:0/3618994894 >> v1:172.21.15.111:6789/0 conn(0x7fece80b8510 legacy=0x7fece806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fece7fff700 1 -- 172.21.15.111:0/3618994894 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece80cd8f0 con 0x7fece80cd460 2023-11-28T12:59:20.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fc4e7fff700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7fc4f015a920 0x7fc4f00cb2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:42726/0 (socket says 172.21.15.111:42726) 2023-11-28T12:59:20.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fc4e7fff700 1 -- 172.21.15.111:0/332453057 learned_addr learned my addr 172.21.15.111:0/332453057 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:20.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fece7fff700 1 --2- 172.21.15.111:0/3618994894 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fece80cd460 0x7fece806c080 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009840 tx=0x7fecdc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=646b5c8e77675bf1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7f0475d9b700 1 --2- 172.21.15.111:0/3135548943 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f04580683c0 0x7f045806a870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0478149b00 tx=0x7f0468006c80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fc4eeffd700 1 --2- 172.21.15.111:0/332453057 >> v2:172.21.15.111:3300/0 conn(0x7fc4f00ce850 0x7fc4f00cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7f4ecbfff700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4eac068390 0x7f4eac06a840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4eb400ad60 tx=0x7f4eb4012000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fd6211b4700 1 --2- 172.21.15.111:0/2355794779 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6000682f0 0x7fd60006a7a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd60c000f60 tx=0x7fd60c007a40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fc4eeffd700 1 -- 172.21.15.111:0/332453057 >> v1:172.21.15.149:6789/0 conn(0x7fc4f015a920 legacy=0x7fc4f00cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fc4eeffd700 1 -- 172.21.15.111:0/332453057 >> v1:172.21.15.111:6789/0 conn(0x7fc4f00afba0 legacy=0x7fc4f00cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fc4eeffd700 1 -- 172.21.15.111:0/332453057 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f00addd0 con 0x7fc4f00ce850 2023-11-28T12:59:20.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.828+0000 7fc4eeffd700 1 --2- 172.21.15.111:0/332453057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f00ce850 0x7fc4f00cec20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8009840 tx=0x7fc4d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66829b2077242a4a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7f41d8d2c700 1 Processor -- start 2023-11-28T12:59:20.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7f41d8d2c700 1 -- start start 2023-11-28T12:59:20.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7f41d8d2c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d40c43b0 0x7f41d40c51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7f41d8d2c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d40d13e0 0x7f41d40c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7f41d8d2c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41d4160e20 0x7f41d41f5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7f41d8d2c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d40c5d80 con 0x7f41d40d13e0 2023-11-28T12:59:20.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7f41d8d2c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d4162dd0 con 0x7f41d4160e20 2023-11-28T12:59:20.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7f41d8d2c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d41627c0 con 0x7f41d40c43b0 2023-11-28T12:59:20.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7f41d1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d40d13e0 0x7f41d40c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7f4ec8ff9700 1 -- 172.21.15.111:0/189492702 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ebc051c40 con 0x7f4ecc0cb240 2023-11-28T12:59:20.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7f41d1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d40d13e0 0x7f41d40c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33414/0 (socket says 172.21.15.111:33414) 2023-11-28T12:59:20.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7f41d1d9b700 1 -- 172.21.15.111:0/3476788304 learned_addr learned my addr 172.21.15.111:0/3476788304 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7fc4ee7fc700 1 -- 172.21.15.111:0/332453057 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8011830 con 0x7fc4f00ce850 2023-11-28T12:59:20.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7fc4ee7fc700 1 -- 172.21.15.111:0/332453057 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d8011990 con 0x7fc4f00ce850 2023-11-28T12:59:20.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7fc4ee7fc700 1 -- 172.21.15.111:0/332453057 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8011d20 con 0x7fc4f00ce850 2023-11-28T12:59:20.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7fb544ff9700 1 -- 172.21.15.111:0/1561002981 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c013610 con 0x7fb5480ccf40 2023-11-28T12:59:20.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7fb544ff9700 1 -- 172.21.15.111:0/1561002981 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c013790 con 0x7fb5480ccf40 2023-11-28T12:59:20.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7fb544ff9700 1 -- 172.21.15.111:0/1561002981 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c013b40 con 0x7fb5480ccf40 2023-11-28T12:59:20.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1561002981 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb548165180 con 0x7fb5480ccf40 2023-11-28T12:59:20.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7f41d259c700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d40c43b0 0x7f41d40c51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7f41d1d9b700 1 -- 172.21.15.111:0/3476788304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d40c43b0 msgr2=0x7f41d40c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1561002981 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb548164ea0 con 0x7fb5480ccf40 2023-11-28T12:59:20.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7fece77fe700 1 -- 172.21.15.111:0/3618994894 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc011830 con 0x7fece80cd460 2023-11-28T12:59:20.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7fece77fe700 1 -- 172.21.15.111:0/3618994894 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecdc011990 con 0x7fece80cd460 2023-11-28T12:59:20.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7fece77fe700 1 -- 172.21.15.111:0/3618994894 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc011d20 con 0x7fece80cd460 2023-11-28T12:59:20.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.829+0000 7f9cc3fff700 1 -- 172.21.15.111:0/4241937954 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc0052d0 con 0x7f9cdc0ade90 2023-11-28T12:59:20.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7f9cc3fff700 1 -- 172.21.15.111:0/4241937954 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ccc005430 con 0x7f9cdc0ade90 2023-11-28T12:59:20.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7f9ce185f700 1 -- 172.21.15.111:0/4241937954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cdc164800 con 0x7f9cdc0ade90 2023-11-28T12:59:20.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7f9ce185f700 1 -- 172.21.15.111:0/4241937954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cdc165460 con 0x7f9cdc0ade90 2023-11-28T12:59:20.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7fc4f5981700 1 -- 172.21.15.111:0/332453057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f00ce850 msgr2=0x7fc4f00cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7fc4f5981700 1 --2- 172.21.15.111:0/332453057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f00ce850 0x7fc4f00cec20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8009840 tx=0x7fc4d8009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:20.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7f9cc3fff700 1 -- 172.21.15.111:0/4241937954 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc0038a0 con 0x7f9cdc0ade90 2023-11-28T12:59:20.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7fc4f5981700 1 -- 172.21.15.111:0/332453057 shutdown_connections 2023-11-28T12:59:20.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7fc4f5981700 1 --2- 172.21.15.111:0/332453057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f00ce850 0x7fc4f00cec20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7fc4f5981700 1 -- 172.21.15.111:0/332453057 >> 172.21.15.111:0/332453057 conn(0x7fc4f009a040 msgr2=0x7fc4f00d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7fd6127fc700 1 -- 172.21.15.111:0/2355794779 <== mon.1 v2:172.21.15.149:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd618054200 con 0x7fd61c06cc70 2023-11-28T12:59:20.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7feceebba700 1 -- 172.21.15.111:0/3618994894 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fece80cd460 msgr2=0x7fece806c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7feceebba700 1 --2- 172.21.15.111:0/3618994894 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fece80cd460 0x7fece806c080 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009840 tx=0x7fecdc009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:20.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7f41d1d9b700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d40c43b0 0x7f41d40c51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7f41d1d9b700 1 -- 172.21.15.111:0/3476788304 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41d4160e20 msgr2=0x7f41d41f5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7f41d1d9b700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41d4160e20 0x7f41d41f5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7f41d1d9b700 1 -- 172.21.15.111:0/3476788304 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c40094a0 con 0x7f41d40d13e0 2023-11-28T12:59:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7f41d259c700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d40c43b0 0x7f41d40c51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.830+0000 7f41d1d9b700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d40d13e0 0x7f41d40c63f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f41c4000c00 tx=0x7f41c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7f41c2ffd700 1 -- 172.21.15.111:0/3476788304 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c4011630 con 0x7f41d40d13e0 2023-11-28T12:59:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7f41c2ffd700 1 -- 172.21.15.111:0/3476788304 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c401e070 con 0x7f41d40d13e0 2023-11-28T12:59:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7f41d8d2c700 1 -- 172.21.15.111:0/3476788304 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d40d05d0 con 0x7f41d40d13e0 2023-11-28T12:59:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7f41c2ffd700 1 -- 172.21.15.111:0/3476788304 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c40119b0 con 0x7f41d40d13e0 2023-11-28T12:59:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7f41d8d2c700 1 -- 172.21.15.111:0/3476788304 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d40d02f0 con 0x7f41d40d13e0 2023-11-28T12:59:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7feceebba700 1 -- 172.21.15.111:0/3618994894 shutdown_connections 2023-11-28T12:59:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7feceebba700 1 --2- 172.21.15.111:0/3618994894 >> v2:172.21.15.149:3300/0 conn(0x7fece80d0d60 0x7fece80d11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7feceebba700 1 --2- 172.21.15.111:0/3618994894 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fece80cd460 0x7fece806c080 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7feceebba700 1 -- 172.21.15.111:0/3618994894 >> 172.21.15.111:0/3618994894 conn(0x7fece80719a0 msgr2=0x7fece8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:20.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7fc4f5981700 1 -- 172.21.15.111:0/332453057 shutdown_connections 2023-11-28T12:59:20.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7fc4f5981700 1 -- 172.21.15.111:0/332453057 wait complete. 2023-11-28T12:59:20.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.831+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1561002981 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb548165460 con 0x7fb5480ccf40 2023-11-28T12:59:20.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.832+0000 7f9ce185f700 1 -- 172.21.15.111:0/4241937954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cbc0052f0 con 0x7f9cdc0ade90 2023-11-28T12:59:20.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.832+0000 7fb544ff9700 1 -- 172.21.15.111:0/1561002981 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb53c023070 con 0x7fb5480ccf40 2023-11-28T12:59:20.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.833+0000 7fc4f5981700 1 Processor -- start 2023-11-28T12:59:20.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.833+0000 7feceebba700 1 -- 172.21.15.111:0/3618994894 shutdown_connections 2023-11-28T12:59:20.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.833+0000 7fc4f5981700 1 -- start start 2023-11-28T12:59:20.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.833+0000 7fb544ff9700 1 --2- 172.21.15.111:0/1561002981 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb5340682f0 0x7fb53406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.833+0000 7f41d8d2c700 1 -- 172.21.15.111:0/3476788304 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d41611f0 con 0x7f41d40d13e0 2023-11-28T12:59:20.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.833+0000 7fb544ff9700 1 -- 172.21.15.111:0/1561002981 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb53c085b40 con 0x7fb5480ccf40 2023-11-28T12:59:20.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.833+0000 7f0466ffd700 1 -- 172.21.15.111:0/3135548943 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0470053110 con 0x7f047806a270 2023-11-28T12:59:20.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.833+0000 7feceebba700 1 -- 172.21.15.111:0/3618994894 wait complete. 2023-11-28T12:59:20.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.833+0000 7fb547fff700 1 --2- 172.21.15.111:0/1561002981 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb5340682f0 0x7fb53406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fb547fff700 1 --2- 172.21.15.111:0/1561002981 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb5340682f0 0x7fb53406a7a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb540005c90 tx=0x7fb540009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4f5981700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4f00afba0 0x7fc4f00c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4f5981700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f00ce850 0x7fc4f006fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4f5981700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4f0070290 0x7fc4f00b2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4f5981700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f0070730 con 0x7fc4f00ce850 2023-11-28T12:59:20.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4f5981700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f0153f60 con 0x7fc4f0070290 2023-11-28T12:59:20.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4f5981700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f0154200 con 0x7fc4f00afba0 2023-11-28T12:59:20.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4e7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f00ce850 0x7fc4f006fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4e7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f00ce850 0x7fc4f006fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33422/0 (socket says 172.21.15.111:33422) 2023-11-28T12:59:20.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4e7fff700 1 -- 172.21.15.111:0/1182226262 learned_addr learned my addr 172.21.15.111:0/1182226262 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4eeffd700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4f00afba0 0x7fc4f00c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4ef7fe700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4f0070290 0x7fc4f00b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4e7fff700 1 -- 172.21.15.111:0/1182226262 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4f00afba0 msgr2=0x7fc4f00c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4e7fff700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4f00afba0 0x7fc4f00c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4e7fff700 1 -- 172.21.15.111:0/1182226262 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4f0070290 msgr2=0x7fc4f00b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4e7fff700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4f0070290 0x7fc4f00b2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.834+0000 7fc4e7fff700 1 -- 172.21.15.111:0/1182226262 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d80094a0 con 0x7fc4f00ce850 2023-11-28T12:59:20.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.835+0000 7fc4e7fff700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f00ce850 0x7fc4f006fd50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc4e000c480 tx=0x7fc4e000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.835+0000 7f41c2ffd700 1 -- 172.21.15.111:0/3476788304 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f41c4017020 con 0x7f41d40d13e0 2023-11-28T12:59:20.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.835+0000 7feceebba700 1 Processor -- start 2023-11-28T12:59:20.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.835+0000 7feceebba700 1 -- start start 2023-11-28T12:59:20.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.835+0000 7f41c2ffd700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41b4068340 0x7f41b406a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.835+0000 7f41d259c700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41b4068340 0x7f41b406a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.835+0000 7f41c2ffd700 1 -- 172.21.15.111:0/3476788304 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f41c4087510 con 0x7f41d40d13e0 2023-11-28T12:59:20.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.836+0000 7f41d259c700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41b4068340 0x7f41b406a7f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f41c8000f60 tx=0x7f41c8007e50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.836+0000 7feceebba700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fece80b8510 0x7fece81ff9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.836+0000 7feceebba700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fece80cd460 0x7fece81f1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.836+0000 7feceebba700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fece80d0d60 0x7fece81f3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.836+0000 7feceebba700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece81f39b0 con 0x7fece80b8510 2023-11-28T12:59:20.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.836+0000 7feceebba700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece81f2ae0 con 0x7fece80d0d60 2023-11-28T12:59:20.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.836+0000 7feceebba700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece81ffee0 con 0x7fece80cd460 2023-11-28T12:59:20.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.836+0000 7fecec956700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fece80b8510 0x7fece81ff9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.836+0000 7fecec956700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fece80b8510 0x7fece81ff9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33428/0 (socket says 172.21.15.111:33428) 2023-11-28T12:59:20.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.836+0000 7fecec956700 1 -- 172.21.15.111:0/2481966096 learned_addr learned my addr 172.21.15.111:0/2481966096 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:20.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.836+0000 7feced157700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fece80d0d60 0x7fece81f3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.836+0000 7fece7fff700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fece80cd460 0x7fece81f1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.837+0000 7fecec956700 1 -- 172.21.15.111:0/2481966096 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fece80cd460 msgr2=0x7fece81f1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.837+0000 7fecec956700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fece80cd460 0x7fece81f1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.837+0000 7fecec956700 1 -- 172.21.15.111:0/2481966096 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fece80d0d60 msgr2=0x7fece81f3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:20.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.837+0000 7fecec956700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fece80d0d60 0x7fece81f3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:20.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.837+0000 7fecec956700 1 -- 172.21.15.111:0/2481966096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecdc0094a0 con 0x7fece80b8510 2023-11-28T12:59:20.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.837+0000 7feced157700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fece80d0d60 0x7fece81f3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:20.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.837+0000 7fecec956700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fece80b8510 0x7fece81ff9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fecd800c520 tx=0x7fecd800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.838+0000 7fb544ff9700 1 -- 172.21.15.111:0/1561002981 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb53c051fc0 con 0x7fb5480ccf40 2023-11-28T12:59:20.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.841+0000 7fece57fa700 1 -- 172.21.15.111:0/2481966096 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8005d70 con 0x7fece80b8510 2023-11-28T12:59:20.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.841+0000 7fc4ecff9700 1 -- 172.21.15.111:0/1182226262 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0005d70 con 0x7fc4f00ce850 2023-11-28T12:59:20.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.841+0000 7feceebba700 1 -- 172.21.15.111:0/2481966096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece80cfff0 con 0x7fece80b8510 2023-11-28T12:59:20.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.841+0000 7feceebba700 1 -- 172.21.15.111:0/2481966096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece80cfd10 con 0x7fece80b8510 2023-11-28T12:59:20.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.843+0000 7feceebba700 1 -- 172.21.15.111:0/2481966096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece81f3c20 con 0x7fece80b8510 2023-11-28T12:59:20.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.844+0000 7f9cc3fff700 1 -- 172.21.15.111:0/4241937954 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ccc019030 con 0x7f9cdc0ade90 2023-11-28T12:59:20.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.844+0000 7f9cc3fff700 1 --2- 172.21.15.111:0/4241937954 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9cc80682f0 0x7f9cc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.844+0000 7f9cc3fff700 1 -- 172.21.15.111:0/4241937954 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9ccc0856e0 con 0x7f9cdc0ade90 2023-11-28T12:59:20.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.844+0000 7f9cdaffd700 1 --2- 172.21.15.111:0/4241937954 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9cc80682f0 0x7f9cc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.845+0000 7f9cdaffd700 1 --2- 172.21.15.111:0/4241937954 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9cc80682f0 0x7f9cc806a7a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0c8f00 tx=0x7f9cd0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.846+0000 7fc4ecff9700 1 -- 172.21.15.111:0/1182226262 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e000ce90 con 0x7fc4f00ce850 2023-11-28T12:59:20.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.846+0000 7fece57fa700 1 -- 172.21.15.111:0/2481966096 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd8001260 con 0x7fece80b8510 2023-11-28T12:59:20.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.846+0000 7fece57fa700 1 -- 172.21.15.111:0/2481966096 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8007270 con 0x7fece80b8510 2023-11-28T12:59:20.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.846+0000 7f41c2ffd700 1 -- 172.21.15.111:0/3476788304 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f41c4053b20 con 0x7f41d40d13e0 2023-11-28T12:59:20.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.846+0000 7fc4ecff9700 1 -- 172.21.15.111:0/1182226262 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e00072d0 con 0x7fc4f00ce850 2023-11-28T12:59:20.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.847+0000 7fece57fa700 1 -- 172.21.15.111:0/2481966096 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fecd8019030 con 0x7fece80b8510 2023-11-28T12:59:20.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.848+0000 7fece57fa700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fecd40682f0 0x7fecd406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.849+0000 7fc4f5981700 1 -- 172.21.15.111:0/1182226262 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f00ae390 con 0x7fc4f00ce850 2023-11-28T12:59:20.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.849+0000 7fc4f5981700 1 -- 172.21.15.111:0/1182226262 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f00ae0b0 con 0x7fc4f00ce850 2023-11-28T12:59:20.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.850+0000 7fc4f5981700 1 -- 172.21.15.111:0/1182226262 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f00ae670 con 0x7fc4f00ce850 2023-11-28T12:59:20.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.853+0000 7fece7fff700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fecd40682f0 0x7fecd406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.853+0000 7fece7fff700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fecd40682f0 0x7fecd406a7a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009b90 tx=0x7fecdc01d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.858+0000 7f9cc3fff700 1 -- 172.21.15.111:0/4241937954 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ccc051c90 con 0x7f9cdc0ade90 2023-11-28T12:59:20.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.862+0000 7fc4ecff9700 1 -- 172.21.15.111:0/1182226262 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc4e0019030 con 0x7fc4f00ce850 2023-11-28T12:59:20.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.862+0000 7fc4ecff9700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4d00682f0 0x7fc4d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:20.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.861+0000 7fece57fa700 1 -- 172.21.15.111:0/2481966096 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fecd8055b70 con 0x7fece80b8510 2023-11-28T12:59:20.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.863+0000 7fece57fa700 1 -- 172.21.15.111:0/2481966096 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fecd8051eb0 con 0x7fece80b8510 2023-11-28T12:59:20.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.863+0000 7fc4eeffd700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4d00682f0 0x7fc4d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:20.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.863+0000 7fc4ecff9700 1 -- 172.21.15.111:0/1182226262 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc4e005d800 con 0x7fc4f00ce850 2023-11-28T12:59:20.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.864+0000 7fc4eeffd700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4d00682f0 0x7fc4d006a7a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80090b0 tx=0x7fc4d8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:20.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:20.871+0000 7fc4ecff9700 1 -- 172.21.15.111:0/1182226262 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc4e0051b60 con 0x7fc4f00ce850 2023-11-28T12:59:21.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.210+0000 7f047cb70700 1 -- 172.21.15.111:0/3135548943 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f045c005870 con 0x7f047806a270 2023-11-28T12:59:21.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.211+0000 7f0466ffd700 1 -- 172.21.15.111:0/3135548943 <== mon.2 v2:172.21.15.149:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0470056730 con 0x7f047806a270 2023-11-28T12:59:21.215 INFO:teuthology.orchestra.run.smithi111.stdout:55834574856 2023-11-28T12:59:21.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.215+0000 7f047cb70700 1 -- 172.21.15.111:0/3135548943 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f04580683c0 msgr2=0x7f045806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.215+0000 7f047cb70700 1 --2- 172.21.15.111:0/3135548943 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f04580683c0 0x7f045806a870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0478149b00 tx=0x7f0468006c80 comp rx=0 tx=0).stop 2023-11-28T12:59:21.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.215+0000 7f047cb70700 1 -- 172.21.15.111:0/3135548943 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 msgr2=0x7f0478152180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.215+0000 7f047cb70700 1 --2- 172.21.15.111:0/3135548943 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 0x7f0478152180 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0470007230 tx=0x7f0470004ad0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.215+0000 7f047cb70700 1 -- 172.21.15.111:0/3135548943 shutdown_connections 2023-11-28T12:59:21.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.215+0000 7f047cb70700 1 --2- 172.21.15.111:0/3135548943 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f04580683c0 0x7f045806a870 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0478149b00 tx=0x7f0468006c80 comp rx=0 tx=0).stop 2023-11-28T12:59:21.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.215+0000 7f047cb70700 1 --2- 172.21.15.111:0/3135548943 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f047815ce70 0x7f0478089400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.215+0000 7f047cb70700 1 --2- 172.21.15.111:0/3135548943 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f047806a270 0x7f0478152180 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.215+0000 7f047cb70700 1 --2- 172.21.15.111:0/3135548943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04780cdee0 0x7f047806d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.215+0000 7f047cb70700 1 -- 172.21.15.111:0/3135548943 >> 172.21.15.111:0/3135548943 conn(0x7f04780cb490 msgr2=0x7f047806c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:21.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.215+0000 7f047cb70700 1 -- 172.21.15.111:0/3135548943 shutdown_connections 2023-11-28T12:59:21.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.215+0000 7f047cb70700 1 -- 172.21.15.111:0/3135548943 wait complete. 2023-11-28T12:59:21.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.219+0000 7f4ed2337700 1 -- 172.21.15.111:0/189492702 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4ecc0aeb90 con 0x7f4ecc0cb240 2023-11-28T12:59:21.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.221+0000 7fd623c19700 1 -- 172.21.15.111:0/2355794779 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd61c067880 con 0x7fd61c06cc70 2023-11-28T12:59:21.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.221+0000 7fd6127fc700 1 -- 172.21.15.111:0/2355794779 <== mon.1 v2:172.21.15.149:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd618057820 con 0x7fd61c06cc70 2023-11-28T12:59:21.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.223+0000 7f4ec8ff9700 1 -- 172.21.15.111:0/189492702 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4ebc055260 con 0x7f4ecc0cb240 2023-11-28T12:59:21.225 INFO:teuthology.orchestra.run.smithi111.stdout:55834574858 2023-11-28T12:59:21.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.223+0000 7f41d8d2c700 1 -- 172.21.15.111:0/3476788304 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f41d41611f0 con 0x7f41d40d13e0 2023-11-28T12:59:21.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.223+0000 7f41c2ffd700 1 -- 172.21.15.111:0/3476788304 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f41c4020020 con 0x7f41d40d13e0 2023-11-28T12:59:21.225 INFO:teuthology.orchestra.run.smithi111.stdout:55834574857 2023-11-28T12:59:21.228 INFO:teuthology.orchestra.run.smithi111.stdout:55834574857 2023-11-28T12:59:21.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.228+0000 7f41d8d2c700 1 -- 172.21.15.111:0/3476788304 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41b4068340 msgr2=0x7f41b406a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.228+0000 7f41d8d2c700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41b4068340 0x7f41b406a7f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f41c8000f60 tx=0x7f41c8007e50 comp rx=0 tx=0).stop 2023-11-28T12:59:21.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.228+0000 7f41d8d2c700 1 -- 172.21.15.111:0/3476788304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d40d13e0 msgr2=0x7f41d40c63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.228+0000 7f41d8d2c700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d40d13e0 0x7f41d40c63f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f41c4000c00 tx=0x7f41c4004750 comp rx=0 tx=0).stop 2023-11-28T12:59:21.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7f41d8d2c700 1 -- 172.21.15.111:0/3476788304 shutdown_connections 2023-11-28T12:59:21.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7f41d8d2c700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41b4068340 0x7f41b406a7f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7f41d8d2c700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41d4160e20 0x7f41d41f5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7f41d8d2c700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d40c43b0 0x7f41d40c51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7f41d8d2c700 1 --2- 172.21.15.111:0/3476788304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d40d13e0 0x7f41d40c63f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7f41d8d2c700 1 -- 172.21.15.111:0/3476788304 >> 172.21.15.111:0/3476788304 conn(0x7f41d409a020 msgr2=0x7f41d40b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:21.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7f41d8d2c700 1 -- 172.21.15.111:0/3476788304 shutdown_connections 2023-11-28T12:59:21.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7f41d8d2c700 1 -- 172.21.15.111:0/3476788304 wait complete. 2023-11-28T12:59:21.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7fd623c19700 1 -- 172.21.15.111:0/2355794779 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6000682f0 msgr2=0x7fd60006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7fd623c19700 1 --2- 172.21.15.111:0/2355794779 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6000682f0 0x7fd60006a7a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd60c000f60 tx=0x7fd60c007a40 comp rx=0 tx=0).stop 2023-11-28T12:59:21.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7fd623c19700 1 -- 172.21.15.111:0/2355794779 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd61c06cc70 msgr2=0x7fd61c157ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7fd623c19700 1 --2- 172.21.15.111:0/2355794779 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd61c06cc70 0x7fd61c157ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd618000c00 tx=0x7fd618004750 comp rx=0 tx=0).stop 2023-11-28T12:59:21.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7fd623c19700 1 -- 172.21.15.111:0/2355794779 shutdown_connections 2023-11-28T12:59:21.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7fd623c19700 1 --2- 172.21.15.111:0/2355794779 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd61c158b70 0x7fd61c1724e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7fd623c19700 1 --2- 172.21.15.111:0/2355794779 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6000682f0 0x7fd60006a7a0 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd60c000f60 tx=0x7fd60c007a40 comp rx=0 tx=0).stop 2023-11-28T12:59:21.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7fd623c19700 1 --2- 172.21.15.111:0/2355794779 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd61c06cc70 0x7fd61c157ed0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7fd623c19700 1 --2- 172.21.15.111:0/2355794779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd61c070910 0x7fd61c0c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.229+0000 7fd623c19700 1 -- 172.21.15.111:0/2355794779 >> 172.21.15.111:0/2355794779 conn(0x7fd61c0cb490 msgr2=0x7fd61c0b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:21.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.233+0000 7fd623c19700 1 -- 172.21.15.111:0/2355794779 shutdown_connections 2023-11-28T12:59:21.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.233+0000 7f4ed2337700 1 -- 172.21.15.111:0/189492702 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4eac068390 msgr2=0x7f4eac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.233+0000 7f4ed2337700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4eac068390 0x7f4eac06a840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4eb400ad60 tx=0x7f4eb4012000 comp rx=0 tx=0).stop 2023-11-28T12:59:21.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.233+0000 7f4ed2337700 1 -- 172.21.15.111:0/189492702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ecc0cb240 msgr2=0x7f4ecc0c6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.233+0000 7f4ed2337700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ecc0cb240 0x7f4ecc0c6390 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00c3c0 tx=0x7f4ebc00c6d0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.233+0000 7fd623c19700 1 -- 172.21.15.111:0/2355794779 wait complete. 2023-11-28T12:59:21.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.233+0000 7f4ed2337700 1 -- 172.21.15.111:0/189492702 shutdown_connections 2023-11-28T12:59:21.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.233+0000 7f4ed2337700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4eac068390 0x7f4eac06a840 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.233+0000 7f4ed2337700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ecc15d270 0x7f4ecc0af3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.233+0000 7f4ed2337700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ecc06e730 0x7f4ecc0c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.233+0000 7f4ed2337700 1 --2- 172.21.15.111:0/189492702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ecc0cb240 0x7f4ecc0c6390 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.233+0000 7f4ed2337700 1 -- 172.21.15.111:0/189492702 >> 172.21.15.111:0/189492702 conn(0x7f4ecc0b0db0 msgr2=0x7f4ecc0cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:21.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.234+0000 7f4ed2337700 1 -- 172.21.15.111:0/189492702 shutdown_connections 2023-11-28T12:59:21.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.234+0000 7f4ed2337700 1 -- 172.21.15.111:0/189492702 wait complete. 2023-11-28T12:59:21.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.234+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1561002981 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb548067880 con 0x7fb5480ccf40 2023-11-28T12:59:21.238 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.7 2023-11-28T12:59:21.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.235+0000 7fb544ff9700 1 -- 172.21.15.111:0/1561002981 <== mon.2 v2:172.21.15.149:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb53c009440 con 0x7fb5480ccf40 2023-11-28T12:59:21.239 INFO:teuthology.orchestra.run.smithi111.stdout:55834574857 2023-11-28T12:59:21.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.238+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1561002981 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb5340682f0 msgr2=0x7fb53406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.238+0000 7fb54e8c7700 1 --2- 172.21.15.111:0/1561002981 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb5340682f0 0x7fb53406a7a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb540005c90 tx=0x7fb540009040 comp rx=0 tx=0).stop 2023-11-28T12:59:21.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.238+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1561002981 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb5480ccf40 msgr2=0x7fb548151d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.238+0000 7fb54e8c7700 1 --2- 172.21.15.111:0/1561002981 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb5480ccf40 0x7fb548151d20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00e810 tx=0x7fb53c00eb20 comp rx=0 tx=0).stop 2023-11-28T12:59:21.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.242+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1561002981 shutdown_connections 2023-11-28T12:59:21.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.242+0000 7fb54e8c7700 1 --2- 172.21.15.111:0/1561002981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb548160e10 0x7fb548152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.242+0000 7fb54e8c7700 1 --2- 172.21.15.111:0/1561002981 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb5340682f0 0x7fb53406a7a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.242+0000 7fb54e8c7700 1 --2- 172.21.15.111:0/1561002981 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb54806a270 0x7fb5481611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.242+0000 7fb54e8c7700 1 --2- 172.21.15.111:0/1561002981 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb5480ccf40 0x7fb548151d20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.242+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1561002981 >> 172.21.15.111:0/1561002981 conn(0x7fb5480cb490 msgr2=0x7fb548160080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:21.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.242+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1561002981 shutdown_connections 2023-11-28T12:59:21.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.242+0000 7fb54e8c7700 1 -- 172.21.15.111:0/1561002981 wait complete. 2023-11-28T12:59:21.244 INFO:teuthology.orchestra.run.smithi111.stdout:55834574855 2023-11-28T12:59:21.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.242+0000 7f9ce185f700 1 -- 172.21.15.111:0/4241937954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9cbc005870 con 0x7f9cdc0ade90 2023-11-28T12:59:21.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.242+0000 7f9cc3fff700 1 -- 172.21.15.111:0/4241937954 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9ccc0552b0 con 0x7f9cdc0ade90 2023-11-28T12:59:21.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.246+0000 7f9ce185f700 1 -- 172.21.15.111:0/4241937954 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9cc80682f0 msgr2=0x7f9cc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.246+0000 7f9ce185f700 1 --2- 172.21.15.111:0/4241937954 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9cc80682f0 0x7f9cc806a7a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0c8f00 tx=0x7f9cd0008040 comp rx=0 tx=0).stop 2023-11-28T12:59:21.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.246+0000 7f9ce185f700 1 -- 172.21.15.111:0/4241937954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cdc0ade90 msgr2=0x7f9cdc150370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.246+0000 7f9ce185f700 1 --2- 172.21.15.111:0/4241937954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cdc0ade90 0x7f9cdc150370 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00c4e0 tx=0x7f9ccc00c8a0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.247+0000 7f9ce185f700 1 -- 172.21.15.111:0/4241937954 shutdown_connections 2023-11-28T12:59:21.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.247+0000 7f9ce185f700 1 --2- 172.21.15.111:0/4241937954 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cdc160e10 0x7f9cdc152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.247+0000 7f9ce185f700 1 --2- 172.21.15.111:0/4241937954 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9cc80682f0 0x7f9cc806a7a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.247+0000 7f9ce185f700 1 --2- 172.21.15.111:0/4241937954 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9cdc06a980 0x7f9cdc148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.247+0000 7f9ce185f700 1 --2- 172.21.15.111:0/4241937954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cdc0ade90 0x7f9cdc150370 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.247+0000 7f9ce185f700 1 -- 172.21.15.111:0/4241937954 >> 172.21.15.111:0/4241937954 conn(0x7f9cdc0cb490 msgr2=0x7f9cdc0717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:21.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.249+0000 7f9ce185f700 1 -- 172.21.15.111:0/4241937954 shutdown_connections 2023-11-28T12:59:21.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.249+0000 7feceebba700 1 -- 172.21.15.111:0/2481966096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fece80d2940 con 0x7fece80b8510 2023-11-28T12:59:21.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.250+0000 7fece57fa700 1 -- 172.21.15.111:0/2481966096 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fecd8021020 con 0x7fece80b8510 2023-11-28T12:59:21.252 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.5 2023-11-28T12:59:21.252 INFO:teuthology.orchestra.run.smithi111.stdout:55834574857 2023-11-28T12:59:21.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.253+0000 7f9ce185f700 1 -- 172.21.15.111:0/4241937954 wait complete. 2023-11-28T12:59:21.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.253+0000 7fc4f5981700 1 -- 172.21.15.111:0/1182226262 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc4f0067880 con 0x7fc4f00ce850 2023-11-28T12:59:21.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.254+0000 7fc4ecff9700 1 -- 172.21.15.111:0/1182226262 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc4e0021090 con 0x7fc4f00ce850 2023-11-28T12:59:21.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.253+0000 7feceebba700 1 -- 172.21.15.111:0/2481966096 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fecd40682f0 msgr2=0x7fecd406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.253+0000 7feceebba700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fecd40682f0 0x7fecd406a7a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009b90 tx=0x7fecdc01d040 comp rx=0 tx=0).stop 2023-11-28T12:59:21.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.253+0000 7feceebba700 1 -- 172.21.15.111:0/2481966096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fece80b8510 msgr2=0x7fece81ff9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.253+0000 7feceebba700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fece80b8510 0x7fece81ff9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fecd800c520 tx=0x7fecd800c830 comp rx=0 tx=0).stop 2023-11-28T12:59:21.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.254+0000 7feceebba700 1 -- 172.21.15.111:0/2481966096 shutdown_connections 2023-11-28T12:59:21.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.254+0000 7feceebba700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fecd40682f0 0x7fecd406a7a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.254+0000 7feceebba700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fece80d0d60 0x7fece81f3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.254+0000 7feceebba700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fece80b8510 0x7fece81ff9a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.254+0000 7feceebba700 1 --2- 172.21.15.111:0/2481966096 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fece80cd460 0x7fece81f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.254+0000 7feceebba700 1 -- 172.21.15.111:0/2481966096 >> 172.21.15.111:0/2481966096 conn(0x7fece80719a0 msgr2=0x7fece80b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:21.256 INFO:teuthology.orchestra.run.smithi111.stdout:55834574857 2023-11-28T12:59:21.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.254+0000 7feceebba700 1 -- 172.21.15.111:0/2481966096 shutdown_connections 2023-11-28T12:59:21.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.254+0000 7feceebba700 1 -- 172.21.15.111:0/2481966096 wait complete. 2023-11-28T12:59:21.258 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.6 2023-11-28T12:59:21.259 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.1 2023-11-28T12:59:21.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.257+0000 7fc4f5981700 1 -- 172.21.15.111:0/1182226262 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4d00682f0 msgr2=0x7fc4d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.257+0000 7fc4f5981700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4d00682f0 0x7fc4d006a7a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80090b0 tx=0x7fc4d8008030 comp rx=0 tx=0).stop 2023-11-28T12:59:21.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.257+0000 7fc4f5981700 1 -- 172.21.15.111:0/1182226262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f00ce850 msgr2=0x7fc4f006fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:21.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.257+0000 7fc4f5981700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f00ce850 0x7fc4f006fd50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc4e000c480 tx=0x7fc4e000c790 comp rx=0 tx=0).stop 2023-11-28T12:59:21.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.257+0000 7fc4f5981700 1 -- 172.21.15.111:0/1182226262 shutdown_connections 2023-11-28T12:59:21.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.257+0000 7fc4f5981700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4d00682f0 0x7fc4d006a7a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.257+0000 7fc4f5981700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4f0070290 0x7fc4f00b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.257+0000 7fc4f5981700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4f00afba0 0x7fc4f00c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.257+0000 7fc4f5981700 1 --2- 172.21.15.111:0/1182226262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4f00ce850 0x7fc4f006fd50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:21.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.257+0000 7fc4f5981700 1 -- 172.21.15.111:0/1182226262 >> 172.21.15.111:0/1182226262 conn(0x7fc4f009a040 msgr2=0x7fc4f00cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:21.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.257+0000 7fc4f5981700 1 -- 172.21.15.111:0/1182226262 shutdown_connections 2023-11-28T12:59:21.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:21.257+0000 7fc4f5981700 1 -- 172.21.15.111:0/1182226262 wait complete. 2023-11-28T12:59:21.271 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.2 2023-11-28T12:59:21.272 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2023-11-28T12:59:21.276 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.0 2023-11-28T12:59:21.277 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.3 2023-11-28T12:59:22.239 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-28T12:59:22.253 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-28T12:59:22.259 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-28T12:59:22.259 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-28T12:59:22.272 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-28T12:59:22.273 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-28T12:59:22.276 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-28T12:59:22.278 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-28T12:59:22.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.449+0000 7fa8e77ef700 1 Processor -- start 2023-11-28T12:59:22.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.450+0000 7fa8e77ef700 1 -- start start 2023-11-28T12:59:22.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.451+0000 7fa8e77ef700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa8e00c9ed0 0x7fa8e006a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.451+0000 7fa8e77ef700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa8e00cdee0 0x7fa8e006c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.451+0000 7fa8e77ef700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e006ab80 con 0x7fa8e006a270 2023-11-28T12:59:22.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.451+0000 7fa8e77ef700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e00ccb60 con 0x7fa8e00cdee0 2023-11-28T12:59:22.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.451+0000 7fa8e77ef700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00cce40 con 0x7fa8e00c9ed0 2023-11-28T12:59:22.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.452+0000 7fa8e4d8a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa8e00c9ed0 0x7fa8e006a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.452+0000 7fa8e4d8a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa8e00c9ed0 0x7fa8e006a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55190/0 (socket says 172.21.15.111:55190) 2023-11-28T12:59:22.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.452+0000 7fa8e4d8a700 1 -- 172.21.15.111:0/4274780358 learned_addr learned my addr 172.21.15.111:0/4274780358 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:22.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.453+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240969885 0 0) 0x7fa8e006ab80 con 0x7fa8e006a270 2023-11-28T12:59:22.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.453+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8cc0035b0 con 0x7fa8e006a270 2023-11-28T12:59:22.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1974769348 0 0) 0x7fa8cc0035b0 con 0x7fa8e006a270 2023-11-28T12:59:22.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa8e00c9ed0 msgr2=0x7fa8e006a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:22.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 --2- 172.21.15.111:0/4274780358 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa8e00c9ed0 0x7fa8e006a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 >> v2:172.21.15.111:3300/0 conn(0x7fa8e00cdee0 msgr2=0x7fa8e006c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:22.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 --2- 172.21.15.111:0/4274780358 >> v2:172.21.15.111:3300/0 conn(0x7fa8e00cdee0 0x7fa8e006c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e00cd120 con 0x7fa8e006a270 2023-11-28T12:59:22.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fa8d80019f0 con 0x7fa8e006a270 2023-11-28T12:59:22.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 >> v1:172.21.15.111:6789/0 conn(0x7fa8e006a270 legacy=0x7fa8e015ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 --2- 172.21.15.111:0/4274780358 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa8cc003a00 0x7fa8cc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 --2- 172.21.15.111:0/4274780358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8cc004850 0x7fa8cc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 --2- 172.21.15.111:0/4274780358 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa8cc009a00 0x7fa8cc005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00ccb60 con 0x7fa8cc004850 2023-11-28T12:59:22.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00cce40 con 0x7fa8cc003a00 2023-11-28T12:59:22.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.454+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc0148f0 con 0x7fa8cc009a00 2023-11-28T12:59:22.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.455+0000 7fa8e4d8a700 1 --2- 172.21.15.111:0/4274780358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8cc004850 0x7fa8cc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.455+0000 7fa8e4d8a700 1 -- 172.21.15.111:0/4274780358 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa8cc009a00 msgr2=0x7fa8cc005220 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T12:59:22.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.455+0000 7fa8e4d8a700 1 --2- 172.21.15.111:0/4274780358 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa8cc009a00 0x7fa8cc005220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.455+0000 7fa8e4d8a700 1 -- 172.21.15.111:0/4274780358 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa8cc003a00 msgr2=0x7fa8cc003e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T12:59:22.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.455+0000 7fa8e4d8a700 1 --2- 172.21.15.111:0/4274780358 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa8cc003a00 0x7fa8cc003e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.455+0000 7fa8e4d8a700 1 -- 172.21.15.111:0/4274780358 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e00cd120 con 0x7fa8cc004850 2023-11-28T12:59:22.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.456+0000 7fa8e4d8a700 1 --2- 172.21.15.111:0/4274780358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8cc004850 0x7fa8cc004cc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00cab70 tx=0x7fa8cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49cfa7ff18ff5f05 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.456+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc010070 con 0x7fa8cc004850 2023-11-28T12:59:22.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.456+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc0087c0 con 0x7fa8cc004850 2023-11-28T12:59:22.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.456+0000 7fa8d7fff700 1 -- 172.21.15.111:0/4274780358 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc008b50 con 0x7fa8cc004850 2023-11-28T12:59:22.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.457+0000 7fa8e77ef700 1 -- 172.21.15.111:0/4274780358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8cc004850 msgr2=0x7fa8cc004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.457+0000 7fa8e77ef700 1 --2- 172.21.15.111:0/4274780358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8cc004850 0x7fa8cc004cc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00cab70 tx=0x7fa8cc005880 comp rx=0 tx=0).stop 2023-11-28T12:59:22.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.459+0000 7fa8e5d8c700 1 -- 172.21.15.111:0/4274780358 reap_dead start 2023-11-28T12:59:22.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.459+0000 7fa8e77ef700 1 -- 172.21.15.111:0/4274780358 shutdown_connections 2023-11-28T12:59:22.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.459+0000 7fa8e77ef700 1 -- 172.21.15.111:0/4274780358 >> 172.21.15.111:0/4274780358 conn(0x7fa8e00cb490 msgr2=0x7fa8e00cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.459+0000 7fa8e77ef700 1 -- 172.21.15.111:0/4274780358 shutdown_connections 2023-11-28T12:59:22.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.459+0000 7fa8e77ef700 1 -- 172.21.15.111:0/4274780358 wait complete. 2023-11-28T12:59:22.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.461+0000 7fa8e77ef700 1 Processor -- start 2023-11-28T12:59:22.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.461+0000 7fa8e77ef700 1 -- start start 2023-11-28T12:59:22.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.462+0000 7fa8e77ef700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa8e01ed720 0x7fa8e00b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.462+0000 7fa8e77ef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8e01edaf0 0x7fa8e00af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.462+0000 7fa8e77ef700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa8e00afc30 0x7fa8e00b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.462+0000 7fa8e77ef700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00b0d70 con 0x7fa8e01edaf0 2023-11-28T12:59:22.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.462+0000 7fa8e77ef700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00ae370 con 0x7fa8e00afc30 2023-11-28T12:59:22.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.462+0000 7fa8e77ef700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e00ae650 con 0x7fa8e01ed720 2023-11-28T12:59:22.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.463+0000 7fa8e4d8a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8e01edaf0 0x7fa8e00af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.463+0000 7fa8e4d8a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8e01edaf0 0x7fa8e00af6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33454/0 (socket says 172.21.15.111:33454) 2023-11-28T12:59:22.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.463+0000 7fa8e4d8a700 1 -- 172.21.15.111:0/1352709261 learned_addr learned my addr 172.21.15.111:0/1352709261 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:22.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.463+0000 7fa8e558b700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa8e01ed720 0x7fa8e00b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.463+0000 7fa8e5d8c700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa8e00afc30 0x7fa8e00b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.463+0000 7fa8e4d8a700 1 -- 172.21.15.111:0/1352709261 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa8e01ed720 msgr2=0x7fa8e00b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.463+0000 7fa8e4d8a700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa8e01ed720 0x7fa8e00b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.463+0000 7fa8e4d8a700 1 -- 172.21.15.111:0/1352709261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa8e00afc30 msgr2=0x7fa8e00b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.463+0000 7fa8e4d8a700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa8e00afc30 0x7fa8e00b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.463+0000 7fa8e4d8a700 1 -- 172.21.15.111:0/1352709261 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e00cd6e0 con 0x7fa8e01edaf0 2023-11-28T12:59:22.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.464+0000 7fa8e4d8a700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8e01edaf0 0x7fa8e00af6f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc014840 tx=0x7fa8dc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.464+0000 7fa8d5ffb700 1 -- 172.21.15.111:0/1352709261 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc004020 con 0x7fa8e01edaf0 2023-11-28T12:59:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.464+0000 7fa8d5ffb700 1 -- 172.21.15.111:0/1352709261 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc004180 con 0x7fa8e01edaf0 2023-11-28T12:59:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.464+0000 7fa8d5ffb700 1 -- 172.21.15.111:0/1352709261 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc004530 con 0x7fa8e01edaf0 2023-11-28T12:59:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.464+0000 7fa8e77ef700 1 -- 172.21.15.111:0/1352709261 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e00cd400 con 0x7fa8e01edaf0 2023-11-28T12:59:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.465+0000 7fa8e77ef700 1 -- 172.21.15.111:0/1352709261 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e00cd9c0 con 0x7fa8e01edaf0 2023-11-28T12:59:22.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.466+0000 7fa8e77ef700 1 -- 172.21.15.111:0/1352709261 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e0067880 con 0x7fa8e01edaf0 2023-11-28T12:59:22.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.476+0000 7fa8d5ffb700 1 -- 172.21.15.111:0/1352709261 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa8dc02f070 con 0x7fa8e01edaf0 2023-11-28T12:59:22.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.477+0000 7fa8d5ffb700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa8cc07ac80 0x7fa8cc07d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.477+0000 7fa8d5ffb700 1 -- 172.21.15.111:0/1352709261 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa8dc087560 con 0x7fa8e01edaf0 2023-11-28T12:59:22.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.477+0000 7fa8d5ffb700 1 -- 172.21.15.111:0/1352709261 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa8dc0879e0 con 0x7fa8e01edaf0 2023-11-28T12:59:22.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.478+0000 7fa8e558b700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa8cc07ac80 0x7fa8cc07d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.478+0000 7fa8e558b700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa8cc07ac80 0x7fa8cc07d130 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8001b20 tx=0x7fa8d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.495+0000 7f0c0baf5700 1 Processor -- start 2023-11-28T12:59:22.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.495+0000 7f0c0baf5700 1 -- start start 2023-11-28T12:59:22.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.496+0000 7f0c0baf5700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f0c0406a270 0x7f0c0406a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.496+0000 7f0c0baf5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0c040ccf40 0x7f0c040cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.496+0000 7f0c0baf5700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c04165740 con 0x7f0c04160e10 2023-11-28T12:59:22.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.496+0000 7f0c0baf5700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0406ac30 con 0x7f0c0406a270 2023-11-28T12:59:22.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.496+0000 7f0c0baf5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040ae060 con 0x7f0c040ccf40 2023-11-28T12:59:22.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.497+0000 7f0c0a092700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0c040ccf40 0x7f0c040cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.497+0000 7f0c09891700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f0c04160e10 0x7f0c040ad950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:42778/0 (socket says 172.21.15.111:42778) 2023-11-28T12:59:22.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.497+0000 7f0c09891700 1 -- 172.21.15.111:0/68373751 learned_addr learned my addr 172.21.15.111:0/68373751 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:22.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.497+0000 7f0c09090700 1 --2- 172.21.15.111:0/68373751 >> v2:172.21.15.149:3300/0 conn(0x7f0c0406a270 0x7f0c0406a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.497+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683758761 0 0) 0x7f0c04165740 con 0x7f0c04160e10 2023-11-28T12:59:22.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.497+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf00035b0 con 0x7f0c04160e10 2023-11-28T12:59:22.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 <== mon.1 v1:172.21.15.149:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3674089047 0 0) 0x7f0bf00035b0 con 0x7f0c04160e10 2023-11-28T12:59:22.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0c040ccf40 msgr2=0x7f0c040cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:22.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 --2- 172.21.15.111:0/68373751 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0c040ccf40 0x7f0c040cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 >> v2:172.21.15.149:3300/0 conn(0x7f0c0406a270 msgr2=0x7f0c0406a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:22.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 --2- 172.21.15.111:0/68373751 >> v2:172.21.15.149:3300/0 conn(0x7f0c0406a270 0x7f0c0406a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 --> v1:172.21.15.149:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c04164bc0 con 0x7f0c04160e10 2023-11-28T12:59:22.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 <== mon.1 v1:172.21.15.149:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f0c00001670 con 0x7f0c04160e10 2023-11-28T12:59:22.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 >> v1:172.21.15.149:6789/0 conn(0x7f0c04160e10 legacy=0x7f0c040ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 --2- 172.21.15.111:0/68373751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bf0003a00 0x7f0bf0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 --2- 172.21.15.111:0/68373751 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0bf0004850 0x7f0bf0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 --2- 172.21.15.111:0/68373751 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0bf0005200 0x7f0bf000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0406ac30 con 0x7f0bf0003a00 2023-11-28T12:59:22.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040ae060 con 0x7f0bf0005200 2023-11-28T12:59:22.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.498+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf000c000 con 0x7f0bf0004850 2023-11-28T12:59:22.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.499+0000 7f0c09891700 1 --2- 172.21.15.111:0/68373751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bf0003a00 0x7f0bf0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.500+0000 7f0c09891700 1 -- 172.21.15.111:0/68373751 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0bf0004850 msgr2=0x7f0bf0004cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T12:59:22.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.500+0000 7f0c09891700 1 --2- 172.21.15.111:0/68373751 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0bf0004850 0x7f0bf0004cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.500+0000 7f0c09891700 1 -- 172.21.15.111:0/68373751 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0bf0005200 msgr2=0x7f0bf000ba10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T12:59:22.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.500+0000 7f0c09891700 1 --2- 172.21.15.111:0/68373751 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0bf0005200 0x7f0bf000ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.500+0000 7f0c09891700 1 -- 172.21.15.111:0/68373751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c04164bc0 con 0x7f0bf0003a00 2023-11-28T12:59:22.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.500+0000 7f0c09891700 1 --2- 172.21.15.111:0/68373751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bf0003a00 0x7f0bf0003e10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0c0000c3f0 tx=0x7f0bf0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b74f325484767f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.501+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c00005960 con 0x7f0bf0003a00 2023-11-28T12:59:22.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.501+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c00005ac0 con 0x7f0bf0003a00 2023-11-28T12:59:22.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.501+0000 7f0c0888f700 1 -- 172.21.15.111:0/68373751 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c00005e50 con 0x7f0bf0003a00 2023-11-28T12:59:22.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.502+0000 7f0c0baf5700 1 -- 172.21.15.111:0/68373751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bf0003a00 msgr2=0x7f0bf0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.502+0000 7f0c0baf5700 1 --2- 172.21.15.111:0/68373751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bf0003a00 0x7f0bf0003e10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0c0000c3f0 tx=0x7f0bf0005880 comp rx=0 tx=0).stop 2023-11-28T12:59:22.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.502+0000 7f0c0a092700 1 -- 172.21.15.111:0/68373751 reap_dead start 2023-11-28T12:59:22.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.502+0000 7f0c0baf5700 1 -- 172.21.15.111:0/68373751 shutdown_connections 2023-11-28T12:59:22.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.502+0000 7f0c0baf5700 1 -- 172.21.15.111:0/68373751 >> 172.21.15.111:0/68373751 conn(0x7f0c040cb490 msgr2=0x7f0c040cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.502+0000 7fa5e6338700 1 Processor -- start 2023-11-28T12:59:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.503+0000 7f0c0baf5700 1 -- 172.21.15.111:0/68373751 shutdown_connections 2023-11-28T12:59:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.503+0000 7f0c0baf5700 1 -- 172.21.15.111:0/68373751 wait complete. 2023-11-28T12:59:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.503+0000 7fa5e6338700 1 -- start start 2023-11-28T12:59:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.504+0000 7fa5e6338700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fa5e006a270 0x7fa5e00c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.504+0000 7fa5e6338700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5e015ce70 0x7fa5e006a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.504+0000 7fa5e6338700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e006ab80 con 0x7fa5e00cdee0 2023-11-28T12:59:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.504+0000 7fa5e6338700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e00ccb60 con 0x7fa5e006a270 2023-11-28T12:59:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.504+0000 7fa5e6338700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e00cce40 con 0x7fa5e015ce70 2023-11-28T12:59:22.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.503+0000 7fb34c21b700 1 Processor -- start 2023-11-28T12:59:22.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.503+0000 7fb34c21b700 1 -- start start 2023-11-28T12:59:22.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.504+0000 7f0c0baf5700 1 Processor -- start 2023-11-28T12:59:22.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.505+0000 7fa5e48d5700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa5e00cdee0 0x7fa5e006c6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34254/0 (socket says 172.21.15.111:34254) 2023-11-28T12:59:22.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.505+0000 7fa5e48d5700 1 -- 172.21.15.111:0/2847435413 learned_addr learned my addr 172.21.15.111:0/2847435413 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:22.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.505+0000 7fb34c21b700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb3440ae8c0 0x7fb344159960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.505+0000 7fb34c21b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3440ad420 con 0x7fb3440cdd90 2023-11-28T12:59:22.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.505+0000 7fb34c21b700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3440ac810 con 0x7fb3440aecc0 2023-11-28T12:59:22.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.505+0000 7fb34c21b700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3440af090 con 0x7fb3440ae8c0 2023-11-28T12:59:22.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7fa5df7fe700 1 --2- 172.21.15.111:0/2847435413 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5e015ce70 0x7fa5e006a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.504+0000 7f0c0baf5700 1 -- start start 2023-11-28T12:59:22.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.505+0000 7f0c0baf5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0bf0003a00 0x7f0c040c14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f0c0baf5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c0415fb60 0x7f0c040bed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f0c0baf5700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0c040c24f0 0x7f0c04144c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f0c0baf5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c04143c10 con 0x7f0c0415fb60 2023-11-28T12:59:22.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f0c0baf5700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c04143e60 con 0x7f0c040c24f0 2023-11-28T12:59:22.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f0c0baf5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c04144140 con 0x7f0bf0003a00 2023-11-28T12:59:22.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7fb3497b6700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb3440cdd90 0x7fb34406a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34266/0 (socket says 172.21.15.111:34266) 2023-11-28T12:59:22.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7fb3497b6700 1 -- 172.21.15.111:0/2061773070 learned_addr learned my addr 172.21.15.111:0/2061773070 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:22.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f0c09891700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0bf0003a00 0x7f0c040c14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f0c09891700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0bf0003a00 0x7f0c040c14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55232/0 (socket says 172.21.15.111:55232) 2023-11-28T12:59:22.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f0c09891700 1 -- 172.21.15.111:0/711179556 learned_addr learned my addr 172.21.15.111:0/711179556 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:22.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f0c0a092700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0c040c24f0 0x7f0c04144c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365664032 0 0) 0x7fb3440ad420 con 0x7fb3440cdd90 2023-11-28T12:59:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3300035b0 con 0x7fb3440cdd90 2023-11-28T12:59:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7fa5df7fe700 1 -- 172.21.15.111:0/2847435413 >> v2:172.21.15.149:3300/0 conn(0x7fa5e006a270 msgr2=0x7fa5e00c9ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T12:59:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7fa5df7fe700 1 --2- 172.21.15.111:0/2847435413 >> v2:172.21.15.149:3300/0 conn(0x7fa5e006a270 0x7fa5e00c9ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7fa5df7fe700 1 -- 172.21.15.111:0/2847435413 >> v1:172.21.15.111:6789/0 conn(0x7fa5e00cdee0 legacy=0x7fa5e006c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7fa5df7fe700 1 -- 172.21.15.111:0/2847435413 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e00cd120 con 0x7fa5e015ce70 2023-11-28T12:59:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f0c09090700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c0415fb60 0x7f0c040bed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7f0c09891700 1 -- 172.21.15.111:0/711179556 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0c040c24f0 msgr2=0x7f0c04144c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7f0c09891700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0c040c24f0 0x7f0c04144c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7f0c09891700 1 -- 172.21.15.111:0/711179556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c0415fb60 msgr2=0x7f0c040bed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7f0c09891700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c0415fb60 0x7f0c040bed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7f0c09891700 1 -- 172.21.15.111:0/711179556 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c04165180 con 0x7f0bf0003a00 2023-11-28T12:59:22.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fa5df7fe700 1 --2- 172.21.15.111:0/2847435413 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5e015ce70 0x7fa5e006a640 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa5d000a9e0 tx=0x7fa5d000acf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=608ab7bee42143d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879833232 0 0) 0x7fb3440ac810 con 0x7fb3440aecc0 2023-11-28T12:59:22.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3440ad420 con 0x7fb3440aecc0 2023-11-28T12:59:22.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 521691652 0 0) 0x7fb3300035b0 con 0x7fb3440cdd90 2023-11-28T12:59:22.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 >> v2:172.21.15.149:3300/0 conn(0x7fb3440ae8c0 msgr2=0x7fb344159960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T12:59:22.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 --2- 172.21.15.111:0/2061773070 >> v2:172.21.15.149:3300/0 conn(0x7fb3440ae8c0 0x7fb344159960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 >> v1:172.21.15.149:6789/0 conn(0x7fb3440aecc0 legacy=0x7fb3440cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3441603e0 con 0x7fb3440cdd90 2023-11-28T12:59:22.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fb334001670 con 0x7fb3440cdd90 2023-11-28T12:59:22.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 >> v1:172.21.15.111:6789/0 conn(0x7fb3440cdd90 legacy=0x7fb34406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 --2- 172.21.15.111:0/2061773070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb330003b90 0x7fb330003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 --2- 172.21.15.111:0/2061773070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb330004850 0x7fb330004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 --2- 172.21.15.111:0/2061773070 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb330005250 0x7fb33000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3440af090 con 0x7fb330003b90 2023-11-28T12:59:22.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33000c190 con 0x7fb330004850 2023-11-28T12:59:22.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33000c350 con 0x7fb330005250 2023-11-28T12:59:22.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7f0c09891700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0bf0003a00 0x7f0c040c14b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0c0000c9e0 tx=0x7f0c00001930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.504+0000 7f23e425f700 1 Processor -- start 2023-11-28T12:59:22.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.504+0000 7f23e425f700 1 -- start start 2023-11-28T12:59:22.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.505+0000 7f23e425f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23dc0adda0 0x7f23dc06a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.505+0000 7f23e425f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f23dc089280 0x7f23dc06a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.505+0000 7f23e425f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23dc16df50 con 0x7f23dc0897d0 2023-11-28T12:59:22.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.505+0000 7f23e425f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23dc16d1d0 con 0x7f23dc089280 2023-11-28T12:59:22.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.505+0000 7f23e425f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc0cdc00 con 0x7f23dc0adda0 2023-11-28T12:59:22.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f23e27fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f23dc089280 0x7f23dc06a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f23e27fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f23dc089280 0x7f23dc06a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33480/0 (socket says 172.21.15.111:33480) 2023-11-28T12:59:22.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f23e27fc700 1 -- 172.21.15.111:0/2033843891 learned_addr learned my addr 172.21.15.111:0/2033843891 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:22.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.506+0000 7f23e0ff9700 1 -- 172.21.15.111:0/2033843891 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227518384 0 0) 0x7f23dc16df50 con 0x7f23dc0897d0 2023-11-28T12:59:22.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7f23e0ff9700 1 -- 172.21.15.111:0/2033843891 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c8003590 con 0x7f23dc0897d0 2023-11-28T12:59:22.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.508+0000 7f23e27fc700 1 -- 172.21.15.111:0/2033843891 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23dc0adda0 msgr2=0x7f23dc06a270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T12:59:22.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.508+0000 7f23e27fc700 1 --2- 172.21.15.111:0/2033843891 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23dc0adda0 0x7f23dc06a270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.508+0000 7f23e27fc700 1 -- 172.21.15.111:0/2033843891 >> v1:172.21.15.111:6789/0 conn(0x7f23dc0897d0 legacy=0x7f23dc070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.508+0000 7f23e27fc700 1 -- 172.21.15.111:0/2033843891 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23dc16d470 con 0x7f23dc089280 2023-11-28T12:59:22.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.508+0000 7f23e27fc700 1 --2- 172.21.15.111:0/2033843891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23dc089280 0x7f23dc06a7b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f23d400c2e0 tx=0x7f23d400c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db8e2a0936afa37 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.508+0000 7f23e0ff9700 1 -- 172.21.15.111:0/2033843891 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d400f070 con 0x7f23dc089280 2023-11-28T12:59:22.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.508+0000 7f23e0ff9700 1 -- 172.21.15.111:0/2033843891 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d400a0d0 con 0x7f23dc089280 2023-11-28T12:59:22.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.508+0000 7f23e0ff9700 1 -- 172.21.15.111:0/2033843891 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d400a460 con 0x7f23dc089280 2023-11-28T12:59:22.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7f0bfa7fc700 1 -- 172.21.15.111:0/711179556 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c000072d0 con 0x7f0bf0003a00 2023-11-28T12:59:22.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7f0c0baf5700 1 -- 172.21.15.111:0/711179556 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c04164ea0 con 0x7f0bf0003a00 2023-11-28T12:59:22.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7f0c0baf5700 1 -- 172.21.15.111:0/711179556 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c04165460 con 0x7f0bf0003a00 2023-11-28T12:59:22.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7f0bfa7fc700 1 -- 172.21.15.111:0/711179556 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c00001a50 con 0x7f0bf0003a00 2023-11-28T12:59:22.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.507+0000 7f0bfa7fc700 1 -- 172.21.15.111:0/711179556 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c00007630 con 0x7f0bf0003a00 2023-11-28T12:59:22.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.508+0000 7fb349fb7700 1 --2- 172.21.15.111:0/2061773070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb330003b90 0x7fb330003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.508+0000 7fa5deffd700 1 -- 172.21.15.111:0/2847435413 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0004230 con 0x7fa5e015ce70 2023-11-28T12:59:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.508+0000 7fa5deffd700 1 -- 172.21.15.111:0/2847435413 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d0004390 con 0x7fa5e015ce70 2023-11-28T12:59:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.508+0000 7fa5deffd700 1 -- 172.21.15.111:0/2847435413 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d000c870 con 0x7fa5e015ce70 2023-11-28T12:59:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fa5e6338700 1 -- 172.21.15.111:0/2847435413 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5e015ce70 msgr2=0x7fa5e006a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fa5e6338700 1 --2- 172.21.15.111:0/2847435413 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5e015ce70 0x7fa5e006a640 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa5d000a9e0 tx=0x7fa5d000acf0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fa5e6338700 1 -- 172.21.15.111:0/2847435413 shutdown_connections 2023-11-28T12:59:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fa5e6338700 1 --2- 172.21.15.111:0/2847435413 >> v2:172.21.15.149:3300/0 conn(0x7fa5e006a270 0x7fa5e00c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fa5e6338700 1 --2- 172.21.15.111:0/2847435413 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5e015ce70 0x7fa5e006a640 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fa5e6338700 1 -- 172.21.15.111:0/2847435413 >> 172.21.15.111:0/2847435413 conn(0x7fa5e00cb490 msgr2=0x7fa5e00cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fb349fb7700 1 -- 172.21.15.111:0/2061773070 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb330005250 msgr2=0x7fb33000ba10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T12:59:22.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fb349fb7700 1 --2- 172.21.15.111:0/2061773070 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb330005250 0x7fb33000ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fb349fb7700 1 -- 172.21.15.111:0/2061773070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb330004850 msgr2=0x7fb330004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:22.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fb349fb7700 1 --2- 172.21.15.111:0/2061773070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb330004850 0x7fb330004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fb349fb7700 1 -- 172.21.15.111:0/2061773070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3441603e0 con 0x7fb330003b90 2023-11-28T12:59:22.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fb349fb7700 1 --2- 172.21.15.111:0/2061773070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb330003b90 0x7fb330003f60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb330005830 tx=0x7fb330005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df814ab9c4c6b4d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340016070 con 0x7fb330003b90 2023-11-28T12:59:22.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb340006860 con 0x7fb330003b90 2023-11-28T12:59:22.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fb348fb5700 1 -- 172.21.15.111:0/2061773070 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340006bf0 con 0x7fb330003b90 2023-11-28T12:59:22.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7f23e425f700 1 -- 172.21.15.111:0/2033843891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23dc089280 msgr2=0x7f23dc06a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7f23e425f700 1 --2- 172.21.15.111:0/2033843891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23dc089280 0x7f23dc06a7b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f23d400c2e0 tx=0x7f23d400c5f0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7fb34a7b8700 1 -- 172.21.15.111:0/2061773070 reap_dead start 2023-11-28T12:59:22.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7f23e425f700 1 -- 172.21.15.111:0/2033843891 shutdown_connections 2023-11-28T12:59:22.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7f23e425f700 1 --2- 172.21.15.111:0/2033843891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23dc089280 0x7f23dc06a7b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7f23e425f700 1 --2- 172.21.15.111:0/2033843891 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23dc0adda0 0x7f23dc06a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7f23e425f700 1 -- 172.21.15.111:0/2033843891 >> 172.21.15.111:0/2033843891 conn(0x7f23dc0cb490 msgr2=0x7f23dc0cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.510+0000 7fa5e6338700 1 -- 172.21.15.111:0/2847435413 shutdown_connections 2023-11-28T12:59:22.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.510+0000 7f23e425f700 1 -- 172.21.15.111:0/2033843891 shutdown_connections 2023-11-28T12:59:22.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.509+0000 7f0bfa7fc700 1 -- 172.21.15.111:0/711179556 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0c00010070 con 0x7f0bf0003a00 2023-11-28T12:59:22.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.510+0000 7f23e425f700 1 -- 172.21.15.111:0/2033843891 wait complete. 2023-11-28T12:59:22.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.510+0000 7f0bfa7fc700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0be80682f0 0x7f0be806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.510+0000 7f0bfa7fc700 1 -- 172.21.15.111:0/711179556 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0c00087870 con 0x7f0bf0003a00 2023-11-28T12:59:22.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.510+0000 7f0c09090700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0be80682f0 0x7f0be806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.510+0000 7fb34c21b700 1 -- 172.21.15.111:0/2061773070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb330003b90 msgr2=0x7fb330003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.510+0000 7fb34c21b700 1 --2- 172.21.15.111:0/2061773070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb330003b90 0x7fb330003f60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb330005830 tx=0x7fb330005880 comp rx=0 tx=0).stop 2023-11-28T12:59:22.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.510+0000 7f0c0baf5700 1 -- 172.21.15.111:0/711179556 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bec0052f0 con 0x7f0bf0003a00 2023-11-28T12:59:22.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.511+0000 7fb34c21b700 1 -- 172.21.15.111:0/2061773070 shutdown_connections 2023-11-28T12:59:22.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.511+0000 7fb34c21b700 1 --2- 172.21.15.111:0/2061773070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb330003b90 0x7fb330003f60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.511+0000 7fb34c21b700 1 -- 172.21.15.111:0/2061773070 >> 172.21.15.111:0/2061773070 conn(0x7fb344098ee0 msgr2=0x7fb34406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.510+0000 7fa5e6338700 1 -- 172.21.15.111:0/2847435413 wait complete. 2023-11-28T12:59:22.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.512+0000 7fa5e6338700 1 Processor -- start 2023-11-28T12:59:22.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.512+0000 7f23e425f700 1 Processor -- start 2023-11-28T12:59:22.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.512+0000 7f23e425f700 1 -- start start 2023-11-28T12:59:22.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.511+0000 7fb34c21b700 1 -- 172.21.15.111:0/2061773070 shutdown_connections 2023-11-28T12:59:22.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.511+0000 7fb34c21b700 1 -- 172.21.15.111:0/2061773070 wait complete. 2023-11-28T12:59:22.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7fb34c21b700 1 Processor -- start 2023-11-28T12:59:22.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.512+0000 7f880739c700 1 Processor -- start 2023-11-28T12:59:22.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.512+0000 7f880739c700 1 -- start start 2023-11-28T12:59:22.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f880739c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88000d2df0 0x7f88000cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f880739c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f88000b4fc0 0x7f88000d31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f880739c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88000d1950 con 0x7f88000b1c60 2023-11-28T12:59:22.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f880739c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88000d3d30 con 0x7f88000b4fc0 2023-11-28T12:59:22.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f880739c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88000b2500 con 0x7f88000d2df0 2023-11-28T12:59:22.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.512+0000 7f087d7cb700 1 Processor -- start 2023-11-28T12:59:22.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.512+0000 7f087d7cb700 1 -- start start 2023-11-28T12:59:22.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f087d7cb700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f087806a270 0x7f087806a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f087d7cb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f08780ccf40 0x7f08780cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f087d7cb700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087806ac00 con 0x7f0878160e10 2023-11-28T12:59:22.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f087d7cb700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0878165740 con 0x7f08780ccf40 2023-11-28T12:59:22.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f087d7cb700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08780ae060 con 0x7f087806a270 2023-11-28T12:59:22.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f08777fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f08780ccf40 0x7f08780cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f08777fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f08780ccf40 0x7f08780cd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33508/0 (socket says 172.21.15.111:33508) 2023-11-28T12:59:22.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f08777fe700 1 -- 172.21.15.111:0/1352356827 learned_addr learned my addr 172.21.15.111:0/1352356827 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:22.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.512+0000 7fa5e6338700 1 -- start start 2023-11-28T12:59:22.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7fa5e6338700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e00cdee0 0x7fa5e0148de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7fa5e6338700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa5e015ce70 0x7fa5e00c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7fa5e6338700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5e015afd0 0x7fa5e0153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7fa5e6338700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e0159fc0 con 0x7fa5e00cdee0 2023-11-28T12:59:22.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7fa5e6338700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e015a210 con 0x7fa5e015ce70 2023-11-28T12:59:22.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7fa5e6338700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e015a4f0 con 0x7fa5e015afd0 2023-11-28T12:59:22.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7fa5dffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e00cdee0 0x7fa5e0148de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7fa5dffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e00cdee0 0x7fa5e0148de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33504/0 (socket says 172.21.15.111:33504) 2023-11-28T12:59:22.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7fa5dffff700 1 -- 172.21.15.111:0/2439070217 learned_addr learned my addr 172.21.15.111:0/2439070217 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:22.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7fa5df7fe700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa5e015ce70 0x7fa5e00c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7fb34c21b700 1 -- start start 2023-11-28T12:59:22.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f0c09090700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0be80682f0 0x7f0be806a7a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40097d0 tx=0x7f0bf4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f08777fe700 1 -- 172.21.15.111:0/1352356827 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f087806a270 msgr2=0x7f087806a6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T12:59:22.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f08777fe700 1 --2- 172.21.15.111:0/1352356827 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f087806a270 0x7f087806a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f08777fe700 1 -- 172.21.15.111:0/1352356827 >> v1:172.21.15.111:6789/0 conn(0x7f0878160e10 legacy=0x7f08780ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:22.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f08777fe700 1 -- 172.21.15.111:0/1352356827 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0878164bc0 con 0x7f08780ccf40 2023-11-28T12:59:22.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7fa5e48d5700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5e015afd0 0x7fa5e0153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7fb34c21b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb330003b90 0x7fb3441fe6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7fb34c21b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34406a270 0x7fb34406a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7fb34c21b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb34415ae30 0x7fb34415b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f08777fe700 1 --2- 172.21.15.111:0/1352356827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08780ccf40 0x7f08780cd3a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f086c0098f0 tx=0x7f086c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608a9272d10221d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7fb34c21b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3441577d0 con 0x7fb34406a270 2023-11-28T12:59:22.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7fb34c21b700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34406ac20 con 0x7fb34415ae30 2023-11-28T12:59:22.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7fb34c21b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34415c790 con 0x7fb330003b90 2023-11-28T12:59:22.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7fb349fb7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb330003b90 0x7fb3441fe6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7fb349fb7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb330003b90 0x7fb3441fe6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55264/0 (socket says 172.21.15.111:55264) 2023-11-28T12:59:22.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7fb349fb7700 1 -- 172.21.15.111:0/3304406649 learned_addr learned my addr 172.21.15.111:0/3304406649 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:22.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7fb34a7b8700 1 --2- 172.21.15.111:0/3304406649 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb34415ae30 0x7fb34415b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fb349fb7700 1 -- 172.21.15.111:0/3304406649 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb34415ae30 msgr2=0x7fb34415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fb349fb7700 1 --2- 172.21.15.111:0/3304406649 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb34415ae30 0x7fb34415b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fb349fb7700 1 -- 172.21.15.111:0/3304406649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34406a270 msgr2=0x7fb34406a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fb349fb7700 1 --2- 172.21.15.111:0/3304406649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34406a270 0x7fb34406a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fb349fb7700 1 -- 172.21.15.111:0/3304406649 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3440acdd0 con 0x7fb330003b90 2023-11-28T12:59:22.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7f0875ffb700 1 -- 172.21.15.111:0/1352356827 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c004230 con 0x7f08780ccf40 2023-11-28T12:59:22.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fb349fb7700 1 --2- 172.21.15.111:0/3304406649 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb330003b90 0x7fb3441fe6b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb33000c160 tx=0x7fb340001300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fa5dffff700 1 -- 172.21.15.111:0/2439070217 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5e015afd0 msgr2=0x7fa5e0153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fa5dffff700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5e015afd0 0x7fa5e0153880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fa5dffff700 1 -- 172.21.15.111:0/2439070217 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa5e015ce70 msgr2=0x7fa5e00c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fa5dffff700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa5e015ce70 0x7fa5e00c8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fa5dffff700 1 -- 172.21.15.111:0/2439070217 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d000a640 con 0x7fa5e00cdee0 2023-11-28T12:59:22.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fa5dffff700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e00cdee0 0x7fa5e0148de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa5d400b4a0 tx=0x7fa5d400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fa5dcff9700 1 -- 172.21.15.111:0/2439070217 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d4011830 con 0x7fa5e00cdee0 2023-11-28T12:59:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fa5dcff9700 1 -- 172.21.15.111:0/2439070217 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d40119b0 con 0x7fa5e00cdee0 2023-11-28T12:59:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fa5e6338700 1 -- 172.21.15.111:0/2439070217 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e00cd6e0 con 0x7fa5e00cdee0 2023-11-28T12:59:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7fa5e6338700 1 -- 172.21.15.111:0/2439070217 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e00cd400 con 0x7fa5e00cdee0 2023-11-28T12:59:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f23e425f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23dc089280 0x7f23dc088790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f23e425f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23dc0897d0 0x7f23dc0c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f23e425f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23dc16b760 0x7f23dc2015a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f23e425f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc0c8850 con 0x7f23dc16b760 2023-11-28T12:59:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f23e425f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc0c7ce0 con 0x7f23dc089280 2023-11-28T12:59:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.513+0000 7f23e425f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc0c7fb0 con 0x7f23dc0897d0 2023-11-28T12:59:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f23e17fa700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23dc0897d0 0x7f23dc0c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f23e17fa700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23dc0897d0 0x7f23dc0c77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55248/0 (socket says 172.21.15.111:55248) 2023-11-28T12:59:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f23e17fa700 1 -- 172.21.15.111:0/2618588320 learned_addr learned my addr 172.21.15.111:0/2618588320 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f23e27fc700 1 --2- 172.21.15.111:0/2618588320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23dc16b760 0x7f23dc2015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f23e17fa700 1 -- 172.21.15.111:0/2618588320 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23dc089280 msgr2=0x7f23dc088790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f23e17fa700 1 --2- 172.21.15.111:0/2618588320 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23dc089280 0x7f23dc088790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f23e17fa700 1 -- 172.21.15.111:0/2618588320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23dc16b760 msgr2=0x7f23dc2015a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f23e17fa700 1 --2- 172.21.15.111:0/2618588320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23dc16b760 0x7f23dc2015a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.514+0000 7f23e17fa700 1 -- 172.21.15.111:0/2618588320 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d400c050 con 0x7f23dc0897d0 2023-11-28T12:59:22.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7fb33a7fc700 1 -- 172.21.15.111:0/3304406649 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340016040 con 0x7fb330003b90 2023-11-28T12:59:22.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7fb33a7fc700 1 -- 172.21.15.111:0/3304406649 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3400054e0 con 0x7fb330003b90 2023-11-28T12:59:22.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7fb33a7fc700 1 -- 172.21.15.111:0/3304406649 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340007260 con 0x7fb330003b90 2023-11-28T12:59:22.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7fb34c21b700 1 -- 172.21.15.111:0/3304406649 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3440acaf0 con 0x7fb330003b90 2023-11-28T12:59:22.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7fb34c21b700 1 -- 172.21.15.111:0/3304406649 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3440ad0b0 con 0x7fb330003b90 2023-11-28T12:59:22.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7f8805138700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88000d2df0 0x7f88000cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7f8805138700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88000d2df0 0x7f88000cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55270/0 (socket says 172.21.15.111:55270) 2023-11-28T12:59:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7f8805138700 1 -- 172.21.15.111:0/3365310564 learned_addr learned my addr 172.21.15.111:0/3365310564 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.517+0000 7f8804937700 1 --2- 172.21.15.111:0/3365310564 >> v2:172.21.15.111:3300/0 conn(0x7f88000b4fc0 0x7f88000d31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7f0875ffb700 1 -- 172.21.15.111:0/1352356827 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c004390 con 0x7f08780ccf40 2023-11-28T12:59:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.515+0000 7f0875ffb700 1 -- 172.21.15.111:0/1352356827 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c0036a0 con 0x7f08780ccf40 2023-11-28T12:59:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7f087d7cb700 1 -- 172.21.15.111:0/1352356827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08780ccf40 msgr2=0x7f08780cd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7f087d7cb700 1 --2- 172.21.15.111:0/1352356827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08780ccf40 0x7f08780cd3a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f086c0098f0 tx=0x7f086c009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7f087d7cb700 1 -- 172.21.15.111:0/1352356827 shutdown_connections 2023-11-28T12:59:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7f087d7cb700 1 --2- 172.21.15.111:0/1352356827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08780ccf40 0x7f08780cd3a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7f087d7cb700 1 --2- 172.21.15.111:0/1352356827 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f087806a270 0x7f087806a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7f087d7cb700 1 -- 172.21.15.111:0/1352356827 >> 172.21.15.111:0/1352356827 conn(0x7f08780cb490 msgr2=0x7f08780cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7f087d7cb700 1 -- 172.21.15.111:0/1352356827 shutdown_connections 2023-11-28T12:59:22.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.516+0000 7f087d7cb700 1 -- 172.21.15.111:0/1352356827 wait complete. 2023-11-28T12:59:22.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.517+0000 7fb34c21b700 1 -- 172.21.15.111:0/3304406649 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3440c3700 con 0x7fb330003b90 2023-11-28T12:59:22.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.517+0000 7f0bfa7fc700 1 -- 172.21.15.111:0/711179556 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0c00053c70 con 0x7f0bf0003a00 2023-11-28T12:59:22.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.517+0000 7fa5e6338700 1 -- 172.21.15.111:0/2439070217 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e00cd9c0 con 0x7fa5e00cdee0 2023-11-28T12:59:22.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.517+0000 7f8805138700 1 -- 172.21.15.111:0/3365310564 >> v2:172.21.15.111:3300/0 conn(0x7f88000b4fc0 msgr2=0x7f88000d31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:22.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.517+0000 7f8805138700 1 --2- 172.21.15.111:0/3365310564 >> v2:172.21.15.111:3300/0 conn(0x7f88000b4fc0 0x7f88000d31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.517+0000 7f8805138700 1 -- 172.21.15.111:0/3365310564 >> v1:172.21.15.149:6789/0 conn(0x7f88000b1c60 legacy=0x7f88000cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.517+0000 7f8805138700 1 -- 172.21.15.111:0/3365310564 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88000d3a50 con 0x7f88000d2df0 2023-11-28T12:59:22.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.517+0000 7f8804937700 1 --2- 172.21.15.111:0/3365310564 >> v2:172.21.15.111:3300/0 conn(0x7f88000b4fc0 0x7f88000d31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:22.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.519+0000 7fb33a7fc700 1 -- 172.21.15.111:0/3304406649 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb340010050 con 0x7fb330003b90 2023-11-28T12:59:22.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.518+0000 7f8805138700 1 --2- 172.21.15.111:0/3365310564 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88000d2df0 0x7f88000cd620 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0098f0 tx=0x7f87fc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=235655fb64f4f319 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.518+0000 7fa5dcff9700 1 -- 172.21.15.111:0/2439070217 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d4011d60 con 0x7fa5e00cdee0 2023-11-28T12:59:22.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.518+0000 7fa5dcff9700 1 -- 172.21.15.111:0/2439070217 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa5d400f050 con 0x7fa5e00cdee0 2023-11-28T12:59:22.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.518+0000 7fa5dcff9700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5cc0683f0 0x7fa5cc06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.518+0000 7fa5dcff9700 1 -- 172.21.15.111:0/2439070217 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa5d4085020 con 0x7fa5e00cdee0 2023-11-28T12:59:22.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.518+0000 7fa5df7fe700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5cc0683f0 0x7fa5cc06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.519+0000 7fb33a7fc700 1 --2- 172.21.15.111:0/3304406649 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb330069ea0 0x7fb33006c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.519+0000 7fb33a7fc700 1 -- 172.21.15.111:0/3304406649 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb340086e70 con 0x7fb330003b90 2023-11-28T12:59:22.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.519+0000 7fb3497b6700 1 --2- 172.21.15.111:0/3304406649 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb330069ea0 0x7fb33006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.519+0000 7fa5df7fe700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5cc0683f0 0x7fa5cc06a8a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa5d000ad30 tx=0x7fa5d0012000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.520+0000 7fb3497b6700 1 --2- 172.21.15.111:0/3304406649 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb330069ea0 0x7fb33006c350 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb334001050 tx=0x7fb334001580 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.518+0000 7f087d7cb700 1 Processor -- start 2023-11-28T12:59:22.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.520+0000 7f087d7cb700 1 -- start start 2023-11-28T12:59:22.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.521+0000 7f23e17fa700 1 --2- 172.21.15.111:0/2618588320 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23dc0897d0 0x7f23dc0c77a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f23cc00c520 tx=0x7f23cc00c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.521+0000 7f087d7cb700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f087806a270 0x7f08780c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.521+0000 7f087d7cb700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08780ccf40 0x7f08781611e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.521+0000 7f087d7cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0878160e10 0x7f0878161720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.521+0000 7f087d7cb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0878150850 con 0x7f0878160e10 2023-11-28T12:59:22.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.521+0000 7f087d7cb700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0878149b30 con 0x7f087806a270 2023-11-28T12:59:22.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.521+0000 7f087d7cb700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08781634c0 con 0x7f08780ccf40 2023-11-28T12:59:22.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.521+0000 7f0876ffd700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f087806a270 0x7f08780c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.521+0000 7f0876ffd700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f087806a270 0x7f08780c34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:51432/0 (socket says 172.21.15.111:51432) 2023-11-28T12:59:22.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.521+0000 7f0876ffd700 1 -- 172.21.15.111:0/3171354828 learned_addr learned my addr 172.21.15.111:0/3171354828 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:22.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.521+0000 7f08767fc700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08780ccf40 0x7f08781611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.521+0000 7f08777fe700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0878160e10 0x7f0878161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7ff063c10700 1 Processor -- start 2023-11-28T12:59:22.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f08767fc700 1 -- 172.21.15.111:0/3171354828 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f087806a270 msgr2=0x7f08780c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f08767fc700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f087806a270 0x7f08780c34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f08767fc700 1 -- 172.21.15.111:0/3171354828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0878160e10 msgr2=0x7f0878161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f08767fc700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0878160e10 0x7f0878161720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f08767fc700 1 -- 172.21.15.111:0/3171354828 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c009550 con 0x7f08780ccf40 2023-11-28T12:59:22.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f0876ffd700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f087806a270 0x7f08780c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:22.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f87f7fff700 1 -- 172.21.15.111:0/3365310564 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc004230 con 0x7f88000d2df0 2023-11-28T12:59:22.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f87f7fff700 1 -- 172.21.15.111:0/3365310564 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87fc004390 con 0x7f88000d2df0 2023-11-28T12:59:22.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f87f7fff700 1 -- 172.21.15.111:0/3365310564 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc0036a0 con 0x7f88000d2df0 2023-11-28T12:59:22.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f08777fe700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0878160e10 0x7f0878161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:22.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f08767fc700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08780ccf40 0x7f08781611e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f087000bd50 tx=0x7f087000bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7ff063c10700 1 -- start start 2023-11-28T12:59:22.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f085f7fe700 1 -- 172.21.15.111:0/3171354828 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087000cbb0 con 0x7f08780ccf40 2023-11-28T12:59:22.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f085f7fe700 1 -- 172.21.15.111:0/3171354828 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087000cd30 con 0x7f08780ccf40 2023-11-28T12:59:22.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f085f7fe700 1 -- 172.21.15.111:0/3171354828 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0870014680 con 0x7f08780ccf40 2023-11-28T12:59:22.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7fb33a7fc700 1 -- 172.21.15.111:0/3304406649 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb340056eb0 con 0x7fb330003b90 2023-11-28T12:59:22.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.522+0000 7f087d7cb700 1 -- 172.21.15.111:0/3171354828 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0878165180 con 0x7f08780ccf40 2023-11-28T12:59:22.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.523+0000 7ff063c10700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff05c06a4e0 0x7ff05c0b0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.523+0000 7ff063c10700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff05c0b2300 0x7ff05c06a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.523+0000 7ff063c10700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff05c0afdc0 con 0x7ff05c06adf0 2023-11-28T12:59:22.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.523+0000 7ff063c10700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff05c0c2d30 con 0x7ff05c0b2300 2023-11-28T12:59:22.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.523+0000 7ff063c10700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff05c0cf750 con 0x7ff05c06a4e0 2023-11-28T12:59:22.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.523+0000 7ff0611ab700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff05c0b2300 0x7ff05c06a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.524+0000 7ff0611ab700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff05c0b2300 0x7ff05c06a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33540/0 (socket says 172.21.15.111:33540) 2023-11-28T12:59:22.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.524+0000 7ff0611ab700 1 -- 172.21.15.111:0/3833806151 learned_addr learned my addr 172.21.15.111:0/3833806151 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:22.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.524+0000 7ff0611ab700 1 -- 172.21.15.111:0/3833806151 >> v2:172.21.15.149:3300/0 conn(0x7ff05c06a4e0 msgr2=0x7ff05c0b0250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T12:59:22.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.524+0000 7ff0611ab700 1 --2- 172.21.15.111:0/3833806151 >> v2:172.21.15.149:3300/0 conn(0x7ff05c06a4e0 0x7ff05c0b0250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.524+0000 7ff0611ab700 1 -- 172.21.15.111:0/3833806151 >> v1:172.21.15.111:6789/0 conn(0x7ff05c06adf0 legacy=0x7ff05c0cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.524+0000 7ff0611ab700 1 -- 172.21.15.111:0/3833806151 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c0cf470 con 0x7ff05c0b2300 2023-11-28T12:59:22.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.524+0000 7ff0611ab700 1 --2- 172.21.15.111:0/3833806151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff05c0b2300 0x7ff05c06a8b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff05c06d490 tx=0x7ff058009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6e1f4e87de0c3b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.523+0000 7f087d7cb700 1 -- 172.21.15.111:0/3171354828 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0878164ea0 con 0x7f08780ccf40 2023-11-28T12:59:22.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.524+0000 7f087d7cb700 1 -- 172.21.15.111:0/3171354828 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0878165460 con 0x7f08780ccf40 2023-11-28T12:59:22.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.525+0000 7f085f7fe700 1 -- 172.21.15.111:0/3171354828 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f087001b030 con 0x7f08780ccf40 2023-11-28T12:59:22.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.525+0000 7f085f7fe700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f08640682f0 0x7f086406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.525+0000 7f085f7fe700 1 -- 172.21.15.111:0/3171354828 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0870085ae0 con 0x7f08780ccf40 2023-11-28T12:59:22.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.525+0000 7f0876ffd700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f08640682f0 0x7f086406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.525+0000 7f0876ffd700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f08640682f0 0x7f086406a7a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0878164710 tx=0x7f0868009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.527+0000 7fa5dcff9700 1 -- 172.21.15.111:0/2439070217 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa5d408a060 con 0x7fa5e00cdee0 2023-11-28T12:59:22.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.528+0000 7f880739c700 1 -- 172.21.15.111:0/3365310564 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88000d2df0 msgr2=0x7f88000cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.528+0000 7f880739c700 1 --2- 172.21.15.111:0/3365310564 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88000d2df0 0x7f88000cd620 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0098f0 tx=0x7f87fc009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:22.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.528+0000 7f880739c700 1 -- 172.21.15.111:0/3365310564 shutdown_connections 2023-11-28T12:59:22.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.528+0000 7f880739c700 1 --2- 172.21.15.111:0/3365310564 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88000d2df0 0x7f88000cd620 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.528+0000 7f880739c700 1 --2- 172.21.15.111:0/3365310564 >> v2:172.21.15.111:3300/0 conn(0x7f88000b4fc0 0x7f88000d31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.528+0000 7f880739c700 1 -- 172.21.15.111:0/3365310564 >> 172.21.15.111:0/3365310564 conn(0x7f88000719a0 msgr2=0x7f8800071da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.529+0000 7f23d27fc700 1 -- 172.21.15.111:0/2618588320 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc0052d0 con 0x7f23dc0897d0 2023-11-28T12:59:22.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.529+0000 7f23d27fc700 1 -- 172.21.15.111:0/2618588320 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23cc005430 con 0x7f23dc0897d0 2023-11-28T12:59:22.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.529+0000 7f23d27fc700 1 -- 172.21.15.111:0/2618588320 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc0038c0 con 0x7f23dc0897d0 2023-11-28T12:59:22.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.529+0000 7f23e425f700 1 -- 172.21.15.111:0/2618588320 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23dc16da30 con 0x7f23dc0897d0 2023-11-28T12:59:22.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.529+0000 7f23e425f700 1 -- 172.21.15.111:0/2618588320 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23dc16d750 con 0x7f23dc0897d0 2023-11-28T12:59:22.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.529+0000 7f085f7fe700 1 -- 172.21.15.111:0/3171354828 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0870052000 con 0x7f08780ccf40 2023-11-28T12:59:22.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.529+0000 7ff0609aa700 1 -- 172.21.15.111:0/3833806151 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05800b870 con 0x7ff05c0b2300 2023-11-28T12:59:22.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.529+0000 7ff0609aa700 1 -- 172.21.15.111:0/3833806151 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05800b9d0 con 0x7ff05c0b2300 2023-11-28T12:59:22.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.529+0000 7ff0609aa700 1 -- 172.21.15.111:0/3833806151 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05800bd60 con 0x7ff05c0b2300 2023-11-28T12:59:22.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.530+0000 7ff063c10700 1 -- 172.21.15.111:0/3833806151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff05c0b2300 msgr2=0x7ff05c06a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.530+0000 7ff063c10700 1 --2- 172.21.15.111:0/3833806151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff05c0b2300 0x7ff05c06a8b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff05c06d490 tx=0x7ff058009bd0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.531+0000 7f23e425f700 1 -- 172.21.15.111:0/2618588320 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23dc0c9d50 con 0x7f23dc0897d0 2023-11-28T12:59:22.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.532+0000 7f23d27fc700 1 -- 172.21.15.111:0/2618588320 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f23cc019030 con 0x7f23dc0897d0 2023-11-28T12:59:22.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.532+0000 7f23d27fc700 1 --2- 172.21.15.111:0/2618588320 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f23c00682f0 0x7f23c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.532+0000 7f23d27fc700 1 -- 172.21.15.111:0/2618588320 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f23cc0853a0 con 0x7f23dc0897d0 2023-11-28T12:59:22.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.532+0000 7f23e1ffb700 1 --2- 172.21.15.111:0/2618588320 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f23c00682f0 0x7f23c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.533+0000 7f23e1ffb700 1 --2- 172.21.15.111:0/2618588320 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f23c00682f0 0x7f23c006a7a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f23dc166a80 tx=0x7f23d8006cd0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.533+0000 7ff063c10700 1 -- 172.21.15.111:0/3833806151 shutdown_connections 2023-11-28T12:59:22.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.533+0000 7ff063c10700 1 --2- 172.21.15.111:0/3833806151 >> v2:172.21.15.149:3300/0 conn(0x7ff05c06a4e0 0x7ff05c0b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.533+0000 7ff063c10700 1 --2- 172.21.15.111:0/3833806151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff05c0b2300 0x7ff05c06a8b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.533+0000 7ff063c10700 1 -- 172.21.15.111:0/3833806151 >> 172.21.15.111:0/3833806151 conn(0x7ff05c098ee0 msgr2=0x7ff05c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.534+0000 7ff063c10700 1 -- 172.21.15.111:0/3833806151 shutdown_connections 2023-11-28T12:59:22.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.534+0000 7ff063c10700 1 -- 172.21.15.111:0/3833806151 wait complete. 2023-11-28T12:59:22.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.535+0000 7ff063c10700 1 Processor -- start 2023-11-28T12:59:22.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.536+0000 7f23d27fc700 1 -- 172.21.15.111:0/2618588320 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f23cc0518b0 con 0x7f23dc0897d0 2023-11-28T12:59:22.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.535+0000 7ff063c10700 1 -- start start 2023-11-28T12:59:22.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.537+0000 7ff063c10700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff05c06a4e0 0x7ff05c154950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.537+0000 7ff063c10700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff05c06adf0 0x7ff05c1583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.537+0000 7ff063c10700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff05c0b2300 0x7ff05c0cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.537+0000 7ff063c10700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c1f9d70 con 0x7ff05c06a4e0 2023-11-28T12:59:22.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.537+0000 7ff063c10700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c154e90 con 0x7ff05c06adf0 2023-11-28T12:59:22.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.537+0000 7ff063c10700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c1f9fe0 con 0x7ff05c0b2300 2023-11-28T12:59:22.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.541+0000 7ff0621ad700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff05c0b2300 0x7ff05c0cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.541+0000 7ff0619ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff05c06a4e0 0x7ff05c154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.541+0000 7ff0621ad700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff05c0b2300 0x7ff05c0cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55298/0 (socket says 172.21.15.111:55298) 2023-11-28T12:59:22.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.541+0000 7ff0621ad700 1 -- 172.21.15.111:0/963027373 learned_addr learned my addr 172.21.15.111:0/963027373 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:22.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.541+0000 7ff0621ad700 1 -- 172.21.15.111:0/963027373 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff05c06adf0 msgr2=0x7ff05c1583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.541+0000 7ff0621ad700 1 --2- 172.21.15.111:0/963027373 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff05c06adf0 0x7ff05c1583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.541+0000 7ff0621ad700 1 -- 172.21.15.111:0/963027373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff05c06a4e0 msgr2=0x7ff05c154950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.541+0000 7ff0621ad700 1 --2- 172.21.15.111:0/963027373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff05c06a4e0 0x7ff05c154950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.541+0000 7ff0621ad700 1 -- 172.21.15.111:0/963027373 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff058009550 con 0x7ff05c0b2300 2023-11-28T12:59:22.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.542+0000 7f880739c700 1 -- 172.21.15.111:0/3365310564 shutdown_connections 2023-11-28T12:59:22.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.544+0000 7ff0621ad700 1 --2- 172.21.15.111:0/963027373 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff05c0b2300 0x7ff05c0cb690 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff05400cb40 tx=0x7ff05400ce50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.544+0000 7ff0527fc700 1 -- 172.21.15.111:0/963027373 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054003c00 con 0x7ff05c0b2300 2023-11-28T12:59:22.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.544+0000 7ff0527fc700 1 -- 172.21.15.111:0/963027373 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff054003da0 con 0x7ff05c0b2300 2023-11-28T12:59:22.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.544+0000 7ff0527fc700 1 -- 172.21.15.111:0/963027373 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05400aae0 con 0x7ff05c0b2300 2023-11-28T12:59:22.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.545+0000 7ff063c10700 1 -- 172.21.15.111:0/963027373 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff05c0af770 con 0x7ff05c0b2300 2023-11-28T12:59:22.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.545+0000 7ff063c10700 1 -- 172.21.15.111:0/963027373 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05c0af490 con 0x7ff05c0b2300 2023-11-28T12:59:22.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.545+0000 7f880739c700 1 -- 172.21.15.111:0/3365310564 wait complete. 2023-11-28T12:59:22.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.546+0000 7f880739c700 1 Processor -- start 2023-11-28T12:59:22.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.547+0000 7f880739c700 1 -- start start 2023-11-28T12:59:22.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.548+0000 7f880739c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f88000b1c60 0x7f88001e5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.548+0000 7f880739c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88000b4fc0 0x7f88001e5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.548+0000 7f880739c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88000d2df0 0x7f88000b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.548+0000 7f880739c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88001e7ac0 con 0x7f88000b4fc0 2023-11-28T12:59:22.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.548+0000 7f880739c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88001e7cd0 con 0x7f88000b1c60 2023-11-28T12:59:22.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.548+0000 7f880739c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88001e7fb0 con 0x7f88000d2df0 2023-11-28T12:59:22.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.548+0000 7f8805138700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f88000b1c60 0x7f88001e5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.548+0000 7f8805138700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f88000b1c60 0x7f88001e5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:51452/0 (socket says 172.21.15.111:51452) 2023-11-28T12:59:22.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.548+0000 7f8805138700 1 -- 172.21.15.111:0/4193448040 learned_addr learned my addr 172.21.15.111:0/4193448040 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:22.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.548+0000 7f8804937700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88000b4fc0 0x7f88001e5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.548+0000 7f8805939700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88000d2df0 0x7f88000b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.549+0000 7f8805939700 1 -- 172.21.15.111:0/4193448040 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f88000b1c60 msgr2=0x7f88001e5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.549+0000 7f8805939700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f88000b1c60 0x7f88001e5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.549+0000 7f8805939700 1 -- 172.21.15.111:0/4193448040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88000b4fc0 msgr2=0x7f88001e5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.549+0000 7f8805939700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88000b4fc0 0x7f88001e5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.549+0000 7f8805939700 1 -- 172.21.15.111:0/4193448040 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc009550 con 0x7f88000d2df0 2023-11-28T12:59:22.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.549+0000 7f8805939700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88000d2df0 0x7f88000b6a50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f87f800c7f0 tx=0x7f87f800cb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.550+0000 7ff0527fc700 1 -- 172.21.15.111:0/963027373 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff054010050 con 0x7ff05c0b2300 2023-11-28T12:59:22.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.551+0000 7ff0527fc700 1 --2- 172.21.15.111:0/963027373 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff0400682c0 0x7ff04006a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.551+0000 7ff0527fc700 1 -- 172.21.15.111:0/963027373 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff054085d70 con 0x7ff05c0b2300 2023-11-28T12:59:22.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.556+0000 7ff0619ac700 1 --2- 172.21.15.111:0/963027373 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff0400682c0 0x7ff04006a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.556+0000 7ff0619ac700 1 --2- 172.21.15.111:0/963027373 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff0400682c0 0x7ff04006a770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0099f0 tx=0x7ff04c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.560+0000 7ff063c10700 1 -- 172.21.15.111:0/963027373 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0440052f0 con 0x7ff05c0b2300 2023-11-28T12:59:22.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.561+0000 7f87f5ffb700 1 -- 172.21.15.111:0/4193448040 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f8003e30 con 0x7f88000d2df0 2023-11-28T12:59:22.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.561+0000 7f87f5ffb700 1 -- 172.21.15.111:0/4193448040 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f8005070 con 0x7f88000d2df0 2023-11-28T12:59:22.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.561+0000 7f87f5ffb700 1 -- 172.21.15.111:0/4193448040 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f8005400 con 0x7f88000d2df0 2023-11-28T12:59:22.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.561+0000 7f880739c700 1 -- 172.21.15.111:0/4193448040 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88000b88f0 con 0x7f88000d2df0 2023-11-28T12:59:22.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.561+0000 7f880739c700 1 -- 172.21.15.111:0/4193448040 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88000d15e0 con 0x7f88000d2df0 2023-11-28T12:59:22.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.564+0000 7f880739c700 1 -- 172.21.15.111:0/4193448040 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88000b21b0 con 0x7f88000d2df0 2023-11-28T12:59:22.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.565+0000 7ff0527fc700 1 -- 172.21.15.111:0/963027373 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff054052320 con 0x7ff05c0b2300 2023-11-28T12:59:22.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.582+0000 7f87f5ffb700 1 -- 172.21.15.111:0/4193448040 <== mon.2 v2:172.21.15.149:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f87f8019030 con 0x7f88000d2df0 2023-11-28T12:59:22.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.583+0000 7f87f5ffb700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87e40683c0 0x7f87e406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:22.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.583+0000 7f87f5ffb700 1 -- 172.21.15.111:0/4193448040 <== mon.2 v2:172.21.15.149:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f87f80858e0 con 0x7f88000d2df0 2023-11-28T12:59:22.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.583+0000 7f87f5ffb700 1 -- 172.21.15.111:0/4193448040 <== mon.2 v2:172.21.15.149:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f87f80b0680 con 0x7f88000d2df0 2023-11-28T12:59:22.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.583+0000 7f8805138700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87e40683c0 0x7f87e406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:22.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.584+0000 7f8805138700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87e40683c0 0x7f87e406a870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f87fc004720 tx=0x7f87fc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:22.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.839+0000 7fa8e77ef700 1 -- 172.21.15.111:0/1352709261 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fa8e0159320 con 0x7fa8e01edaf0 2023-11-28T12:59:22.844 INFO:teuthology.orchestra.run.smithi111.stdout:55834574858 2023-11-28T12:59:22.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.840+0000 7fa8d5ffb700 1 -- 172.21.15.111:0/1352709261 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa8dc053b10 con 0x7fa8e01edaf0 2023-11-28T12:59:22.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.848+0000 7fa8e77ef700 1 -- 172.21.15.111:0/1352709261 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa8cc07ac80 msgr2=0x7fa8cc07d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.848+0000 7fa8e77ef700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa8cc07ac80 0x7fa8cc07d130 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8001b20 tx=0x7fa8d8009040 comp rx=0 tx=0).stop 2023-11-28T12:59:22.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.848+0000 7fa8e77ef700 1 -- 172.21.15.111:0/1352709261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8e01edaf0 msgr2=0x7fa8e00af6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.848+0000 7fa8e77ef700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8e01edaf0 0x7fa8e00af6f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc014840 tx=0x7fa8dc0047c0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.848+0000 7fa8e77ef700 1 -- 172.21.15.111:0/1352709261 shutdown_connections 2023-11-28T12:59:22.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.848+0000 7fa8e77ef700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa8cc07ac80 0x7fa8cc07d130 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.848+0000 7fa8e77ef700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa8e00afc30 0x7fa8e00b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.848+0000 7fa8e77ef700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa8e01ed720 0x7fa8e00b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.848+0000 7fa8e77ef700 1 --2- 172.21.15.111:0/1352709261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8e01edaf0 0x7fa8e00af6f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.848+0000 7fa8e77ef700 1 -- 172.21.15.111:0/1352709261 >> 172.21.15.111:0/1352709261 conn(0x7fa8e00cb490 msgr2=0x7fa8e0070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.848+0000 7fa8e77ef700 1 -- 172.21.15.111:0/1352709261 shutdown_connections 2023-11-28T12:59:22.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.850+0000 7fa8e77ef700 1 -- 172.21.15.111:0/1352709261 wait complete. 2023-11-28T12:59:22.869 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574858 for osd.7 2023-11-28T12:59:22.869 DEBUG:teuthology.parallel:result is None 2023-11-28T12:59:22.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.899+0000 7f23e425f700 1 -- 172.21.15.111:0/2618588320 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f23dc067880 con 0x7f23dc0897d0 2023-11-28T12:59:22.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.899+0000 7f23d27fc700 1 -- 172.21.15.111:0/2618588320 <== mon.2 v2:172.21.15.149:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f23cc054ed0 con 0x7f23dc0897d0 2023-11-28T12:59:22.901 INFO:teuthology.orchestra.run.smithi111.stdout:55834574859 2023-11-28T12:59:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.903+0000 7f23e425f700 1 -- 172.21.15.111:0/2618588320 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f23c00682f0 msgr2=0x7f23c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.903+0000 7f23e425f700 1 --2- 172.21.15.111:0/2618588320 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f23c00682f0 0x7f23c006a7a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f23dc166a80 tx=0x7f23d8006cd0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.903+0000 7f23e425f700 1 -- 172.21.15.111:0/2618588320 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23dc0897d0 msgr2=0x7f23dc0c77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.903+0000 7f23e425f700 1 --2- 172.21.15.111:0/2618588320 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23dc0897d0 0x7f23dc0c77a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f23cc00c520 tx=0x7f23cc00c8e0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.903+0000 7f23e425f700 1 -- 172.21.15.111:0/2618588320 shutdown_connections 2023-11-28T12:59:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.903+0000 7f23e425f700 1 --2- 172.21.15.111:0/2618588320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23dc16b760 0x7f23dc2015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.903+0000 7f23e425f700 1 --2- 172.21.15.111:0/2618588320 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f23c00682f0 0x7f23c006a7a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.903+0000 7f23e425f700 1 --2- 172.21.15.111:0/2618588320 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23dc089280 0x7f23dc088790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.903+0000 7f23e425f700 1 --2- 172.21.15.111:0/2618588320 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23dc0897d0 0x7f23dc0c77a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.903+0000 7f23e425f700 1 -- 172.21.15.111:0/2618588320 >> 172.21.15.111:0/2618588320 conn(0x7f23dc0cb490 msgr2=0x7f23dc169b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.903+0000 7f23e425f700 1 -- 172.21.15.111:0/2618588320 shutdown_connections 2023-11-28T12:59:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.903+0000 7f23e425f700 1 -- 172.21.15.111:0/2618588320 wait complete. 2023-11-28T12:59:22.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.910+0000 7f0c0baf5700 1 -- 172.21.15.111:0/711179556 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0bec0055a0 con 0x7f0bf0003a00 2023-11-28T12:59:22.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.910+0000 7f0bfa7fc700 1 -- 172.21.15.111:0/711179556 <== mon.2 v2:172.21.15.149:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0c00022020 con 0x7f0bf0003a00 2023-11-28T12:59:22.912 INFO:teuthology.orchestra.run.smithi111.stdout:55834574859 2023-11-28T12:59:22.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.912+0000 7fa5e6338700 1 -- 172.21.15.111:0/2439070217 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa5e0067880 con 0x7fa5e00cdee0 2023-11-28T12:59:22.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.913+0000 7fa5dcff9700 1 -- 172.21.15.111:0/2439070217 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa5d4051580 con 0x7fa5e00cdee0 2023-11-28T12:59:22.915 INFO:teuthology.orchestra.run.smithi111.stdout:55834574860 2023-11-28T12:59:22.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.914+0000 7fb34c21b700 1 -- 172.21.15.111:0/3304406649 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb344067170 con 0x7fb330003b90 2023-11-28T12:59:22.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.915+0000 7f0c0baf5700 1 -- 172.21.15.111:0/711179556 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0be80682f0 msgr2=0x7f0be806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.915+0000 7f0c0baf5700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0be80682f0 0x7f0be806a7a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40097d0 tx=0x7f0bf4008040 comp rx=0 tx=0).stop 2023-11-28T12:59:22.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.915+0000 7f0c0baf5700 1 -- 172.21.15.111:0/711179556 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0bf0003a00 msgr2=0x7f0c040c14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.915+0000 7f0c0baf5700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0bf0003a00 0x7f0c040c14b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0c0000c9e0 tx=0x7f0c00001930 comp rx=0 tx=0).stop 2023-11-28T12:59:22.916 INFO:teuthology.orchestra.run.smithi111.stdout:55834574859 2023-11-28T12:59:22.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.915+0000 7fb33a7fc700 1 -- 172.21.15.111:0/3304406649 <== mon.2 v2:172.21.15.149:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb340056aa0 con 0x7fb330003b90 2023-11-28T12:59:22.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.916+0000 7f087d7cb700 1 -- 172.21.15.111:0/3171354828 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0878067880 con 0x7f08780ccf40 2023-11-28T12:59:22.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.916+0000 7ff063c10700 1 -- 172.21.15.111:0/963027373 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff0440055a0 con 0x7ff05c0b2300 2023-11-28T12:59:22.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.916+0000 7f085f7fe700 1 -- 172.21.15.111:0/3171354828 <== mon.2 v2:172.21.15.149:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0870019020 con 0x7f08780ccf40 2023-11-28T12:59:22.918 INFO:teuthology.orchestra.run.smithi111.stdout:55834574859 2023-11-28T12:59:22.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.917+0000 7ff0527fc700 1 -- 172.21.15.111:0/963027373 <== mon.2 v2:172.21.15.149:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff054055940 con 0x7ff05c0b2300 2023-11-28T12:59:22.919 INFO:teuthology.orchestra.run.smithi111.stdout:55834574859 2023-11-28T12:59:22.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fa5e6338700 1 -- 172.21.15.111:0/2439070217 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5cc0683f0 msgr2=0x7fa5cc06a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fa5e6338700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5cc0683f0 0x7fa5cc06a8a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa5d000ad30 tx=0x7fa5d0012000 comp rx=0 tx=0).stop 2023-11-28T12:59:22.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fa5e6338700 1 -- 172.21.15.111:0/2439070217 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e00cdee0 msgr2=0x7fa5e0148de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fa5e6338700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e00cdee0 0x7fa5e0148de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa5d400b4a0 tx=0x7fa5d400b7b0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fb34c21b700 1 -- 172.21.15.111:0/3304406649 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb330069ea0 msgr2=0x7fb33006c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fb34c21b700 1 --2- 172.21.15.111:0/3304406649 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb330069ea0 0x7fb33006c350 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb334001050 tx=0x7fb334001580 comp rx=0 tx=0).stop 2023-11-28T12:59:22.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fb34c21b700 1 -- 172.21.15.111:0/3304406649 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb330003b90 msgr2=0x7fb3441fe6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fb34c21b700 1 --2- 172.21.15.111:0/3304406649 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb330003b90 0x7fb3441fe6b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb33000c160 tx=0x7fb340001300 comp rx=0 tx=0).stop 2023-11-28T12:59:22.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fb34c21b700 1 -- 172.21.15.111:0/3304406649 shutdown_connections 2023-11-28T12:59:22.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fb34c21b700 1 --2- 172.21.15.111:0/3304406649 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb330069ea0 0x7fb33006c350 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fb34c21b700 1 --2- 172.21.15.111:0/3304406649 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb34415ae30 0x7fb34415b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fb34c21b700 1 --2- 172.21.15.111:0/3304406649 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb330003b90 0x7fb3441fe6b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fb34c21b700 1 --2- 172.21.15.111:0/3304406649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb34406a270 0x7fb34406a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fb34c21b700 1 -- 172.21.15.111:0/3304406649 >> 172.21.15.111:0/3304406649 conn(0x7fb344098ee0 msgr2=0x7fb34406ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fb34c21b700 1 -- 172.21.15.111:0/3304406649 shutdown_connections 2023-11-28T12:59:22.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.918+0000 7fb34c21b700 1 -- 172.21.15.111:0/3304406649 wait complete. 2023-11-28T12:59:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.919+0000 7fa5e6338700 1 -- 172.21.15.111:0/2439070217 shutdown_connections 2023-11-28T12:59:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.919+0000 7fa5e6338700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5e015afd0 0x7fa5e0153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.919+0000 7fa5e6338700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e00cdee0 0x7fa5e0148de0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.919+0000 7fa5e6338700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5cc0683f0 0x7fa5cc06a8a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.919+0000 7fa5e6338700 1 --2- 172.21.15.111:0/2439070217 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa5e015ce70 0x7fa5e00c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.919+0000 7fa5e6338700 1 -- 172.21.15.111:0/2439070217 >> 172.21.15.111:0/2439070217 conn(0x7fa5e00cb490 msgr2=0x7fa5e006c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.919+0000 7fa5e6338700 1 -- 172.21.15.111:0/2439070217 shutdown_connections 2023-11-28T12:59:22.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.919+0000 7fa5e6338700 1 -- 172.21.15.111:0/2439070217 wait complete. 2023-11-28T12:59:22.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.919+0000 7f087d7cb700 1 -- 172.21.15.111:0/3171354828 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f08640682f0 msgr2=0x7f086406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.919+0000 7f087d7cb700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f08640682f0 0x7f086406a7a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0878164710 tx=0x7f0868009190 comp rx=0 tx=0).stop 2023-11-28T12:59:22.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.919+0000 7f087d7cb700 1 -- 172.21.15.111:0/3171354828 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08780ccf40 msgr2=0x7f08781611e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.919+0000 7f087d7cb700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08780ccf40 0x7f08781611e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f087000bd50 tx=0x7f087000bd80 comp rx=0 tx=0).stop 2023-11-28T12:59:22.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.920+0000 7ff063c10700 1 -- 172.21.15.111:0/963027373 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff0400682c0 msgr2=0x7ff04006a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.920+0000 7ff063c10700 1 --2- 172.21.15.111:0/963027373 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff0400682c0 0x7ff04006a770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0099f0 tx=0x7ff04c009190 comp rx=0 tx=0).stop 2023-11-28T12:59:22.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.920+0000 7ff063c10700 1 -- 172.21.15.111:0/963027373 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff05c0b2300 msgr2=0x7ff05c0cb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.920+0000 7ff063c10700 1 --2- 172.21.15.111:0/963027373 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff05c0b2300 0x7ff05c0cb690 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff05400cb40 tx=0x7ff05400ce50 comp rx=0 tx=0).stop 2023-11-28T12:59:22.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.920+0000 7ff063c10700 1 -- 172.21.15.111:0/963027373 shutdown_connections 2023-11-28T12:59:22.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.920+0000 7ff063c10700 1 --2- 172.21.15.111:0/963027373 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff05c0b2300 0x7ff05c0cb690 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.920+0000 7ff063c10700 1 --2- 172.21.15.111:0/963027373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff05c06a4e0 0x7ff05c154950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.920+0000 7ff063c10700 1 --2- 172.21.15.111:0/963027373 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff0400682c0 0x7ff04006a770 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.920+0000 7ff063c10700 1 --2- 172.21.15.111:0/963027373 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff05c06adf0 0x7ff05c1583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.920+0000 7ff063c10700 1 -- 172.21.15.111:0/963027373 >> 172.21.15.111:0/963027373 conn(0x7ff05c098ee0 msgr2=0x7ff05c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.920+0000 7ff063c10700 1 -- 172.21.15.111:0/963027373 shutdown_connections 2023-11-28T12:59:22.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.920+0000 7ff063c10700 1 -- 172.21.15.111:0/963027373 wait complete. 2023-11-28T12:59:22.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.922+0000 7f0c0baf5700 1 -- 172.21.15.111:0/711179556 shutdown_connections 2023-11-28T12:59:22.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.922+0000 7f0c0baf5700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0be80682f0 0x7f0be806a7a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.922+0000 7f0c0baf5700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0c040c24f0 0x7f0c04144c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.922+0000 7f0c0baf5700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0bf0003a00 0x7f0c040c14b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.922+0000 7f0c0baf5700 1 --2- 172.21.15.111:0/711179556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c0415fb60 0x7f0c040bed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.922+0000 7f0c0baf5700 1 -- 172.21.15.111:0/711179556 >> 172.21.15.111:0/711179556 conn(0x7f0c040cb490 msgr2=0x7f0c040b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.922+0000 7f087d7cb700 1 -- 172.21.15.111:0/3171354828 shutdown_connections 2023-11-28T12:59:22.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.922+0000 7f087d7cb700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0878160e10 0x7f0878161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.922+0000 7f087d7cb700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f08640682f0 0x7f086406a7a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.922+0000 7f087d7cb700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f087806a270 0x7f08780c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.922+0000 7f087d7cb700 1 --2- 172.21.15.111:0/3171354828 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08780ccf40 0x7f08781611e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.922+0000 7f087d7cb700 1 -- 172.21.15.111:0/3171354828 >> 172.21.15.111:0/3171354828 conn(0x7f08780cb490 msgr2=0x7f0878070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.922+0000 7f087d7cb700 1 -- 172.21.15.111:0/3171354828 shutdown_connections 2023-11-28T12:59:22.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.926+0000 7f087d7cb700 1 -- 172.21.15.111:0/3171354828 wait complete. 2023-11-28T12:59:22.928 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.1 2023-11-28T12:59:22.928 DEBUG:teuthology.parallel:result is None 2023-11-28T12:59:22.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.932+0000 7f0c0baf5700 1 -- 172.21.15.111:0/711179556 shutdown_connections 2023-11-28T12:59:22.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.932+0000 7f0c0baf5700 1 -- 172.21.15.111:0/711179556 wait complete. 2023-11-28T12:59:22.936 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.2 2023-11-28T12:59:22.936 DEBUG:teuthology.parallel:result is None 2023-11-28T12:59:22.943 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.3 2023-11-28T12:59:22.943 DEBUG:teuthology.parallel:result is None 2023-11-28T12:59:22.944 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574860 for osd.6 2023-11-28T12:59:22.944 DEBUG:teuthology.parallel:result is None 2023-11-28T12:59:22.947 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.0 2023-11-28T12:59:22.947 DEBUG:teuthology.parallel:result is None 2023-11-28T12:59:22.950 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.5 2023-11-28T12:59:22.950 DEBUG:teuthology.parallel:result is None 2023-11-28T12:59:22.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.951+0000 7f880739c700 1 -- 172.21.15.111:0/4193448040 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f88001e80f0 con 0x7f88000d2df0 2023-11-28T12:59:22.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.951+0000 7f87f5ffb700 1 -- 172.21.15.111:0/4193448040 <== mon.2 v2:172.21.15.149:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f88001e80f0 con 0x7f88000d2df0 2023-11-28T12:59:22.953 INFO:teuthology.orchestra.run.smithi111.stdout:55834574857 2023-11-28T12:59:22.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.953+0000 7f880739c700 1 -- 172.21.15.111:0/4193448040 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87e40683c0 msgr2=0x7f87e406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.953+0000 7f880739c700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87e40683c0 0x7f87e406a870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f87fc004720 tx=0x7f87fc008030 comp rx=0 tx=0).stop 2023-11-28T12:59:22.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.954+0000 7f880739c700 1 -- 172.21.15.111:0/4193448040 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88000d2df0 msgr2=0x7f88000b6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:22.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.954+0000 7f880739c700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88000d2df0 0x7f88000b6a50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f87f800c7f0 tx=0x7f87f800cb00 comp rx=0 tx=0).stop 2023-11-28T12:59:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.954+0000 7f880739c700 1 -- 172.21.15.111:0/4193448040 shutdown_connections 2023-11-28T12:59:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.954+0000 7f880739c700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88000d2df0 0x7f88000b6a50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.954+0000 7f880739c700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87e40683c0 0x7f87e406a870 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.954+0000 7f880739c700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f88000b1c60 0x7f88001e5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.954+0000 7f880739c700 1 --2- 172.21.15.111:0/4193448040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88000b4fc0 0x7f88001e5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.954+0000 7f880739c700 1 -- 172.21.15.111:0/4193448040 >> 172.21.15.111:0/4193448040 conn(0x7f88000719a0 msgr2=0x7f88000b2820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.954+0000 7f880739c700 1 -- 172.21.15.111:0/4193448040 shutdown_connections 2023-11-28T12:59:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:22.954+0000 7f880739c700 1 -- 172.21.15.111:0/4193448040 wait complete. 2023-11-28T12:59:22.967 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.4 2023-11-28T12:59:22.967 DEBUG:teuthology.parallel:result is None 2023-11-28T12:59:22.967 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-28T12:59:22.967 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T12:59:23.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.125+0000 7f6062a7f700 1 Processor -- start 2023-11-28T12:59:23.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.125+0000 7f6062a7f700 1 -- start start 2023-11-28T12:59:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f6062a7f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f605c0cef60 0x7f605c0ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f6062a7f700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f605c0b0160 0x7f605c0adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f6062a7f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605c0cc390 con 0x7f605c15a5d0 2023-11-28T12:59:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f6062a7f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605c0cc1b0 con 0x7f605c0cef60 2023-11-28T12:59:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f6062a7f700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605c0cc670 con 0x7f605c0b0160 2023-11-28T12:59:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f605bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f605c0cef60 0x7f605c0ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f606081b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f605c15a5d0 0x7f605c0ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34340/0 (socket says 172.21.15.111:34340) 2023-11-28T12:59:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f606081b700 1 -- 172.21.15.111:0/3481145635 learned_addr learned my addr 172.21.15.111:0/3481145635 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f606101c700 1 --2- 172.21.15.111:0/3481145635 >> v2:172.21.15.149:3300/0 conn(0x7f605c0b0160 0x7f605c0adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f605bfff700 1 -- 172.21.15.111:0/3481145635 >> v2:172.21.15.149:3300/0 conn(0x7f605c0b0160 msgr2=0x7f605c0adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f605bfff700 1 --2- 172.21.15.111:0/3481145635 >> v2:172.21.15.149:3300/0 conn(0x7f605c0b0160 0x7f605c0adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f605bfff700 1 -- 172.21.15.111:0/3481145635 >> v1:172.21.15.111:6789/0 conn(0x7f605c15a5d0 legacy=0x7f605c0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f605b7fe700 1 -- 172.21.15.111:0/3481145635 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114901660 0 0) 0x7f605c0cc390 con 0x7f605c15a5d0 2023-11-28T12:59:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f605bfff700 1 -- 172.21.15.111:0/3481145635 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c0cc950 con 0x7f605c0cef60 2023-11-28T12:59:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.126+0000 7f605bfff700 1 --2- 172.21.15.111:0/3481145635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605c0cef60 0x7f605c0ad8a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f605c08a070 tx=0x7f6048009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=335723b6ee77afdc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.127+0000 7f605b7fe700 1 -- 172.21.15.111:0/3481145635 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6048011830 con 0x7f605c0cef60 2023-11-28T12:59:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.127+0000 7f605b7fe700 1 -- 172.21.15.111:0/3481145635 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6048011990 con 0x7f605c0cef60 2023-11-28T12:59:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.127+0000 7f605b7fe700 1 -- 172.21.15.111:0/3481145635 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6048011d20 con 0x7f605c0cef60 2023-11-28T12:59:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.127+0000 7f6062a7f700 1 -- 172.21.15.111:0/3481145635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605c0cef60 msgr2=0x7f605c0ad8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.127+0000 7f6062a7f700 1 --2- 172.21.15.111:0/3481145635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605c0cef60 0x7f605c0ad8a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f605c08a070 tx=0x7f6048009b20 comp rx=0 tx=0).stop 2023-11-28T12:59:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.127+0000 7f6062a7f700 1 -- 172.21.15.111:0/3481145635 shutdown_connections 2023-11-28T12:59:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.127+0000 7f6062a7f700 1 --2- 172.21.15.111:0/3481145635 >> v2:172.21.15.149:3300/0 conn(0x7f605c0b0160 0x7f605c0adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.127+0000 7f6062a7f700 1 --2- 172.21.15.111:0/3481145635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605c0cef60 0x7f605c0ad8a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.127+0000 7f6062a7f700 1 -- 172.21.15.111:0/3481145635 >> 172.21.15.111:0/3481145635 conn(0x7f605c09a040 msgr2=0x7f605c0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:23.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.127+0000 7f6062a7f700 1 -- 172.21.15.111:0/3481145635 shutdown_connections 2023-11-28T12:59:23.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.127+0000 7f6062a7f700 1 -- 172.21.15.111:0/3481145635 wait complete. 2023-11-28T12:59:23.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.128+0000 7f6062a7f700 1 Processor -- start 2023-11-28T12:59:23.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.128+0000 7f6062a7f700 1 -- start start 2023-11-28T12:59:23.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.129+0000 7f6062a7f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605c0b0160 0x7f605c150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.129+0000 7f6062a7f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f605c0cef60 0x7f605c0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.129+0000 7f6062a7f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f605c15a5d0 0x7f605c0b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.129+0000 7f6062a7f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c0c7d50 con 0x7f605c0b0160 2023-11-28T12:59:23.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.129+0000 7f6062a7f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c0b2000 con 0x7f605c0cef60 2023-11-28T12:59:23.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.129+0000 7f6062a7f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605c0b22e0 con 0x7f605c15a5d0 2023-11-28T12:59:23.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.129+0000 7f606081b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605c0b0160 0x7f605c150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.129+0000 7f606081b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605c0b0160 0x7f605c150130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33568/0 (socket says 172.21.15.111:33568) 2023-11-28T12:59:23.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.129+0000 7f606081b700 1 -- 172.21.15.111:0/1735013695 learned_addr learned my addr 172.21.15.111:0/1735013695 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:23.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.130+0000 7f605bfff700 1 --2- 172.21.15.111:0/1735013695 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f605c0cef60 0x7f605c0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.130+0000 7f606081b700 1 -- 172.21.15.111:0/1735013695 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f605c15a5d0 msgr2=0x7f605c0b17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.130+0000 7f606081b700 1 --2- 172.21.15.111:0/1735013695 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f605c15a5d0 0x7f605c0b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.130+0000 7f606081b700 1 -- 172.21.15.111:0/1735013695 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f605c0cef60 msgr2=0x7f605c0b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.130+0000 7f606081b700 1 --2- 172.21.15.111:0/1735013695 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f605c0cef60 0x7f605c0b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.130+0000 7f606081b700 1 -- 172.21.15.111:0/1735013695 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60480094a0 con 0x7f605c0b0160 2023-11-28T12:59:23.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.130+0000 7f606081b700 1 --2- 172.21.15.111:0/1735013695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605c0b0160 0x7f605c150130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f605000c4a0 tx=0x7f605000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:23.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.130+0000 7f60597fa700 1 -- 172.21.15.111:0/1735013695 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60500074e0 con 0x7f605c0b0160 2023-11-28T12:59:23.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.130+0000 7f60597fa700 1 -- 172.21.15.111:0/1735013695 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6050007640 con 0x7f605c0b0160 2023-11-28T12:59:23.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.130+0000 7f6062a7f700 1 -- 172.21.15.111:0/1735013695 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605c0ccf10 con 0x7f605c0b0160 2023-11-28T12:59:23.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.130+0000 7f60597fa700 1 -- 172.21.15.111:0/1735013695 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6050005a70 con 0x7f605c0b0160 2023-11-28T12:59:23.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.130+0000 7f6062a7f700 1 -- 172.21.15.111:0/1735013695 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605c0ccc30 con 0x7f605c0b0160 2023-11-28T12:59:23.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.131+0000 7f6062a7f700 1 -- 172.21.15.111:0/1735013695 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605c15a9a0 con 0x7f605c0b0160 2023-11-28T12:59:23.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.131+0000 7f60597fa700 1 -- 172.21.15.111:0/1735013695 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6050019030 con 0x7f605c0b0160 2023-11-28T12:59:23.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.132+0000 7f60597fa700 1 --2- 172.21.15.111:0/1735013695 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f60400682f0 0x7f604006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.132+0000 7f60597fa700 1 -- 172.21.15.111:0/1735013695 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6050085350 con 0x7f605c0b0160 2023-11-28T12:59:23.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.132+0000 7f605bfff700 1 --2- 172.21.15.111:0/1735013695 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f60400682f0 0x7f604006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.132+0000 7f605bfff700 1 --2- 172.21.15.111:0/1735013695 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f60400682f0 0x7f604006a7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f605c0cf350 tx=0x7f6048008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:23.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.136+0000 7f60597fa700 1 -- 172.21.15.111:0/1735013695 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f60500518c0 con 0x7f605c0b0160 2023-11-28T12:59:23.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.315+0000 7f6062a7f700 1 -- 172.21.15.111:0/1735013695 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f605c0b2450 con 0x7f60400682f0 2023-11-28T12:59:23.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.325+0000 7f60597fa700 1 -- 172.21.15.111:0/1735013695 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367319 (secure 0 0 0) 0x7f605c0b2450 con 0x7f60400682f0 2023-11-28T12:59:23.330 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:23.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.328+0000 7f6062a7f700 1 -- 172.21.15.111:0/1735013695 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f60400682f0 msgr2=0x7f604006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.328+0000 7f6062a7f700 1 --2- 172.21.15.111:0/1735013695 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f60400682f0 0x7f604006a7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f605c0cf350 tx=0x7f6048008030 comp rx=0 tx=0).stop 2023-11-28T12:59:23.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.328+0000 7f6062a7f700 1 -- 172.21.15.111:0/1735013695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605c0b0160 msgr2=0x7f605c150130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.328+0000 7f6062a7f700 1 --2- 172.21.15.111:0/1735013695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605c0b0160 0x7f605c150130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f605000c4a0 tx=0x7f605000c860 comp rx=0 tx=0).stop 2023-11-28T12:59:23.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.328+0000 7f6062a7f700 1 -- 172.21.15.111:0/1735013695 shutdown_connections 2023-11-28T12:59:23.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.328+0000 7f6062a7f700 1 --2- 172.21.15.111:0/1735013695 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f605c15a5d0 0x7f605c0b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.328+0000 7f6062a7f700 1 --2- 172.21.15.111:0/1735013695 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f605c0b0160 0x7f605c150130 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.328+0000 7f6062a7f700 1 --2- 172.21.15.111:0/1735013695 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f60400682f0 0x7f604006a7a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.328+0000 7f6062a7f700 1 --2- 172.21.15.111:0/1735013695 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f605c0cef60 0x7f605c0b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.328+0000 7f6062a7f700 1 -- 172.21.15.111:0/1735013695 >> 172.21.15.111:0/1735013695 conn(0x7f605c09a040 msgr2=0x7f605c06ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:23.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.328+0000 7f6062a7f700 1 -- 172.21.15.111:0/1735013695 shutdown_connections 2023-11-28T12:59:23.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.328+0000 7f6062a7f700 1 -- 172.21.15.111:0/1735013695 wait complete. 2023-11-28T12:59:23.334 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T12:59:23.342 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-11-28T12:59:22.339709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594896,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":67400,"kb_used_data":3400,"kb_used_omap":0,"kb_used_meta":63936,"kb_avail":754907320,"statfs":{"total":773094113280,"available":773025095680,"internally_reserved":0,"allocated":3481600,"data_stored":2108544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65470464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003657"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672369+0000","last_change":"2023-11-28T12:58:55.661175+0000","last_active":"2023-11-28T12:58:59.672369+0000","last_peered":"2023-11-28T12:58:59.672369+0000","last_clean":"2023-11-28T12:58:59.672369+0000","last_became_active":"2023-11-28T12:58:55.660919+0000","last_became_peered":"2023-11-28T12:58:55.660919+0000","last_unstale":"2023-11-28T12:58:59.672369+0000","last_undegraded":"2023-11-28T12:58:59.672369+0000","last_fullsized":"2023-11-28T12:58:59.672369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:12.137203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666089+0000","last_change":"2023-11-28T12:58:55.652749+0000","last_active":"2023-11-28T12:58:59.666089+0000","last_peered":"2023-11-28T12:58:59.666089+0000","last_clean":"2023-11-28T12:58:59.666089+0000","last_became_active":"2023-11-28T12:58:55.652544+0000","last_became_peered":"2023-11-28T12:58:55.652544+0000","last_unstale":"2023-11-28T12:58:59.666089+0000","last_undegraded":"2023-11-28T12:58:59.666089+0000","last_fullsized":"2023-11-28T12:58:59.666089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660310+0000","last_change":"2023-11-28T12:58:55.669208+0000","last_active":"2023-11-28T12:58:59.660310+0000","last_peered":"2023-11-28T12:58:59.660310+0000","last_clean":"2023-11-28T12:58:59.660310+0000","last_became_active":"2023-11-28T12:58:55.668930+0000","last_became_peered":"2023-11-28T12:58:55.668930+0000","last_unstale":"2023-11-28T12:58:59.660310+0000","last_undegraded":"2023-11-28T12:58:59.660310+0000","last_fullsized":"2023-11-28T12:58:59.660310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:18:05.065329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657123+0000","last_change":"2023-11-28T12:58:55.653156+0000","last_active":"2023-11-28T12:58:59.657123+0000","last_peered":"2023-11-28T12:58:59.657123+0000","last_clean":"2023-11-28T12:58:59.657123+0000","last_became_active":"2023-11-28T12:58:55.652820+0000","last_became_peered":"2023-11-28T12:58:55.652820+0000","last_unstale":"2023-11-28T12:58:59.657123+0000","last_undegraded":"2023-11-28T12:58:59.657123+0000","last_fullsized":"2023-11-28T12:58:59.657123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:19.178525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669305+0000","last_change":"2023-11-28T12:58:55.664471+0000","last_active":"2023-11-28T12:58:59.669305+0000","last_peered":"2023-11-28T12:58:59.669305+0000","last_clean":"2023-11-28T12:58:59.669305+0000","last_became_active":"2023-11-28T12:58:55.664221+0000","last_became_peered":"2023-11-28T12:58:55.664221+0000","last_unstale":"2023-11-28T12:58:59.669305+0000","last_undegraded":"2023-11-28T12:58:59.669305+0000","last_fullsized":"2023-11-28T12:58:59.669305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:34:15.223824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659490+0000","last_change":"2023-11-28T12:58:55.669109+0000","last_active":"2023-11-28T12:58:59.659490+0000","last_peered":"2023-11-28T12:58:59.659490+0000","last_clean":"2023-11-28T12:58:59.659490+0000","last_became_active":"2023-11-28T12:58:55.668857+0000","last_became_peered":"2023-11-28T12:58:55.668857+0000","last_unstale":"2023-11-28T12:58:59.659490+0000","last_undegraded":"2023-11-28T12:58:59.659490+0000","last_fullsized":"2023-11-28T12:58:59.659490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:14:02.504692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.663977+0000","last_change":"2023-11-28T12:58:55.662854+0000","last_active":"2023-11-28T12:58:59.663977+0000","last_peered":"2023-11-28T12:58:59.663977+0000","last_clean":"2023-11-28T12:58:59.663977+0000","last_became_active":"2023-11-28T12:58:55.662539+0000","last_became_peered":"2023-11-28T12:58:55.662539+0000","last_unstale":"2023-11-28T12:58:59.663977+0000","last_undegraded":"2023-11-28T12:58:59.663977+0000","last_fullsized":"2023-11-28T12:58:59.663977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661304+0000","last_change":"2023-11-28T12:58:55.657958+0000","last_active":"2023-11-28T12:58:59.661304+0000","last_peered":"2023-11-28T12:58:59.661304+0000","last_clean":"2023-11-28T12:58:59.661304+0000","last_became_active":"2023-11-28T12:58:55.657681+0000","last_became_peered":"2023-11-28T12:58:55.657681+0000","last_unstale":"2023-11-28T12:58:59.661304+0000","last_undegraded":"2023-11-28T12:58:59.661304+0000","last_fullsized":"2023-11-28T12:58:59.661304+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674142+0000","last_change":"2023-11-28T12:58:55.666485+0000","last_active":"2023-11-28T12:58:59.674142+0000","last_peered":"2023-11-28T12:58:59.674142+0000","last_clean":"2023-11-28T12:58:59.674142+0000","last_became_active":"2023-11-28T12:58:55.665670+0000","last_became_peered":"2023-11-28T12:58:55.665670+0000","last_unstale":"2023-11-28T12:58:59.674142+0000","last_undegraded":"2023-11-28T12:58:59.674142+0000","last_fullsized":"2023-11-28T12:58:59.674142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668534+0000","last_change":"2023-11-28T12:58:55.655129+0000","last_active":"2023-11-28T12:58:59.668534+0000","last_peered":"2023-11-28T12:58:59.668534+0000","last_clean":"2023-11-28T12:58:59.668534+0000","last_became_active":"2023-11-28T12:58:55.654839+0000","last_became_peered":"2023-11-28T12:58:55.654839+0000","last_unstale":"2023-11-28T12:58:59.668534+0000","last_undegraded":"2023-11-28T12:58:59.668534+0000","last_fullsized":"2023-11-28T12:58:59.668534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660394+0000","last_change":"2023-11-28T12:58:55.663796+0000","last_active":"2023-11-28T12:58:59.660394+0000","last_peered":"2023-11-28T12:58:59.660394+0000","last_clean":"2023-11-28T12:58:59.660394+0000","last_became_active":"2023-11-28T12:58:55.663553+0000","last_became_peered":"2023-11-28T12:58:55.663553+0000","last_unstale":"2023-11-28T12:58:59.660394+0000","last_undegraded":"2023-11-28T12:58:59.660394+0000","last_fullsized":"2023-11-28T12:58:59.660394+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:31:46.923203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672559+0000","last_change":"2023-11-28T12:58:55.666858+0000","last_active":"2023-11-28T12:58:59.672559+0000","last_peered":"2023-11-28T12:58:59.672559+0000","last_clean":"2023-11-28T12:58:59.672559+0000","last_became_active":"2023-11-28T12:58:55.666597+0000","last_became_peered":"2023-11-28T12:58:55.666597+0000","last_unstale":"2023-11-28T12:58:59.672559+0000","last_undegraded":"2023-11-28T12:58:59.672559+0000","last_fullsized":"2023-11-28T12:58:59.672559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:50.239136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.673902+0000","last_change":"2023-11-28T12:58:55.658497+0000","last_active":"2023-11-28T12:58:59.673902+0000","last_peered":"2023-11-28T12:58:59.673902+0000","last_clean":"2023-11-28T12:58:59.673902+0000","last_became_active":"2023-11-28T12:58:55.658258+0000","last_became_peered":"2023-11-28T12:58:55.658258+0000","last_unstale":"2023-11-28T12:58:59.673902+0000","last_undegraded":"2023-11-28T12:58:59.673902+0000","last_fullsized":"2023-11-28T12:58:59.673902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:38:46.935297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664436+0000","last_change":"2023-11-28T12:58:55.648629+0000","last_active":"2023-11-28T12:58:59.664436+0000","last_peered":"2023-11-28T12:58:59.664436+0000","last_clean":"2023-11-28T12:58:59.664436+0000","last_became_active":"2023-11-28T12:58:55.648143+0000","last_became_peered":"2023-11-28T12:58:55.648143+0000","last_unstale":"2023-11-28T12:58:59.664436+0000","last_undegraded":"2023-11-28T12:58:59.664436+0000","last_fullsized":"2023-11-28T12:58:59.664436+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:06:39.770893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.663795+0000","last_change":"2023-11-28T12:58:55.665560+0000","last_active":"2023-11-28T12:58:59.663795+0000","last_peered":"2023-11-28T12:58:59.663795+0000","last_clean":"2023-11-28T12:58:59.663795+0000","last_became_active":"2023-11-28T12:58:55.665304+0000","last_became_peered":"2023-11-28T12:58:55.665304+0000","last_unstale":"2023-11-28T12:58:59.663795+0000","last_undegraded":"2023-11-28T12:58:59.663795+0000","last_fullsized":"2023-11-28T12:58:59.663795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.658903+0000","last_change":"2023-11-28T12:58:55.669222+0000","last_active":"2023-11-28T12:58:59.658903+0000","last_peered":"2023-11-28T12:58:59.658903+0000","last_clean":"2023-11-28T12:58:59.658903+0000","last_became_active":"2023-11-28T12:58:55.668931+0000","last_became_peered":"2023-11-28T12:58:55.668931+0000","last_unstale":"2023-11-28T12:58:59.658903+0000","last_undegraded":"2023-11-28T12:58:59.658903+0000","last_fullsized":"2023-11-28T12:58:59.658903+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:12:28.351912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674186+0000","last_change":"2023-11-28T12:58:55.666643+0000","last_active":"2023-11-28T12:58:59.674186+0000","last_peered":"2023-11-28T12:58:59.674186+0000","last_clean":"2023-11-28T12:58:59.674186+0000","last_became_active":"2023-11-28T12:58:55.666224+0000","last_became_peered":"2023-11-28T12:58:55.666224+0000","last_unstale":"2023-11-28T12:58:59.674186+0000","last_undegraded":"2023-11-28T12:58:59.674186+0000","last_fullsized":"2023-11-28T12:58:59.674186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660392+0000","last_change":"2023-11-28T12:58:55.660396+0000","last_active":"2023-11-28T12:58:59.660392+0000","last_peered":"2023-11-28T12:58:59.660392+0000","last_clean":"2023-11-28T12:58:59.660392+0000","last_became_active":"2023-11-28T12:58:55.659518+0000","last_became_peered":"2023-11-28T12:58:55.659518+0000","last_unstale":"2023-11-28T12:58:59.660392+0000","last_undegraded":"2023-11-28T12:58:59.660392+0000","last_fullsized":"2023-11-28T12:58:59.660392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:49:50.356082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672413+0000","last_change":"2023-11-28T12:58:55.650573+0000","last_active":"2023-11-28T12:58:59.672413+0000","last_peered":"2023-11-28T12:58:59.672413+0000","last_clean":"2023-11-28T12:58:59.672413+0000","last_became_active":"2023-11-28T12:58:55.650382+0000","last_became_peered":"2023-11-28T12:58:55.650382+0000","last_unstale":"2023-11-28T12:58:59.672413+0000","last_undegraded":"2023-11-28T12:58:59.672413+0000","last_fullsized":"2023-11-28T12:58:59.672413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:54:07.154200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659749+0000","last_change":"2023-11-28T12:58:55.663920+0000","last_active":"2023-11-28T12:58:59.659749+0000","last_peered":"2023-11-28T12:58:59.659749+0000","last_clean":"2023-11-28T12:58:59.659749+0000","last_became_active":"2023-11-28T12:58:55.663347+0000","last_became_peered":"2023-11-28T12:58:55.663347+0000","last_unstale":"2023-11-28T12:58:59.659749+0000","last_undegraded":"2023-11-28T12:58:59.659749+0000","last_fullsized":"2023-11-28T12:58:59.659749+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664143+0000","last_change":"2023-11-28T12:58:55.661755+0000","last_active":"2023-11-28T12:58:59.664143+0000","last_peered":"2023-11-28T12:58:59.664143+0000","last_clean":"2023-11-28T12:58:59.664143+0000","last_became_active":"2023-11-28T12:58:55.661496+0000","last_became_peered":"2023-11-28T12:58:55.661496+0000","last_unstale":"2023-11-28T12:58:59.664143+0000","last_undegraded":"2023-11-28T12:58:59.664143+0000","last_fullsized":"2023-11-28T12:58:59.664143+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669008+0000","last_change":"2023-11-28T12:58:55.666002+0000","last_active":"2023-11-28T12:58:59.669008+0000","last_peered":"2023-11-28T12:58:59.669008+0000","last_clean":"2023-11-28T12:58:59.669008+0000","last_became_active":"2023-11-28T12:58:55.657322+0000","last_became_peered":"2023-11-28T12:58:55.657322+0000","last_unstale":"2023-11-28T12:58:59.669008+0000","last_undegraded":"2023-11-28T12:58:59.669008+0000","last_fullsized":"2023-11-28T12:58:59.669008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655373+0000","last_change":"2023-11-28T12:58:55.669042+0000","last_active":"2023-11-28T12:58:59.655373+0000","last_peered":"2023-11-28T12:58:59.655373+0000","last_clean":"2023-11-28T12:58:59.655373+0000","last_became_active":"2023-11-28T12:58:55.668810+0000","last_became_peered":"2023-11-28T12:58:55.668810+0000","last_unstale":"2023-11-28T12:58:59.655373+0000","last_undegraded":"2023-11-28T12:58:59.655373+0000","last_fullsized":"2023-11-28T12:58:59.655373+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:49:50.356082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666243+0000","last_change":"2023-11-28T12:58:55.658006+0000","last_active":"2023-11-28T12:58:59.666243+0000","last_peered":"2023-11-28T12:58:59.666243+0000","last_clean":"2023-11-28T12:58:59.666243+0000","last_became_active":"2023-11-28T12:58:55.657791+0000","last_became_peered":"2023-11-28T12:58:55.657791+0000","last_unstale":"2023-11-28T12:58:59.666243+0000","last_undegraded":"2023-11-28T12:58:59.666243+0000","last_fullsized":"2023-11-28T12:58:59.666243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667574+0000","last_change":"2023-11-28T12:58:55.658052+0000","last_active":"2023-11-28T12:58:59.667574+0000","last_peered":"2023-11-28T12:58:59.667574+0000","last_clean":"2023-11-28T12:58:59.667574+0000","last_became_active":"2023-11-28T12:58:55.657752+0000","last_became_peered":"2023-11-28T12:58:55.657752+0000","last_unstale":"2023-11-28T12:58:59.667574+0000","last_undegraded":"2023-11-28T12:58:59.667574+0000","last_fullsized":"2023-11-28T12:58:59.667574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665870+0000","last_change":"2023-11-28T12:58:55.657589+0000","last_active":"2023-11-28T12:58:59.665870+0000","last_peered":"2023-11-28T12:58:59.665870+0000","last_clean":"2023-11-28T12:58:59.665870+0000","last_became_active":"2023-11-28T12:58:55.654770+0000","last_became_peered":"2023-11-28T12:58:55.654770+0000","last_unstale":"2023-11-28T12:58:59.665870+0000","last_undegraded":"2023-11-28T12:58:59.665870+0000","last_fullsized":"2023-11-28T12:58:59.665870+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:06:39.770893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665748+0000","last_change":"2023-11-28T12:58:55.657244+0000","last_active":"2023-11-28T12:58:59.665748+0000","last_peered":"2023-11-28T12:58:59.665748+0000","last_clean":"2023-11-28T12:58:59.665748+0000","last_became_active":"2023-11-28T12:58:55.656984+0000","last_became_peered":"2023-11-28T12:58:55.656984+0000","last_unstale":"2023-11-28T12:58:59.665748+0000","last_undegraded":"2023-11-28T12:58:59.665748+0000","last_fullsized":"2023-11-28T12:58:59.665748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059204+0000","last_change":"2023-11-28T12:58:59.681861+0000","last_active":"2023-11-28T12:59:00.059204+0000","last_peered":"2023-11-28T12:59:00.059204+0000","last_clean":"2023-11-28T12:59:00.059204+0000","last_became_active":"2023-11-28T12:58:59.681503+0000","last_became_peered":"2023-11-28T12:58:59.681503+0000","last_unstale":"2023-11-28T12:59:00.059204+0000","last_undegraded":"2023-11-28T12:59:00.059204+0000","last_fullsized":"2023-11-28T12:59:00.059204+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:27.673942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659368+0000","last_change":"2023-11-28T12:58:55.248299+0000","last_active":"2023-11-28T12:58:59.659368+0000","last_peered":"2023-11-28T12:58:59.659368+0000","last_clean":"2023-11-28T12:58:59.659368+0000","last_became_active":"2023-11-28T12:58:55.247956+0000","last_became_peered":"2023-11-28T12:58:55.247956+0000","last_unstale":"2023-11-28T12:58:59.659368+0000","last_undegraded":"2023-11-28T12:58:59.659368+0000","last_fullsized":"2023-11-28T12:58:59.659368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:25:52.356939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655726+0000","last_change":"2023-11-28T12:58:55.660586+0000","last_active":"2023-11-28T12:58:59.655726+0000","last_peered":"2023-11-28T12:58:59.655726+0000","last_clean":"2023-11-28T12:58:59.655726+0000","last_became_active":"2023-11-28T12:58:55.658146+0000","last_became_peered":"2023-11-28T12:58:55.658146+0000","last_unstale":"2023-11-28T12:58:59.655726+0000","last_undegraded":"2023-11-28T12:58:59.655726+0000","last_fullsized":"2023-11-28T12:58:59.655726+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058649+0000","last_change":"2023-11-28T12:58:59.672932+0000","last_active":"2023-11-28T12:59:00.058649+0000","last_peered":"2023-11-28T12:59:00.058649+0000","last_clean":"2023-11-28T12:59:00.058649+0000","last_became_active":"2023-11-28T12:58:59.672540+0000","last_became_peered":"2023-11-28T12:58:59.672540+0000","last_unstale":"2023-11-28T12:59:00.058649+0000","last_undegraded":"2023-11-28T12:59:00.058649+0000","last_fullsized":"2023-11-28T12:59:00.058649+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:12:31.778491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661524+0000","last_change":"2023-11-28T12:58:55.247493+0000","last_active":"2023-11-28T12:58:59.661524+0000","last_peered":"2023-11-28T12:58:59.661524+0000","last_clean":"2023-11-28T12:58:59.661524+0000","last_became_active":"2023-11-28T12:58:55.247218+0000","last_became_peered":"2023-11-28T12:58:55.247218+0000","last_unstale":"2023-11-28T12:58:59.661524+0000","last_undegraded":"2023-11-28T12:58:59.661524+0000","last_fullsized":"2023-11-28T12:58:59.661524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:38:45.929395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674242+0000","last_change":"2023-11-28T12:58:55.666790+0000","last_active":"2023-11-28T12:58:59.674242+0000","last_peered":"2023-11-28T12:58:59.674242+0000","last_clean":"2023-11-28T12:58:59.674242+0000","last_became_active":"2023-11-28T12:58:55.666541+0000","last_became_peered":"2023-11-28T12:58:55.666541+0000","last_unstale":"2023-11-28T12:58:59.674242+0000","last_undegraded":"2023-11-28T12:58:59.674242+0000","last_fullsized":"2023-11-28T12:58:59.674242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313692+0000","last_change":"2023-11-28T12:58:59.685314+0000","last_active":"2023-11-28T12:59:00.313692+0000","last_peered":"2023-11-28T12:59:00.313692+0000","last_clean":"2023-11-28T12:59:00.313692+0000","last_became_active":"2023-11-28T12:58:59.684940+0000","last_became_peered":"2023-11-28T12:58:59.684940+0000","last_unstale":"2023-11-28T12:59:00.313692+0000","last_undegraded":"2023-11-28T12:59:00.313692+0000","last_fullsized":"2023-11-28T12:59:00.313692+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:53.458898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657621+0000","last_change":"2023-11-28T12:58:55.256308+0000","last_active":"2023-11-28T12:58:59.657621+0000","last_peered":"2023-11-28T12:58:59.657621+0000","last_clean":"2023-11-28T12:58:59.657621+0000","last_became_active":"2023-11-28T12:58:55.256020+0000","last_became_peered":"2023-11-28T12:58:55.256020+0000","last_unstale":"2023-11-28T12:58:59.657621+0000","last_undegraded":"2023-11-28T12:58:59.657621+0000","last_fullsized":"2023-11-28T12:58:59.657621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:49.233234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659104+0000","last_change":"2023-11-28T12:58:55.658284+0000","last_active":"2023-11-28T12:58:59.659104+0000","last_peered":"2023-11-28T12:58:59.659104+0000","last_clean":"2023-11-28T12:58:59.659104+0000","last_became_active":"2023-11-28T12:58:55.657995+0000","last_became_peered":"2023-11-28T12:58:55.657995+0000","last_unstale":"2023-11-28T12:58:59.659104+0000","last_undegraded":"2023-11-28T12:58:59.659104+0000","last_fullsized":"2023-11-28T12:58:59.659104+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:11:49.546022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058116+0000","last_change":"2023-11-28T12:58:59.669835+0000","last_active":"2023-11-28T12:59:00.058116+0000","last_peered":"2023-11-28T12:59:00.058116+0000","last_clean":"2023-11-28T12:59:00.058116+0000","last_became_active":"2023-11-28T12:58:59.669428+0000","last_became_peered":"2023-11-28T12:58:59.669428+0000","last_unstale":"2023-11-28T12:59:00.058116+0000","last_undegraded":"2023-11-28T12:59:00.058116+0000","last_fullsized":"2023-11-28T12:59:00.058116+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:18:08.491909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674258+0000","last_change":"2023-11-28T12:58:55.242328+0000","last_active":"2023-11-28T12:58:59.674258+0000","last_peered":"2023-11-28T12:58:59.674258+0000","last_clean":"2023-11-28T12:58:59.674258+0000","last_became_active":"2023-11-28T12:58:55.242028+0000","last_became_peered":"2023-11-28T12:58:55.242028+0000","last_unstale":"2023-11-28T12:58:59.674258+0000","last_undegraded":"2023-11-28T12:58:59.674258+0000","last_fullsized":"2023-11-28T12:58:59.674258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:42.300078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655600+0000","last_change":"2023-11-28T12:58:55.655732+0000","last_active":"2023-11-28T12:58:59.655600+0000","last_peered":"2023-11-28T12:58:59.655600+0000","last_clean":"2023-11-28T12:58:59.655600+0000","last_became_active":"2023-11-28T12:58:55.655490+0000","last_became_peered":"2023-11-28T12:58:55.655490+0000","last_unstale":"2023-11-28T12:58:59.655600+0000","last_undegraded":"2023-11-28T12:58:59.655600+0000","last_fullsized":"2023-11-28T12:58:59.655600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829559+0000","last_change":"2023-11-28T12:58:59.676365+0000","last_active":"2023-11-28T12:59:00.829559+0000","last_peered":"2023-11-28T12:59:00.829559+0000","last_clean":"2023-11-28T12:59:00.829559+0000","last_became_active":"2023-11-28T12:58:59.675934+0000","last_became_peered":"2023-11-28T12:58:59.675934+0000","last_unstale":"2023-11-28T12:59:00.829559+0000","last_undegraded":"2023-11-28T12:59:00.829559+0000","last_fullsized":"2023-11-28T12:59:00.829559+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:04:35.226907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668457+0000","last_change":"2023-11-28T12:58:55.244542+0000","last_active":"2023-11-28T12:58:59.668457+0000","last_peered":"2023-11-28T12:58:59.668457+0000","last_clean":"2023-11-28T12:58:59.668457+0000","last_became_active":"2023-11-28T12:58:55.244251+0000","last_became_peered":"2023-11-28T12:58:55.244251+0000","last_unstale":"2023-11-28T12:58:59.668457+0000","last_undegraded":"2023-11-28T12:58:59.668457+0000","last_fullsized":"2023-11-28T12:58:59.668457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:06.134459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655451+0000","last_change":"2023-11-28T12:58:55.661735+0000","last_active":"2023-11-28T12:58:59.655451+0000","last_peered":"2023-11-28T12:58:59.655451+0000","last_clean":"2023-11-28T12:58:59.655451+0000","last_became_active":"2023-11-28T12:58:55.661468+0000","last_became_peered":"2023-11-28T12:58:55.661468+0000","last_unstale":"2023-11-28T12:58:59.655451+0000","last_undegraded":"2023-11-28T12:58:59.655451+0000","last_fullsized":"2023-11-28T12:58:59.655451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:34.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059839+0000","last_change":"2023-11-28T12:58:59.677031+0000","last_active":"2023-11-28T12:59:00.059839+0000","last_peered":"2023-11-28T12:59:00.059839+0000","last_clean":"2023-11-28T12:59:00.059839+0000","last_became_active":"2023-11-28T12:58:59.676510+0000","last_became_peered":"2023-11-28T12:58:59.676510+0000","last_unstale":"2023-11-28T12:59:00.059839+0000","last_undegraded":"2023-11-28T12:59:00.059839+0000","last_fullsized":"2023-11-28T12:59:00.059839+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:03:33.042937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"21'3","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664803+0000","last_change":"2023-11-28T12:58:55.254041+0000","last_active":"2023-11-28T12:58:59.664803+0000","last_peered":"2023-11-28T12:58:59.664803+0000","last_clean":"2023-11-28T12:58:59.664803+0000","last_became_active":"2023-11-28T12:58:55.253797+0000","last_became_peered":"2023-11-28T12:58:55.253797+0000","last_unstale":"2023-11-28T12:58:59.664803+0000","last_undegraded":"2023-11-28T12:58:59.664803+0000","last_fullsized":"2023-11-28T12:58:59.664803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:42.442743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664769+0000","last_change":"2023-11-28T12:58:55.648632+0000","last_active":"2023-11-28T12:58:59.664769+0000","last_peered":"2023-11-28T12:58:59.664769+0000","last_clean":"2023-11-28T12:58:59.664769+0000","last_became_active":"2023-11-28T12:58:55.648367+0000","last_became_peered":"2023-11-28T12:58:55.648367+0000","last_unstale":"2023-11-28T12:58:59.664769+0000","last_undegraded":"2023-11-28T12:58:59.664769+0000","last_fullsized":"2023-11-28T12:58:59.664769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.315401+0000","last_change":"2023-11-28T12:58:59.675361+0000","last_active":"2023-11-28T12:59:00.315401+0000","last_peered":"2023-11-28T12:59:00.315401+0000","last_clean":"2023-11-28T12:59:00.315401+0000","last_became_active":"2023-11-28T12:58:59.674991+0000","last_became_peered":"2023-11-28T12:58:59.674991+0000","last_unstale":"2023-11-28T12:59:00.315401+0000","last_undegraded":"2023-11-28T12:59:00.315401+0000","last_fullsized":"2023-11-28T12:59:00.315401+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:21:52.978291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"26'9","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:04.555380+0000","last_change":"2023-11-28T12:58:55.245189+0000","last_active":"2023-11-28T12:59:04.555380+0000","last_peered":"2023-11-28T12:59:04.555380+0000","last_clean":"2023-11-28T12:59:04.555380+0000","last_became_active":"2023-11-28T12:58:55.244896+0000","last_became_peered":"2023-11-28T12:58:55.244896+0000","last_unstale":"2023-11-28T12:59:04.555380+0000","last_undegraded":"2023-11-28T12:59:04.555380+0000","last_fullsized":"2023-11-28T12:59:04.555380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3746,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659437+0000","last_change":"2023-11-28T12:58:55.658816+0000","last_active":"2023-11-28T12:58:59.659437+0000","last_peered":"2023-11-28T12:58:59.659437+0000","last_clean":"2023-11-28T12:58:59.659437+0000","last_became_active":"2023-11-28T12:58:55.657238+0000","last_became_peered":"2023-11-28T12:58:55.657238+0000","last_unstale":"2023-11-28T12:58:59.659437+0000","last_undegraded":"2023-11-28T12:58:59.659437+0000","last_fullsized":"2023-11-28T12:58:59.659437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:14:28.821164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830316+0000","last_change":"2023-11-28T12:58:59.685273+0000","last_active":"2023-11-28T12:59:00.830316+0000","last_peered":"2023-11-28T12:59:00.830316+0000","last_clean":"2023-11-28T12:59:00.830316+0000","last_became_active":"2023-11-28T12:58:59.684917+0000","last_became_peered":"2023-11-28T12:58:59.684917+0000","last_unstale":"2023-11-28T12:59:00.830316+0000","last_undegraded":"2023-11-28T12:59:00.830316+0000","last_fullsized":"2023-11-28T12:59:00.830316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:31.632801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664060+0000","last_change":"2023-11-28T12:58:55.249201+0000","last_active":"2023-11-28T12:58:59.664060+0000","last_peered":"2023-11-28T12:58:59.664060+0000","last_clean":"2023-11-28T12:58:59.664060+0000","last_became_active":"2023-11-28T12:58:55.248828+0000","last_became_peered":"2023-11-28T12:58:55.248828+0000","last_unstale":"2023-11-28T12:58:59.664060+0000","last_undegraded":"2023-11-28T12:58:59.664060+0000","last_fullsized":"2023-11-28T12:58:59.664060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:11:48.540120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655687+0000","last_change":"2023-11-28T12:58:55.660397+0000","last_active":"2023-11-28T12:58:59.655687+0000","last_peered":"2023-11-28T12:58:59.655687+0000","last_clean":"2023-11-28T12:58:59.655687+0000","last_became_active":"2023-11-28T12:58:55.657972+0000","last_became_peered":"2023-11-28T12:58:55.657972+0000","last_unstale":"2023-11-28T12:58:59.655687+0000","last_undegraded":"2023-11-28T12:58:59.655687+0000","last_fullsized":"2023-11-28T12:58:59.655687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059788+0000","last_change":"2023-11-28T12:58:59.681226+0000","last_active":"2023-11-28T12:59:00.059788+0000","last_peered":"2023-11-28T12:59:00.059788+0000","last_clean":"2023-11-28T12:59:00.059788+0000","last_became_active":"2023-11-28T12:58:59.680846+0000","last_became_peered":"2023-11-28T12:58:59.680846+0000","last_unstale":"2023-11-28T12:59:00.059788+0000","last_undegraded":"2023-11-28T12:59:00.059788+0000","last_fullsized":"2023-11-28T12:59:00.059788+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:59.373707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668909+0000","last_change":"2023-11-28T12:58:55.246343+0000","last_active":"2023-11-28T12:58:59.668909+0000","last_peered":"2023-11-28T12:58:59.668909+0000","last_clean":"2023-11-28T12:58:59.668909+0000","last_became_active":"2023-11-28T12:58:55.244840+0000","last_became_peered":"2023-11-28T12:58:55.244840+0000","last_unstale":"2023-11-28T12:58:59.668909+0000","last_undegraded":"2023-11-28T12:58:59.668909+0000","last_fullsized":"2023-11-28T12:58:59.668909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665665+0000","last_change":"2023-11-28T12:58:55.656948+0000","last_active":"2023-11-28T12:58:59.665665+0000","last_peered":"2023-11-28T12:58:59.665665+0000","last_clean":"2023-11-28T12:58:59.665665+0000","last_became_active":"2023-11-28T12:58:55.656678+0000","last_became_peered":"2023-11-28T12:58:55.656678+0000","last_unstale":"2023-11-28T12:58:59.665665+0000","last_undegraded":"2023-11-28T12:58:59.665665+0000","last_fullsized":"2023-11-28T12:58:59.665665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.060115+0000","last_change":"2023-11-28T12:58:59.677384+0000","last_active":"2023-11-28T12:59:00.060115+0000","last_peered":"2023-11-28T12:59:00.060115+0000","last_clean":"2023-11-28T12:59:00.060115+0000","last_became_active":"2023-11-28T12:58:59.676997+0000","last_became_peered":"2023-11-28T12:58:59.676997+0000","last_unstale":"2023-11-28T12:59:00.060115+0000","last_undegraded":"2023-11-28T12:59:00.060115+0000","last_fullsized":"2023-11-28T12:59:00.060115+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:31.632801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.663889+0000","last_change":"2023-11-28T12:58:55.249928+0000","last_active":"2023-11-28T12:58:59.663889+0000","last_peered":"2023-11-28T12:58:59.663889+0000","last_clean":"2023-11-28T12:58:59.663889+0000","last_became_active":"2023-11-28T12:58:55.249500+0000","last_became_peered":"2023-11-28T12:58:55.249500+0000","last_unstale":"2023-11-28T12:58:59.663889+0000","last_undegraded":"2023-11-28T12:58:59.663889+0000","last_fullsized":"2023-11-28T12:58:59.663889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:18.172623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674319+0000","last_change":"2023-11-28T12:58:55.663464+0000","last_active":"2023-11-28T12:58:59.674319+0000","last_peered":"2023-11-28T12:58:59.674319+0000","last_clean":"2023-11-28T12:58:59.674319+0000","last_became_active":"2023-11-28T12:58:55.663205+0000","last_became_peered":"2023-11-28T12:58:55.663205+0000","last_unstale":"2023-11-28T12:58:59.674319+0000","last_undegraded":"2023-11-28T12:58:59.674319+0000","last_fullsized":"2023-11-28T12:58:59.674319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.831228+0000","last_change":"2023-11-28T12:58:59.675282+0000","last_active":"2023-11-28T12:59:00.831228+0000","last_peered":"2023-11-28T12:59:00.831228+0000","last_clean":"2023-11-28T12:59:00.831228+0000","last_became_active":"2023-11-28T12:58:59.674859+0000","last_became_peered":"2023-11-28T12:58:59.674859+0000","last_unstale":"2023-11-28T12:59:00.831228+0000","last_undegraded":"2023-11-28T12:59:00.831228+0000","last_fullsized":"2023-11-28T12:59:00.831228+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:59.373707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"21'8","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.656905+0000","last_change":"2023-11-28T12:58:55.236728+0000","last_active":"2023-11-28T12:58:59.656905+0000","last_peered":"2023-11-28T12:58:59.656905+0000","last_clean":"2023-11-28T12:58:59.656905+0000","last_became_active":"2023-11-28T12:58:55.236452+0000","last_became_peered":"2023-11-28T12:58:55.236452+0000","last_unstale":"2023-11-28T12:58:59.656905+0000","last_undegraded":"2023-11-28T12:58:59.656905+0000","last_fullsized":"2023-11-28T12:58:59.656905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655960+0000","last_change":"2023-11-28T12:58:55.662005+0000","last_active":"2023-11-28T12:58:59.655960+0000","last_peered":"2023-11-28T12:58:59.655960+0000","last_clean":"2023-11-28T12:58:59.655960+0000","last_became_active":"2023-11-28T12:58:55.661531+0000","last_became_peered":"2023-11-28T12:58:55.661531+0000","last_unstale":"2023-11-28T12:58:59.655960+0000","last_undegraded":"2023-11-28T12:58:59.655960+0000","last_fullsized":"2023-11-28T12:58:59.655960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:34:15.223824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830169+0000","last_change":"2023-11-28T12:58:59.675000+0000","last_active":"2023-11-28T12:59:00.830169+0000","last_peered":"2023-11-28T12:59:00.830169+0000","last_clean":"2023-11-28T12:59:00.830169+0000","last_became_active":"2023-11-28T12:58:59.674646+0000","last_became_peered":"2023-11-28T12:58:59.674646+0000","last_unstale":"2023-11-28T12:59:00.830169+0000","last_undegraded":"2023-11-28T12:59:00.830169+0000","last_fullsized":"2023-11-28T12:59:00.830169+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:46:17.417359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667637+0000","last_change":"2023-11-28T12:58:55.249124+0000","last_active":"2023-11-28T12:58:59.667637+0000","last_peered":"2023-11-28T12:58:59.667637+0000","last_clean":"2023-11-28T12:58:59.667637+0000","last_became_active":"2023-11-28T12:58:55.248501+0000","last_became_peered":"2023-11-28T12:58:55.248501+0000","last_unstale":"2023-11-28T12:58:59.667637+0000","last_undegraded":"2023-11-28T12:58:59.667637+0000","last_fullsized":"2023-11-28T12:58:59.667637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:25.569310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661198+0000","last_change":"2023-11-28T12:58:55.659587+0000","last_active":"2023-11-28T12:58:59.661198+0000","last_peered":"2023-11-28T12:58:59.661198+0000","last_clean":"2023-11-28T12:58:59.661198+0000","last_became_active":"2023-11-28T12:58:55.659319+0000","last_became_peered":"2023-11-28T12:58:55.659319+0000","last_unstale":"2023-11-28T12:58:59.661198+0000","last_undegraded":"2023-11-28T12:58:59.661198+0000","last_fullsized":"2023-11-28T12:58:59.661198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.828936+0000","last_change":"2023-11-28T12:58:59.686412+0000","last_active":"2023-11-28T12:59:00.828936+0000","last_peered":"2023-11-28T12:59:00.828936+0000","last_clean":"2023-11-28T12:59:00.828936+0000","last_became_active":"2023-11-28T12:58:59.686079+0000","last_became_peered":"2023-11-28T12:58:59.686079+0000","last_unstale":"2023-11-28T12:59:00.828936+0000","last_undegraded":"2023-11-28T12:59:00.828936+0000","last_fullsized":"2023-11-28T12:59:00.828936+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:58:48.492561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664149+0000","last_change":"2023-11-28T12:58:55.248876+0000","last_active":"2023-11-28T12:58:59.664149+0000","last_peered":"2023-11-28T12:58:59.664149+0000","last_clean":"2023-11-28T12:58:59.664149+0000","last_became_active":"2023-11-28T12:58:55.248584+0000","last_became_peered":"2023-11-28T12:58:55.248584+0000","last_unstale":"2023-11-28T12:58:59.664149+0000","last_undegraded":"2023-11-28T12:58:59.664149+0000","last_fullsized":"2023-11-28T12:58:59.664149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:47:12.446132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659149+0000","last_change":"2023-11-28T12:58:55.663768+0000","last_active":"2023-11-28T12:58:59.659149+0000","last_peered":"2023-11-28T12:58:59.659149+0000","last_clean":"2023-11-28T12:58:59.659149+0000","last_became_active":"2023-11-28T12:58:55.663476+0000","last_became_peered":"2023-11-28T12:58:55.663476+0000","last_unstale":"2023-11-28T12:58:59.659149+0000","last_undegraded":"2023-11-28T12:58:59.659149+0000","last_fullsized":"2023-11-28T12:58:59.659149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:07:49.345226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829075+0000","last_change":"2023-11-28T12:58:59.685644+0000","last_active":"2023-11-28T12:59:00.829075+0000","last_peered":"2023-11-28T12:59:00.829075+0000","last_clean":"2023-11-28T12:59:00.829075+0000","last_became_active":"2023-11-28T12:58:59.685259+0000","last_became_peered":"2023-11-28T12:58:59.685259+0000","last_unstale":"2023-11-28T12:59:00.829075+0000","last_undegraded":"2023-11-28T12:59:00.829075+0000","last_fullsized":"2023-11-28T12:59:00.829075+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:07:52.771805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664976+0000","last_change":"2023-11-28T12:58:55.247686+0000","last_active":"2023-11-28T12:58:59.664976+0000","last_peered":"2023-11-28T12:58:59.664976+0000","last_clean":"2023-11-28T12:58:59.664976+0000","last_became_active":"2023-11-28T12:58:55.247328+0000","last_became_peered":"2023-11-28T12:58:55.247328+0000","last_unstale":"2023-11-28T12:58:59.664976+0000","last_undegraded":"2023-11-28T12:58:59.664976+0000","last_fullsized":"2023-11-28T12:58:59.664976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:22:00.609577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.656940+0000","last_change":"2023-11-28T12:58:55.649490+0000","last_active":"2023-11-28T12:58:59.656940+0000","last_peered":"2023-11-28T12:58:59.656940+0000","last_clean":"2023-11-28T12:58:59.656940+0000","last_became_active":"2023-11-28T12:58:55.649143+0000","last_became_peered":"2023-11-28T12:58:55.649143+0000","last_unstale":"2023-11-28T12:58:59.656940+0000","last_undegraded":"2023-11-28T12:58:59.656940+0000","last_fullsized":"2023-11-28T12:58:59.656940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:26.575212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829146+0000","last_change":"2023-11-28T12:58:59.679465+0000","last_active":"2023-11-28T12:59:00.829146+0000","last_peered":"2023-11-28T12:59:00.829146+0000","last_clean":"2023-11-28T12:59:00.829146+0000","last_became_active":"2023-11-28T12:58:59.679026+0000","last_became_peered":"2023-11-28T12:58:59.679026+0000","last_unstale":"2023-11-28T12:59:00.829146+0000","last_undegraded":"2023-11-28T12:59:00.829146+0000","last_fullsized":"2023-11-28T12:59:00.829146+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:52.339645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665301+0000","last_change":"2023-11-28T12:58:55.249645+0000","last_active":"2023-11-28T12:58:59.665301+0000","last_peered":"2023-11-28T12:58:59.665301+0000","last_clean":"2023-11-28T12:58:59.665301+0000","last_became_active":"2023-11-28T12:58:55.249284+0000","last_became_peered":"2023-11-28T12:58:55.249284+0000","last_unstale":"2023-11-28T12:58:59.665301+0000","last_undegraded":"2023-11-28T12:58:59.665301+0000","last_fullsized":"2023-11-28T12:58:59.665301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:49.233234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669390+0000","last_change":"2023-11-28T12:58:55.663994+0000","last_active":"2023-11-28T12:58:59.669390+0000","last_peered":"2023-11-28T12:58:59.669390+0000","last_clean":"2023-11-28T12:58:59.669390+0000","last_became_active":"2023-11-28T12:58:55.657280+0000","last_became_peered":"2023-11-28T12:58:55.657280+0000","last_unstale":"2023-11-28T12:58:59.669390+0000","last_undegraded":"2023-11-28T12:58:59.669390+0000","last_fullsized":"2023-11-28T12:58:59.669390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:34.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829349+0000","last_change":"2023-11-28T12:58:59.680307+0000","last_active":"2023-11-28T12:59:00.829349+0000","last_peered":"2023-11-28T12:59:00.829349+0000","last_clean":"2023-11-28T12:59:00.829349+0000","last_became_active":"2023-11-28T12:58:59.679837+0000","last_became_peered":"2023-11-28T12:58:59.679837+0000","last_unstale":"2023-11-28T12:59:00.829349+0000","last_undegraded":"2023-11-28T12:59:00.829349+0000","last_fullsized":"2023-11-28T12:59:00.829349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:18:08.491909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655480+0000","last_change":"2023-11-28T12:58:55.256347+0000","last_active":"2023-11-28T12:58:59.655480+0000","last_peered":"2023-11-28T12:58:59.655480+0000","last_clean":"2023-11-28T12:58:59.655480+0000","last_became_active":"2023-11-28T12:58:55.256027+0000","last_became_peered":"2023-11-28T12:58:55.256027+0000","last_unstale":"2023-11-28T12:58:59.655480+0000","last_undegraded":"2023-11-28T12:58:59.655480+0000","last_fullsized":"2023-11-28T12:58:59.655480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:42.300078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659538+0000","last_change":"2023-11-28T12:58:55.664332+0000","last_active":"2023-11-28T12:58:59.659538+0000","last_peered":"2023-11-28T12:58:59.659538+0000","last_clean":"2023-11-28T12:58:59.659538+0000","last_became_active":"2023-11-28T12:58:55.663913+0000","last_became_peered":"2023-11-28T12:58:55.663913+0000","last_unstale":"2023-11-28T12:58:59.659538+0000","last_undegraded":"2023-11-28T12:58:59.659538+0000","last_fullsized":"2023-11-28T12:58:59.659538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:50.590629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.315498+0000","last_change":"2023-11-28T12:58:59.678114+0000","last_active":"2023-11-28T12:59:00.315498+0000","last_peered":"2023-11-28T12:59:00.315498+0000","last_clean":"2023-11-28T12:59:00.315498+0000","last_became_active":"2023-11-28T12:58:59.677704+0000","last_became_peered":"2023-11-28T12:58:59.677704+0000","last_unstale":"2023-11-28T12:59:00.315498+0000","last_undegraded":"2023-11-28T12:59:00.315498+0000","last_fullsized":"2023-11-28T12:59:00.315498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:53.458898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657011+0000","last_change":"2023-11-28T12:58:55.241207+0000","last_active":"2023-11-28T12:58:59.657011+0000","last_peered":"2023-11-28T12:58:59.657011+0000","last_clean":"2023-11-28T12:58:59.657011+0000","last_became_active":"2023-11-28T12:58:55.240917+0000","last_became_peered":"2023-11-28T12:58:55.240917+0000","last_unstale":"2023-11-28T12:58:59.657011+0000","last_undegraded":"2023-11-28T12:58:59.657011+0000","last_fullsized":"2023-11-28T12:58:59.657011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659347+0000","last_change":"2023-11-28T12:58:55.660463+0000","last_active":"2023-11-28T12:58:59.659347+0000","last_peered":"2023-11-28T12:58:59.659347+0000","last_clean":"2023-11-28T12:58:59.659347+0000","last_became_active":"2023-11-28T12:58:55.660208+0000","last_became_peered":"2023-11-28T12:58:55.660208+0000","last_unstale":"2023-11-28T12:58:59.659347+0000","last_undegraded":"2023-11-28T12:58:59.659347+0000","last_fullsized":"2023-11-28T12:58:59.659347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830829+0000","last_change":"2023-11-28T12:58:59.688100+0000","last_active":"2023-11-28T12:59:00.830829+0000","last_peered":"2023-11-28T12:59:00.830829+0000","last_clean":"2023-11-28T12:59:00.830829+0000","last_became_active":"2023-11-28T12:58:59.687752+0000","last_became_peered":"2023-11-28T12:58:59.687752+0000","last_unstale":"2023-11-28T12:59:00.830829+0000","last_undegraded":"2023-11-28T12:59:00.830829+0000","last_fullsized":"2023-11-28T12:59:00.830829+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:38:05.937526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659014+0000","last_change":"2023-11-28T12:58:55.246754+0000","last_active":"2023-11-28T12:58:59.659014+0000","last_peered":"2023-11-28T12:58:59.659014+0000","last_clean":"2023-11-28T12:58:59.659014+0000","last_became_active":"2023-11-28T12:58:55.246480+0000","last_became_peered":"2023-11-28T12:58:55.246480+0000","last_unstale":"2023-11-28T12:58:59.659014+0000","last_undegraded":"2023-11-28T12:58:59.659014+0000","last_fullsized":"2023-11-28T12:58:59.659014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:08.923409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660472+0000","last_change":"2023-11-28T12:58:55.660027+0000","last_active":"2023-11-28T12:58:59.660472+0000","last_peered":"2023-11-28T12:58:59.660472+0000","last_clean":"2023-11-28T12:58:59.660472+0000","last_became_active":"2023-11-28T12:58:55.659775+0000","last_became_peered":"2023-11-28T12:58:55.659775+0000","last_unstale":"2023-11-28T12:58:59.660472+0000","last_undegraded":"2023-11-28T12:58:59.660472+0000","last_fullsized":"2023-11-28T12:58:59.660472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058758+0000","last_change":"2023-11-28T12:58:59.681700+0000","last_active":"2023-11-28T12:59:00.058758+0000","last_peered":"2023-11-28T12:59:00.058758+0000","last_clean":"2023-11-28T12:59:00.058758+0000","last_became_active":"2023-11-28T12:58:59.681305+0000","last_became_peered":"2023-11-28T12:58:59.681305+0000","last_unstale":"2023-11-28T12:59:00.058758+0000","last_undegraded":"2023-11-28T12:59:00.058758+0000","last_fullsized":"2023-11-28T12:59:00.058758+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:46.532522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655511+0000","last_change":"2023-11-28T12:58:55.249674+0000","last_active":"2023-11-28T12:58:59.655511+0000","last_peered":"2023-11-28T12:58:59.655511+0000","last_clean":"2023-11-28T12:58:59.655511+0000","last_became_active":"2023-11-28T12:58:55.249391+0000","last_became_peered":"2023-11-28T12:58:55.249391+0000","last_unstale":"2023-11-28T12:58:59.655511+0000","last_undegraded":"2023-11-28T12:58:59.655511+0000","last_fullsized":"2023-11-28T12:58:59.655511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667665+0000","last_change":"2023-11-28T12:58:55.661741+0000","last_active":"2023-11-28T12:58:59.667665+0000","last_peered":"2023-11-28T12:58:59.667665+0000","last_clean":"2023-11-28T12:58:59.667665+0000","last_became_active":"2023-11-28T12:58:55.661473+0000","last_became_peered":"2023-11-28T12:58:55.661473+0000","last_unstale":"2023-11-28T12:58:59.667665+0000","last_undegraded":"2023-11-28T12:58:59.667665+0000","last_fullsized":"2023-11-28T12:58:59.667665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:41:27.479117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313283+0000","last_change":"2023-11-28T12:58:59.675425+0000","last_active":"2023-11-28T12:59:00.313283+0000","last_peered":"2023-11-28T12:59:00.313283+0000","last_clean":"2023-11-28T12:59:00.313283+0000","last_became_active":"2023-11-28T12:58:59.675044+0000","last_became_peered":"2023-11-28T12:58:59.675044+0000","last_unstale":"2023-11-28T12:59:00.313283+0000","last_undegraded":"2023-11-28T12:59:00.313283+0000","last_fullsized":"2023-11-28T12:59:00.313283+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:46:17.417359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664599+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T12:58:59.664599+0000","last_peered":"2023-11-28T12:58:59.664599+0000","last_clean":"2023-11-28T12:58:59.664599+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T12:58:59.664599+0000","last_undegraded":"2023-11-28T12:58:59.664599+0000","last_fullsized":"2023-11-28T12:58:59.664599+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665575+0000","last_change":"2023-11-28T12:58:55.247919+0000","last_active":"2023-11-28T12:58:59.665575+0000","last_peered":"2023-11-28T12:58:59.665575+0000","last_clean":"2023-11-28T12:58:59.665575+0000","last_became_active":"2023-11-28T12:58:55.247653+0000","last_became_peered":"2023-11-28T12:58:55.247653+0000","last_unstale":"2023-11-28T12:58:59.665575+0000","last_undegraded":"2023-11-28T12:58:59.665575+0000","last_fullsized":"2023-11-28T12:58:59.665575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:54:06.148298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664140+0000","last_change":"2023-11-28T12:58:55.662430+0000","last_active":"2023-11-28T12:58:59.664140+0000","last_peered":"2023-11-28T12:58:59.664140+0000","last_clean":"2023-11-28T12:58:59.664140+0000","last_became_active":"2023-11-28T12:58:55.657899+0000","last_became_peered":"2023-11-28T12:58:55.657899+0000","last_unstale":"2023-11-28T12:58:59.664140+0000","last_undegraded":"2023-11-28T12:58:59.664140+0000","last_fullsized":"2023-11-28T12:58:59.664140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:22:01.615479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313377+0000","last_change":"2023-11-28T12:58:59.675356+0000","last_active":"2023-11-28T12:59:00.313377+0000","last_peered":"2023-11-28T12:59:00.313377+0000","last_clean":"2023-11-28T12:59:00.313377+0000","last_became_active":"2023-11-28T12:58:59.674917+0000","last_became_peered":"2023-11-28T12:58:59.674917+0000","last_unstale":"2023-11-28T12:59:00.313377+0000","last_undegraded":"2023-11-28T12:59:00.313377+0000","last_fullsized":"2023-11-28T12:59:00.313377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:51:00.263485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661311+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T12:58:59.661311+0000","last_peered":"2023-11-28T12:58:59.661311+0000","last_clean":"2023-11-28T12:58:59.661311+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T12:58:59.661311+0000","last_undegraded":"2023-11-28T12:58:59.661311+0000","last_fullsized":"2023-11-28T12:58:59.661311+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659552+0000","last_change":"2023-11-28T12:58:55.240294+0000","last_active":"2023-11-28T12:58:59.659552+0000","last_peered":"2023-11-28T12:58:59.659552+0000","last_clean":"2023-11-28T12:58:59.659552+0000","last_became_active":"2023-11-28T12:58:55.239947+0000","last_became_peered":"2023-11-28T12:58:55.239947+0000","last_unstale":"2023-11-28T12:58:59.659552+0000","last_undegraded":"2023-11-28T12:58:59.659552+0000","last_fullsized":"2023-11-28T12:58:59.659552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664509+0000","last_change":"2023-11-28T12:58:55.657537+0000","last_active":"2023-11-28T12:58:59.664509+0000","last_peered":"2023-11-28T12:58:59.664509+0000","last_clean":"2023-11-28T12:58:59.664509+0000","last_became_active":"2023-11-28T12:58:55.655666+0000","last_became_peered":"2023-11-28T12:58:55.655666+0000","last_unstale":"2023-11-28T12:58:59.664509+0000","last_undegraded":"2023-11-28T12:58:59.664509+0000","last_fullsized":"2023-11-28T12:58:59.664509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829428+0000","last_change":"2023-11-28T12:58:59.680479+0000","last_active":"2023-11-28T12:59:00.829428+0000","last_peered":"2023-11-28T12:59:00.829428+0000","last_clean":"2023-11-28T12:59:00.829428+0000","last_became_active":"2023-11-28T12:58:59.680089+0000","last_became_peered":"2023-11-28T12:58:59.680089+0000","last_unstale":"2023-11-28T12:59:00.829428+0000","last_undegraded":"2023-11-28T12:59:00.829428+0000","last_fullsized":"2023-11-28T12:59:00.829428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:12:31.778491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.673610+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T12:58:59.673610+0000","last_peered":"2023-11-28T12:58:59.673610+0000","last_clean":"2023-11-28T12:58:59.673610+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T12:58:59.673610+0000","last_undegraded":"2023-11-28T12:58:59.673610+0000","last_fullsized":"2023-11-28T12:58:59.673610+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660849+0000","last_change":"2023-11-28T12:58:55.238872+0000","last_active":"2023-11-28T12:58:59.660849+0000","last_peered":"2023-11-28T12:58:59.660849+0000","last_clean":"2023-11-28T12:58:59.660849+0000","last_became_active":"2023-11-28T12:58:55.238457+0000","last_became_peered":"2023-11-28T12:58:55.238457+0000","last_unstale":"2023-11-28T12:58:59.660849+0000","last_undegraded":"2023-11-28T12:58:59.660849+0000","last_fullsized":"2023-11-28T12:58:59.660849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667641+0000","last_change":"2023-11-28T12:58:55.659175+0000","last_active":"2023-11-28T12:58:59.667641+0000","last_peered":"2023-11-28T12:58:59.667641+0000","last_clean":"2023-11-28T12:58:59.667641+0000","last_became_active":"2023-11-28T12:58:55.658043+0000","last_became_peered":"2023-11-28T12:58:55.658043+0000","last_unstale":"2023-11-28T12:58:59.667641+0000","last_undegraded":"2023-11-28T12:58:59.667641+0000","last_fullsized":"2023-11-28T12:58:59.667641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:43.448645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830642+0000","last_change":"2023-11-28T12:58:59.682758+0000","last_active":"2023-11-28T12:59:00.830642+0000","last_peered":"2023-11-28T12:59:00.830642+0000","last_clean":"2023-11-28T12:59:00.830642+0000","last_became_active":"2023-11-28T12:58:59.682429+0000","last_became_peered":"2023-11-28T12:58:59.682429+0000","last_unstale":"2023-11-28T12:59:00.830642+0000","last_undegraded":"2023-11-28T12:59:00.830642+0000","last_fullsized":"2023-11-28T12:59:00.830642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:59.373707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668370+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T12:58:59.668370+0000","last_peered":"2023-11-28T12:58:59.668370+0000","last_clean":"2023-11-28T12:58:59.668370+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T12:58:59.668370+0000","last_undegraded":"2023-11-28T12:58:59.668370+0000","last_fullsized":"2023-11-28T12:58:59.668370+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666175+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T12:58:59.666175+0000","last_peered":"2023-11-28T12:58:59.666175+0000","last_clean":"2023-11-28T12:58:59.666175+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T12:58:59.666175+0000","last_undegraded":"2023-11-28T12:58:59.666175+0000","last_fullsized":"2023-11-28T12:58:59.666175+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655438+0000","last_change":"2023-11-28T12:58:55.248780+0000","last_active":"2023-11-28T12:58:59.655438+0000","last_peered":"2023-11-28T12:58:59.655438+0000","last_clean":"2023-11-28T12:58:59.655438+0000","last_became_active":"2023-11-28T12:58:55.248456+0000","last_became_peered":"2023-11-28T12:58:55.248456+0000","last_unstale":"2023-11-28T12:58:59.655438+0000","last_undegraded":"2023-11-28T12:58:59.655438+0000","last_fullsized":"2023-11-28T12:58:59.655438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:25:52.356939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657383+0000","last_change":"2023-11-28T12:58:55.661508+0000","last_active":"2023-11-28T12:58:59.657383+0000","last_peered":"2023-11-28T12:58:59.657383+0000","last_clean":"2023-11-28T12:58:59.657383+0000","last_became_active":"2023-11-28T12:58:55.661252+0000","last_became_peered":"2023-11-28T12:58:55.661252+0000","last_unstale":"2023-11-28T12:58:59.657383+0000","last_undegraded":"2023-11-28T12:58:59.657383+0000","last_fullsized":"2023-11-28T12:58:59.657383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:22:01.615479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830736+0000","last_change":"2023-11-28T12:58:59.675557+0000","last_active":"2023-11-28T12:59:00.830736+0000","last_peered":"2023-11-28T12:59:00.830736+0000","last_clean":"2023-11-28T12:59:00.830736+0000","last_became_active":"2023-11-28T12:58:59.675183+0000","last_became_peered":"2023-11-28T12:58:59.675183+0000","last_unstale":"2023-11-28T12:59:00.830736+0000","last_undegraded":"2023-11-28T12:59:00.830736+0000","last_fullsized":"2023-11-28T12:59:00.830736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:17.338033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666288+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T12:58:59.666288+0000","last_peered":"2023-11-28T12:58:59.666288+0000","last_clean":"2023-11-28T12:58:59.666288+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T12:58:59.666288+0000","last_undegraded":"2023-11-28T12:58:59.666288+0000","last_fullsized":"2023-11-28T12:58:59.666288+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668355+0000","last_change":"2023-11-28T12:58:55.246924+0000","last_active":"2023-11-28T12:58:59.668355+0000","last_peered":"2023-11-28T12:58:59.668355+0000","last_clean":"2023-11-28T12:58:59.668355+0000","last_became_active":"2023-11-28T12:58:55.244984+0000","last_became_peered":"2023-11-28T12:58:55.244984+0000","last_unstale":"2023-11-28T12:58:59.668355+0000","last_undegraded":"2023-11-28T12:58:59.668355+0000","last_fullsized":"2023-11-28T12:58:59.668355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674088+0000","last_change":"2023-11-28T12:58:55.666303+0000","last_active":"2023-11-28T12:58:59.674088+0000","last_peered":"2023-11-28T12:58:59.674088+0000","last_clean":"2023-11-28T12:58:59.674088+0000","last_became_active":"2023-11-28T12:58:55.666058+0000","last_became_peered":"2023-11-28T12:58:55.666058+0000","last_unstale":"2023-11-28T12:58:59.674088+0000","last_undegraded":"2023-11-28T12:58:59.674088+0000","last_fullsized":"2023-11-28T12:58:59.674088+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058107+0000","last_change":"2023-11-28T12:58:59.681611+0000","last_active":"2023-11-28T12:59:00.058107+0000","last_peered":"2023-11-28T12:59:00.058107+0000","last_clean":"2023-11-28T12:59:00.058107+0000","last_became_active":"2023-11-28T12:58:59.681195+0000","last_became_peered":"2023-11-28T12:58:59.681195+0000","last_unstale":"2023-11-28T12:59:00.058107+0000","last_undegraded":"2023-11-28T12:59:00.058107+0000","last_fullsized":"2023-11-28T12:59:00.058107+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:58:48.492561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664884+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T12:58:59.664884+0000","last_peered":"2023-11-28T12:58:59.664884+0000","last_clean":"2023-11-28T12:58:59.664884+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T12:58:59.664884+0000","last_undegraded":"2023-11-28T12:58:59.664884+0000","last_fullsized":"2023-11-28T12:58:59.664884+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664779+0000","last_change":"2023-11-28T12:58:55.240088+0000","last_active":"2023-11-28T12:58:59.664779+0000","last_peered":"2023-11-28T12:58:59.664779+0000","last_clean":"2023-11-28T12:58:59.664779+0000","last_became_active":"2023-11-28T12:58:55.237949+0000","last_became_peered":"2023-11-28T12:58:55.237949+0000","last_unstale":"2023-11-28T12:58:59.664779+0000","last_undegraded":"2023-11-28T12:58:59.664779+0000","last_fullsized":"2023-11-28T12:58:59.664779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669188+0000","last_change":"2023-11-28T12:58:55.657049+0000","last_active":"2023-11-28T12:58:59.669188+0000","last_peered":"2023-11-28T12:58:59.669188+0000","last_clean":"2023-11-28T12:58:59.669188+0000","last_became_active":"2023-11-28T12:58:55.656787+0000","last_became_peered":"2023-11-28T12:58:55.656787+0000","last_unstale":"2023-11-28T12:58:59.669188+0000","last_undegraded":"2023-11-28T12:58:59.669188+0000","last_fullsized":"2023-11-28T12:58:59.669188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:06:39.770893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058517+0000","last_change":"2023-11-28T12:58:59.676382+0000","last_active":"2023-11-28T12:59:00.058517+0000","last_peered":"2023-11-28T12:59:00.058517+0000","last_clean":"2023-11-28T12:59:00.058517+0000","last_became_active":"2023-11-28T12:58:59.675939+0000","last_became_peered":"2023-11-28T12:58:59.675939+0000","last_unstale":"2023-11-28T12:59:00.058517+0000","last_undegraded":"2023-11-28T12:59:00.058517+0000","last_fullsized":"2023-11-28T12:59:00.058517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:50:50.704627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.656336+0000","last_change":"2023-11-28T12:58:55.250390+0000","last_active":"2023-11-28T12:58:59.656336+0000","last_peered":"2023-11-28T12:58:59.656336+0000","last_clean":"2023-11-28T12:58:59.656336+0000","last_became_active":"2023-11-28T12:58:55.249919+0000","last_became_peered":"2023-11-28T12:58:55.249919+0000","last_unstale":"2023-11-28T12:58:59.656336+0000","last_undegraded":"2023-11-28T12:58:59.656336+0000","last_fullsized":"2023-11-28T12:58:59.656336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:11.131301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665787+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T12:58:59.665787+0000","last_peered":"2023-11-28T12:58:59.665787+0000","last_clean":"2023-11-28T12:58:59.665787+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T12:58:59.665787+0000","last_undegraded":"2023-11-28T12:58:59.665787+0000","last_fullsized":"2023-11-28T12:58:59.665787+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655406+0000","last_change":"2023-11-28T12:58:55.661761+0000","last_active":"2023-11-28T12:58:59.655406+0000","last_peered":"2023-11-28T12:58:59.655406+0000","last_clean":"2023-11-28T12:58:59.655406+0000","last_became_active":"2023-11-28T12:58:55.661488+0000","last_became_peered":"2023-11-28T12:58:55.661488+0000","last_unstale":"2023-11-28T12:58:59.655406+0000","last_undegraded":"2023-11-28T12:58:59.655406+0000","last_fullsized":"2023-11-28T12:58:59.655406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:14:28.821164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059438+0000","last_change":"2023-11-28T12:58:59.678960+0000","last_active":"2023-11-28T12:59:00.059438+0000","last_peered":"2023-11-28T12:59:00.059438+0000","last_clean":"2023-11-28T12:59:00.059438+0000","last_became_active":"2023-11-28T12:58:59.678568+0000","last_became_peered":"2023-11-28T12:58:59.678568+0000","last_unstale":"2023-11-28T12:59:00.059438+0000","last_undegraded":"2023-11-28T12:59:00.059438+0000","last_fullsized":"2023-11-28T12:59:00.059438+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:17.338033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659020+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T12:58:59.659020+0000","last_peered":"2023-11-28T12:58:59.659020+0000","last_clean":"2023-11-28T12:58:59.659020+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T12:58:59.659020+0000","last_undegraded":"2023-11-28T12:58:59.659020+0000","last_fullsized":"2023-11-28T12:58:59.659020+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"26'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313562+0000","last_change":"2023-11-28T12:58:55.251038+0000","last_active":"2023-11-28T12:59:00.313562+0000","last_peered":"2023-11-28T12:59:00.313562+0000","last_clean":"2023-11-28T12:59:00.313562+0000","last_became_active":"2023-11-28T12:58:55.250328+0000","last_became_peered":"2023-11-28T12:58:55.250328+0000","last_unstale":"2023-11-28T12:59:00.313562+0000","last_undegraded":"2023-11-28T12:59:00.313562+0000","last_fullsized":"2023-11-28T12:59:00.313562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:11.131301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659036+0000","last_change":"2023-11-28T12:58:55.658965+0000","last_active":"2023-11-28T12:58:59.659036+0000","last_peered":"2023-11-28T12:58:59.659036+0000","last_clean":"2023-11-28T12:58:59.659036+0000","last_became_active":"2023-11-28T12:58:55.658709+0000","last_became_peered":"2023-11-28T12:58:55.658709+0000","last_unstale":"2023-11-28T12:58:59.659036+0000","last_undegraded":"2023-11-28T12:58:59.659036+0000","last_fullsized":"2023-11-28T12:58:59.659036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:58:45.065981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.316209+0000","last_change":"2023-11-28T12:58:59.679267+0000","last_active":"2023-11-28T12:59:00.316209+0000","last_peered":"2023-11-28T12:59:00.316209+0000","last_clean":"2023-11-28T12:59:00.316209+0000","last_became_active":"2023-11-28T12:58:59.678822+0000","last_became_peered":"2023-11-28T12:58:59.678822+0000","last_unstale":"2023-11-28T12:59:00.316209+0000","last_undegraded":"2023-11-28T12:59:00.316209+0000","last_fullsized":"2023-11-28T12:59:00.316209+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:38:05.937526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669102+0000","last_change":"2023-11-28T12:58:55.250742+0000","last_active":"2023-11-28T12:58:59.669102+0000","last_peered":"2023-11-28T12:58:59.669102+0000","last_clean":"2023-11-28T12:58:59.669102+0000","last_became_active":"2023-11-28T12:58:55.245578+0000","last_became_peered":"2023-11-28T12:58:55.245578+0000","last_unstale":"2023-11-28T12:58:59.669102+0000","last_undegraded":"2023-11-28T12:58:59.669102+0000","last_fullsized":"2023-11-28T12:58:59.669102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:25:52.356939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672422+0000","last_change":"2023-11-28T12:58:55.647087+0000","last_active":"2023-11-28T12:58:59.672422+0000","last_peered":"2023-11-28T12:58:59.672422+0000","last_clean":"2023-11-28T12:58:59.672422+0000","last_became_active":"2023-11-28T12:58:55.646827+0000","last_became_peered":"2023-11-28T12:58:55.646827+0000","last_unstale":"2023-11-28T12:58:59.672422+0000","last_undegraded":"2023-11-28T12:58:59.672422+0000","last_fullsized":"2023-11-28T12:58:59.672422+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:43.305980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830493+0000","last_change":"2023-11-28T12:58:59.673358+0000","last_active":"2023-11-28T12:59:00.830493+0000","last_peered":"2023-11-28T12:59:00.830493+0000","last_clean":"2023-11-28T12:59:00.830493+0000","last_became_active":"2023-11-28T12:58:59.673010+0000","last_became_peered":"2023-11-28T12:58:59.673010+0000","last_unstale":"2023-11-28T12:59:00.830493+0000","last_undegraded":"2023-11-28T12:59:00.830493+0000","last_fullsized":"2023-11-28T12:59:00.830493+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:53.458898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665394+0000","last_change":"2023-11-28T12:58:55.242678+0000","last_active":"2023-11-28T12:58:59.665394+0000","last_peered":"2023-11-28T12:58:59.665394+0000","last_clean":"2023-11-28T12:58:59.665394+0000","last_became_active":"2023-11-28T12:58:55.242467+0000","last_became_peered":"2023-11-28T12:58:55.242467+0000","last_unstale":"2023-11-28T12:58:59.665394+0000","last_undegraded":"2023-11-28T12:58:59.665394+0000","last_fullsized":"2023-11-28T12:58:59.665394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:11.131301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659707+0000","last_change":"2023-11-28T12:58:55.664310+0000","last_active":"2023-11-28T12:58:59.659707+0000","last_peered":"2023-11-28T12:58:59.659707+0000","last_clean":"2023-11-28T12:58:59.659707+0000","last_became_active":"2023-11-28T12:58:55.664036+0000","last_became_peered":"2023-11-28T12:58:55.664036+0000","last_unstale":"2023-11-28T12:58:59.659707+0000","last_undegraded":"2023-11-28T12:58:59.659707+0000","last_fullsized":"2023-11-28T12:58:59.659707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:38:36.781235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830051+0000","last_change":"2023-11-28T12:58:59.685443+0000","last_active":"2023-11-28T12:59:00.830051+0000","last_peered":"2023-11-28T12:59:00.830051+0000","last_clean":"2023-11-28T12:59:00.830051+0000","last_became_active":"2023-11-28T12:58:59.685042+0000","last_became_peered":"2023-11-28T12:58:59.685042+0000","last_unstale":"2023-11-28T12:59:00.830051+0000","last_undegraded":"2023-11-28T12:59:00.830051+0000","last_fullsized":"2023-11-28T12:59:00.830051+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:03:33.042937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665495+0000","last_change":"2023-11-28T12:58:55.241170+0000","last_active":"2023-11-28T12:58:59.665495+0000","last_peered":"2023-11-28T12:58:59.665495+0000","last_clean":"2023-11-28T12:58:59.665495+0000","last_became_active":"2023-11-28T12:58:55.240880+0000","last_became_peered":"2023-11-28T12:58:55.240880+0000","last_unstale":"2023-11-28T12:58:59.665495+0000","last_undegraded":"2023-11-28T12:58:59.665495+0000","last_fullsized":"2023-11-28T12:58:59.665495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:42.300078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659284+0000","last_change":"2023-11-28T12:58:55.664483+0000","last_active":"2023-11-28T12:58:59.659284+0000","last_peered":"2023-11-28T12:58:59.659284+0000","last_clean":"2023-11-28T12:58:59.659284+0000","last_became_active":"2023-11-28T12:58:55.662151+0000","last_became_peered":"2023-11-28T12:58:55.662151+0000","last_unstale":"2023-11-28T12:58:59.659284+0000","last_undegraded":"2023-11-28T12:58:59.659284+0000","last_fullsized":"2023-11-28T12:58:59.659284+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:00:06.262366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058024+0000","last_change":"2023-11-28T12:58:59.679533+0000","last_active":"2023-11-28T12:59:00.058024+0000","last_peered":"2023-11-28T12:59:00.058024+0000","last_clean":"2023-11-28T12:59:00.058024+0000","last_became_active":"2023-11-28T12:58:59.679174+0000","last_became_peered":"2023-11-28T12:58:59.679174+0000","last_unstale":"2023-11-28T12:59:00.058024+0000","last_undegraded":"2023-11-28T12:59:00.058024+0000","last_fullsized":"2023-11-28T12:59:00.058024+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:34:44.322352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655854+0000","last_change":"2023-11-28T12:58:55.246420+0000","last_active":"2023-11-28T12:58:59.655854+0000","last_peered":"2023-11-28T12:58:59.655854+0000","last_clean":"2023-11-28T12:58:59.655854+0000","last_became_active":"2023-11-28T12:58:55.246075+0000","last_became_peered":"2023-11-28T12:58:55.246075+0000","last_unstale":"2023-11-28T12:58:59.655854+0000","last_undegraded":"2023-11-28T12:58:59.655854+0000","last_fullsized":"2023-11-28T12:58:59.655854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:38:45.929395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655474+0000","last_change":"2023-11-28T12:58:55.660837+0000","last_active":"2023-11-28T12:58:59.655474+0000","last_peered":"2023-11-28T12:58:59.655474+0000","last_clean":"2023-11-28T12:58:59.655474+0000","last_became_active":"2023-11-28T12:58:55.657932+0000","last_became_peered":"2023-11-28T12:58:55.657932+0000","last_unstale":"2023-11-28T12:58:59.655474+0000","last_undegraded":"2023-11-28T12:58:59.655474+0000","last_fullsized":"2023-11-28T12:58:59.655474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.315551+0000","last_change":"2023-11-28T12:58:59.683087+0000","last_active":"2023-11-28T12:59:00.315551+0000","last_peered":"2023-11-28T12:59:00.315551+0000","last_clean":"2023-11-28T12:59:00.315551+0000","last_became_active":"2023-11-28T12:58:59.682678+0000","last_became_peered":"2023-11-28T12:58:59.682678+0000","last_unstale":"2023-11-28T12:59:00.315551+0000","last_undegraded":"2023-11-28T12:59:00.315551+0000","last_fullsized":"2023-11-28T12:59:00.315551+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:07:52.771805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"21'4","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674325+0000","last_change":"2023-11-28T12:58:55.246094+0000","last_active":"2023-11-28T12:58:59.674325+0000","last_peered":"2023-11-28T12:58:59.674325+0000","last_clean":"2023-11-28T12:58:59.674325+0000","last_became_active":"2023-11-28T12:58:55.245800+0000","last_became_peered":"2023-11-28T12:58:55.245800+0000","last_unstale":"2023-11-28T12:58:59.674325+0000","last_undegraded":"2023-11-28T12:58:59.674325+0000","last_fullsized":"2023-11-28T12:58:59.674325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:49.233234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.658861+0000","last_change":"2023-11-28T12:58:55.660995+0000","last_active":"2023-11-28T12:58:59.658861+0000","last_peered":"2023-11-28T12:58:59.658861+0000","last_clean":"2023-11-28T12:58:59.658861+0000","last_became_active":"2023-11-28T12:58:55.660761+0000","last_became_peered":"2023-11-28T12:58:55.660761+0000","last_unstale":"2023-11-28T12:58:59.658861+0000","last_undegraded":"2023-11-28T12:58:59.658861+0000","last_fullsized":"2023-11-28T12:58:59.658861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657014+0000","last_change":"2023-11-28T12:58:55.650780+0000","last_active":"2023-11-28T12:58:59.657014+0000","last_peered":"2023-11-28T12:58:59.657014+0000","last_clean":"2023-11-28T12:58:59.657014+0000","last_became_active":"2023-11-28T12:58:55.650511+0000","last_became_peered":"2023-11-28T12:58:55.650511+0000","last_unstale":"2023-11-28T12:58:59.657014+0000","last_undegraded":"2023-11-28T12:58:59.657014+0000","last_fullsized":"2023-11-28T12:58:59.657014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:43.448645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655863+0000","last_change":"2023-11-28T12:58:55.663313+0000","last_active":"2023-11-28T12:58:59.655863+0000","last_peered":"2023-11-28T12:58:59.655863+0000","last_clean":"2023-11-28T12:58:59.655863+0000","last_became_active":"2023-11-28T12:58:55.663036+0000","last_became_peered":"2023-11-28T12:58:55.663036+0000","last_unstale":"2023-11-28T12:58:59.655863+0000","last_undegraded":"2023-11-28T12:58:59.655863+0000","last_fullsized":"2023-11-28T12:58:59.655863+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659437+0000","last_change":"2023-11-28T12:58:55.661206+0000","last_active":"2023-11-28T12:58:59.659437+0000","last_peered":"2023-11-28T12:58:59.659437+0000","last_clean":"2023-11-28T12:58:59.659437+0000","last_became_active":"2023-11-28T12:58:55.660952+0000","last_became_peered":"2023-11-28T12:58:55.660952+0000","last_unstale":"2023-11-28T12:58:59.659437+0000","last_undegraded":"2023-11-28T12:58:59.659437+0000","last_fullsized":"2023-11-28T12:58:59.659437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:34.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655629+0000","last_change":"2023-11-28T12:58:55.661675+0000","last_active":"2023-11-28T12:58:59.655629+0000","last_peered":"2023-11-28T12:58:59.655629+0000","last_clean":"2023-11-28T12:58:59.655629+0000","last_became_active":"2023-11-28T12:58:55.661418+0000","last_became_peered":"2023-11-28T12:58:55.661418+0000","last_unstale":"2023-11-28T12:58:59.655629+0000","last_undegraded":"2023-11-28T12:58:59.655629+0000","last_fullsized":"2023-11-28T12:58:59.655629+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664699+0000","last_change":"2023-11-28T12:58:55.651538+0000","last_active":"2023-11-28T12:58:59.664699+0000","last_peered":"2023-11-28T12:58:59.664699+0000","last_clean":"2023-11-28T12:58:59.664699+0000","last_became_active":"2023-11-28T12:58:55.651297+0000","last_became_peered":"2023-11-28T12:58:55.651297+0000","last_unstale":"2023-11-28T12:58:59.664699+0000","last_undegraded":"2023-11-28T12:58:59.664699+0000","last_fullsized":"2023-11-28T12:58:59.664699+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:56:55.140717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4528,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9272,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362568,"statfs":{"total":96636764160,"available":96627269632,"internally_reserved":0,"allocated":901120,"data_stored":709174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8088,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363752,"statfs":{"total":96636764160,"available":96628482048,"internally_reserved":0,"allocated":278528,"data_stored":114844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8152,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363688,"statfs":{"total":96636764160,"available":96628416512,"internally_reserved":0,"allocated":278528,"data_stored":114844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9248,"kb_used_data":856,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362592,"statfs":{"total":96636764160,"available":96627294208,"internally_reserved":0,"allocated":876544,"data_stored":705778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8088,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363752,"statfs":{"total":96636764160,"available":96628482048,"internally_reserved":0,"allocated":278528,"data_stored":114844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8156,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363684,"statfs":{"total":96636764160,"available":96628412416,"internally_reserved":0,"allocated":282624,"data_stored":114934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8236,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363604,"statfs":{"total":96636764160,"available":96628330496,"internally_reserved":0,"allocated":299008,"data_stored":118646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8160,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363680,"statfs":{"total":96636764160,"available":96628408320,"internally_reserved":0,"allocated":286720,"data_stored":115480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T12:59:23.348 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T12:59:23.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.500+0000 7f08896de700 1 Processor -- start 2023-11-28T12:59:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.500+0000 7f08896de700 1 -- start start 2023-11-28T12:59:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.501+0000 7f08896de700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f08840b0d70 0x7f08841604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.501+0000 7f08896de700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f08840cc550 0x7f08840cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.501+0000 7f08896de700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f088415cd90 con 0x7f08840b1140 2023-11-28T12:59:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.501+0000 7f08896de700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08840cedd0 con 0x7f08840cc550 2023-11-28T12:59:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.501+0000 7f08896de700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08840cced0 con 0x7f08840b0d70 2023-11-28T12:59:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.501+0000 7f08837fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f08840cc550 0x7f08840cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.501+0000 7f08837fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f08840cc550 0x7f08840cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33578/0 (socket says 172.21.15.111:33578) 2023-11-28T12:59:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.501+0000 7f08837fe700 1 -- 172.21.15.111:0/2258268955 learned_addr learned my addr 172.21.15.111:0/2258268955 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.502+0000 7f0882ffd700 1 --2- 172.21.15.111:0/2258268955 >> v2:172.21.15.149:3300/0 conn(0x7f08840b0d70 0x7f08841604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.502+0000 7f08837fe700 1 -- 172.21.15.111:0/2258268955 >> v2:172.21.15.149:3300/0 conn(0x7f08840b0d70 msgr2=0x7f08841604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:23.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.502+0000 7f08837fe700 1 --2- 172.21.15.111:0/2258268955 >> v2:172.21.15.149:3300/0 conn(0x7f08840b0d70 0x7f08841604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.502+0000 7f08837fe700 1 -- 172.21.15.111:0/2258268955 >> v1:172.21.15.149:6789/0 conn(0x7f08840b1140 legacy=0x7f08840ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.502+0000 7f08837fe700 1 -- 172.21.15.111:0/2258268955 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08841613e0 con 0x7f08840cc550 2023-11-28T12:59:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.502+0000 7f0882ffd700 1 --2- 172.21.15.111:0/2258268955 >> v2:172.21.15.149:3300/0 conn(0x7f08840b0d70 0x7f08841604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.502+0000 7f08837fe700 1 --2- 172.21.15.111:0/2258268955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08840cc550 0x7f08840cc990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0884071960 tx=0x7f087c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb02158cbfa480d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.502+0000 7f0881ffb700 1 -- 172.21.15.111:0/2258268955 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c00e070 con 0x7f08840cc550 2023-11-28T12:59:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.502+0000 7f0881ffb700 1 -- 172.21.15.111:0/2258268955 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087c003a20 con 0x7f08840cc550 2023-11-28T12:59:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.502+0000 7f0881ffb700 1 -- 172.21.15.111:0/2258268955 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c003dd0 con 0x7f08840cc550 2023-11-28T12:59:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.502+0000 7f08896de700 1 -- 172.21.15.111:0/2258268955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08840cc550 msgr2=0x7f08840cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.502+0000 7f08896de700 1 --2- 172.21.15.111:0/2258268955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08840cc550 0x7f08840cc990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0884071960 tx=0x7f087c00b330 comp rx=0 tx=0).stop 2023-11-28T12:59:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.503+0000 7f08896de700 1 -- 172.21.15.111:0/2258268955 shutdown_connections 2023-11-28T12:59:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.503+0000 7f08896de700 1 --2- 172.21.15.111:0/2258268955 >> v2:172.21.15.149:3300/0 conn(0x7f08840b0d70 0x7f08841604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.503+0000 7f08896de700 1 --2- 172.21.15.111:0/2258268955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08840cc550 0x7f08840cc990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.503+0000 7f08896de700 1 -- 172.21.15.111:0/2258268955 >> 172.21.15.111:0/2258268955 conn(0x7f0884098ee0 msgr2=0x7f088406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:23.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.503+0000 7f08896de700 1 -- 172.21.15.111:0/2258268955 shutdown_connections 2023-11-28T12:59:23.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.503+0000 7f08896de700 1 -- 172.21.15.111:0/2258268955 wait complete. 2023-11-28T12:59:23.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.504+0000 7f08896de700 1 Processor -- start 2023-11-28T12:59:23.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.504+0000 7f08896de700 1 -- start start 2023-11-28T12:59:23.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08896de700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08840b0d70 0x7f08840c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08896de700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f08840b1140 0x7f08840c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08896de700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08840cc550 0x7f08840c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08896de700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08840c61f0 con 0x7f08840cc550 2023-11-28T12:59:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08896de700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08840c55e0 con 0x7f08840b1140 2023-11-28T12:59:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08896de700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08840c58c0 con 0x7f08840b0d70 2023-11-28T12:59:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08837fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08840cc550 0x7f08840c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08837fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08840cc550 0x7f08840c8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33584/0 (socket says 172.21.15.111:33584) 2023-11-28T12:59:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08837fe700 1 -- 172.21.15.111:0/3891635638 learned_addr learned my addr 172.21.15.111:0/3891635638 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f0882ffd700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08840b0d70 0x7f08840c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08827fc700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f08840b1140 0x7f08840c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08837fe700 1 -- 172.21.15.111:0/3891635638 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08840b0d70 msgr2=0x7f08840c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08837fe700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08840b0d70 0x7f08840c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08837fe700 1 -- 172.21.15.111:0/3891635638 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f08840b1140 msgr2=0x7f08840c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08837fe700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f08840b1140 0x7f08840c4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08837fe700 1 -- 172.21.15.111:0/3891635638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c009b30 con 0x7f08840cc550 2023-11-28T12:59:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08827fc700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f08840b1140 0x7f08840c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08837fe700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08840cc550 0x7f08840c8f90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f087c015040 tx=0x7f087c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f086b7fe700 1 -- 172.21.15.111:0/3891635638 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c004440 con 0x7f08840cc550 2023-11-28T12:59:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f086b7fe700 1 -- 172.21.15.111:0/3891635638 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087c0045a0 con 0x7f08840cc550 2023-11-28T12:59:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08896de700 1 -- 172.21.15.111:0/3891635638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0884160e20 con 0x7f08840cc550 2023-11-28T12:59:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f086b7fe700 1 -- 172.21.15.111:0/3891635638 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c004950 con 0x7f08840cc550 2023-11-28T12:59:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.505+0000 7f08896de700 1 -- 172.21.15.111:0/3891635638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0884160b40 con 0x7f08840cc550 2023-11-28T12:59:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.506+0000 7f08896de700 1 -- 172.21.15.111:0/3891635638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0884161100 con 0x7f08840cc550 2023-11-28T12:59:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.507+0000 7f086b7fe700 1 -- 172.21.15.111:0/3891635638 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f087c02c070 con 0x7f08840cc550 2023-11-28T12:59:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.507+0000 7f086b7fe700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f08700682f0 0x7f087006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.507+0000 7f086b7fe700 1 -- 172.21.15.111:0/3891635638 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f087c0870b0 con 0x7f08840cc550 2023-11-28T12:59:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.507+0000 7f0882ffd700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f08700682f0 0x7f087006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.508+0000 7f0882ffd700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f08700682f0 0x7f087006a7a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f087800abe0 tx=0x7f0878008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.512+0000 7f086b7fe700 1 -- 172.21.15.111:0/3891635638 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f087c053620 con 0x7f08840cc550 2023-11-28T12:59:23.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.693+0000 7f08896de700 1 -- 172.21.15.111:0/3891635638 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f088415cd90 con 0x7f08700682f0 2023-11-28T12:59:23.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.703+0000 7f086b7fe700 1 -- 172.21.15.111:0/3891635638 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367319 (secure 0 0 0) 0x7f088415cd90 con 0x7f08700682f0 2023-11-28T12:59:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:23.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.708+0000 7f08896de700 1 -- 172.21.15.111:0/3891635638 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f08700682f0 msgr2=0x7f087006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.708+0000 7f08896de700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f08700682f0 0x7f087006a7a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f087800abe0 tx=0x7f0878008040 comp rx=0 tx=0).stop 2023-11-28T12:59:23.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.708+0000 7f08896de700 1 -- 172.21.15.111:0/3891635638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08840cc550 msgr2=0x7f08840c8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.708+0000 7f08896de700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08840cc550 0x7f08840c8f90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f087c015040 tx=0x7f087c004ad0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.708+0000 7f08896de700 1 -- 172.21.15.111:0/3891635638 shutdown_connections 2023-11-28T12:59:23.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.708+0000 7f08896de700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08840cc550 0x7f08840c8f90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.708+0000 7f08896de700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08840b0d70 0x7f08840c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.708+0000 7f08896de700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f08700682f0 0x7f087006a7a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.708+0000 7f08896de700 1 --2- 172.21.15.111:0/3891635638 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f08840b1140 0x7f08840c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.708+0000 7f08896de700 1 -- 172.21.15.111:0/3891635638 >> 172.21.15.111:0/3891635638 conn(0x7f0884098ee0 msgr2=0x7f0884072cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:23.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.708+0000 7f08896de700 1 -- 172.21.15.111:0/3891635638 shutdown_connections 2023-11-28T12:59:23.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.708+0000 7f08896de700 1 -- 172.21.15.111:0/3891635638 wait complete. 2023-11-28T12:59:23.713 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T12:59:23.721 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-11-28T12:59:22.339709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594896,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":67400,"kb_used_data":3400,"kb_used_omap":0,"kb_used_meta":63936,"kb_avail":754907320,"statfs":{"total":773094113280,"available":773025095680,"internally_reserved":0,"allocated":3481600,"data_stored":2108544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65470464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003657"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672369+0000","last_change":"2023-11-28T12:58:55.661175+0000","last_active":"2023-11-28T12:58:59.672369+0000","last_peered":"2023-11-28T12:58:59.672369+0000","last_clean":"2023-11-28T12:58:59.672369+0000","last_became_active":"2023-11-28T12:58:55.660919+0000","last_became_peered":"2023-11-28T12:58:55.660919+0000","last_unstale":"2023-11-28T12:58:59.672369+0000","last_undegraded":"2023-11-28T12:58:59.672369+0000","last_fullsized":"2023-11-28T12:58:59.672369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:12.137203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666089+0000","last_change":"2023-11-28T12:58:55.652749+0000","last_active":"2023-11-28T12:58:59.666089+0000","last_peered":"2023-11-28T12:58:59.666089+0000","last_clean":"2023-11-28T12:58:59.666089+0000","last_became_active":"2023-11-28T12:58:55.652544+0000","last_became_peered":"2023-11-28T12:58:55.652544+0000","last_unstale":"2023-11-28T12:58:59.666089+0000","last_undegraded":"2023-11-28T12:58:59.666089+0000","last_fullsized":"2023-11-28T12:58:59.666089+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660310+0000","last_change":"2023-11-28T12:58:55.669208+0000","last_active":"2023-11-28T12:58:59.660310+0000","last_peered":"2023-11-28T12:58:59.660310+0000","last_clean":"2023-11-28T12:58:59.660310+0000","last_became_active":"2023-11-28T12:58:55.668930+0000","last_became_peered":"2023-11-28T12:58:55.668930+0000","last_unstale":"2023-11-28T12:58:59.660310+0000","last_undegraded":"2023-11-28T12:58:59.660310+0000","last_fullsized":"2023-11-28T12:58:59.660310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:18:05.065329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657123+0000","last_change":"2023-11-28T12:58:55.653156+0000","last_active":"2023-11-28T12:58:59.657123+0000","last_peered":"2023-11-28T12:58:59.657123+0000","last_clean":"2023-11-28T12:58:59.657123+0000","last_became_active":"2023-11-28T12:58:55.652820+0000","last_became_peered":"2023-11-28T12:58:55.652820+0000","last_unstale":"2023-11-28T12:58:59.657123+0000","last_undegraded":"2023-11-28T12:58:59.657123+0000","last_fullsized":"2023-11-28T12:58:59.657123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:19.178525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669305+0000","last_change":"2023-11-28T12:58:55.664471+0000","last_active":"2023-11-28T12:58:59.669305+0000","last_peered":"2023-11-28T12:58:59.669305+0000","last_clean":"2023-11-28T12:58:59.669305+0000","last_became_active":"2023-11-28T12:58:55.664221+0000","last_became_peered":"2023-11-28T12:58:55.664221+0000","last_unstale":"2023-11-28T12:58:59.669305+0000","last_undegraded":"2023-11-28T12:58:59.669305+0000","last_fullsized":"2023-11-28T12:58:59.669305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:34:15.223824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659490+0000","last_change":"2023-11-28T12:58:55.669109+0000","last_active":"2023-11-28T12:58:59.659490+0000","last_peered":"2023-11-28T12:58:59.659490+0000","last_clean":"2023-11-28T12:58:59.659490+0000","last_became_active":"2023-11-28T12:58:55.668857+0000","last_became_peered":"2023-11-28T12:58:55.668857+0000","last_unstale":"2023-11-28T12:58:59.659490+0000","last_undegraded":"2023-11-28T12:58:59.659490+0000","last_fullsized":"2023-11-28T12:58:59.659490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:14:02.504692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.663977+0000","last_change":"2023-11-28T12:58:55.662854+0000","last_active":"2023-11-28T12:58:59.663977+0000","last_peered":"2023-11-28T12:58:59.663977+0000","last_clean":"2023-11-28T12:58:59.663977+0000","last_became_active":"2023-11-28T12:58:55.662539+0000","last_became_peered":"2023-11-28T12:58:55.662539+0000","last_unstale":"2023-11-28T12:58:59.663977+0000","last_undegraded":"2023-11-28T12:58:59.663977+0000","last_fullsized":"2023-11-28T12:58:59.663977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661304+0000","last_change":"2023-11-28T12:58:55.657958+0000","last_active":"2023-11-28T12:58:59.661304+0000","last_peered":"2023-11-28T12:58:59.661304+0000","last_clean":"2023-11-28T12:58:59.661304+0000","last_became_active":"2023-11-28T12:58:55.657681+0000","last_became_peered":"2023-11-28T12:58:55.657681+0000","last_unstale":"2023-11-28T12:58:59.661304+0000","last_undegraded":"2023-11-28T12:58:59.661304+0000","last_fullsized":"2023-11-28T12:58:59.661304+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674142+0000","last_change":"2023-11-28T12:58:55.666485+0000","last_active":"2023-11-28T12:58:59.674142+0000","last_peered":"2023-11-28T12:58:59.674142+0000","last_clean":"2023-11-28T12:58:59.674142+0000","last_became_active":"2023-11-28T12:58:55.665670+0000","last_became_peered":"2023-11-28T12:58:55.665670+0000","last_unstale":"2023-11-28T12:58:59.674142+0000","last_undegraded":"2023-11-28T12:58:59.674142+0000","last_fullsized":"2023-11-28T12:58:59.674142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668534+0000","last_change":"2023-11-28T12:58:55.655129+0000","last_active":"2023-11-28T12:58:59.668534+0000","last_peered":"2023-11-28T12:58:59.668534+0000","last_clean":"2023-11-28T12:58:59.668534+0000","last_became_active":"2023-11-28T12:58:55.654839+0000","last_became_peered":"2023-11-28T12:58:55.654839+0000","last_unstale":"2023-11-28T12:58:59.668534+0000","last_undegraded":"2023-11-28T12:58:59.668534+0000","last_fullsized":"2023-11-28T12:58:59.668534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660394+0000","last_change":"2023-11-28T12:58:55.663796+0000","last_active":"2023-11-28T12:58:59.660394+0000","last_peered":"2023-11-28T12:58:59.660394+0000","last_clean":"2023-11-28T12:58:59.660394+0000","last_became_active":"2023-11-28T12:58:55.663553+0000","last_became_peered":"2023-11-28T12:58:55.663553+0000","last_unstale":"2023-11-28T12:58:59.660394+0000","last_undegraded":"2023-11-28T12:58:59.660394+0000","last_fullsized":"2023-11-28T12:58:59.660394+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:31:46.923203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672559+0000","last_change":"2023-11-28T12:58:55.666858+0000","last_active":"2023-11-28T12:58:59.672559+0000","last_peered":"2023-11-28T12:58:59.672559+0000","last_clean":"2023-11-28T12:58:59.672559+0000","last_became_active":"2023-11-28T12:58:55.666597+0000","last_became_peered":"2023-11-28T12:58:55.666597+0000","last_unstale":"2023-11-28T12:58:59.672559+0000","last_undegraded":"2023-11-28T12:58:59.672559+0000","last_fullsized":"2023-11-28T12:58:59.672559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:50.239136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.673902+0000","last_change":"2023-11-28T12:58:55.658497+0000","last_active":"2023-11-28T12:58:59.673902+0000","last_peered":"2023-11-28T12:58:59.673902+0000","last_clean":"2023-11-28T12:58:59.673902+0000","last_became_active":"2023-11-28T12:58:55.658258+0000","last_became_peered":"2023-11-28T12:58:55.658258+0000","last_unstale":"2023-11-28T12:58:59.673902+0000","last_undegraded":"2023-11-28T12:58:59.673902+0000","last_fullsized":"2023-11-28T12:58:59.673902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:38:46.935297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664436+0000","last_change":"2023-11-28T12:58:55.648629+0000","last_active":"2023-11-28T12:58:59.664436+0000","last_peered":"2023-11-28T12:58:59.664436+0000","last_clean":"2023-11-28T12:58:59.664436+0000","last_became_active":"2023-11-28T12:58:55.648143+0000","last_became_peered":"2023-11-28T12:58:55.648143+0000","last_unstale":"2023-11-28T12:58:59.664436+0000","last_undegraded":"2023-11-28T12:58:59.664436+0000","last_fullsized":"2023-11-28T12:58:59.664436+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:06:39.770893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.663795+0000","last_change":"2023-11-28T12:58:55.665560+0000","last_active":"2023-11-28T12:58:59.663795+0000","last_peered":"2023-11-28T12:58:59.663795+0000","last_clean":"2023-11-28T12:58:59.663795+0000","last_became_active":"2023-11-28T12:58:55.665304+0000","last_became_peered":"2023-11-28T12:58:55.665304+0000","last_unstale":"2023-11-28T12:58:59.663795+0000","last_undegraded":"2023-11-28T12:58:59.663795+0000","last_fullsized":"2023-11-28T12:58:59.663795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.658903+0000","last_change":"2023-11-28T12:58:55.669222+0000","last_active":"2023-11-28T12:58:59.658903+0000","last_peered":"2023-11-28T12:58:59.658903+0000","last_clean":"2023-11-28T12:58:59.658903+0000","last_became_active":"2023-11-28T12:58:55.668931+0000","last_became_peered":"2023-11-28T12:58:55.668931+0000","last_unstale":"2023-11-28T12:58:59.658903+0000","last_undegraded":"2023-11-28T12:58:59.658903+0000","last_fullsized":"2023-11-28T12:58:59.658903+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:12:28.351912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674186+0000","last_change":"2023-11-28T12:58:55.666643+0000","last_active":"2023-11-28T12:58:59.674186+0000","last_peered":"2023-11-28T12:58:59.674186+0000","last_clean":"2023-11-28T12:58:59.674186+0000","last_became_active":"2023-11-28T12:58:55.666224+0000","last_became_peered":"2023-11-28T12:58:55.666224+0000","last_unstale":"2023-11-28T12:58:59.674186+0000","last_undegraded":"2023-11-28T12:58:59.674186+0000","last_fullsized":"2023-11-28T12:58:59.674186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660392+0000","last_change":"2023-11-28T12:58:55.660396+0000","last_active":"2023-11-28T12:58:59.660392+0000","last_peered":"2023-11-28T12:58:59.660392+0000","last_clean":"2023-11-28T12:58:59.660392+0000","last_became_active":"2023-11-28T12:58:55.659518+0000","last_became_peered":"2023-11-28T12:58:55.659518+0000","last_unstale":"2023-11-28T12:58:59.660392+0000","last_undegraded":"2023-11-28T12:58:59.660392+0000","last_fullsized":"2023-11-28T12:58:59.660392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:49:50.356082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672413+0000","last_change":"2023-11-28T12:58:55.650573+0000","last_active":"2023-11-28T12:58:59.672413+0000","last_peered":"2023-11-28T12:58:59.672413+0000","last_clean":"2023-11-28T12:58:59.672413+0000","last_became_active":"2023-11-28T12:58:55.650382+0000","last_became_peered":"2023-11-28T12:58:55.650382+0000","last_unstale":"2023-11-28T12:58:59.672413+0000","last_undegraded":"2023-11-28T12:58:59.672413+0000","last_fullsized":"2023-11-28T12:58:59.672413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:54:07.154200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659749+0000","last_change":"2023-11-28T12:58:55.663920+0000","last_active":"2023-11-28T12:58:59.659749+0000","last_peered":"2023-11-28T12:58:59.659749+0000","last_clean":"2023-11-28T12:58:59.659749+0000","last_became_active":"2023-11-28T12:58:55.663347+0000","last_became_peered":"2023-11-28T12:58:55.663347+0000","last_unstale":"2023-11-28T12:58:59.659749+0000","last_undegraded":"2023-11-28T12:58:59.659749+0000","last_fullsized":"2023-11-28T12:58:59.659749+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664143+0000","last_change":"2023-11-28T12:58:55.661755+0000","last_active":"2023-11-28T12:58:59.664143+0000","last_peered":"2023-11-28T12:58:59.664143+0000","last_clean":"2023-11-28T12:58:59.664143+0000","last_became_active":"2023-11-28T12:58:55.661496+0000","last_became_peered":"2023-11-28T12:58:55.661496+0000","last_unstale":"2023-11-28T12:58:59.664143+0000","last_undegraded":"2023-11-28T12:58:59.664143+0000","last_fullsized":"2023-11-28T12:58:59.664143+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669008+0000","last_change":"2023-11-28T12:58:55.666002+0000","last_active":"2023-11-28T12:58:59.669008+0000","last_peered":"2023-11-28T12:58:59.669008+0000","last_clean":"2023-11-28T12:58:59.669008+0000","last_became_active":"2023-11-28T12:58:55.657322+0000","last_became_peered":"2023-11-28T12:58:55.657322+0000","last_unstale":"2023-11-28T12:58:59.669008+0000","last_undegraded":"2023-11-28T12:58:59.669008+0000","last_fullsized":"2023-11-28T12:58:59.669008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655373+0000","last_change":"2023-11-28T12:58:55.669042+0000","last_active":"2023-11-28T12:58:59.655373+0000","last_peered":"2023-11-28T12:58:59.655373+0000","last_clean":"2023-11-28T12:58:59.655373+0000","last_became_active":"2023-11-28T12:58:55.668810+0000","last_became_peered":"2023-11-28T12:58:55.668810+0000","last_unstale":"2023-11-28T12:58:59.655373+0000","last_undegraded":"2023-11-28T12:58:59.655373+0000","last_fullsized":"2023-11-28T12:58:59.655373+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:49:50.356082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666243+0000","last_change":"2023-11-28T12:58:55.658006+0000","last_active":"2023-11-28T12:58:59.666243+0000","last_peered":"2023-11-28T12:58:59.666243+0000","last_clean":"2023-11-28T12:58:59.666243+0000","last_became_active":"2023-11-28T12:58:55.657791+0000","last_became_peered":"2023-11-28T12:58:55.657791+0000","last_unstale":"2023-11-28T12:58:59.666243+0000","last_undegraded":"2023-11-28T12:58:59.666243+0000","last_fullsized":"2023-11-28T12:58:59.666243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667574+0000","last_change":"2023-11-28T12:58:55.658052+0000","last_active":"2023-11-28T12:58:59.667574+0000","last_peered":"2023-11-28T12:58:59.667574+0000","last_clean":"2023-11-28T12:58:59.667574+0000","last_became_active":"2023-11-28T12:58:55.657752+0000","last_became_peered":"2023-11-28T12:58:55.657752+0000","last_unstale":"2023-11-28T12:58:59.667574+0000","last_undegraded":"2023-11-28T12:58:59.667574+0000","last_fullsized":"2023-11-28T12:58:59.667574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665870+0000","last_change":"2023-11-28T12:58:55.657589+0000","last_active":"2023-11-28T12:58:59.665870+0000","last_peered":"2023-11-28T12:58:59.665870+0000","last_clean":"2023-11-28T12:58:59.665870+0000","last_became_active":"2023-11-28T12:58:55.654770+0000","last_became_peered":"2023-11-28T12:58:55.654770+0000","last_unstale":"2023-11-28T12:58:59.665870+0000","last_undegraded":"2023-11-28T12:58:59.665870+0000","last_fullsized":"2023-11-28T12:58:59.665870+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:06:39.770893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665748+0000","last_change":"2023-11-28T12:58:55.657244+0000","last_active":"2023-11-28T12:58:59.665748+0000","last_peered":"2023-11-28T12:58:59.665748+0000","last_clean":"2023-11-28T12:58:59.665748+0000","last_became_active":"2023-11-28T12:58:55.656984+0000","last_became_peered":"2023-11-28T12:58:55.656984+0000","last_unstale":"2023-11-28T12:58:59.665748+0000","last_undegraded":"2023-11-28T12:58:59.665748+0000","last_fullsized":"2023-11-28T12:58:59.665748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059204+0000","last_change":"2023-11-28T12:58:59.681861+0000","last_active":"2023-11-28T12:59:00.059204+0000","last_peered":"2023-11-28T12:59:00.059204+0000","last_clean":"2023-11-28T12:59:00.059204+0000","last_became_active":"2023-11-28T12:58:59.681503+0000","last_became_peered":"2023-11-28T12:58:59.681503+0000","last_unstale":"2023-11-28T12:59:00.059204+0000","last_undegraded":"2023-11-28T12:59:00.059204+0000","last_fullsized":"2023-11-28T12:59:00.059204+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:27.673942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659368+0000","last_change":"2023-11-28T12:58:55.248299+0000","last_active":"2023-11-28T12:58:59.659368+0000","last_peered":"2023-11-28T12:58:59.659368+0000","last_clean":"2023-11-28T12:58:59.659368+0000","last_became_active":"2023-11-28T12:58:55.247956+0000","last_became_peered":"2023-11-28T12:58:55.247956+0000","last_unstale":"2023-11-28T12:58:59.659368+0000","last_undegraded":"2023-11-28T12:58:59.659368+0000","last_fullsized":"2023-11-28T12:58:59.659368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:25:52.356939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655726+0000","last_change":"2023-11-28T12:58:55.660586+0000","last_active":"2023-11-28T12:58:59.655726+0000","last_peered":"2023-11-28T12:58:59.655726+0000","last_clean":"2023-11-28T12:58:59.655726+0000","last_became_active":"2023-11-28T12:58:55.658146+0000","last_became_peered":"2023-11-28T12:58:55.658146+0000","last_unstale":"2023-11-28T12:58:59.655726+0000","last_undegraded":"2023-11-28T12:58:59.655726+0000","last_fullsized":"2023-11-28T12:58:59.655726+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058649+0000","last_change":"2023-11-28T12:58:59.672932+0000","last_active":"2023-11-28T12:59:00.058649+0000","last_peered":"2023-11-28T12:59:00.058649+0000","last_clean":"2023-11-28T12:59:00.058649+0000","last_became_active":"2023-11-28T12:58:59.672540+0000","last_became_peered":"2023-11-28T12:58:59.672540+0000","last_unstale":"2023-11-28T12:59:00.058649+0000","last_undegraded":"2023-11-28T12:59:00.058649+0000","last_fullsized":"2023-11-28T12:59:00.058649+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:12:31.778491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661524+0000","last_change":"2023-11-28T12:58:55.247493+0000","last_active":"2023-11-28T12:58:59.661524+0000","last_peered":"2023-11-28T12:58:59.661524+0000","last_clean":"2023-11-28T12:58:59.661524+0000","last_became_active":"2023-11-28T12:58:55.247218+0000","last_became_peered":"2023-11-28T12:58:55.247218+0000","last_unstale":"2023-11-28T12:58:59.661524+0000","last_undegraded":"2023-11-28T12:58:59.661524+0000","last_fullsized":"2023-11-28T12:58:59.661524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:38:45.929395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674242+0000","last_change":"2023-11-28T12:58:55.666790+0000","last_active":"2023-11-28T12:58:59.674242+0000","last_peered":"2023-11-28T12:58:59.674242+0000","last_clean":"2023-11-28T12:58:59.674242+0000","last_became_active":"2023-11-28T12:58:55.666541+0000","last_became_peered":"2023-11-28T12:58:55.666541+0000","last_unstale":"2023-11-28T12:58:59.674242+0000","last_undegraded":"2023-11-28T12:58:59.674242+0000","last_fullsized":"2023-11-28T12:58:59.674242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313692+0000","last_change":"2023-11-28T12:58:59.685314+0000","last_active":"2023-11-28T12:59:00.313692+0000","last_peered":"2023-11-28T12:59:00.313692+0000","last_clean":"2023-11-28T12:59:00.313692+0000","last_became_active":"2023-11-28T12:58:59.684940+0000","last_became_peered":"2023-11-28T12:58:59.684940+0000","last_unstale":"2023-11-28T12:59:00.313692+0000","last_undegraded":"2023-11-28T12:59:00.313692+0000","last_fullsized":"2023-11-28T12:59:00.313692+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:53.458898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657621+0000","last_change":"2023-11-28T12:58:55.256308+0000","last_active":"2023-11-28T12:58:59.657621+0000","last_peered":"2023-11-28T12:58:59.657621+0000","last_clean":"2023-11-28T12:58:59.657621+0000","last_became_active":"2023-11-28T12:58:55.256020+0000","last_became_peered":"2023-11-28T12:58:55.256020+0000","last_unstale":"2023-11-28T12:58:59.657621+0000","last_undegraded":"2023-11-28T12:58:59.657621+0000","last_fullsized":"2023-11-28T12:58:59.657621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:49.233234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659104+0000","last_change":"2023-11-28T12:58:55.658284+0000","last_active":"2023-11-28T12:58:59.659104+0000","last_peered":"2023-11-28T12:58:59.659104+0000","last_clean":"2023-11-28T12:58:59.659104+0000","last_became_active":"2023-11-28T12:58:55.657995+0000","last_became_peered":"2023-11-28T12:58:55.657995+0000","last_unstale":"2023-11-28T12:58:59.659104+0000","last_undegraded":"2023-11-28T12:58:59.659104+0000","last_fullsized":"2023-11-28T12:58:59.659104+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:11:49.546022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058116+0000","last_change":"2023-11-28T12:58:59.669835+0000","last_active":"2023-11-28T12:59:00.058116+0000","last_peered":"2023-11-28T12:59:00.058116+0000","last_clean":"2023-11-28T12:59:00.058116+0000","last_became_active":"2023-11-28T12:58:59.669428+0000","last_became_peered":"2023-11-28T12:58:59.669428+0000","last_unstale":"2023-11-28T12:59:00.058116+0000","last_undegraded":"2023-11-28T12:59:00.058116+0000","last_fullsized":"2023-11-28T12:59:00.058116+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:18:08.491909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674258+0000","last_change":"2023-11-28T12:58:55.242328+0000","last_active":"2023-11-28T12:58:59.674258+0000","last_peered":"2023-11-28T12:58:59.674258+0000","last_clean":"2023-11-28T12:58:59.674258+0000","last_became_active":"2023-11-28T12:58:55.242028+0000","last_became_peered":"2023-11-28T12:58:55.242028+0000","last_unstale":"2023-11-28T12:58:59.674258+0000","last_undegraded":"2023-11-28T12:58:59.674258+0000","last_fullsized":"2023-11-28T12:58:59.674258+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:42.300078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655600+0000","last_change":"2023-11-28T12:58:55.655732+0000","last_active":"2023-11-28T12:58:59.655600+0000","last_peered":"2023-11-28T12:58:59.655600+0000","last_clean":"2023-11-28T12:58:59.655600+0000","last_became_active":"2023-11-28T12:58:55.655490+0000","last_became_peered":"2023-11-28T12:58:55.655490+0000","last_unstale":"2023-11-28T12:58:59.655600+0000","last_undegraded":"2023-11-28T12:58:59.655600+0000","last_fullsized":"2023-11-28T12:58:59.655600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829559+0000","last_change":"2023-11-28T12:58:59.676365+0000","last_active":"2023-11-28T12:59:00.829559+0000","last_peered":"2023-11-28T12:59:00.829559+0000","last_clean":"2023-11-28T12:59:00.829559+0000","last_became_active":"2023-11-28T12:58:59.675934+0000","last_became_peered":"2023-11-28T12:58:59.675934+0000","last_unstale":"2023-11-28T12:59:00.829559+0000","last_undegraded":"2023-11-28T12:59:00.829559+0000","last_fullsized":"2023-11-28T12:59:00.829559+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:04:35.226907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668457+0000","last_change":"2023-11-28T12:58:55.244542+0000","last_active":"2023-11-28T12:58:59.668457+0000","last_peered":"2023-11-28T12:58:59.668457+0000","last_clean":"2023-11-28T12:58:59.668457+0000","last_became_active":"2023-11-28T12:58:55.244251+0000","last_became_peered":"2023-11-28T12:58:55.244251+0000","last_unstale":"2023-11-28T12:58:59.668457+0000","last_undegraded":"2023-11-28T12:58:59.668457+0000","last_fullsized":"2023-11-28T12:58:59.668457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:45:06.134459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655451+0000","last_change":"2023-11-28T12:58:55.661735+0000","last_active":"2023-11-28T12:58:59.655451+0000","last_peered":"2023-11-28T12:58:59.655451+0000","last_clean":"2023-11-28T12:58:59.655451+0000","last_became_active":"2023-11-28T12:58:55.661468+0000","last_became_peered":"2023-11-28T12:58:55.661468+0000","last_unstale":"2023-11-28T12:58:59.655451+0000","last_undegraded":"2023-11-28T12:58:59.655451+0000","last_fullsized":"2023-11-28T12:58:59.655451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:34.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059839+0000","last_change":"2023-11-28T12:58:59.677031+0000","last_active":"2023-11-28T12:59:00.059839+0000","last_peered":"2023-11-28T12:59:00.059839+0000","last_clean":"2023-11-28T12:59:00.059839+0000","last_became_active":"2023-11-28T12:58:59.676510+0000","last_became_peered":"2023-11-28T12:58:59.676510+0000","last_unstale":"2023-11-28T12:59:00.059839+0000","last_undegraded":"2023-11-28T12:59:00.059839+0000","last_fullsized":"2023-11-28T12:59:00.059839+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:03:33.042937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"21'3","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664803+0000","last_change":"2023-11-28T12:58:55.254041+0000","last_active":"2023-11-28T12:58:59.664803+0000","last_peered":"2023-11-28T12:58:59.664803+0000","last_clean":"2023-11-28T12:58:59.664803+0000","last_became_active":"2023-11-28T12:58:55.253797+0000","last_became_peered":"2023-11-28T12:58:55.253797+0000","last_unstale":"2023-11-28T12:58:59.664803+0000","last_undegraded":"2023-11-28T12:58:59.664803+0000","last_fullsized":"2023-11-28T12:58:59.664803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:42.442743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664769+0000","last_change":"2023-11-28T12:58:55.648632+0000","last_active":"2023-11-28T12:58:59.664769+0000","last_peered":"2023-11-28T12:58:59.664769+0000","last_clean":"2023-11-28T12:58:59.664769+0000","last_became_active":"2023-11-28T12:58:55.648367+0000","last_became_peered":"2023-11-28T12:58:55.648367+0000","last_unstale":"2023-11-28T12:58:59.664769+0000","last_undegraded":"2023-11-28T12:58:59.664769+0000","last_fullsized":"2023-11-28T12:58:59.664769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.315401+0000","last_change":"2023-11-28T12:58:59.675361+0000","last_active":"2023-11-28T12:59:00.315401+0000","last_peered":"2023-11-28T12:59:00.315401+0000","last_clean":"2023-11-28T12:59:00.315401+0000","last_became_active":"2023-11-28T12:58:59.674991+0000","last_became_peered":"2023-11-28T12:58:59.674991+0000","last_unstale":"2023-11-28T12:59:00.315401+0000","last_undegraded":"2023-11-28T12:59:00.315401+0000","last_fullsized":"2023-11-28T12:59:00.315401+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:21:52.978291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"26'9","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:04.555380+0000","last_change":"2023-11-28T12:58:55.245189+0000","last_active":"2023-11-28T12:59:04.555380+0000","last_peered":"2023-11-28T12:59:04.555380+0000","last_clean":"2023-11-28T12:59:04.555380+0000","last_became_active":"2023-11-28T12:58:55.244896+0000","last_became_peered":"2023-11-28T12:58:55.244896+0000","last_unstale":"2023-11-28T12:59:04.555380+0000","last_undegraded":"2023-11-28T12:59:04.555380+0000","last_fullsized":"2023-11-28T12:59:04.555380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3746,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659437+0000","last_change":"2023-11-28T12:58:55.658816+0000","last_active":"2023-11-28T12:58:59.659437+0000","last_peered":"2023-11-28T12:58:59.659437+0000","last_clean":"2023-11-28T12:58:59.659437+0000","last_became_active":"2023-11-28T12:58:55.657238+0000","last_became_peered":"2023-11-28T12:58:55.657238+0000","last_unstale":"2023-11-28T12:58:59.659437+0000","last_undegraded":"2023-11-28T12:58:59.659437+0000","last_fullsized":"2023-11-28T12:58:59.659437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:14:28.821164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830316+0000","last_change":"2023-11-28T12:58:59.685273+0000","last_active":"2023-11-28T12:59:00.830316+0000","last_peered":"2023-11-28T12:59:00.830316+0000","last_clean":"2023-11-28T12:59:00.830316+0000","last_became_active":"2023-11-28T12:58:59.684917+0000","last_became_peered":"2023-11-28T12:58:59.684917+0000","last_unstale":"2023-11-28T12:59:00.830316+0000","last_undegraded":"2023-11-28T12:59:00.830316+0000","last_fullsized":"2023-11-28T12:59:00.830316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:31.632801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664060+0000","last_change":"2023-11-28T12:58:55.249201+0000","last_active":"2023-11-28T12:58:59.664060+0000","last_peered":"2023-11-28T12:58:59.664060+0000","last_clean":"2023-11-28T12:58:59.664060+0000","last_became_active":"2023-11-28T12:58:55.248828+0000","last_became_peered":"2023-11-28T12:58:55.248828+0000","last_unstale":"2023-11-28T12:58:59.664060+0000","last_undegraded":"2023-11-28T12:58:59.664060+0000","last_fullsized":"2023-11-28T12:58:59.664060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:11:48.540120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655687+0000","last_change":"2023-11-28T12:58:55.660397+0000","last_active":"2023-11-28T12:58:59.655687+0000","last_peered":"2023-11-28T12:58:59.655687+0000","last_clean":"2023-11-28T12:58:59.655687+0000","last_became_active":"2023-11-28T12:58:55.657972+0000","last_became_peered":"2023-11-28T12:58:55.657972+0000","last_unstale":"2023-11-28T12:58:59.655687+0000","last_undegraded":"2023-11-28T12:58:59.655687+0000","last_fullsized":"2023-11-28T12:58:59.655687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059788+0000","last_change":"2023-11-28T12:58:59.681226+0000","last_active":"2023-11-28T12:59:00.059788+0000","last_peered":"2023-11-28T12:59:00.059788+0000","last_clean":"2023-11-28T12:59:00.059788+0000","last_became_active":"2023-11-28T12:58:59.680846+0000","last_became_peered":"2023-11-28T12:58:59.680846+0000","last_unstale":"2023-11-28T12:59:00.059788+0000","last_undegraded":"2023-11-28T12:59:00.059788+0000","last_fullsized":"2023-11-28T12:59:00.059788+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:59.373707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668909+0000","last_change":"2023-11-28T12:58:55.246343+0000","last_active":"2023-11-28T12:58:59.668909+0000","last_peered":"2023-11-28T12:58:59.668909+0000","last_clean":"2023-11-28T12:58:59.668909+0000","last_became_active":"2023-11-28T12:58:55.244840+0000","last_became_peered":"2023-11-28T12:58:55.244840+0000","last_unstale":"2023-11-28T12:58:59.668909+0000","last_undegraded":"2023-11-28T12:58:59.668909+0000","last_fullsized":"2023-11-28T12:58:59.668909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665665+0000","last_change":"2023-11-28T12:58:55.656948+0000","last_active":"2023-11-28T12:58:59.665665+0000","last_peered":"2023-11-28T12:58:59.665665+0000","last_clean":"2023-11-28T12:58:59.665665+0000","last_became_active":"2023-11-28T12:58:55.656678+0000","last_became_peered":"2023-11-28T12:58:55.656678+0000","last_unstale":"2023-11-28T12:58:59.665665+0000","last_undegraded":"2023-11-28T12:58:59.665665+0000","last_fullsized":"2023-11-28T12:58:59.665665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.060115+0000","last_change":"2023-11-28T12:58:59.677384+0000","last_active":"2023-11-28T12:59:00.060115+0000","last_peered":"2023-11-28T12:59:00.060115+0000","last_clean":"2023-11-28T12:59:00.060115+0000","last_became_active":"2023-11-28T12:58:59.676997+0000","last_became_peered":"2023-11-28T12:58:59.676997+0000","last_unstale":"2023-11-28T12:59:00.060115+0000","last_undegraded":"2023-11-28T12:59:00.060115+0000","last_fullsized":"2023-11-28T12:59:00.060115+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:27:31.632801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.663889+0000","last_change":"2023-11-28T12:58:55.249928+0000","last_active":"2023-11-28T12:58:59.663889+0000","last_peered":"2023-11-28T12:58:59.663889+0000","last_clean":"2023-11-28T12:58:59.663889+0000","last_became_active":"2023-11-28T12:58:55.249500+0000","last_became_peered":"2023-11-28T12:58:55.249500+0000","last_unstale":"2023-11-28T12:58:59.663889+0000","last_undegraded":"2023-11-28T12:58:59.663889+0000","last_fullsized":"2023-11-28T12:58:59.663889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:18.172623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674319+0000","last_change":"2023-11-28T12:58:55.663464+0000","last_active":"2023-11-28T12:58:59.674319+0000","last_peered":"2023-11-28T12:58:59.674319+0000","last_clean":"2023-11-28T12:58:59.674319+0000","last_became_active":"2023-11-28T12:58:55.663205+0000","last_became_peered":"2023-11-28T12:58:55.663205+0000","last_unstale":"2023-11-28T12:58:59.674319+0000","last_undegraded":"2023-11-28T12:58:59.674319+0000","last_fullsized":"2023-11-28T12:58:59.674319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.831228+0000","last_change":"2023-11-28T12:58:59.675282+0000","last_active":"2023-11-28T12:59:00.831228+0000","last_peered":"2023-11-28T12:59:00.831228+0000","last_clean":"2023-11-28T12:59:00.831228+0000","last_became_active":"2023-11-28T12:58:59.674859+0000","last_became_peered":"2023-11-28T12:58:59.674859+0000","last_unstale":"2023-11-28T12:59:00.831228+0000","last_undegraded":"2023-11-28T12:59:00.831228+0000","last_fullsized":"2023-11-28T12:59:00.831228+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:59.373707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"21'8","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.656905+0000","last_change":"2023-11-28T12:58:55.236728+0000","last_active":"2023-11-28T12:58:59.656905+0000","last_peered":"2023-11-28T12:58:59.656905+0000","last_clean":"2023-11-28T12:58:59.656905+0000","last_became_active":"2023-11-28T12:58:55.236452+0000","last_became_peered":"2023-11-28T12:58:55.236452+0000","last_unstale":"2023-11-28T12:58:59.656905+0000","last_undegraded":"2023-11-28T12:58:59.656905+0000","last_fullsized":"2023-11-28T12:58:59.656905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655960+0000","last_change":"2023-11-28T12:58:55.662005+0000","last_active":"2023-11-28T12:58:59.655960+0000","last_peered":"2023-11-28T12:58:59.655960+0000","last_clean":"2023-11-28T12:58:59.655960+0000","last_became_active":"2023-11-28T12:58:55.661531+0000","last_became_peered":"2023-11-28T12:58:55.661531+0000","last_unstale":"2023-11-28T12:58:59.655960+0000","last_undegraded":"2023-11-28T12:58:59.655960+0000","last_fullsized":"2023-11-28T12:58:59.655960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:34:15.223824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830169+0000","last_change":"2023-11-28T12:58:59.675000+0000","last_active":"2023-11-28T12:59:00.830169+0000","last_peered":"2023-11-28T12:59:00.830169+0000","last_clean":"2023-11-28T12:59:00.830169+0000","last_became_active":"2023-11-28T12:58:59.674646+0000","last_became_peered":"2023-11-28T12:58:59.674646+0000","last_unstale":"2023-11-28T12:59:00.830169+0000","last_undegraded":"2023-11-28T12:59:00.830169+0000","last_fullsized":"2023-11-28T12:59:00.830169+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:46:17.417359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667637+0000","last_change":"2023-11-28T12:58:55.249124+0000","last_active":"2023-11-28T12:58:59.667637+0000","last_peered":"2023-11-28T12:58:59.667637+0000","last_clean":"2023-11-28T12:58:59.667637+0000","last_became_active":"2023-11-28T12:58:55.248501+0000","last_became_peered":"2023-11-28T12:58:55.248501+0000","last_unstale":"2023-11-28T12:58:59.667637+0000","last_undegraded":"2023-11-28T12:58:59.667637+0000","last_fullsized":"2023-11-28T12:58:59.667637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:25.569310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661198+0000","last_change":"2023-11-28T12:58:55.659587+0000","last_active":"2023-11-28T12:58:59.661198+0000","last_peered":"2023-11-28T12:58:59.661198+0000","last_clean":"2023-11-28T12:58:59.661198+0000","last_became_active":"2023-11-28T12:58:55.659319+0000","last_became_peered":"2023-11-28T12:58:55.659319+0000","last_unstale":"2023-11-28T12:58:59.661198+0000","last_undegraded":"2023-11-28T12:58:59.661198+0000","last_fullsized":"2023-11-28T12:58:59.661198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.828936+0000","last_change":"2023-11-28T12:58:59.686412+0000","last_active":"2023-11-28T12:59:00.828936+0000","last_peered":"2023-11-28T12:59:00.828936+0000","last_clean":"2023-11-28T12:59:00.828936+0000","last_became_active":"2023-11-28T12:58:59.686079+0000","last_became_peered":"2023-11-28T12:58:59.686079+0000","last_unstale":"2023-11-28T12:59:00.828936+0000","last_undegraded":"2023-11-28T12:59:00.828936+0000","last_fullsized":"2023-11-28T12:59:00.828936+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:58:48.492561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664149+0000","last_change":"2023-11-28T12:58:55.248876+0000","last_active":"2023-11-28T12:58:59.664149+0000","last_peered":"2023-11-28T12:58:59.664149+0000","last_clean":"2023-11-28T12:58:59.664149+0000","last_became_active":"2023-11-28T12:58:55.248584+0000","last_became_peered":"2023-11-28T12:58:55.248584+0000","last_unstale":"2023-11-28T12:58:59.664149+0000","last_undegraded":"2023-11-28T12:58:59.664149+0000","last_fullsized":"2023-11-28T12:58:59.664149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:47:12.446132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659149+0000","last_change":"2023-11-28T12:58:55.663768+0000","last_active":"2023-11-28T12:58:59.659149+0000","last_peered":"2023-11-28T12:58:59.659149+0000","last_clean":"2023-11-28T12:58:59.659149+0000","last_became_active":"2023-11-28T12:58:55.663476+0000","last_became_peered":"2023-11-28T12:58:55.663476+0000","last_unstale":"2023-11-28T12:58:59.659149+0000","last_undegraded":"2023-11-28T12:58:59.659149+0000","last_fullsized":"2023-11-28T12:58:59.659149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:07:49.345226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829075+0000","last_change":"2023-11-28T12:58:59.685644+0000","last_active":"2023-11-28T12:59:00.829075+0000","last_peered":"2023-11-28T12:59:00.829075+0000","last_clean":"2023-11-28T12:59:00.829075+0000","last_became_active":"2023-11-28T12:58:59.685259+0000","last_became_peered":"2023-11-28T12:58:59.685259+0000","last_unstale":"2023-11-28T12:59:00.829075+0000","last_undegraded":"2023-11-28T12:59:00.829075+0000","last_fullsized":"2023-11-28T12:59:00.829075+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:07:52.771805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664976+0000","last_change":"2023-11-28T12:58:55.247686+0000","last_active":"2023-11-28T12:58:59.664976+0000","last_peered":"2023-11-28T12:58:59.664976+0000","last_clean":"2023-11-28T12:58:59.664976+0000","last_became_active":"2023-11-28T12:58:55.247328+0000","last_became_peered":"2023-11-28T12:58:55.247328+0000","last_unstale":"2023-11-28T12:58:59.664976+0000","last_undegraded":"2023-11-28T12:58:59.664976+0000","last_fullsized":"2023-11-28T12:58:59.664976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:22:00.609577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.656940+0000","last_change":"2023-11-28T12:58:55.649490+0000","last_active":"2023-11-28T12:58:59.656940+0000","last_peered":"2023-11-28T12:58:59.656940+0000","last_clean":"2023-11-28T12:58:59.656940+0000","last_became_active":"2023-11-28T12:58:55.649143+0000","last_became_peered":"2023-11-28T12:58:55.649143+0000","last_unstale":"2023-11-28T12:58:59.656940+0000","last_undegraded":"2023-11-28T12:58:59.656940+0000","last_fullsized":"2023-11-28T12:58:59.656940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:26.575212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829146+0000","last_change":"2023-11-28T12:58:59.679465+0000","last_active":"2023-11-28T12:59:00.829146+0000","last_peered":"2023-11-28T12:59:00.829146+0000","last_clean":"2023-11-28T12:59:00.829146+0000","last_became_active":"2023-11-28T12:58:59.679026+0000","last_became_peered":"2023-11-28T12:58:59.679026+0000","last_unstale":"2023-11-28T12:59:00.829146+0000","last_undegraded":"2023-11-28T12:59:00.829146+0000","last_fullsized":"2023-11-28T12:59:00.829146+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:52.339645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665301+0000","last_change":"2023-11-28T12:58:55.249645+0000","last_active":"2023-11-28T12:58:59.665301+0000","last_peered":"2023-11-28T12:58:59.665301+0000","last_clean":"2023-11-28T12:58:59.665301+0000","last_became_active":"2023-11-28T12:58:55.249284+0000","last_became_peered":"2023-11-28T12:58:55.249284+0000","last_unstale":"2023-11-28T12:58:59.665301+0000","last_undegraded":"2023-11-28T12:58:59.665301+0000","last_fullsized":"2023-11-28T12:58:59.665301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:49.233234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669390+0000","last_change":"2023-11-28T12:58:55.663994+0000","last_active":"2023-11-28T12:58:59.669390+0000","last_peered":"2023-11-28T12:58:59.669390+0000","last_clean":"2023-11-28T12:58:59.669390+0000","last_became_active":"2023-11-28T12:58:55.657280+0000","last_became_peered":"2023-11-28T12:58:55.657280+0000","last_unstale":"2023-11-28T12:58:59.669390+0000","last_undegraded":"2023-11-28T12:58:59.669390+0000","last_fullsized":"2023-11-28T12:58:59.669390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:34.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829349+0000","last_change":"2023-11-28T12:58:59.680307+0000","last_active":"2023-11-28T12:59:00.829349+0000","last_peered":"2023-11-28T12:59:00.829349+0000","last_clean":"2023-11-28T12:59:00.829349+0000","last_became_active":"2023-11-28T12:58:59.679837+0000","last_became_peered":"2023-11-28T12:58:59.679837+0000","last_unstale":"2023-11-28T12:59:00.829349+0000","last_undegraded":"2023-11-28T12:59:00.829349+0000","last_fullsized":"2023-11-28T12:59:00.829349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:18:08.491909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655480+0000","last_change":"2023-11-28T12:58:55.256347+0000","last_active":"2023-11-28T12:58:59.655480+0000","last_peered":"2023-11-28T12:58:59.655480+0000","last_clean":"2023-11-28T12:58:59.655480+0000","last_became_active":"2023-11-28T12:58:55.256027+0000","last_became_peered":"2023-11-28T12:58:55.256027+0000","last_unstale":"2023-11-28T12:58:59.655480+0000","last_undegraded":"2023-11-28T12:58:59.655480+0000","last_fullsized":"2023-11-28T12:58:59.655480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:42.300078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659538+0000","last_change":"2023-11-28T12:58:55.664332+0000","last_active":"2023-11-28T12:58:59.659538+0000","last_peered":"2023-11-28T12:58:59.659538+0000","last_clean":"2023-11-28T12:58:59.659538+0000","last_became_active":"2023-11-28T12:58:55.663913+0000","last_became_peered":"2023-11-28T12:58:55.663913+0000","last_unstale":"2023-11-28T12:58:59.659538+0000","last_undegraded":"2023-11-28T12:58:59.659538+0000","last_fullsized":"2023-11-28T12:58:59.659538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:50.590629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.315498+0000","last_change":"2023-11-28T12:58:59.678114+0000","last_active":"2023-11-28T12:59:00.315498+0000","last_peered":"2023-11-28T12:59:00.315498+0000","last_clean":"2023-11-28T12:59:00.315498+0000","last_became_active":"2023-11-28T12:58:59.677704+0000","last_became_peered":"2023-11-28T12:58:59.677704+0000","last_unstale":"2023-11-28T12:59:00.315498+0000","last_undegraded":"2023-11-28T12:59:00.315498+0000","last_fullsized":"2023-11-28T12:59:00.315498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:53.458898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657011+0000","last_change":"2023-11-28T12:58:55.241207+0000","last_active":"2023-11-28T12:58:59.657011+0000","last_peered":"2023-11-28T12:58:59.657011+0000","last_clean":"2023-11-28T12:58:59.657011+0000","last_became_active":"2023-11-28T12:58:55.240917+0000","last_became_peered":"2023-11-28T12:58:55.240917+0000","last_unstale":"2023-11-28T12:58:59.657011+0000","last_undegraded":"2023-11-28T12:58:59.657011+0000","last_fullsized":"2023-11-28T12:58:59.657011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659347+0000","last_change":"2023-11-28T12:58:55.660463+0000","last_active":"2023-11-28T12:58:59.659347+0000","last_peered":"2023-11-28T12:58:59.659347+0000","last_clean":"2023-11-28T12:58:59.659347+0000","last_became_active":"2023-11-28T12:58:55.660208+0000","last_became_peered":"2023-11-28T12:58:55.660208+0000","last_unstale":"2023-11-28T12:58:59.659347+0000","last_undegraded":"2023-11-28T12:58:59.659347+0000","last_fullsized":"2023-11-28T12:58:59.659347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830829+0000","last_change":"2023-11-28T12:58:59.688100+0000","last_active":"2023-11-28T12:59:00.830829+0000","last_peered":"2023-11-28T12:59:00.830829+0000","last_clean":"2023-11-28T12:59:00.830829+0000","last_became_active":"2023-11-28T12:58:59.687752+0000","last_became_peered":"2023-11-28T12:58:59.687752+0000","last_unstale":"2023-11-28T12:59:00.830829+0000","last_undegraded":"2023-11-28T12:59:00.830829+0000","last_fullsized":"2023-11-28T12:59:00.830829+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:38:05.937526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659014+0000","last_change":"2023-11-28T12:58:55.246754+0000","last_active":"2023-11-28T12:58:59.659014+0000","last_peered":"2023-11-28T12:58:59.659014+0000","last_clean":"2023-11-28T12:58:59.659014+0000","last_became_active":"2023-11-28T12:58:55.246480+0000","last_became_peered":"2023-11-28T12:58:55.246480+0000","last_unstale":"2023-11-28T12:58:59.659014+0000","last_undegraded":"2023-11-28T12:58:59.659014+0000","last_fullsized":"2023-11-28T12:58:59.659014+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:27:08.923409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660472+0000","last_change":"2023-11-28T12:58:55.660027+0000","last_active":"2023-11-28T12:58:59.660472+0000","last_peered":"2023-11-28T12:58:59.660472+0000","last_clean":"2023-11-28T12:58:59.660472+0000","last_became_active":"2023-11-28T12:58:55.659775+0000","last_became_peered":"2023-11-28T12:58:55.659775+0000","last_unstale":"2023-11-28T12:58:59.660472+0000","last_undegraded":"2023-11-28T12:58:59.660472+0000","last_fullsized":"2023-11-28T12:58:59.660472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:06:07.909615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058758+0000","last_change":"2023-11-28T12:58:59.681700+0000","last_active":"2023-11-28T12:59:00.058758+0000","last_peered":"2023-11-28T12:59:00.058758+0000","last_clean":"2023-11-28T12:59:00.058758+0000","last_became_active":"2023-11-28T12:58:59.681305+0000","last_became_peered":"2023-11-28T12:58:59.681305+0000","last_unstale":"2023-11-28T12:59:00.058758+0000","last_undegraded":"2023-11-28T12:59:00.058758+0000","last_fullsized":"2023-11-28T12:59:00.058758+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:53:46.532522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655511+0000","last_change":"2023-11-28T12:58:55.249674+0000","last_active":"2023-11-28T12:58:59.655511+0000","last_peered":"2023-11-28T12:58:59.655511+0000","last_clean":"2023-11-28T12:58:59.655511+0000","last_became_active":"2023-11-28T12:58:55.249391+0000","last_became_peered":"2023-11-28T12:58:55.249391+0000","last_unstale":"2023-11-28T12:58:59.655511+0000","last_undegraded":"2023-11-28T12:58:59.655511+0000","last_fullsized":"2023-11-28T12:58:59.655511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667665+0000","last_change":"2023-11-28T12:58:55.661741+0000","last_active":"2023-11-28T12:58:59.667665+0000","last_peered":"2023-11-28T12:58:59.667665+0000","last_clean":"2023-11-28T12:58:59.667665+0000","last_became_active":"2023-11-28T12:58:55.661473+0000","last_became_peered":"2023-11-28T12:58:55.661473+0000","last_unstale":"2023-11-28T12:58:59.667665+0000","last_undegraded":"2023-11-28T12:58:59.667665+0000","last_fullsized":"2023-11-28T12:58:59.667665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:41:27.479117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313283+0000","last_change":"2023-11-28T12:58:59.675425+0000","last_active":"2023-11-28T12:59:00.313283+0000","last_peered":"2023-11-28T12:59:00.313283+0000","last_clean":"2023-11-28T12:59:00.313283+0000","last_became_active":"2023-11-28T12:58:59.675044+0000","last_became_peered":"2023-11-28T12:58:59.675044+0000","last_unstale":"2023-11-28T12:59:00.313283+0000","last_undegraded":"2023-11-28T12:59:00.313283+0000","last_fullsized":"2023-11-28T12:59:00.313283+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:46:17.417359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664599+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T12:58:59.664599+0000","last_peered":"2023-11-28T12:58:59.664599+0000","last_clean":"2023-11-28T12:58:59.664599+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T12:58:59.664599+0000","last_undegraded":"2023-11-28T12:58:59.664599+0000","last_fullsized":"2023-11-28T12:58:59.664599+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665575+0000","last_change":"2023-11-28T12:58:55.247919+0000","last_active":"2023-11-28T12:58:59.665575+0000","last_peered":"2023-11-28T12:58:59.665575+0000","last_clean":"2023-11-28T12:58:59.665575+0000","last_became_active":"2023-11-28T12:58:55.247653+0000","last_became_peered":"2023-11-28T12:58:55.247653+0000","last_unstale":"2023-11-28T12:58:59.665575+0000","last_undegraded":"2023-11-28T12:58:59.665575+0000","last_fullsized":"2023-11-28T12:58:59.665575+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:54:06.148298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664140+0000","last_change":"2023-11-28T12:58:55.662430+0000","last_active":"2023-11-28T12:58:59.664140+0000","last_peered":"2023-11-28T12:58:59.664140+0000","last_clean":"2023-11-28T12:58:59.664140+0000","last_became_active":"2023-11-28T12:58:55.657899+0000","last_became_peered":"2023-11-28T12:58:55.657899+0000","last_unstale":"2023-11-28T12:58:59.664140+0000","last_undegraded":"2023-11-28T12:58:59.664140+0000","last_fullsized":"2023-11-28T12:58:59.664140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:22:01.615479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313377+0000","last_change":"2023-11-28T12:58:59.675356+0000","last_active":"2023-11-28T12:59:00.313377+0000","last_peered":"2023-11-28T12:59:00.313377+0000","last_clean":"2023-11-28T12:59:00.313377+0000","last_became_active":"2023-11-28T12:58:59.674917+0000","last_became_peered":"2023-11-28T12:58:59.674917+0000","last_unstale":"2023-11-28T12:59:00.313377+0000","last_undegraded":"2023-11-28T12:59:00.313377+0000","last_fullsized":"2023-11-28T12:59:00.313377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:51:00.263485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.661311+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T12:58:59.661311+0000","last_peered":"2023-11-28T12:58:59.661311+0000","last_clean":"2023-11-28T12:58:59.661311+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T12:58:59.661311+0000","last_undegraded":"2023-11-28T12:58:59.661311+0000","last_fullsized":"2023-11-28T12:58:59.661311+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659552+0000","last_change":"2023-11-28T12:58:55.240294+0000","last_active":"2023-11-28T12:58:59.659552+0000","last_peered":"2023-11-28T12:58:59.659552+0000","last_clean":"2023-11-28T12:58:59.659552+0000","last_became_active":"2023-11-28T12:58:55.239947+0000","last_became_peered":"2023-11-28T12:58:55.239947+0000","last_unstale":"2023-11-28T12:58:59.659552+0000","last_undegraded":"2023-11-28T12:58:59.659552+0000","last_fullsized":"2023-11-28T12:58:59.659552+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664509+0000","last_change":"2023-11-28T12:58:55.657537+0000","last_active":"2023-11-28T12:58:59.664509+0000","last_peered":"2023-11-28T12:58:59.664509+0000","last_clean":"2023-11-28T12:58:59.664509+0000","last_became_active":"2023-11-28T12:58:55.655666+0000","last_became_peered":"2023-11-28T12:58:55.655666+0000","last_unstale":"2023-11-28T12:58:59.664509+0000","last_undegraded":"2023-11-28T12:58:59.664509+0000","last_fullsized":"2023-11-28T12:58:59.664509+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.829428+0000","last_change":"2023-11-28T12:58:59.680479+0000","last_active":"2023-11-28T12:59:00.829428+0000","last_peered":"2023-11-28T12:59:00.829428+0000","last_clean":"2023-11-28T12:59:00.829428+0000","last_became_active":"2023-11-28T12:58:59.680089+0000","last_became_peered":"2023-11-28T12:58:59.680089+0000","last_unstale":"2023-11-28T12:59:00.829428+0000","last_undegraded":"2023-11-28T12:59:00.829428+0000","last_fullsized":"2023-11-28T12:59:00.829428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:12:31.778491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.673610+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T12:58:59.673610+0000","last_peered":"2023-11-28T12:58:59.673610+0000","last_clean":"2023-11-28T12:58:59.673610+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T12:58:59.673610+0000","last_undegraded":"2023-11-28T12:58:59.673610+0000","last_fullsized":"2023-11-28T12:58:59.673610+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.660849+0000","last_change":"2023-11-28T12:58:55.238872+0000","last_active":"2023-11-28T12:58:59.660849+0000","last_peered":"2023-11-28T12:58:59.660849+0000","last_clean":"2023-11-28T12:58:59.660849+0000","last_became_active":"2023-11-28T12:58:55.238457+0000","last_became_peered":"2023-11-28T12:58:55.238457+0000","last_unstale":"2023-11-28T12:58:59.660849+0000","last_undegraded":"2023-11-28T12:58:59.660849+0000","last_fullsized":"2023-11-28T12:58:59.660849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.667641+0000","last_change":"2023-11-28T12:58:55.659175+0000","last_active":"2023-11-28T12:58:59.667641+0000","last_peered":"2023-11-28T12:58:59.667641+0000","last_clean":"2023-11-28T12:58:59.667641+0000","last_became_active":"2023-11-28T12:58:55.658043+0000","last_became_peered":"2023-11-28T12:58:55.658043+0000","last_unstale":"2023-11-28T12:58:59.667641+0000","last_undegraded":"2023-11-28T12:58:59.667641+0000","last_fullsized":"2023-11-28T12:58:59.667641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:43.448645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830642+0000","last_change":"2023-11-28T12:58:59.682758+0000","last_active":"2023-11-28T12:59:00.830642+0000","last_peered":"2023-11-28T12:59:00.830642+0000","last_clean":"2023-11-28T12:59:00.830642+0000","last_became_active":"2023-11-28T12:58:59.682429+0000","last_became_peered":"2023-11-28T12:58:59.682429+0000","last_unstale":"2023-11-28T12:59:00.830642+0000","last_undegraded":"2023-11-28T12:59:00.830642+0000","last_fullsized":"2023-11-28T12:59:00.830642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:59.373707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668370+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T12:58:59.668370+0000","last_peered":"2023-11-28T12:58:59.668370+0000","last_clean":"2023-11-28T12:58:59.668370+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T12:58:59.668370+0000","last_undegraded":"2023-11-28T12:58:59.668370+0000","last_fullsized":"2023-11-28T12:58:59.668370+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666175+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T12:58:59.666175+0000","last_peered":"2023-11-28T12:58:59.666175+0000","last_clean":"2023-11-28T12:58:59.666175+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T12:58:59.666175+0000","last_undegraded":"2023-11-28T12:58:59.666175+0000","last_fullsized":"2023-11-28T12:58:59.666175+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655438+0000","last_change":"2023-11-28T12:58:55.248780+0000","last_active":"2023-11-28T12:58:59.655438+0000","last_peered":"2023-11-28T12:58:59.655438+0000","last_clean":"2023-11-28T12:58:59.655438+0000","last_became_active":"2023-11-28T12:58:55.248456+0000","last_became_peered":"2023-11-28T12:58:55.248456+0000","last_unstale":"2023-11-28T12:58:59.655438+0000","last_undegraded":"2023-11-28T12:58:59.655438+0000","last_fullsized":"2023-11-28T12:58:59.655438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:25:52.356939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657383+0000","last_change":"2023-11-28T12:58:55.661508+0000","last_active":"2023-11-28T12:58:59.657383+0000","last_peered":"2023-11-28T12:58:59.657383+0000","last_clean":"2023-11-28T12:58:59.657383+0000","last_became_active":"2023-11-28T12:58:55.661252+0000","last_became_peered":"2023-11-28T12:58:55.661252+0000","last_unstale":"2023-11-28T12:58:59.657383+0000","last_undegraded":"2023-11-28T12:58:59.657383+0000","last_fullsized":"2023-11-28T12:58:59.657383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:22:01.615479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830736+0000","last_change":"2023-11-28T12:58:59.675557+0000","last_active":"2023-11-28T12:59:00.830736+0000","last_peered":"2023-11-28T12:59:00.830736+0000","last_clean":"2023-11-28T12:59:00.830736+0000","last_became_active":"2023-11-28T12:58:59.675183+0000","last_became_peered":"2023-11-28T12:58:59.675183+0000","last_unstale":"2023-11-28T12:59:00.830736+0000","last_undegraded":"2023-11-28T12:59:00.830736+0000","last_fullsized":"2023-11-28T12:59:00.830736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:17.338033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.666288+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T12:58:59.666288+0000","last_peered":"2023-11-28T12:58:59.666288+0000","last_clean":"2023-11-28T12:58:59.666288+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T12:58:59.666288+0000","last_undegraded":"2023-11-28T12:58:59.666288+0000","last_fullsized":"2023-11-28T12:58:59.666288+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.668355+0000","last_change":"2023-11-28T12:58:55.246924+0000","last_active":"2023-11-28T12:58:59.668355+0000","last_peered":"2023-11-28T12:58:59.668355+0000","last_clean":"2023-11-28T12:58:59.668355+0000","last_became_active":"2023-11-28T12:58:55.244984+0000","last_became_peered":"2023-11-28T12:58:55.244984+0000","last_unstale":"2023-11-28T12:58:59.668355+0000","last_undegraded":"2023-11-28T12:58:59.668355+0000","last_fullsized":"2023-11-28T12:58:59.668355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:15.207455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674088+0000","last_change":"2023-11-28T12:58:55.666303+0000","last_active":"2023-11-28T12:58:59.674088+0000","last_peered":"2023-11-28T12:58:59.674088+0000","last_clean":"2023-11-28T12:58:59.674088+0000","last_became_active":"2023-11-28T12:58:55.666058+0000","last_became_peered":"2023-11-28T12:58:55.666058+0000","last_unstale":"2023-11-28T12:58:59.674088+0000","last_undegraded":"2023-11-28T12:58:59.674088+0000","last_fullsized":"2023-11-28T12:58:59.674088+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058107+0000","last_change":"2023-11-28T12:58:59.681611+0000","last_active":"2023-11-28T12:59:00.058107+0000","last_peered":"2023-11-28T12:59:00.058107+0000","last_clean":"2023-11-28T12:59:00.058107+0000","last_became_active":"2023-11-28T12:58:59.681195+0000","last_became_peered":"2023-11-28T12:58:59.681195+0000","last_unstale":"2023-11-28T12:59:00.058107+0000","last_undegraded":"2023-11-28T12:59:00.058107+0000","last_fullsized":"2023-11-28T12:59:00.058107+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:58:48.492561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664884+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T12:58:59.664884+0000","last_peered":"2023-11-28T12:58:59.664884+0000","last_clean":"2023-11-28T12:58:59.664884+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T12:58:59.664884+0000","last_undegraded":"2023-11-28T12:58:59.664884+0000","last_fullsized":"2023-11-28T12:58:59.664884+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664779+0000","last_change":"2023-11-28T12:58:55.240088+0000","last_active":"2023-11-28T12:58:59.664779+0000","last_peered":"2023-11-28T12:58:59.664779+0000","last_clean":"2023-11-28T12:58:59.664779+0000","last_became_active":"2023-11-28T12:58:55.237949+0000","last_became_peered":"2023-11-28T12:58:55.237949+0000","last_unstale":"2023-11-28T12:58:59.664779+0000","last_undegraded":"2023-11-28T12:58:59.664779+0000","last_fullsized":"2023-11-28T12:58:59.664779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:51.653524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669188+0000","last_change":"2023-11-28T12:58:55.657049+0000","last_active":"2023-11-28T12:58:59.669188+0000","last_peered":"2023-11-28T12:58:59.669188+0000","last_clean":"2023-11-28T12:58:59.669188+0000","last_became_active":"2023-11-28T12:58:55.656787+0000","last_became_peered":"2023-11-28T12:58:55.656787+0000","last_unstale":"2023-11-28T12:58:59.669188+0000","last_undegraded":"2023-11-28T12:58:59.669188+0000","last_fullsized":"2023-11-28T12:58:59.669188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:06:39.770893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058517+0000","last_change":"2023-11-28T12:58:59.676382+0000","last_active":"2023-11-28T12:59:00.058517+0000","last_peered":"2023-11-28T12:59:00.058517+0000","last_clean":"2023-11-28T12:59:00.058517+0000","last_became_active":"2023-11-28T12:58:59.675939+0000","last_became_peered":"2023-11-28T12:58:59.675939+0000","last_unstale":"2023-11-28T12:59:00.058517+0000","last_undegraded":"2023-11-28T12:59:00.058517+0000","last_fullsized":"2023-11-28T12:59:00.058517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:50:50.704627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.656336+0000","last_change":"2023-11-28T12:58:55.250390+0000","last_active":"2023-11-28T12:58:59.656336+0000","last_peered":"2023-11-28T12:58:59.656336+0000","last_clean":"2023-11-28T12:58:59.656336+0000","last_became_active":"2023-11-28T12:58:55.249919+0000","last_became_peered":"2023-11-28T12:58:55.249919+0000","last_unstale":"2023-11-28T12:58:59.656336+0000","last_undegraded":"2023-11-28T12:58:59.656336+0000","last_fullsized":"2023-11-28T12:58:59.656336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:11.131301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665787+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T12:58:59.665787+0000","last_peered":"2023-11-28T12:58:59.665787+0000","last_clean":"2023-11-28T12:58:59.665787+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T12:58:59.665787+0000","last_undegraded":"2023-11-28T12:58:59.665787+0000","last_fullsized":"2023-11-28T12:58:59.665787+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655406+0000","last_change":"2023-11-28T12:58:55.661761+0000","last_active":"2023-11-28T12:58:59.655406+0000","last_peered":"2023-11-28T12:58:59.655406+0000","last_clean":"2023-11-28T12:58:59.655406+0000","last_became_active":"2023-11-28T12:58:55.661488+0000","last_became_peered":"2023-11-28T12:58:55.661488+0000","last_unstale":"2023-11-28T12:58:59.655406+0000","last_undegraded":"2023-11-28T12:58:59.655406+0000","last_fullsized":"2023-11-28T12:58:59.655406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:14:28.821164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.059438+0000","last_change":"2023-11-28T12:58:59.678960+0000","last_active":"2023-11-28T12:59:00.059438+0000","last_peered":"2023-11-28T12:59:00.059438+0000","last_clean":"2023-11-28T12:59:00.059438+0000","last_became_active":"2023-11-28T12:58:59.678568+0000","last_became_peered":"2023-11-28T12:58:59.678568+0000","last_unstale":"2023-11-28T12:59:00.059438+0000","last_undegraded":"2023-11-28T12:59:00.059438+0000","last_fullsized":"2023-11-28T12:59:00.059438+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:17.338033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659020+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T12:58:59.659020+0000","last_peered":"2023-11-28T12:58:59.659020+0000","last_clean":"2023-11-28T12:58:59.659020+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T12:58:59.659020+0000","last_undegraded":"2023-11-28T12:58:59.659020+0000","last_fullsized":"2023-11-28T12:58:59.659020+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"26'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.313562+0000","last_change":"2023-11-28T12:58:55.251038+0000","last_active":"2023-11-28T12:59:00.313562+0000","last_peered":"2023-11-28T12:59:00.313562+0000","last_clean":"2023-11-28T12:59:00.313562+0000","last_became_active":"2023-11-28T12:58:55.250328+0000","last_became_peered":"2023-11-28T12:58:55.250328+0000","last_unstale":"2023-11-28T12:59:00.313562+0000","last_undegraded":"2023-11-28T12:59:00.313562+0000","last_fullsized":"2023-11-28T12:59:00.313562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:11.131301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659036+0000","last_change":"2023-11-28T12:58:55.658965+0000","last_active":"2023-11-28T12:58:59.659036+0000","last_peered":"2023-11-28T12:58:59.659036+0000","last_clean":"2023-11-28T12:58:59.659036+0000","last_became_active":"2023-11-28T12:58:55.658709+0000","last_became_peered":"2023-11-28T12:58:55.658709+0000","last_unstale":"2023-11-28T12:58:59.659036+0000","last_undegraded":"2023-11-28T12:58:59.659036+0000","last_fullsized":"2023-11-28T12:58:59.659036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:58:45.065981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.316209+0000","last_change":"2023-11-28T12:58:59.679267+0000","last_active":"2023-11-28T12:59:00.316209+0000","last_peered":"2023-11-28T12:59:00.316209+0000","last_clean":"2023-11-28T12:59:00.316209+0000","last_became_active":"2023-11-28T12:58:59.678822+0000","last_became_peered":"2023-11-28T12:58:59.678822+0000","last_unstale":"2023-11-28T12:59:00.316209+0000","last_undegraded":"2023-11-28T12:59:00.316209+0000","last_fullsized":"2023-11-28T12:59:00.316209+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:38:05.937526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.669102+0000","last_change":"2023-11-28T12:58:55.250742+0000","last_active":"2023-11-28T12:58:59.669102+0000","last_peered":"2023-11-28T12:58:59.669102+0000","last_clean":"2023-11-28T12:58:59.669102+0000","last_became_active":"2023-11-28T12:58:55.245578+0000","last_became_peered":"2023-11-28T12:58:55.245578+0000","last_unstale":"2023-11-28T12:58:59.669102+0000","last_undegraded":"2023-11-28T12:58:59.669102+0000","last_fullsized":"2023-11-28T12:58:59.669102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:25:52.356939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.672422+0000","last_change":"2023-11-28T12:58:55.647087+0000","last_active":"2023-11-28T12:58:59.672422+0000","last_peered":"2023-11-28T12:58:59.672422+0000","last_clean":"2023-11-28T12:58:59.672422+0000","last_became_active":"2023-11-28T12:58:55.646827+0000","last_became_peered":"2023-11-28T12:58:55.646827+0000","last_unstale":"2023-11-28T12:58:59.672422+0000","last_undegraded":"2023-11-28T12:58:59.672422+0000","last_fullsized":"2023-11-28T12:58:59.672422+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:43.305980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830493+0000","last_change":"2023-11-28T12:58:59.673358+0000","last_active":"2023-11-28T12:59:00.830493+0000","last_peered":"2023-11-28T12:59:00.830493+0000","last_clean":"2023-11-28T12:59:00.830493+0000","last_became_active":"2023-11-28T12:58:59.673010+0000","last_became_peered":"2023-11-28T12:58:59.673010+0000","last_unstale":"2023-11-28T12:59:00.830493+0000","last_undegraded":"2023-11-28T12:59:00.830493+0000","last_fullsized":"2023-11-28T12:59:00.830493+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:52:53.458898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665394+0000","last_change":"2023-11-28T12:58:55.242678+0000","last_active":"2023-11-28T12:58:59.665394+0000","last_peered":"2023-11-28T12:58:59.665394+0000","last_clean":"2023-11-28T12:58:59.665394+0000","last_became_active":"2023-11-28T12:58:55.242467+0000","last_became_peered":"2023-11-28T12:58:55.242467+0000","last_unstale":"2023-11-28T12:58:59.665394+0000","last_undegraded":"2023-11-28T12:58:59.665394+0000","last_fullsized":"2023-11-28T12:58:59.665394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:51:11.131301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659707+0000","last_change":"2023-11-28T12:58:55.664310+0000","last_active":"2023-11-28T12:58:59.659707+0000","last_peered":"2023-11-28T12:58:59.659707+0000","last_clean":"2023-11-28T12:58:59.659707+0000","last_became_active":"2023-11-28T12:58:55.664036+0000","last_became_peered":"2023-11-28T12:58:55.664036+0000","last_unstale":"2023-11-28T12:58:59.659707+0000","last_undegraded":"2023-11-28T12:58:59.659707+0000","last_fullsized":"2023-11-28T12:58:59.659707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:38:36.781235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.830051+0000","last_change":"2023-11-28T12:58:59.685443+0000","last_active":"2023-11-28T12:59:00.830051+0000","last_peered":"2023-11-28T12:59:00.830051+0000","last_clean":"2023-11-28T12:59:00.830051+0000","last_became_active":"2023-11-28T12:58:59.685042+0000","last_became_peered":"2023-11-28T12:58:59.685042+0000","last_unstale":"2023-11-28T12:59:00.830051+0000","last_undegraded":"2023-11-28T12:59:00.830051+0000","last_fullsized":"2023-11-28T12:59:00.830051+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:03:33.042937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.665495+0000","last_change":"2023-11-28T12:58:55.241170+0000","last_active":"2023-11-28T12:58:59.665495+0000","last_peered":"2023-11-28T12:58:59.665495+0000","last_clean":"2023-11-28T12:58:59.665495+0000","last_became_active":"2023-11-28T12:58:55.240880+0000","last_became_peered":"2023-11-28T12:58:55.240880+0000","last_unstale":"2023-11-28T12:58:59.665495+0000","last_undegraded":"2023-11-28T12:58:59.665495+0000","last_fullsized":"2023-11-28T12:58:59.665495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:42.300078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659284+0000","last_change":"2023-11-28T12:58:55.664483+0000","last_active":"2023-11-28T12:58:59.659284+0000","last_peered":"2023-11-28T12:58:59.659284+0000","last_clean":"2023-11-28T12:58:59.659284+0000","last_became_active":"2023-11-28T12:58:55.662151+0000","last_became_peered":"2023-11-28T12:58:55.662151+0000","last_unstale":"2023-11-28T12:58:59.659284+0000","last_undegraded":"2023-11-28T12:58:59.659284+0000","last_fullsized":"2023-11-28T12:58:59.659284+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:00:06.262366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.058024+0000","last_change":"2023-11-28T12:58:59.679533+0000","last_active":"2023-11-28T12:59:00.058024+0000","last_peered":"2023-11-28T12:59:00.058024+0000","last_clean":"2023-11-28T12:59:00.058024+0000","last_became_active":"2023-11-28T12:58:59.679174+0000","last_became_peered":"2023-11-28T12:58:59.679174+0000","last_unstale":"2023-11-28T12:59:00.058024+0000","last_undegraded":"2023-11-28T12:59:00.058024+0000","last_fullsized":"2023-11-28T12:59:00.058024+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:34:44.322352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655854+0000","last_change":"2023-11-28T12:58:55.246420+0000","last_active":"2023-11-28T12:58:59.655854+0000","last_peered":"2023-11-28T12:58:59.655854+0000","last_clean":"2023-11-28T12:58:59.655854+0000","last_became_active":"2023-11-28T12:58:55.246075+0000","last_became_peered":"2023-11-28T12:58:55.246075+0000","last_unstale":"2023-11-28T12:58:59.655854+0000","last_undegraded":"2023-11-28T12:58:59.655854+0000","last_fullsized":"2023-11-28T12:58:59.655854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:38:45.929395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655474+0000","last_change":"2023-11-28T12:58:55.660837+0000","last_active":"2023-11-28T12:58:59.655474+0000","last_peered":"2023-11-28T12:58:59.655474+0000","last_clean":"2023-11-28T12:58:59.655474+0000","last_became_active":"2023-11-28T12:58:55.657932+0000","last_became_peered":"2023-11-28T12:58:55.657932+0000","last_unstale":"2023-11-28T12:58:59.655474+0000","last_undegraded":"2023-11-28T12:58:59.655474+0000","last_fullsized":"2023-11-28T12:58:59.655474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:07:50.343956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-11-28T12:59:00.315551+0000","last_change":"2023-11-28T12:58:59.683087+0000","last_active":"2023-11-28T12:59:00.315551+0000","last_peered":"2023-11-28T12:59:00.315551+0000","last_clean":"2023-11-28T12:59:00.315551+0000","last_became_active":"2023-11-28T12:58:59.682678+0000","last_became_peered":"2023-11-28T12:58:59.682678+0000","last_unstale":"2023-11-28T12:59:00.315551+0000","last_undegraded":"2023-11-28T12:59:00.315551+0000","last_fullsized":"2023-11-28T12:59:00.315551+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:58.646623+0000","last_clean_scrub_stamp":"2023-11-28T12:58:58.646623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:07:52.771805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"21'4","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.674325+0000","last_change":"2023-11-28T12:58:55.246094+0000","last_active":"2023-11-28T12:58:59.674325+0000","last_peered":"2023-11-28T12:58:59.674325+0000","last_clean":"2023-11-28T12:58:59.674325+0000","last_became_active":"2023-11-28T12:58:55.245800+0000","last_became_peered":"2023-11-28T12:58:55.245800+0000","last_unstale":"2023-11-28T12:58:59.674325+0000","last_undegraded":"2023-11-28T12:58:59.674325+0000","last_fullsized":"2023-11-28T12:58:59.674325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:54.214141+0000","last_clean_scrub_stamp":"2023-11-28T12:58:54.214141+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:49.233234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.658861+0000","last_change":"2023-11-28T12:58:55.660995+0000","last_active":"2023-11-28T12:58:59.658861+0000","last_peered":"2023-11-28T12:58:59.658861+0000","last_clean":"2023-11-28T12:58:59.658861+0000","last_became_active":"2023-11-28T12:58:55.660761+0000","last_became_peered":"2023-11-28T12:58:55.660761+0000","last_unstale":"2023-11-28T12:58:59.658861+0000","last_undegraded":"2023-11-28T12:58:59.658861+0000","last_fullsized":"2023-11-28T12:58:59.658861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:54:59.199525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.657014+0000","last_change":"2023-11-28T12:58:55.650780+0000","last_active":"2023-11-28T12:58:59.657014+0000","last_peered":"2023-11-28T12:58:59.657014+0000","last_clean":"2023-11-28T12:58:59.657014+0000","last_became_active":"2023-11-28T12:58:55.650511+0000","last_became_peered":"2023-11-28T12:58:55.650511+0000","last_unstale":"2023-11-28T12:58:59.657014+0000","last_undegraded":"2023-11-28T12:58:59.657014+0000","last_fullsized":"2023-11-28T12:58:59.657014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:43.448645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655863+0000","last_change":"2023-11-28T12:58:55.663313+0000","last_active":"2023-11-28T12:58:59.655863+0000","last_peered":"2023-11-28T12:58:59.655863+0000","last_clean":"2023-11-28T12:58:59.655863+0000","last_became_active":"2023-11-28T12:58:55.663036+0000","last_became_peered":"2023-11-28T12:58:55.663036+0000","last_unstale":"2023-11-28T12:58:59.655863+0000","last_undegraded":"2023-11-28T12:58:59.655863+0000","last_fullsized":"2023-11-28T12:58:59.655863+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:21:13.458963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.659437+0000","last_change":"2023-11-28T12:58:55.661206+0000","last_active":"2023-11-28T12:58:59.659437+0000","last_peered":"2023-11-28T12:58:59.659437+0000","last_clean":"2023-11-28T12:58:59.659437+0000","last_became_active":"2023-11-28T12:58:55.660952+0000","last_became_peered":"2023-11-28T12:58:55.660952+0000","last_unstale":"2023-11-28T12:58:59.659437+0000","last_undegraded":"2023-11-28T12:58:59.659437+0000","last_fullsized":"2023-11-28T12:58:59.659437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:34.592151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.655629+0000","last_change":"2023-11-28T12:58:55.661675+0000","last_active":"2023-11-28T12:58:59.655629+0000","last_peered":"2023-11-28T12:58:59.655629+0000","last_clean":"2023-11-28T12:58:59.655629+0000","last_became_active":"2023-11-28T12:58:55.661418+0000","last_became_peered":"2023-11-28T12:58:55.661418+0000","last_unstale":"2023-11-28T12:58:59.655629+0000","last_undegraded":"2023-11-28T12:58:59.655629+0000","last_fullsized":"2023-11-28T12:58:59.655629+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:24:48.913065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-11-28T12:58:59.664699+0000","last_change":"2023-11-28T12:58:55.651538+0000","last_active":"2023-11-28T12:58:59.664699+0000","last_peered":"2023-11-28T12:58:59.664699+0000","last_clean":"2023-11-28T12:58:59.664699+0000","last_became_active":"2023-11-28T12:58:55.651297+0000","last_became_peered":"2023-11-28T12:58:55.651297+0000","last_unstale":"2023-11-28T12:58:59.664699+0000","last_undegraded":"2023-11-28T12:58:59.664699+0000","last_fullsized":"2023-11-28T12:58:59.664699+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:55.220043+0000","last_clean_scrub_stamp":"2023-11-28T12:58:55.220043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:56:55.140717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4528,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9272,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362568,"statfs":{"total":96636764160,"available":96627269632,"internally_reserved":0,"allocated":901120,"data_stored":709174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8088,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363752,"statfs":{"total":96636764160,"available":96628482048,"internally_reserved":0,"allocated":278528,"data_stored":114844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8152,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363688,"statfs":{"total":96636764160,"available":96628416512,"internally_reserved":0,"allocated":278528,"data_stored":114844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9248,"kb_used_data":856,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362592,"statfs":{"total":96636764160,"available":96627294208,"internally_reserved":0,"allocated":876544,"data_stored":705778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8088,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363752,"statfs":{"total":96636764160,"available":96628482048,"internally_reserved":0,"allocated":278528,"data_stored":114844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8156,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363684,"statfs":{"total":96636764160,"available":96628412416,"internally_reserved":0,"allocated":282624,"data_stored":114934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8236,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363604,"statfs":{"total":96636764160,"available":96628330496,"internally_reserved":0,"allocated":299008,"data_stored":118646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8160,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363680,"statfs":{"total":96636764160,"available":96628408320,"internally_reserved":0,"allocated":286720,"data_stored":115480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T12:59:23.726 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-28T12:59:23.726 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-28T12:59:23.726 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-28T12:59:23.726 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T12:59:23.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.879+0000 7fb71ffcc700 1 Processor -- start 2023-11-28T12:59:23.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.879+0000 7fb71ffcc700 1 -- start start 2023-11-28T12:59:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.880+0000 7fb71ffcc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7180cdd90 0x7fb7180b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.880+0000 7fb71ffcc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb7180aecc0 0x7fb7180ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.880+0000 7fb71ffcc700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7180ad420 con 0x7fb718159960 2023-11-28T12:59:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.880+0000 7fb71ffcc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb718159d30 con 0x7fb7180aecc0 2023-11-28T12:59:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.880+0000 7fb71ffcc700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7180ac810 con 0x7fb7180cdd90 2023-11-28T12:59:23.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.880+0000 7fb71e569700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb7180aecc0 0x7fb7180ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.880+0000 7fb71e569700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb7180aecc0 0x7fb7180ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33592/0 (socket says 172.21.15.111:33592) 2023-11-28T12:59:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.880+0000 7fb71e569700 1 -- 172.21.15.111:0/69675278 learned_addr learned my addr 172.21.15.111:0/69675278 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.880+0000 7fb71d567700 1 --2- 172.21.15.111:0/69675278 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7180cdd90 0x7fb7180b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71e569700 1 -- 172.21.15.111:0/69675278 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7180cdd90 msgr2=0x7fb7180b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71e569700 1 --2- 172.21.15.111:0/69675278 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7180cdd90 0x7fb7180b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71e569700 1 -- 172.21.15.111:0/69675278 >> v1:172.21.15.149:6789/0 conn(0x7fb718159960 legacy=0x7fb7180cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71e569700 1 -- 172.21.15.111:0/69675278 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7180acaf0 con 0x7fb7180aecc0 2023-11-28T12:59:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71e569700 1 --2- 172.21.15.111:0/69675278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7180aecc0 0x7fb7180ad8b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb71000d150 tx=0x7fb71000d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b619aa1fc594f38 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71cd66700 1 -- 172.21.15.111:0/69675278 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb710010070 con 0x7fb7180aecc0 2023-11-28T12:59:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71cd66700 1 -- 172.21.15.111:0/69675278 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71000ddc0 con 0x7fb7180aecc0 2023-11-28T12:59:23.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71cd66700 1 -- 172.21.15.111:0/69675278 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71000b310 con 0x7fb7180aecc0 2023-11-28T12:59:23.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71ffcc700 1 -- 172.21.15.111:0/69675278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7180aecc0 msgr2=0x7fb7180ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71ffcc700 1 --2- 172.21.15.111:0/69675278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7180aecc0 0x7fb7180ad8b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb71000d150 tx=0x7fb71000d460 comp rx=0 tx=0).stop 2023-11-28T12:59:23.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71ffcc700 1 -- 172.21.15.111:0/69675278 shutdown_connections 2023-11-28T12:59:23.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71ffcc700 1 --2- 172.21.15.111:0/69675278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7180aecc0 0x7fb7180ad8b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71ffcc700 1 --2- 172.21.15.111:0/69675278 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7180cdd90 0x7fb7180b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.881+0000 7fb71ffcc700 1 -- 172.21.15.111:0/69675278 >> 172.21.15.111:0/69675278 conn(0x7fb718098ee0 msgr2=0x7fb71806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:23.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.882+0000 7fb71ffcc700 1 -- 172.21.15.111:0/69675278 shutdown_connections 2023-11-28T12:59:23.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.882+0000 7fb71ffcc700 1 -- 172.21.15.111:0/69675278 wait complete. 2023-11-28T12:59:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.883+0000 7fb71ffcc700 1 Processor -- start 2023-11-28T12:59:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.883+0000 7fb71ffcc700 1 -- start start 2023-11-28T12:59:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.883+0000 7fb71ffcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7180aecc0 0x7fb7180b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.883+0000 7fb71ffcc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7180cdd90 0x7fb718154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.883+0000 7fb71ffcc700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb718159960 0x7fb718158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.883+0000 7fb71ffcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb718157330 con 0x7fb7180aecc0 2023-11-28T12:59:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.883+0000 7fb71ffcc700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7181f1940 con 0x7fb718159960 2023-11-28T12:59:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.883+0000 7fb71ffcc700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7181f1c20 con 0x7fb7180cdd90 2023-11-28T12:59:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.883+0000 7fb71dd68700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7180aecc0 0x7fb7180b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb71dd68700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7180aecc0 0x7fb7180b2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33594/0 (socket says 172.21.15.111:33594) 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb71dd68700 1 -- 172.21.15.111:0/652285278 learned_addr learned my addr 172.21.15.111:0/652285278 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb71dd68700 1 -- 172.21.15.111:0/652285278 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7180cdd90 msgr2=0x7fb718154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb71e569700 1 --2- 172.21.15.111:0/652285278 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb718159960 0x7fb718158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb71dd68700 1 --2- 172.21.15.111:0/652285278 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7180cdd90 0x7fb718154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb71dd68700 1 -- 172.21.15.111:0/652285278 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb718159960 msgr2=0x7fb718158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb71dd68700 1 --2- 172.21.15.111:0/652285278 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb718159960 0x7fb718158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb71dd68700 1 -- 172.21.15.111:0/652285278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71000bd40 con 0x7fb7180aecc0 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb71dd68700 1 --2- 172.21.15.111:0/652285278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7180aecc0 0x7fb7180b2970 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb70800c4d0 tx=0x7fb70800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb70e7fc700 1 -- 172.21.15.111:0/652285278 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7080052d0 con 0x7fb7180aecc0 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb70e7fc700 1 -- 172.21.15.111:0/652285278 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb708005430 con 0x7fb7180aecc0 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb71ffcc700 1 -- 172.21.15.111:0/652285278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7180ad0b0 con 0x7fb7180aecc0 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb70e7fc700 1 -- 172.21.15.111:0/652285278 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7080038f0 con 0x7fb7180aecc0 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.884+0000 7fb71ffcc700 1 -- 172.21.15.111:0/652285278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7180acdd0 con 0x7fb7180aecc0 2023-11-28T12:59:23.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.885+0000 7fb71ffcc700 1 -- 172.21.15.111:0/652285278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7180addf0 con 0x7fb7180aecc0 2023-11-28T12:59:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.885+0000 7fb70e7fc700 1 -- 172.21.15.111:0/652285278 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb708019030 con 0x7fb7180aecc0 2023-11-28T12:59:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.886+0000 7fb70e7fc700 1 --2- 172.21.15.111:0/652285278 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb6fc068320 0x7fb6fc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.886+0000 7fb70e7fc700 1 -- 172.21.15.111:0/652285278 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb7080852e0 con 0x7fb7180aecc0 2023-11-28T12:59:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.886+0000 7fb71d567700 1 --2- 172.21.15.111:0/652285278 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb6fc068320 0x7fb6fc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:23.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.886+0000 7fb71d567700 1 --2- 172.21.15.111:0/652285278 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb6fc068320 0x7fb6fc06a7d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb718154690 tx=0x7fb714008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:23.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:23.891+0000 7fb70e7fc700 1 -- 172.21.15.111:0/652285278 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb708051850 con 0x7fb7180aecc0 2023-11-28T12:59:24.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.111+0000 7fb71ffcc700 1 -- 172.21.15.111:0/652285278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb718067170 con 0x7fb7180aecc0 2023-11-28T12:59:24.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.112+0000 7fb70e7fc700 1 -- 172.21.15.111:0/652285278 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fb708054e70 con 0x7fb7180aecc0 2023-11-28T12:59:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:24.114 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-28T12:59:24.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.114+0000 7fb71ffcc700 1 -- 172.21.15.111:0/652285278 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb6fc068320 msgr2=0x7fb6fc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.114+0000 7fb71ffcc700 1 --2- 172.21.15.111:0/652285278 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb6fc068320 0x7fb6fc06a7d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb718154690 tx=0x7fb714008040 comp rx=0 tx=0).stop 2023-11-28T12:59:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.114+0000 7fb71ffcc700 1 -- 172.21.15.111:0/652285278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7180aecc0 msgr2=0x7fb7180b2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.114+0000 7fb71ffcc700 1 --2- 172.21.15.111:0/652285278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7180aecc0 0x7fb7180b2970 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb70800c4d0 tx=0x7fb70800c890 comp rx=0 tx=0).stop 2023-11-28T12:59:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.114+0000 7fb71ffcc700 1 -- 172.21.15.111:0/652285278 shutdown_connections 2023-11-28T12:59:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.114+0000 7fb71ffcc700 1 --2- 172.21.15.111:0/652285278 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb6fc068320 0x7fb6fc06a7d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.114+0000 7fb71ffcc700 1 --2- 172.21.15.111:0/652285278 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb718159960 0x7fb718158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.114+0000 7fb71ffcc700 1 --2- 172.21.15.111:0/652285278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7180aecc0 0x7fb7180b2970 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.114+0000 7fb71ffcc700 1 --2- 172.21.15.111:0/652285278 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7180cdd90 0x7fb718154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.114+0000 7fb71ffcc700 1 -- 172.21.15.111:0/652285278 >> 172.21.15.111:0/652285278 conn(0x7fb718098ee0 msgr2=0x7fb718072cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.114+0000 7fb71ffcc700 1 -- 172.21.15.111:0/652285278 shutdown_connections 2023-11-28T12:59:24.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.114+0000 7fb71ffcc700 1 -- 172.21.15.111:0/652285278 wait complete. 2023-11-28T12:59:24.127 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-28T12:59:24.128 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:24.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.245+0000 7f6f30c6a700 1 Processor -- start 2023-11-28T12:59:24.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.245+0000 7f6f30c6a700 1 -- start start 2023-11-28T12:59:24.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.246+0000 7f6f30c6a700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f6f2c06b7f0 0x7f6f2c0b5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.246+0000 7f6f30c6a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2c0b5300 con 0x7f6f2c06b380 2023-11-28T12:59:24.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.246+0000 7f6f30c6a700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2c0cfbb0 con 0x7f6f2c0b20f0 2023-11-28T12:59:24.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.246+0000 7f6f30c6a700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f2c0b46f0 con 0x7f6f2c06b7f0 2023-11-28T12:59:24.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.246+0000 7f6f29d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6f2c06b380 0x7f6f2c0b5790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34382/0 (socket says 172.21.15.111:34382) 2023-11-28T12:59:24.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.246+0000 7f6f29d9b700 1 -- 172.21.15.111:0/1200170024 learned_addr learned my addr 172.21.15.111:0/1200170024 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:24.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.246+0000 7f6f2ad9d700 1 --2- 172.21.15.111:0/1200170024 >> v2:172.21.15.149:3300/0 conn(0x7f6f2c06b7f0 0x7f6f2c0b5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883799671 0 0) 0x7f6f2c0b5300 con 0x7f6f2c06b380 2023-11-28T12:59:24.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f140035b0 con 0x7f6f2c06b380 2023-11-28T12:59:24.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1659798148 0 0) 0x7f6f140035b0 con 0x7f6f2c06b380 2023-11-28T12:59:24.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 >> v2:172.21.15.149:3300/0 conn(0x7f6f2c06b7f0 msgr2=0x7f6f2c0b5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:24.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 --2- 172.21.15.111:0/1200170024 >> v2:172.21.15.149:3300/0 conn(0x7f6f2c06b7f0 0x7f6f2c0b5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 >> v1:172.21.15.149:6789/0 conn(0x7f6f2c0b20f0 legacy=0x7f6f2c0b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c0b49d0 con 0x7f6f2c06b380 2023-11-28T12:59:24.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f6f1c001670 con 0x7f6f2c06b380 2023-11-28T12:59:24.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 >> v1:172.21.15.111:6789/0 conn(0x7f6f2c06b380 legacy=0x7f6f2c0b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 --2- 172.21.15.111:0/1200170024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f14003a00 0x7f6f14003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 --2- 172.21.15.111:0/1200170024 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6f14004850 0x7f6f14004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 --2- 172.21.15.111:0/1200170024 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6f14005200 0x7f6f1400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c0b46f0 con 0x7f6f14003a00 2023-11-28T12:59:24.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1400c080 con 0x7f6f14004850 2023-11-28T12:59:24.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1400c240 con 0x7f6f14005200 2023-11-28T12:59:24.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2a59c700 1 --2- 172.21.15.111:0/1200170024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f14003a00 0x7f6f14003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f29d9b700 1 --2- 172.21.15.111:0/1200170024 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6f14004850 0x7f6f14004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2ad9d700 1 --2- 172.21.15.111:0/1200170024 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6f14005200 0x7f6f1400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2a59c700 1 -- 172.21.15.111:0/1200170024 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6f14005200 msgr2=0x7f6f1400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:24.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2a59c700 1 --2- 172.21.15.111:0/1200170024 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6f14005200 0x7f6f1400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2a59c700 1 -- 172.21.15.111:0/1200170024 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6f14004850 msgr2=0x7f6f14004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:24.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2a59c700 1 --2- 172.21.15.111:0/1200170024 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6f14004850 0x7f6f14004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2a59c700 1 -- 172.21.15.111:0/1200170024 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f140035b0 con 0x7f6f14003a00 2023-11-28T12:59:24.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2ad9d700 1 --2- 172.21.15.111:0/1200170024 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6f14005200 0x7f6f1400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:24.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2ad9d700 1 -- 172.21.15.111:0/1200170024 reap_dead start 2023-11-28T12:59:24.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2a59c700 1 --2- 172.21.15.111:0/1200170024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f14003a00 0x7f6f14003e10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6f200017c0 tx=0x7f6f14005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f79dbf4f4c855c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:24.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f200052d0 con 0x7f6f14003a00 2023-11-28T12:59:24.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.247+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f20005430 con 0x7f6f14003a00 2023-11-28T12:59:24.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.248+0000 7f6f2959a700 1 -- 172.21.15.111:0/1200170024 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f200038c0 con 0x7f6f14003a00 2023-11-28T12:59:24.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.248+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1200170024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f14003a00 msgr2=0x7f6f14003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.248+0000 7f6f30c6a700 1 --2- 172.21.15.111:0/1200170024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f14003a00 0x7f6f14003e10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6f200017c0 tx=0x7f6f14005880 comp rx=0 tx=0).stop 2023-11-28T12:59:24.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.248+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1200170024 shutdown_connections 2023-11-28T12:59:24.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.248+0000 7f6f30c6a700 1 --2- 172.21.15.111:0/1200170024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f14003a00 0x7f6f14003e10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.248+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1200170024 >> 172.21.15.111:0/1200170024 conn(0x7f6f2c09a040 msgr2=0x7f6f2c0d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:24.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.248+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1200170024 shutdown_connections 2023-11-28T12:59:24.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.248+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1200170024 wait complete. 2023-11-28T12:59:24.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.250+0000 7f6f30c6a700 1 Processor -- start 2023-11-28T12:59:24.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.250+0000 7f6f30c6a700 1 -- start start 2023-11-28T12:59:24.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f30c6a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f14003a00 0x7f6f2c14d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f30c6a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6f2c0ad980 0x7f6f2c0b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f30c6a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6f2c0b0700 0x7f6f2c0d0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f30c6a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c0b17b0 con 0x7f6f14003a00 2023-11-28T12:59:24.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f30c6a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c0b0c30 con 0x7f6f2c0b0700 2023-11-28T12:59:24.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f30c6a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c0b0f10 con 0x7f6f2c0ad980 2023-11-28T12:59:24.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f2a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f14003a00 0x7f6f2c14d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f2a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f14003a00 0x7f6f2c14d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33614/0 (socket says 172.21.15.111:33614) 2023-11-28T12:59:24.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f2a59c700 1 -- 172.21.15.111:0/1922672507 learned_addr learned my addr 172.21.15.111:0/1922672507 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:24.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f2a59c700 1 -- 172.21.15.111:0/1922672507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6f2c0ad980 msgr2=0x7f6f2c0b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f2a59c700 1 --2- 172.21.15.111:0/1922672507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6f2c0ad980 0x7f6f2c0b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f2a59c700 1 -- 172.21.15.111:0/1922672507 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6f2c0b0700 msgr2=0x7f6f2c0d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f2a59c700 1 --2- 172.21.15.111:0/1922672507 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6f2c0b0700 0x7f6f2c0d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f2a59c700 1 -- 172.21.15.111:0/1922672507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c0b4f90 con 0x7f6f14003a00 2023-11-28T12:59:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f2a59c700 1 --2- 172.21.15.111:0/1922672507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f14003a00 0x7f6f2c14d060 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6f20005040 tx=0x7f6f20001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f1affd700 1 -- 172.21.15.111:0/1922672507 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20003e70 con 0x7f6f14003a00 2023-11-28T12:59:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f1affd700 1 -- 172.21.15.111:0/1922672507 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f20001290 con 0x7f6f14003a00 2023-11-28T12:59:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.251+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1922672507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f2c0b4cb0 con 0x7f6f14003a00 2023-11-28T12:59:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.252+0000 7f6f1affd700 1 -- 172.21.15.111:0/1922672507 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2001d6c0 con 0x7f6f14003a00 2023-11-28T12:59:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.252+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1922672507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f2c06bc70 con 0x7f6f14003a00 2023-11-28T12:59:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.252+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1922672507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f2c067170 con 0x7f6f14003a00 2023-11-28T12:59:24.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.253+0000 7f6f1affd700 1 -- 172.21.15.111:0/1922672507 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6f20025030 con 0x7f6f14003a00 2023-11-28T12:59:24.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.253+0000 7f6f1affd700 1 --2- 172.21.15.111:0/1922672507 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6f0c0682f0 0x7f6f0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.253+0000 7f6f1affd700 1 -- 172.21.15.111:0/1922672507 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6f20086f90 con 0x7f6f14003a00 2023-11-28T12:59:24.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.253+0000 7f6f29d9b700 1 --2- 172.21.15.111:0/1922672507 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6f0c0682f0 0x7f6f0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.254+0000 7f6f29d9b700 1 --2- 172.21.15.111:0/1922672507 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6f0c0682f0 0x7f6f0c06a7a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c001b40 tx=0x7f6f1c001ab0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:24.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.258+0000 7f6f1affd700 1 -- 172.21.15.111:0/1922672507 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6f20053500 con 0x7f6f14003a00 2023-11-28T12:59:24.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.441+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1922672507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f2c09ce20 con 0x7f6f14003a00 2023-11-28T12:59:24.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.442+0000 7f6f1affd700 1 -- 172.21.15.111:0/1922672507 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19796 (secure 0 0 0) 0x7f6f20017020 con 0x7f6f14003a00 2023-11-28T12:59:24.444 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:24.444 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:00.053943+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:24.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.444+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1922672507 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6f0c0682f0 msgr2=0x7f6f0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.444+0000 7f6f30c6a700 1 --2- 172.21.15.111:0/1922672507 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6f0c0682f0 0x7f6f0c06a7a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c001b40 tx=0x7f6f1c001ab0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.444+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1922672507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f14003a00 msgr2=0x7f6f2c14d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.444+0000 7f6f30c6a700 1 --2- 172.21.15.111:0/1922672507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f14003a00 0x7f6f2c14d060 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6f20005040 tx=0x7f6f20001120 comp rx=0 tx=0).stop 2023-11-28T12:59:24.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.444+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1922672507 shutdown_connections 2023-11-28T12:59:24.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.444+0000 7f6f30c6a700 1 --2- 172.21.15.111:0/1922672507 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6f0c0682f0 0x7f6f0c06a7a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.444+0000 7f6f30c6a700 1 --2- 172.21.15.111:0/1922672507 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6f2c0b0700 0x7f6f2c0d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.444+0000 7f6f30c6a700 1 --2- 172.21.15.111:0/1922672507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f14003a00 0x7f6f2c14d060 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.444+0000 7f6f30c6a700 1 --2- 172.21.15.111:0/1922672507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6f2c0ad980 0x7f6f2c0b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.444+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1922672507 >> 172.21.15.111:0/1922672507 conn(0x7f6f2c09a040 msgr2=0x7f6f2c0ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:24.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.444+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1922672507 shutdown_connections 2023-11-28T12:59:24.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.444+0000 7f6f30c6a700 1 -- 172.21.15.111:0/1922672507 wait complete. 2023-11-28T12:59:24.458 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T12:58:53.275206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T12:58:54.588499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T12:58:57.666033+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T12:59:24.459 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:59:24.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.581+0000 7fd7451e4700 1 Processor -- start 2023-11-28T12:59:24.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.581+0000 7fd7451e4700 1 -- start start 2023-11-28T12:59:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd7451e4700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd7400b0d70 0x7fd7401604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd7451e4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400b1140 0x7fd7400ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd7451e4700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd74015cd90 con 0x7fd7400b15b0 2023-11-28T12:59:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd7451e4700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7401613e0 con 0x7fd7400b0d70 2023-11-28T12:59:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd7451e4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd740160b40 con 0x7fd7400b1140 2023-11-28T12:59:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73f59e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd7400b15b0 0x7fd7400cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34404/0 (socket says 172.21.15.111:34404) 2023-11-28T12:59:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73f59e700 1 -- 172.21.15.111:0/1294889885 learned_addr learned my addr 172.21.15.111:0/1294889885 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125983504 0 0) 0x7fd738001490 con 0x7fd7400b15b0 2023-11-28T12:59:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73e59c700 1 --2- 172.21.15.111:0/1294889885 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400b1140 0x7fd7400ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73ed9d700 1 --2- 172.21.15.111:0/1294889885 >> v2:172.21.15.149:3300/0 conn(0x7fd7400b0d70 0x7fd7401604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7280035b0 con 0x7fd7400b15b0 2023-11-28T12:59:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3282737572 0 0) 0x7fd7280035b0 con 0x7fd7400b15b0 2023-11-28T12:59:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400b1140 msgr2=0x7fd7400ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 --2- 172.21.15.111:0/1294889885 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400b1140 0x7fd7400ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 >> v2:172.21.15.149:3300/0 conn(0x7fd7400b0d70 msgr2=0x7fd7401604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 --2- 172.21.15.111:0/1294889885 >> v2:172.21.15.149:3300/0 conn(0x7fd7400b0d70 0x7fd7401604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd740160e20 con 0x7fd7400b15b0 2023-11-28T12:59:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fd738001b80 con 0x7fd7400b15b0 2023-11-28T12:59:24.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 >> v1:172.21.15.111:6789/0 conn(0x7fd7400b15b0 legacy=0x7fd7400cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 --2- 172.21.15.111:0/1294889885 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400b1140 0x7fd728003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 --2- 172.21.15.111:0/1294889885 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd728003f80 0x7fd728004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 --2- 172.21.15.111:0/1294889885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd728004d90 0x7fd728005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7401613e0 con 0x7fd728004d90 2023-11-28T12:59:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd740160b40 con 0x7fd728003f80 2023-11-28T12:59:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72800ba10 con 0x7fd7400b1140 2023-11-28T12:59:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73f59e700 1 --2- 172.21.15.111:0/1294889885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd728004d90 0x7fd728005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73ed9d700 1 --2- 172.21.15.111:0/1294889885 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400b1140 0x7fd728003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.582+0000 7fd73e59c700 1 --2- 172.21.15.111:0/1294889885 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd728003f80 0x7fd728004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.583+0000 7fd73f59e700 1 -- 172.21.15.111:0/1294889885 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400b1140 msgr2=0x7fd728003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.583+0000 7fd73f59e700 1 --2- 172.21.15.111:0/1294889885 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400b1140 0x7fd728003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.583+0000 7fd73f59e700 1 -- 172.21.15.111:0/1294889885 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd728003f80 msgr2=0x7fd728004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.583+0000 7fd73f59e700 1 --2- 172.21.15.111:0/1294889885 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd728003f80 0x7fd728004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.583+0000 7fd73f59e700 1 -- 172.21.15.111:0/1294889885 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7280035b0 con 0x7fd728004d90 2023-11-28T12:59:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.583+0000 7fd73f59e700 1 --2- 172.21.15.111:0/1294889885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd728004d90 0x7fd728005200 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd740070330 tx=0x7fd728005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618b49f1f91ba09b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.583+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73800a4c0 con 0x7fd728004d90 2023-11-28T12:59:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.583+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd738012600 con 0x7fd728004d90 2023-11-28T12:59:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.583+0000 7fd73dd9b700 1 -- 172.21.15.111:0/1294889885 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7380129b0 con 0x7fd728004d90 2023-11-28T12:59:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.583+0000 7fd7451e4700 1 -- 172.21.15.111:0/1294889885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd728004d90 msgr2=0x7fd728005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.583+0000 7fd7451e4700 1 --2- 172.21.15.111:0/1294889885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd728004d90 0x7fd728005200 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd740070330 tx=0x7fd728005880 comp rx=0 tx=0).stop 2023-11-28T12:59:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.583+0000 7fd73f59e700 1 -- 172.21.15.111:0/1294889885 reap_dead start 2023-11-28T12:59:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.584+0000 7fd7451e4700 1 -- 172.21.15.111:0/1294889885 shutdown_connections 2023-11-28T12:59:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.584+0000 7fd7451e4700 1 -- 172.21.15.111:0/1294889885 >> 172.21.15.111:0/1294889885 conn(0x7fd740098ee0 msgr2=0x7fd74006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.584+0000 7fd7451e4700 1 -- 172.21.15.111:0/1294889885 shutdown_connections 2023-11-28T12:59:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.584+0000 7fd7451e4700 1 -- 172.21.15.111:0/1294889885 wait complete. 2023-11-28T12:59:24.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.585+0000 7fd7451e4700 1 Processor -- start 2023-11-28T12:59:24.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.585+0000 7fd7451e4700 1 -- start start 2023-11-28T12:59:24.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.586+0000 7fd7451e4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd740159410 0x7fd7401f2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.586+0000 7fd7451e4700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7400b2c10 0x7fd7400ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.586+0000 7fd7451e4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400adc00 0x7fd7401e6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.586+0000 7fd7451e4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7400ae0a0 con 0x7fd740159410 2023-11-28T12:59:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.586+0000 7fd7451e4700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7401e7360 con 0x7fd7400b2c10 2023-11-28T12:59:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.586+0000 7fd7451e4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7400b3440 con 0x7fd7400adc00 2023-11-28T12:59:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.586+0000 7fd73ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd740159410 0x7fd7401f2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd73ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd740159410 0x7fd7401f2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33634/0 (socket says 172.21.15.111:33634) 2023-11-28T12:59:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd73ed9d700 1 -- 172.21.15.111:0/2406417161 learned_addr learned my addr 172.21.15.111:0/2406417161 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd73f59e700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400adc00 0x7fd7401e6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd73e59c700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7400b2c10 0x7fd7400ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd73ed9d700 1 -- 172.21.15.111:0/2406417161 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400adc00 msgr2=0x7fd7401e6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd73ed9d700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400adc00 0x7fd7401e6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd73ed9d700 1 -- 172.21.15.111:0/2406417161 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7400b2c10 msgr2=0x7fd7400ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd73ed9d700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7400b2c10 0x7fd7400ad6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd73ed9d700 1 -- 172.21.15.111:0/2406417161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7400af340 con 0x7fd740159410 2023-11-28T12:59:24.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd73f59e700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400adc00 0x7fd7401e6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:24.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd73ed9d700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd740159410 0x7fd7401f2e80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd730011a70 tx=0x7fd730011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:24.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd7277fe700 1 -- 172.21.15.111:0/2406417161 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730005a20 con 0x7fd740159410 2023-11-28T12:59:24.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd7277fe700 1 -- 172.21.15.111:0/2406417161 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd730005b80 con 0x7fd740159410 2023-11-28T12:59:24.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd7277fe700 1 -- 172.21.15.111:0/2406417161 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73000f9a0 con 0x7fd740159410 2023-11-28T12:59:24.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd7451e4700 1 -- 172.21.15.111:0/2406417161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd740161100 con 0x7fd740159410 2023-11-28T12:59:24.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.587+0000 7fd7451e4700 1 -- 172.21.15.111:0/2406417161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7400ccc60 con 0x7fd740159410 2023-11-28T12:59:24.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.588+0000 7fd7451e4700 1 -- 172.21.15.111:0/2406417161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd740067170 con 0x7fd740159410 2023-11-28T12:59:24.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.588+0000 7fd7277fe700 1 -- 172.21.15.111:0/2406417161 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd730024070 con 0x7fd740159410 2023-11-28T12:59:24.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.589+0000 7fd7277fe700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd72c0682f0 0x7fd72c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.589+0000 7fd73e59c700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd72c0682f0 0x7fd72c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.589+0000 7fd7277fe700 1 -- 172.21.15.111:0/2406417161 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd730086260 con 0x7fd740159410 2023-11-28T12:59:24.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.589+0000 7fd73e59c700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd72c0682f0 0x7fd72c06a7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd7400aedb0 tx=0x7fd734008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:24.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.593+0000 7fd7277fe700 1 -- 172.21.15.111:0/2406417161 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd7300527d0 con 0x7fd740159410 2023-11-28T12:59:24.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.773+0000 7fd7451e4700 1 -- 172.21.15.111:0/2406417161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd74006d9a0 con 0x7fd740159410 2023-11-28T12:59:24.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.774+0000 7fd7277fe700 1 -- 172.21.15.111:0/2406417161 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd7300220a0 con 0x7fd740159410 2023-11-28T12:59:24.774 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:59:24.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.776+0000 7fd7451e4700 1 -- 172.21.15.111:0/2406417161 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd72c0682f0 msgr2=0x7fd72c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.776+0000 7fd7451e4700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd72c0682f0 0x7fd72c06a7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd7400aedb0 tx=0x7fd734008040 comp rx=0 tx=0).stop 2023-11-28T12:59:24.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.776+0000 7fd7451e4700 1 -- 172.21.15.111:0/2406417161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd740159410 msgr2=0x7fd7401f2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.776+0000 7fd7451e4700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd740159410 0x7fd7401f2e80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd730011a70 tx=0x7fd730011e30 comp rx=0 tx=0).stop 2023-11-28T12:59:24.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.776+0000 7fd7451e4700 1 -- 172.21.15.111:0/2406417161 shutdown_connections 2023-11-28T12:59:24.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.776+0000 7fd7451e4700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7400adc00 0x7fd7401e6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.776+0000 7fd7451e4700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd740159410 0x7fd7401f2e80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.776+0000 7fd7451e4700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd72c0682f0 0x7fd72c06a7a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.776+0000 7fd7451e4700 1 --2- 172.21.15.111:0/2406417161 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7400b2c10 0x7fd7400ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.776+0000 7fd7451e4700 1 -- 172.21.15.111:0/2406417161 >> 172.21.15.111:0/2406417161 conn(0x7fd740098ee0 msgr2=0x7fd7400ca2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:24.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.776+0000 7fd7451e4700 1 -- 172.21.15.111:0/2406417161 shutdown_connections 2023-11-28T12:59:24.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.776+0000 7fd7451e4700 1 -- 172.21.15.111:0/2406417161 wait complete. 2023-11-28T12:59:24.788 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:59:24.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.911+0000 7fc7c94ec700 1 Processor -- start 2023-11-28T12:59:24.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.911+0000 7fc7c94ec700 1 -- start start 2023-11-28T12:59:24.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.912+0000 7fc7c94ec700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc7c415c7b0 0x7fc7c40cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.912+0000 7fc7c94ec700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c40d0490 0x7fc7c40ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.912+0000 7fc7c94ec700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c40cb810 con 0x7fc7c406b380 2023-11-28T12:59:24.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.912+0000 7fc7c94ec700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7c40cc650 con 0x7fc7c415c7b0 2023-11-28T12:59:24.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.912+0000 7fc7c94ec700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c40cba40 con 0x7fc7c40d0490 2023-11-28T12:59:24.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.912+0000 7fc7c37fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc7c406b380 0x7fc7c40ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34418/0 (socket says 172.21.15.111:34418) 2023-11-28T12:59:24.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.912+0000 7fc7c37fe700 1 -- 172.21.15.111:0/288608721 learned_addr learned my addr 172.21.15.111:0/288608721 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:24.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.912+0000 7fc7c2ffd700 1 --2- 172.21.15.111:0/288608721 >> v2:172.21.15.149:3300/0 conn(0x7fc7c415c7b0 0x7fc7c40cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c27fc700 1 --2- 172.21.15.111:0/288608721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c40d0490 0x7fc7c40ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.912+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201807276 0 0) 0x7fc7c40cb810 con 0x7fc7c406b380 2023-11-28T12:59:24.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7ac0035b0 con 0x7fc7c406b380 2023-11-28T12:59:24.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2003440320 0 0) 0x7fc7ac0035b0 con 0x7fc7c406b380 2023-11-28T12:59:24.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c40d0490 msgr2=0x7fc7c40ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:24.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 --2- 172.21.15.111:0/288608721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c40d0490 0x7fc7c40ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 >> v2:172.21.15.149:3300/0 conn(0x7fc7c415c7b0 msgr2=0x7fc7c40cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:24.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 --2- 172.21.15.111:0/288608721 >> v2:172.21.15.149:3300/0 conn(0x7fc7c415c7b0 0x7fc7c40cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c40cbd20 con 0x7fc7c406b380 2023-11-28T12:59:24.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fc7bc0019f0 con 0x7fc7c406b380 2023-11-28T12:59:24.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 >> v1:172.21.15.111:6789/0 conn(0x7fc7c406b380 legacy=0x7fc7c40ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 --2- 172.21.15.111:0/288608721 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7ac003a00 0x7fc7ac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 --2- 172.21.15.111:0/288608721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c40d0490 0x7fc7ac004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 --2- 172.21.15.111:0/288608721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7ac004d90 0x7fc7ac005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c40cc650 con 0x7fc7ac004d90 2023-11-28T12:59:24.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c40cba40 con 0x7fc7ac003a00 2023-11-28T12:59:24.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac00ba10 con 0x7fc7c40d0490 2023-11-28T12:59:24.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c37fe700 1 --2- 172.21.15.111:0/288608721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7ac004d90 0x7fc7ac005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c2ffd700 1 --2- 172.21.15.111:0/288608721 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7ac003a00 0x7fc7ac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c27fc700 1 --2- 172.21.15.111:0/288608721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c40d0490 0x7fc7ac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c37fe700 1 -- 172.21.15.111:0/288608721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c40d0490 msgr2=0x7fc7ac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:24.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c37fe700 1 --2- 172.21.15.111:0/288608721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c40d0490 0x7fc7ac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c37fe700 1 -- 172.21.15.111:0/288608721 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7ac003a00 msgr2=0x7fc7ac003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:24.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c37fe700 1 --2- 172.21.15.111:0/288608721 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7ac003a00 0x7fc7ac003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c37fe700 1 -- 172.21.15.111:0/288608721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac0035b0 con 0x7fc7ac004d90 2023-11-28T12:59:24.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.913+0000 7fc7c37fe700 1 --2- 172.21.15.111:0/288608721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7ac004d90 0x7fc7ac005200 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc00a830 tx=0x7fc7ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3e86b59503343 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:24.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.914+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc012600 con 0x7fc7ac004d90 2023-11-28T12:59:24.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.914+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7bc012760 con 0x7fc7ac004d90 2023-11-28T12:59:24.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.914+0000 7fc7c1ffb700 1 -- 172.21.15.111:0/288608721 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc012b10 con 0x7fc7ac004d90 2023-11-28T12:59:24.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.914+0000 7fc7c94ec700 1 -- 172.21.15.111:0/288608721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7ac004d90 msgr2=0x7fc7ac005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.914+0000 7fc7c94ec700 1 --2- 172.21.15.111:0/288608721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7ac004d90 0x7fc7ac005200 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc00a830 tx=0x7fc7ac005880 comp rx=0 tx=0).stop 2023-11-28T12:59:24.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.914+0000 7fc7c37fe700 1 -- 172.21.15.111:0/288608721 reap_dead start 2023-11-28T12:59:24.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.914+0000 7fc7c94ec700 1 -- 172.21.15.111:0/288608721 shutdown_connections 2023-11-28T12:59:24.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.914+0000 7fc7c94ec700 1 -- 172.21.15.111:0/288608721 >> 172.21.15.111:0/288608721 conn(0x7fc7c409a040 msgr2=0x7fc7c40d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:24.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.914+0000 7fc7c94ec700 1 -- 172.21.15.111:0/288608721 shutdown_connections 2023-11-28T12:59:24.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.914+0000 7fc7c94ec700 1 -- 172.21.15.111:0/288608721 wait complete. 2023-11-28T12:59:24.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.915+0000 7fc7c94ec700 1 Processor -- start 2023-11-28T12:59:24.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.916+0000 7fc7c94ec700 1 -- start start 2023-11-28T12:59:24.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.916+0000 7fc7c94ec700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c41f18e0 0x7fc7c415dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.916+0000 7fc7c94ec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c41f1cb0 0x7fc7c40b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.916+0000 7fc7c94ec700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7c415e250 0x7fc7c40b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.916+0000 7fc7c94ec700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c40b0670 con 0x7fc7c41f1cb0 2023-11-28T12:59:24.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.916+0000 7fc7c94ec700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c40b0950 con 0x7fc7c415e250 2023-11-28T12:59:24.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.916+0000 7fc7c94ec700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c40b0c30 con 0x7fc7c41f18e0 2023-11-28T12:59:24.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.916+0000 7fc7c27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c41f1cb0 0x7fc7c40b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.916+0000 7fc7c27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c41f1cb0 0x7fc7c40b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33650/0 (socket says 172.21.15.111:33650) 2023-11-28T12:59:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.916+0000 7fc7c27fc700 1 -- 172.21.15.111:0/2331887046 learned_addr learned my addr 172.21.15.111:0/2331887046 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.916+0000 7fc7c2ffd700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c41f18e0 0x7fc7c415dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.916+0000 7fc7c37fe700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7c415e250 0x7fc7c40b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.917+0000 7fc7c27fc700 1 -- 172.21.15.111:0/2331887046 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c41f18e0 msgr2=0x7fc7c415dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.917+0000 7fc7c27fc700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c41f18e0 0x7fc7c415dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.917+0000 7fc7c27fc700 1 -- 172.21.15.111:0/2331887046 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7c415e250 msgr2=0x7fc7c40b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.917+0000 7fc7c27fc700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7c415e250 0x7fc7c40b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.917+0000 7fc7c27fc700 1 -- 172.21.15.111:0/2331887046 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c40cc2e0 con 0x7fc7c41f1cb0 2023-11-28T12:59:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.917+0000 7fc7c37fe700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7c415e250 0x7fc7c40b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.917+0000 7fc7c2ffd700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c41f18e0 0x7fc7c415dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.917+0000 7fc7c27fc700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c41f1cb0 0x7fc7c40b4570 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4011aa0 tx=0x7fc7b4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.917+0000 7fc7ab7fe700 1 -- 172.21.15.111:0/2331887046 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b400f980 con 0x7fc7c41f1cb0 2023-11-28T12:59:24.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.917+0000 7fc7ab7fe700 1 -- 172.21.15.111:0/2331887046 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b400fae0 con 0x7fc7c41f1cb0 2023-11-28T12:59:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.917+0000 7fc7c94ec700 1 -- 172.21.15.111:0/2331887046 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c40cc000 con 0x7fc7c41f1cb0 2023-11-28T12:59:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.917+0000 7fc7ab7fe700 1 -- 172.21.15.111:0/2331887046 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b400fe90 con 0x7fc7c41f1cb0 2023-11-28T12:59:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.917+0000 7fc7c94ec700 1 -- 172.21.15.111:0/2331887046 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c415cb80 con 0x7fc7c41f1cb0 2023-11-28T12:59:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.918+0000 7fc7c94ec700 1 -- 172.21.15.111:0/2331887046 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c4067170 con 0x7fc7c41f1cb0 2023-11-28T12:59:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.918+0000 7fc7ab7fe700 1 -- 172.21.15.111:0/2331887046 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc7b4014440 con 0x7fc7c41f1cb0 2023-11-28T12:59:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.919+0000 7fc7ab7fe700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7b00682f0 0x7fc7b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.919+0000 7fc7ab7fe700 1 -- 172.21.15.111:0/2331887046 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc7b40868a0 con 0x7fc7c41f1cb0 2023-11-28T12:59:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.919+0000 7fc7c2ffd700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7b00682f0 0x7fc7b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:24.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.919+0000 7fc7c2ffd700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7b00682f0 0x7fc7b006a7a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8009cc0 tx=0x7fc7b8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:24.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:24.923+0000 7fc7ab7fe700 1 -- 172.21.15.111:0/2331887046 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc7b4052e70 con 0x7fc7c41f1cb0 2023-11-28T12:59:25.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.105+0000 7fc7c94ec700 1 -- 172.21.15.111:0/2331887046 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc7c415a620 con 0x7fc7c41f1cb0 2023-11-28T12:59:25.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.106+0000 7fc7ab7fe700 1 -- 172.21.15.111:0/2331887046 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc7b4056490 con 0x7fc7c41f1cb0 2023-11-28T12:59:25.106 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:59:25.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.108+0000 7fc7c94ec700 1 -- 172.21.15.111:0/2331887046 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7b00682f0 msgr2=0x7fc7b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.108+0000 7fc7c94ec700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7b00682f0 0x7fc7b006a7a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8009cc0 tx=0x7fc7b8008040 comp rx=0 tx=0).stop 2023-11-28T12:59:25.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.108+0000 7fc7c94ec700 1 -- 172.21.15.111:0/2331887046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c41f1cb0 msgr2=0x7fc7c40b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.108+0000 7fc7c94ec700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c41f1cb0 0x7fc7c40b4570 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4011aa0 tx=0x7fc7b4011e60 comp rx=0 tx=0).stop 2023-11-28T12:59:25.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.108+0000 7fc7c94ec700 1 -- 172.21.15.111:0/2331887046 shutdown_connections 2023-11-28T12:59:25.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.108+0000 7fc7c94ec700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7b00682f0 0x7fc7b006a7a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.108+0000 7fc7c94ec700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7c415e250 0x7fc7c40b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.108+0000 7fc7c94ec700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7c41f18e0 0x7fc7c415dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.108+0000 7fc7c94ec700 1 --2- 172.21.15.111:0/2331887046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7c41f1cb0 0x7fc7c40b4570 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.108+0000 7fc7c94ec700 1 -- 172.21.15.111:0/2331887046 >> 172.21.15.111:0/2331887046 conn(0x7fc7c409a040 msgr2=0x7fc7c4070380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:25.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.108+0000 7fc7c94ec700 1 -- 172.21.15.111:0/2331887046 shutdown_connections 2023-11-28T12:59:25.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.108+0000 7fc7c94ec700 1 -- 172.21.15.111:0/2331887046 wait complete. 2023-11-28T12:59:25.120 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T12:59:25.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.242+0000 7fd4b20e2700 1 Processor -- start 2023-11-28T12:59:25.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.242+0000 7fd4b20e2700 1 -- start start 2023-11-28T12:59:25.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.242+0000 7fd4b20e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd4ac0cdef0 0x7fd4ac0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4b20e2700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd4ac0cc5a0 0x7fd4ac0ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4b20e2700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac0d05a0 con 0x7fd4ac0cfc90 2023-11-28T12:59:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4b20e2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac0ca260 con 0x7fd4ac0cdef0 2023-11-28T12:59:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4b20e2700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac0ccaa0 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4aaffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd4ac0cdef0 0x7fd4ac0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4ab7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd4ac0cfc90 0x7fd4ac0cd7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34434/0 (socket says 172.21.15.111:34434) 2023-11-28T12:59:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4ab7fe700 1 -- 172.21.15.111:0/3890813732 learned_addr learned my addr 172.21.15.111:0/3890813732 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4abfff700 1 --2- 172.21.15.111:0/3890813732 >> v2:172.21.15.149:3300/0 conn(0x7fd4ac0cc5a0 0x7fd4ac0ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4aaffd700 1 -- 172.21.15.111:0/3890813732 >> v2:172.21.15.149:3300/0 conn(0x7fd4ac0cc5a0 msgr2=0x7fd4ac0ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4aaffd700 1 --2- 172.21.15.111:0/3890813732 >> v2:172.21.15.149:3300/0 conn(0x7fd4ac0cc5a0 0x7fd4ac0ca6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4aaffd700 1 -- 172.21.15.111:0/3890813732 >> v1:172.21.15.111:6789/0 conn(0x7fd4ac0cfc90 legacy=0x7fd4ac0cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4aaffd700 1 -- 172.21.15.111:0/3890813732 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac0ccd80 con 0x7fd4ac0cdef0 2023-11-28T12:59:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4aa7fc700 1 -- 172.21.15.111:0/3890813732 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265071315 0 0) 0x7fd49c001110 con 0x7fd4ac0cfc90 2023-11-28T12:59:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4aaffd700 1 --2- 172.21.15.111:0/3890813732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4ac0cdef0 0x7fd4ac0d0060 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac0af2f0 tx=0x7fd4a0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f9a4d1c583e081 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4aa7fc700 1 -- 172.21.15.111:0/3890813732 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0011830 con 0x7fd4ac0cdef0 2023-11-28T12:59:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4aa7fc700 1 -- 172.21.15.111:0/3890813732 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a0011990 con 0x7fd4ac0cdef0 2023-11-28T12:59:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.243+0000 7fd4aa7fc700 1 -- 172.21.15.111:0/3890813732 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0011d20 con 0x7fd4ac0cdef0 2023-11-28T12:59:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.244+0000 7fd4b20e2700 1 -- 172.21.15.111:0/3890813732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4ac0cdef0 msgr2=0x7fd4ac0d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.244+0000 7fd4b20e2700 1 --2- 172.21.15.111:0/3890813732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4ac0cdef0 0x7fd4ac0d0060 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac0af2f0 tx=0x7fd4a0009b20 comp rx=0 tx=0).stop 2023-11-28T12:59:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.244+0000 7fd4b20e2700 1 -- 172.21.15.111:0/3890813732 shutdown_connections 2023-11-28T12:59:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.244+0000 7fd4b20e2700 1 --2- 172.21.15.111:0/3890813732 >> v2:172.21.15.149:3300/0 conn(0x7fd4ac0cc5a0 0x7fd4ac0ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.244+0000 7fd4b20e2700 1 --2- 172.21.15.111:0/3890813732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4ac0cdef0 0x7fd4ac0d0060 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.244+0000 7fd4b20e2700 1 -- 172.21.15.111:0/3890813732 >> 172.21.15.111:0/3890813732 conn(0x7fd4ac098ee0 msgr2=0x7fd4ac06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:25.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.244+0000 7fd4b20e2700 1 -- 172.21.15.111:0/3890813732 shutdown_connections 2023-11-28T12:59:25.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.244+0000 7fd4b20e2700 1 -- 172.21.15.111:0/3890813732 wait complete. 2023-11-28T12:59:25.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.245+0000 7fd4b20e2700 1 Processor -- start 2023-11-28T12:59:25.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.245+0000 7fd4b20e2700 1 -- start start 2023-11-28T12:59:25.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4b20e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4ac0cc5a0 0x7fd4ac0c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4b20e2700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4ac0cdef0 0x7fd4ac15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4b20e2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4ac0cfc90 0x7fd4ac15c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4b20e2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac159070 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4b20e2700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0b01d0 con 0x7fd4ac0cdef0 2023-11-28T12:59:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4b20e2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0c9110 con 0x7fd4ac0cfc90 2023-11-28T12:59:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4ab7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4ac0cc5a0 0x7fd4ac0c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4ab7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4ac0cc5a0 0x7fd4ac0c87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33662/0 (socket says 172.21.15.111:33662) 2023-11-28T12:59:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4ab7fe700 1 -- 172.21.15.111:0/625913290 learned_addr learned my addr 172.21.15.111:0/625913290 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4aaffd700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4ac0cdef0 0x7fd4ac15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4abfff700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4ac0cfc90 0x7fd4ac15c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4ab7fe700 1 -- 172.21.15.111:0/625913290 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4ac0cfc90 msgr2=0x7fd4ac15c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4ab7fe700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4ac0cfc90 0x7fd4ac15c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4ab7fe700 1 -- 172.21.15.111:0/625913290 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4ac0cdef0 msgr2=0x7fd4ac15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4ab7fe700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4ac0cdef0 0x7fd4ac15bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4ab7fe700 1 -- 172.21.15.111:0/625913290 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a00094a0 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.246+0000 7fd4ab7fe700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4ac0cc5a0 0x7fd4ac0c87e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00c550 tx=0x7fd49c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:25.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.247+0000 7fd493fff700 1 -- 172.21.15.111:0/625913290 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c004c80 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.247+0000 7fd493fff700 1 -- 172.21.15.111:0/625913290 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49c004e00 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.247+0000 7fd4b20e2700 1 -- 172.21.15.111:0/625913290 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ac0cd340 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.247+0000 7fd493fff700 1 -- 172.21.15.111:0/625913290 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c007310 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.247+0000 7fd4b20e2700 1 -- 172.21.15.111:0/625913290 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ac0cd060 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.248+0000 7fd4b20e2700 1 -- 172.21.15.111:0/625913290 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ac143280 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.248+0000 7fd493fff700 1 -- 172.21.15.111:0/625913290 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd49c01a020 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.248+0000 7fd493fff700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4980682f0 0x7fd49806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.248+0000 7fd493fff700 1 -- 172.21.15.111:0/625913290 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd49c085810 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.248+0000 7fd4aaffd700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4980682f0 0x7fd49806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.249+0000 7fd4aaffd700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4980682f0 0x7fd49806a7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac0ce2e0 tx=0x7fd4a0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:25.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.253+0000 7fd493fff700 1 -- 172.21.15.111:0/625913290 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd49c051d80 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.436+0000 7fd4b20e2700 1 -- 172.21.15.111:0/625913290 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd4ac0d05a0 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.436+0000 7fd493fff700 1 -- 172.21.15.111:0/625913290 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd4ac0d05a0 con 0x7fd4ac0cc5a0 2023-11-28T12:59:25.437 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:25.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.438+0000 7fd4b20e2700 1 -- 172.21.15.111:0/625913290 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4980682f0 msgr2=0x7fd49806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.438+0000 7fd4b20e2700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4980682f0 0x7fd49806a7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac0ce2e0 tx=0x7fd4a0008030 comp rx=0 tx=0).stop 2023-11-28T12:59:25.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.438+0000 7fd4b20e2700 1 -- 172.21.15.111:0/625913290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4ac0cc5a0 msgr2=0x7fd4ac0c87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.438+0000 7fd4b20e2700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4ac0cc5a0 0x7fd4ac0c87e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00c550 tx=0x7fd49c00c860 comp rx=0 tx=0).stop 2023-11-28T12:59:25.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.438+0000 7fd4b20e2700 1 -- 172.21.15.111:0/625913290 shutdown_connections 2023-11-28T12:59:25.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.438+0000 7fd4b20e2700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4ac0cfc90 0x7fd4ac15c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.438+0000 7fd4b20e2700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4ac0cc5a0 0x7fd4ac0c87e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.438+0000 7fd4b20e2700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4980682f0 0x7fd49806a7a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.438+0000 7fd4b20e2700 1 --2- 172.21.15.111:0/625913290 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4ac0cdef0 0x7fd4ac15bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.438+0000 7fd4b20e2700 1 -- 172.21.15.111:0/625913290 >> 172.21.15.111:0/625913290 conn(0x7fd4ac098ee0 msgr2=0x7fd4ac0b4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:25.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.438+0000 7fd4b20e2700 1 -- 172.21.15.111:0/625913290 shutdown_connections 2023-11-28T12:59:25.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.439+0000 7fd4b20e2700 1 -- 172.21.15.111:0/625913290 wait complete. 2023-11-28T12:59:25.451 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T12:59:25.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.573+0000 7fb803aaf700 1 Processor -- start 2023-11-28T12:59:25.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.574+0000 7fb803aaf700 1 -- start start 2023-11-28T12:59:25.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.574+0000 7fb803aaf700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb7fc0b0d70 0x7fb7fc1604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.574+0000 7fb803aaf700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb7fc0cc550 0x7fb7fc0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.574+0000 7fb803aaf700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc15cd90 con 0x7fb7fc0b1140 2023-11-28T12:59:25.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.574+0000 7fb803aaf700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7fc0cedd0 con 0x7fb7fc0cc550 2023-11-28T12:59:25.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.574+0000 7fb803aaf700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7fc0cced0 con 0x7fb7fc0b0d70 2023-11-28T12:59:25.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.574+0000 7fb80104a700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb7fc0b1140 0x7fb7fc0ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34448/0 (socket says 172.21.15.111:34448) 2023-11-28T12:59:25.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.574+0000 7fb80204c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb7fc0cc550 0x7fb7fc0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.574+0000 7fb80104a700 1 -- 172.21.15.111:0/261591972 learned_addr learned my addr 172.21.15.111:0/261591972 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:25.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.575+0000 7fb80184b700 1 --2- 172.21.15.111:0/261591972 >> v2:172.21.15.149:3300/0 conn(0x7fb7fc0b0d70 0x7fb7fc1604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.575+0000 7fb80204c700 1 -- 172.21.15.111:0/261591972 >> v2:172.21.15.149:3300/0 conn(0x7fb7fc0b0d70 msgr2=0x7fb7fc1604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:25.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.575+0000 7fb80204c700 1 --2- 172.21.15.111:0/261591972 >> v2:172.21.15.149:3300/0 conn(0x7fb7fc0b0d70 0x7fb7fc1604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.575+0000 7fb80204c700 1 -- 172.21.15.111:0/261591972 >> v1:172.21.15.111:6789/0 conn(0x7fb7fc0b1140 legacy=0x7fb7fc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.575+0000 7fb80204c700 1 -- 172.21.15.111:0/261591972 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc1613e0 con 0x7fb7fc0cc550 2023-11-28T12:59:25.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.575+0000 7fb800849700 1 -- 172.21.15.111:0/261591972 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250532024 0 0) 0x7fb7f8001110 con 0x7fb7fc0b1140 2023-11-28T12:59:25.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.575+0000 7fb80204c700 1 --2- 172.21.15.111:0/261591972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7fc0cc550 0x7fb7fc0cc990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc071960 tx=0x7fb7f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c2b3e06a57a149 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:25.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.575+0000 7fb800849700 1 -- 172.21.15.111:0/261591972 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f400e070 con 0x7fb7fc0cc550 2023-11-28T12:59:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.575+0000 7fb800849700 1 -- 172.21.15.111:0/261591972 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f400bbd0 con 0x7fb7fc0cc550 2023-11-28T12:59:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.575+0000 7fb800849700 1 -- 172.21.15.111:0/261591972 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4012430 con 0x7fb7fc0cc550 2023-11-28T12:59:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.576+0000 7fb803aaf700 1 -- 172.21.15.111:0/261591972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7fc0cc550 msgr2=0x7fb7fc0cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.576+0000 7fb803aaf700 1 --2- 172.21.15.111:0/261591972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7fc0cc550 0x7fb7fc0cc990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc071960 tx=0x7fb7f400b330 comp rx=0 tx=0).stop 2023-11-28T12:59:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.576+0000 7fb803aaf700 1 -- 172.21.15.111:0/261591972 shutdown_connections 2023-11-28T12:59:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.576+0000 7fb803aaf700 1 --2- 172.21.15.111:0/261591972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7fc0cc550 0x7fb7fc0cc990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.576+0000 7fb803aaf700 1 --2- 172.21.15.111:0/261591972 >> v2:172.21.15.149:3300/0 conn(0x7fb7fc0b0d70 0x7fb7fc1604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.576+0000 7fb803aaf700 1 -- 172.21.15.111:0/261591972 >> 172.21.15.111:0/261591972 conn(0x7fb7fc098ee0 msgr2=0x7fb7fc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.576+0000 7fb803aaf700 1 -- 172.21.15.111:0/261591972 shutdown_connections 2023-11-28T12:59:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.576+0000 7fb803aaf700 1 -- 172.21.15.111:0/261591972 wait complete. 2023-11-28T12:59:25.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.577+0000 7fb803aaf700 1 Processor -- start 2023-11-28T12:59:25.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.577+0000 7fb803aaf700 1 -- start start 2023-11-28T12:59:25.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb803aaf700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7fc0b0d70 0x7fb7fc159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb803aaf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7fc0b1140 0x7fb7fc0c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb803aaf700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7fc0cc550 0x7fb7fc15e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb803aaf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc0c50e0 con 0x7fb7fc0b1140 2023-11-28T12:59:25.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb803aaf700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc15ecf0 con 0x7fb7fc0cc550 2023-11-28T12:59:25.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb803aaf700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc15efd0 con 0x7fb7fc0b0d70 2023-11-28T12:59:25.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb80104a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7fc0b1140 0x7fb7fc0c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb80104a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7fc0b1140 0x7fb7fc0c4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33678/0 (socket says 172.21.15.111:33678) 2023-11-28T12:59:25.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb80104a700 1 -- 172.21.15.111:0/1702740214 learned_addr learned my addr 172.21.15.111:0/1702740214 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:25.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb80204c700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7fc0cc550 0x7fb7fc15e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb80184b700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7fc0b0d70 0x7fb7fc159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb80104a700 1 -- 172.21.15.111:0/1702740214 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7fc0b0d70 msgr2=0x7fb7fc159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb80104a700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7fc0b0d70 0x7fb7fc159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb80104a700 1 -- 172.21.15.111:0/1702740214 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7fc0cc550 msgr2=0x7fb7fc15e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb80104a700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7fc0cc550 0x7fb7fc15e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb80104a700 1 -- 172.21.15.111:0/1702740214 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f4009a70 con 0x7fb7fc0b1140 2023-11-28T12:59:25.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb80104a700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7fc0b1140 0x7fb7fc0c4ba0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800c730 tx=0x7fb7f800caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:25.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb7f27fc700 1 -- 172.21.15.111:0/1702740214 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8004c80 con 0x7fb7fc0b1140 2023-11-28T12:59:25.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb7f27fc700 1 -- 172.21.15.111:0/1702740214 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f8004e00 con 0x7fb7fc0b1140 2023-11-28T12:59:25.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb803aaf700 1 -- 172.21.15.111:0/1702740214 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7fc160e20 con 0x7fb7fc0b1140 2023-11-28T12:59:25.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb7f27fc700 1 -- 172.21.15.111:0/1702740214 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8007310 con 0x7fb7fc0b1140 2023-11-28T12:59:25.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.578+0000 7fb803aaf700 1 -- 172.21.15.111:0/1702740214 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7fc160b40 con 0x7fb7fc0b1140 2023-11-28T12:59:25.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.579+0000 7fb803aaf700 1 -- 172.21.15.111:0/1702740214 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7fc161100 con 0x7fb7fc0b1140 2023-11-28T12:59:25.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.580+0000 7fb7f27fc700 1 -- 172.21.15.111:0/1702740214 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb7f801a020 con 0x7fb7fc0b1140 2023-11-28T12:59:25.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.580+0000 7fb7f27fc700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7e00682f0 0x7fb7e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.580+0000 7fb7f27fc700 1 -- 172.21.15.111:0/1702740214 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb7f8085a20 con 0x7fb7fc0b1140 2023-11-28T12:59:25.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.580+0000 7fb80184b700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7e00682f0 0x7fb7e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.580+0000 7fb80184b700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7e00682f0 0x7fb7e006a7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec006fd0 tx=0x7fb7ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:25.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.584+0000 7fb7f27fc700 1 -- 172.21.15.111:0/1702740214 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb7f8051f90 con 0x7fb7fc0b1140 2023-11-28T12:59:25.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.766+0000 7fb803aaf700 1 -- 172.21.15.111:0/1702740214 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb7fc067170 con 0x7fb7fc0b1140 2023-11-28T12:59:25.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.767+0000 7fb7f27fc700 1 -- 172.21.15.111:0/1702740214 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb7f80555b0 con 0x7fb7fc0b1140 2023-11-28T12:59:25.768 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T12:59:25.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.768+0000 7fb803aaf700 1 -- 172.21.15.111:0/1702740214 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7e00682f0 msgr2=0x7fb7e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.768+0000 7fb803aaf700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7e00682f0 0x7fb7e006a7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec006fd0 tx=0x7fb7ec008040 comp rx=0 tx=0).stop 2023-11-28T12:59:25.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.768+0000 7fb803aaf700 1 -- 172.21.15.111:0/1702740214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7fc0b1140 msgr2=0x7fb7fc0c4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.768+0000 7fb803aaf700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7fc0b1140 0x7fb7fc0c4ba0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800c730 tx=0x7fb7f800caf0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.769+0000 7fb803aaf700 1 -- 172.21.15.111:0/1702740214 shutdown_connections 2023-11-28T12:59:25.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.769+0000 7fb803aaf700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7e00682f0 0x7fb7e006a7a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.769+0000 7fb803aaf700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7fc0cc550 0x7fb7fc15e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.769+0000 7fb803aaf700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7fc0b0d70 0x7fb7fc159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.769+0000 7fb803aaf700 1 --2- 172.21.15.111:0/1702740214 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7fc0b1140 0x7fb7fc0c4ba0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.769+0000 7fb803aaf700 1 -- 172.21.15.111:0/1702740214 >> 172.21.15.111:0/1702740214 conn(0x7fb7fc098ee0 msgr2=0x7fb7fc06dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:25.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.769+0000 7fb803aaf700 1 -- 172.21.15.111:0/1702740214 shutdown_connections 2023-11-28T12:59:25.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.769+0000 7fb803aaf700 1 -- 172.21.15.111:0/1702740214 wait complete. 2023-11-28T12:59:25.781 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T12:59:25.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.904+0000 7f72fddef700 1 Processor -- start 2023-11-28T12:59:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.904+0000 7f72fddef700 1 -- start start 2023-11-28T12:59:25.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.904+0000 7f72fddef700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72f81604b0 0x7f72f80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.904+0000 7f72fddef700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f815cd90 con 0x7f72f80cc550 2023-11-28T12:59:25.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72fddef700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f81613e0 con 0x7f72f80b0d70 2023-11-28T12:59:25.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72fddef700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f8160950 con 0x7f72f81604b0 2023-11-28T12:59:25.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72f6ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72f81604b0 0x7f72f80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72f7fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f72f80cc550 0x7f72f80af860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34466/0 (socket says 172.21.15.111:34466) 2023-11-28T12:59:25.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72f6ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72f81604b0 0x7f72f80ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33686/0 (socket says 172.21.15.111:33686) 2023-11-28T12:59:25.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72f7fff700 1 -- 172.21.15.111:0/2663164048 learned_addr learned my addr 172.21.15.111:0/2663164048 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:25.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72f6ffd700 1 -- 172.21.15.111:0/2663164048 >> v1:172.21.15.149:6789/0 conn(0x7f72f80b0d70 legacy=0x7f72f80b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:25.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72f6ffd700 1 -- 172.21.15.111:0/2663164048 >> v1:172.21.15.111:6789/0 conn(0x7f72f80cc550 legacy=0x7f72f80af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72f6ffd700 1 -- 172.21.15.111:0/2663164048 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f8160b40 con 0x7f72f81604b0 2023-11-28T12:59:25.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72f67fc700 1 -- 172.21.15.111:0/2663164048 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253000622 0 0) 0x7f72f0001490 con 0x7f72f80cc550 2023-11-28T12:59:25.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72f6ffd700 1 --2- 172.21.15.111:0/2663164048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f81604b0 0x7f72f80ce6c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f72ec009840 tx=0x7f72ec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af172dab7914f89 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:25.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72f67fc700 1 -- 172.21.15.111:0/2663164048 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec004230 con 0x7f72f81604b0 2023-11-28T12:59:25.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72f67fc700 1 -- 172.21.15.111:0/2663164048 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec004390 con 0x7f72f81604b0 2023-11-28T12:59:25.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.905+0000 7f72f67fc700 1 -- 172.21.15.111:0/2663164048 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec0036a0 con 0x7f72f81604b0 2023-11-28T12:59:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.906+0000 7f72fddef700 1 -- 172.21.15.111:0/2663164048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f81604b0 msgr2=0x7f72f80ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.906+0000 7f72fddef700 1 --2- 172.21.15.111:0/2663164048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f81604b0 0x7f72f80ce6c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f72ec009840 tx=0x7f72ec009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.906+0000 7f72fddef700 1 -- 172.21.15.111:0/2663164048 shutdown_connections 2023-11-28T12:59:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.906+0000 7f72fddef700 1 --2- 172.21.15.111:0/2663164048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f81604b0 0x7f72f80ce6c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.906+0000 7f72fddef700 1 -- 172.21.15.111:0/2663164048 >> 172.21.15.111:0/2663164048 conn(0x7f72f8098ee0 msgr2=0x7f72f806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.906+0000 7f72fddef700 1 -- 172.21.15.111:0/2663164048 shutdown_connections 2023-11-28T12:59:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.906+0000 7f72fddef700 1 -- 172.21.15.111:0/2663164048 wait complete. 2023-11-28T12:59:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.907+0000 7f72fddef700 1 Processor -- start 2023-11-28T12:59:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.907+0000 7f72fddef700 1 -- start start 2023-11-28T12:59:25.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72fddef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80b0d70 0x7f72f8154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72fddef700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72f80cc550 0x7f72f815d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72fddef700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72f81604b0 0x7f72f815df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72fddef700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80c5860 con 0x7f72f80b0d70 2023-11-28T12:59:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72fddef700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80c4c50 con 0x7f72f80cc550 2023-11-28T12:59:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72fddef700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80c4f30 con 0x7f72f81604b0 2023-11-28T12:59:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72f77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80b0d70 0x7f72f8154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72f77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80b0d70 0x7f72f8154070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33690/0 (socket says 172.21.15.111:33690) 2023-11-28T12:59:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72f77fe700 1 -- 172.21.15.111:0/2653453110 learned_addr learned my addr 172.21.15.111:0/2653453110 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72f6ffd700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72f80cc550 0x7f72f815d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72f7fff700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72f81604b0 0x7f72f815df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72f77fe700 1 -- 172.21.15.111:0/2653453110 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72f81604b0 msgr2=0x7f72f815df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72f77fe700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72f81604b0 0x7f72f815df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72f77fe700 1 -- 172.21.15.111:0/2653453110 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72f80cc550 msgr2=0x7f72f815d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72f77fe700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72f80cc550 0x7f72f815d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72f77fe700 1 -- 172.21.15.111:0/2653453110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec0094a0 con 0x7f72f80b0d70 2023-11-28T12:59:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72f6ffd700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72f80cc550 0x7f72f815d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72f77fe700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80b0d70 0x7f72f8154070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f72e800c4b0 tx=0x7f72e800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.908+0000 7f72dffff700 1 -- 172.21.15.111:0/2653453110 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8011790 con 0x7f72f80b0d70 2023-11-28T12:59:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.909+0000 7f72dffff700 1 -- 172.21.15.111:0/2653453110 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e80118f0 con 0x7f72f80b0d70 2023-11-28T12:59:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.909+0000 7f72fddef700 1 -- 172.21.15.111:0/2653453110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f8161100 con 0x7f72f80b0d70 2023-11-28T12:59:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.909+0000 7f72dffff700 1 -- 172.21.15.111:0/2653453110 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8011ca0 con 0x7f72f80b0d70 2023-11-28T12:59:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.909+0000 7f72fddef700 1 -- 172.21.15.111:0/2653453110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f8160e20 con 0x7f72f80b0d70 2023-11-28T12:59:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.909+0000 7f72fddef700 1 -- 172.21.15.111:0/2653453110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f80af340 con 0x7f72f80b0d70 2023-11-28T12:59:25.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.910+0000 7f72dffff700 1 -- 172.21.15.111:0/2653453110 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f72e8010050 con 0x7f72f80b0d70 2023-11-28T12:59:25.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.910+0000 7f72dffff700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f72e40682f0 0x7f72e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:25.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.910+0000 7f72dffff700 1 -- 172.21.15.111:0/2653453110 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f72e8085cc0 con 0x7f72f80b0d70 2023-11-28T12:59:25.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.910+0000 7f72f6ffd700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f72e40682f0 0x7f72e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:25.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.911+0000 7f72f6ffd700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f72e40682f0 0x7f72e406a7a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f72f8160000 tx=0x7f72ec013040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:25.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:25.914+0000 7f72dffff700 1 -- 172.21.15.111:0/2653453110 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f72e80521d0 con 0x7f72f80b0d70 2023-11-28T12:59:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.098+0000 7f72fddef700 1 -- 172.21.15.111:0/2653453110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f72f80d0c90 con 0x7f72f80b0d70 2023-11-28T12:59:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.099+0000 7f72dffff700 1 -- 172.21.15.111:0/2653453110 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f72e80557f0 con 0x7f72f80b0d70 2023-11-28T12:59:26.100 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:26.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.101+0000 7f72fddef700 1 -- 172.21.15.111:0/2653453110 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f72e40682f0 msgr2=0x7f72e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:26.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.101+0000 7f72fddef700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f72e40682f0 0x7f72e406a7a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f72f8160000 tx=0x7f72ec013040 comp rx=0 tx=0).stop 2023-11-28T12:59:26.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.101+0000 7f72fddef700 1 -- 172.21.15.111:0/2653453110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80b0d70 msgr2=0x7f72f8154070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:26.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.101+0000 7f72fddef700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80b0d70 0x7f72f8154070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f72e800c4b0 tx=0x7f72e800c870 comp rx=0 tx=0).stop 2023-11-28T12:59:26.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.101+0000 7f72fddef700 1 -- 172.21.15.111:0/2653453110 shutdown_connections 2023-11-28T12:59:26.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.101+0000 7f72fddef700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72f81604b0 0x7f72f815df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:26.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.101+0000 7f72fddef700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80b0d70 0x7f72f8154070 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:26.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.101+0000 7f72fddef700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f72e40682f0 0x7f72e406a7a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:26.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.101+0000 7f72fddef700 1 --2- 172.21.15.111:0/2653453110 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72f80cc550 0x7f72f815d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:26.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.101+0000 7f72fddef700 1 -- 172.21.15.111:0/2653453110 >> 172.21.15.111:0/2653453110 conn(0x7f72f8098ee0 msgr2=0x7f72f815c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:26.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.101+0000 7f72fddef700 1 -- 172.21.15.111:0/2653453110 shutdown_connections 2023-11-28T12:59:26.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.101+0000 7f72fddef700 1 -- 172.21.15.111:0/2653453110 wait complete. 2023-11-28T12:59:26.115 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T12:59:26.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.233+0000 7fe3055eb700 1 Processor -- start 2023-11-28T12:59:26.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.233+0000 7fe3055eb700 1 -- start start 2023-11-28T12:59:26.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.234+0000 7fe3055eb700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3000af9f0 0x7fe3000ca220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:26.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.234+0000 7fe3055eb700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30006ac80 con 0x7fe30015fed0 2023-11-28T12:59:26.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.234+0000 7fe3055eb700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3000cee00 con 0x7fe3000af5a0 2023-11-28T12:59:26.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.234+0000 7fe3055eb700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3000cf9a0 con 0x7fe3000af9f0 2023-11-28T12:59:26.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.234+0000 7fe2feffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe30015fed0 0x7fe3000cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34476/0 (socket says 172.21.15.111:34476) 2023-11-28T12:59:26.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.234+0000 7fe2feffd700 1 -- 172.21.15.111:0/3693131513 learned_addr learned my addr 172.21.15.111:0/3693131513 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:26.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.235+0000 7fe2ff7fe700 1 --2- 172.21.15.111:0/3693131513 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3000af9f0 0x7fe3000ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:26.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.235+0000 7fe2fdffb700 1 -- 172.21.15.111:0/3693131513 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631965962 0 0) 0x7fe30006ac80 con 0x7fe30015fed0 2023-11-28T12:59:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.235+0000 7fe2fdffb700 1 -- 172.21.15.111:0/3693131513 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2e80035b0 con 0x7fe30015fed0 2023-11-28T12:59:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.235+0000 7fe2fdffb700 1 -- 172.21.15.111:0/3693131513 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211934715 0 0) 0x7fe3000cee00 con 0x7fe3000af5a0 2023-11-28T12:59:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.235+0000 7fe2fdffb700 1 -- 172.21.15.111:0/3693131513 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe30006ac80 con 0x7fe3000af5a0 2023-11-28T12:59:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.235+0000 7fe2fdffb700 1 -- 172.21.15.111:0/3693131513 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1226017283 0 0) 0x7fe2e80035b0 con 0x7fe30015fed0 2023-11-28T12:59:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.235+0000 7fe2ff7fe700 1 -- 172.21.15.111:0/3693131513 >> v1:172.21.15.149:6789/0 conn(0x7fe3000af5a0 legacy=0x7fe3000b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.235+0000 7fe2ff7fe700 1 -- 172.21.15.111:0/3693131513 >> v1:172.21.15.111:6789/0 conn(0x7fe30015fed0 legacy=0x7fe3000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.235+0000 7fe2ff7fe700 1 -- 172.21.15.111:0/3693131513 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30006aed0 con 0x7fe3000af9f0 2023-11-28T12:59:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.235+0000 7fe2ff7fe700 1 --2- 172.21.15.111:0/3693131513 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3000af9f0 0x7fe3000ca220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009dc0 tx=0x7fe2f800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0046e6785c991bf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.235+0000 7fe2fdffb700 1 -- 172.21.15.111:0/3693131513 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f800e070 con 0x7fe3000af9f0 2023-11-28T12:59:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.235+0000 7fe2fdffb700 1 -- 172.21.15.111:0/3693131513 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f8003a20 con 0x7fe3000af9f0 2023-11-28T12:59:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.235+0000 7fe2fdffb700 1 -- 172.21.15.111:0/3693131513 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4001670 con 0x7fe3000af9f0 2023-11-28T12:59:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.236+0000 7fe3055eb700 1 -- 172.21.15.111:0/3693131513 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3000af9f0 msgr2=0x7fe3000ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.236+0000 7fe3055eb700 1 --2- 172.21.15.111:0/3693131513 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3000af9f0 0x7fe3000ca220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009dc0 tx=0x7fe2f800b330 comp rx=0 tx=0).stop 2023-11-28T12:59:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.236+0000 7fe3055eb700 1 -- 172.21.15.111:0/3693131513 shutdown_connections 2023-11-28T12:59:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.236+0000 7fe3055eb700 1 --2- 172.21.15.111:0/3693131513 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3000af9f0 0x7fe3000ca220 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.236+0000 7fe3055eb700 1 -- 172.21.15.111:0/3693131513 >> 172.21.15.111:0/3693131513 conn(0x7fe30006bca0 msgr2=0x7fe3000cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.236+0000 7fe3055eb700 1 -- 172.21.15.111:0/3693131513 shutdown_connections 2023-11-28T12:59:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.236+0000 7fe3055eb700 1 -- 172.21.15.111:0/3693131513 wait complete. 2023-11-28T12:59:26.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.237+0000 7fe3055eb700 1 Processor -- start 2023-11-28T12:59:26.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.237+0000 7fe3055eb700 1 -- start start 2023-11-28T12:59:26.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe3055eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3000af5a0 0x7fe3000c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:26.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe3055eb700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3000af9f0 0x7fe3000c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:26.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe3055eb700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe30015fed0 0x7fe3001500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:26.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe3055eb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe300159450 con 0x7fe3000af5a0 2023-11-28T12:59:26.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe3055eb700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe300150620 con 0x7fe30015fed0 2023-11-28T12:59:26.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe3055eb700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3000c41b0 con 0x7fe3000af9f0 2023-11-28T12:59:26.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe2feffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3000af5a0 0x7fe3000c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:26.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe2feffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3000af5a0 0x7fe3000c57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33704/0 (socket says 172.21.15.111:33704) 2023-11-28T12:59:26.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe2feffd700 1 -- 172.21.15.111:0/500239835 learned_addr learned my addr 172.21.15.111:0/500239835 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:26.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe2ff7fe700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe30015fed0 0x7fe3001500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:26.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe2fe7fc700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3000af9f0 0x7fe3000c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:26.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe2feffd700 1 -- 172.21.15.111:0/500239835 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3000af9f0 msgr2=0x7fe3000c6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:26.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe2feffd700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3000af9f0 0x7fe3000c6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:26.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe2feffd700 1 -- 172.21.15.111:0/500239835 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe30015fed0 msgr2=0x7fe3001500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:26.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe2feffd700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe30015fed0 0x7fe3001500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:26.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe2feffd700 1 -- 172.21.15.111:0/500239835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f8009a70 con 0x7fe3000af5a0 2023-11-28T12:59:26.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.238+0000 7fe2feffd700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3000af5a0 0x7fe3000c57a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400c6b0 tx=0x7fe2f400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:26.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.239+0000 7fe2e77fe700 1 -- 172.21.15.111:0/500239835 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4004c80 con 0x7fe3000af5a0 2023-11-28T12:59:26.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.239+0000 7fe2e77fe700 1 -- 172.21.15.111:0/500239835 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f4004e00 con 0x7fe3000af5a0 2023-11-28T12:59:26.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.239+0000 7fe3055eb700 1 -- 172.21.15.111:0/500239835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3001602a0 con 0x7fe3000af5a0 2023-11-28T12:59:26.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.239+0000 7fe2e77fe700 1 -- 172.21.15.111:0/500239835 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4007310 con 0x7fe3000af5a0 2023-11-28T12:59:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.239+0000 7fe3055eb700 1 -- 172.21.15.111:0/500239835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3000cea90 con 0x7fe3000af5a0 2023-11-28T12:59:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.240+0000 7fe3055eb700 1 -- 172.21.15.111:0/500239835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3000ce6b0 con 0x7fe3000af5a0 2023-11-28T12:59:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.240+0000 7fe2e77fe700 1 -- 172.21.15.111:0/500239835 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe2f401a020 con 0x7fe3000af5a0 2023-11-28T12:59:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.240+0000 7fe2e77fe700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2ec0682f0 0x7fe2ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.241+0000 7fe2e77fe700 1 -- 172.21.15.111:0/500239835 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe2f4085d00 con 0x7fe3000af5a0 2023-11-28T12:59:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.241+0000 7fe2fe7fc700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2ec0682f0 0x7fe2ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.241+0000 7fe2fe7fc700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2ec0682f0 0x7fe2ec06a7a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe3001598f0 tx=0x7fe2f0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:26.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.245+0000 7fe2e77fe700 1 -- 172.21.15.111:0/500239835 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2f4089030 con 0x7fe3000af5a0 2023-11-28T12:59:26.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.468+0000 7fe3055eb700 1 -- 172.21.15.111:0/500239835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe300067170 con 0x7fe3000af5a0 2023-11-28T12:59:26.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.468+0000 7fe2e77fe700 1 -- 172.21.15.111:0/500239835 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4656 (secure 0 0 0) 0x7fe2f4052100 con 0x7fe3000af5a0 2023-11-28T12:59:26.470 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:26.470 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2513950989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2513950989},{"type":"v1","addr":"172.21.15.149:6834","nonce":2513950989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1258891714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1258891714},{"type":"v1","addr":"172.21.15.149:6835","nonce":1258891714}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/3923909288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3923909288},{"type":"v1","addr":"172.21.15.111:6836","nonce":3923909288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T12:58:55.633507+0000","modified":"2023-11-28T12:59:17.702148+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/3180164366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":3180164366},{"type":"v1","addr":"172.21.15.111:6837","nonce":3180164366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T12:59:26.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.470+0000 7fe3055eb700 1 -- 172.21.15.111:0/500239835 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2ec0682f0 msgr2=0x7fe2ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:26.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.470+0000 7fe3055eb700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2ec0682f0 0x7fe2ec06a7a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe3001598f0 tx=0x7fe2f0009040 comp rx=0 tx=0).stop 2023-11-28T12:59:26.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.470+0000 7fe3055eb700 1 -- 172.21.15.111:0/500239835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3000af5a0 msgr2=0x7fe3000c57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:26.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.470+0000 7fe3055eb700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3000af5a0 0x7fe3000c57a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400c6b0 tx=0x7fe2f400c9c0 comp rx=0 tx=0).stop 2023-11-28T12:59:26.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.470+0000 7fe3055eb700 1 -- 172.21.15.111:0/500239835 shutdown_connections 2023-11-28T12:59:26.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.470+0000 7fe3055eb700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2ec0682f0 0x7fe2ec06a7a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:26.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.470+0000 7fe3055eb700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe30015fed0 0x7fe3001500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:26.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.470+0000 7fe3055eb700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3000af5a0 0x7fe3000c57a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:26.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.470+0000 7fe3055eb700 1 --2- 172.21.15.111:0/500239835 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3000af9f0 0x7fe3000c6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:26.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.470+0000 7fe3055eb700 1 -- 172.21.15.111:0/500239835 >> 172.21.15.111:0/500239835 conn(0x7fe30006bca0 msgr2=0x7fe30015f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:26.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.470+0000 7fe3055eb700 1 -- 172.21.15.111:0/500239835 shutdown_connections 2023-11-28T12:59:26.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:26.470+0000 7fe3055eb700 1 -- 172.21.15.111:0/500239835 wait complete. 2023-11-28T12:59:26.472 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 9 2023-11-28T12:59:26.886 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T12:58:55.633507+0000', 'modified': '2023-11-28T12:59:17.702148+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4305}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4305': {'gid': 4305, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6837/3180164366', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6835', 'nonce': 3180164366}, {'type': 'v1', 'addr': '172.21.15.111:6837', 'nonce': 3180164366}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T12:59:26.886 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T12:59:26.886 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T12:59:27.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.008+0000 7f3544310700 1 Processor -- start 2023-11-28T12:59:27.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.008+0000 7f3544310700 1 -- start start 2023-11-28T12:59:27.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f3544310700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f353c0cef60 0x7f353c0ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f3544310700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f353c0b0160 0x7f353c0adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f3544310700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353c0cc390 con 0x7f353c15a5d0 2023-11-28T12:59:27.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f3544310700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f353c0cc1b0 con 0x7f353c0b0160 2023-11-28T12:59:27.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f3544310700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353c0cc670 con 0x7f353c0cef60 2023-11-28T12:59:27.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35428ad700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f353c0b0160 0x7f353c0adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35420ac700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f353c15a5d0 0x7f353c0ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34490/0 (socket says 172.21.15.111:34490) 2023-11-28T12:59:27.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35420ac700 1 -- 172.21.15.111:0/1727185897 learned_addr learned my addr 172.21.15.111:0/1727185897 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:27.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35418ab700 1 --2- 172.21.15.111:0/1727185897 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f353c0cef60 0x7f353c0ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35428ad700 1 -- 172.21.15.111:0/1727185897 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f353c0cef60 msgr2=0x7f353c0ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:27.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35428ad700 1 --2- 172.21.15.111:0/1727185897 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f353c0cef60 0x7f353c0ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35428ad700 1 -- 172.21.15.111:0/1727185897 >> v1:172.21.15.111:6789/0 conn(0x7f353c15a5d0 legacy=0x7f353c0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35428ad700 1 -- 172.21.15.111:0/1727185897 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c0cc950 con 0x7f353c0b0160 2023-11-28T12:59:27.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35410aa700 1 -- 172.21.15.111:0/1727185897 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244483384 0 0) 0x7f353c0cc390 con 0x7f353c15a5d0 2023-11-28T12:59:27.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35418ab700 1 --2- 172.21.15.111:0/1727185897 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f353c0cef60 0x7f353c0ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:27.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35428ad700 1 --2- 172.21.15.111:0/1727185897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f353c0b0160 0x7f353c0adde0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3534009d90 tx=0x7f353400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a07f2ccb7a39e0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:27.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35410aa700 1 -- 172.21.15.111:0/1727185897 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353400e070 con 0x7f353c0b0160 2023-11-28T12:59:27.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35410aa700 1 -- 172.21.15.111:0/1727185897 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353400bbd0 con 0x7f353c0b0160 2023-11-28T12:59:27.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.009+0000 7f35410aa700 1 -- 172.21.15.111:0/1727185897 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3534012430 con 0x7f353c0b0160 2023-11-28T12:59:27.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.010+0000 7f3544310700 1 -- 172.21.15.111:0/1727185897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f353c0b0160 msgr2=0x7f353c0adde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.010+0000 7f3544310700 1 --2- 172.21.15.111:0/1727185897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f353c0b0160 0x7f353c0adde0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3534009d90 tx=0x7f353400b330 comp rx=0 tx=0).stop 2023-11-28T12:59:27.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.010+0000 7f3544310700 1 -- 172.21.15.111:0/1727185897 shutdown_connections 2023-11-28T12:59:27.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.010+0000 7f3544310700 1 --2- 172.21.15.111:0/1727185897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f353c0b0160 0x7f353c0adde0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.010+0000 7f3544310700 1 --2- 172.21.15.111:0/1727185897 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f353c0cef60 0x7f353c0ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.010+0000 7f3544310700 1 -- 172.21.15.111:0/1727185897 >> 172.21.15.111:0/1727185897 conn(0x7f353c09a040 msgr2=0x7f353c0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:27.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.010+0000 7f3544310700 1 -- 172.21.15.111:0/1727185897 shutdown_connections 2023-11-28T12:59:27.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.010+0000 7f3544310700 1 -- 172.21.15.111:0/1727185897 wait complete. 2023-11-28T12:59:27.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.011+0000 7f3544310700 1 Processor -- start 2023-11-28T12:59:27.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.011+0000 7f3544310700 1 -- start start 2023-11-28T12:59:27.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f3544310700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f353c0b0160 0x7f353c06cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f3544310700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f353c0cef60 0x7f353c0b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f3544310700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f353c15a5d0 0x7f353c0ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f3544310700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353c0b1f60 con 0x7f353c0b0160 2023-11-28T12:59:27.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f3544310700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353c0cabd0 con 0x7f353c15a5d0 2023-11-28T12:59:27.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f3544310700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353c0b2190 con 0x7f353c0cef60 2023-11-28T12:59:27.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f35420ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f353c0b0160 0x7f353c06cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f35420ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f353c0b0160 0x7f353c06cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33718/0 (socket says 172.21.15.111:33718) 2023-11-28T12:59:27.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f35420ac700 1 -- 172.21.15.111:0/1464995807 learned_addr learned my addr 172.21.15.111:0/1464995807 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:27.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f35418ab700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f353c0cef60 0x7f353c0b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f35428ad700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f353c15a5d0 0x7f353c0ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f35420ac700 1 -- 172.21.15.111:0/1464995807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f353c0cef60 msgr2=0x7f353c0b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f35420ac700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f353c0cef60 0x7f353c0b2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f35420ac700 1 -- 172.21.15.111:0/1464995807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f353c15a5d0 msgr2=0x7f353c0ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f35420ac700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f353c15a5d0 0x7f353c0ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f35420ac700 1 -- 172.21.15.111:0/1464995807 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3534009a70 con 0x7f353c0b0160 2023-11-28T12:59:27.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.012+0000 7f35428ad700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f353c15a5d0 0x7f353c0ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:27.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.013+0000 7f35420ac700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f353c0b0160 0x7f353c06cb60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f353800c700 tx=0x7f353800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:27.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.013+0000 7f3532ffd700 1 -- 172.21.15.111:0/1464995807 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35380074e0 con 0x7f353c0b0160 2023-11-28T12:59:27.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.013+0000 7f3532ffd700 1 -- 172.21.15.111:0/1464995807 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3538007640 con 0x7f353c0b0160 2023-11-28T12:59:27.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.013+0000 7f3544310700 1 -- 172.21.15.111:0/1464995807 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353c0ccf10 con 0x7f353c0b0160 2023-11-28T12:59:27.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.013+0000 7f3532ffd700 1 -- 172.21.15.111:0/1464995807 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3538005a30 con 0x7f353c0b0160 2023-11-28T12:59:27.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.013+0000 7f3544310700 1 -- 172.21.15.111:0/1464995807 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f353c0ccc30 con 0x7f353c0b0160 2023-11-28T12:59:27.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.014+0000 7f3532ffd700 1 -- 172.21.15.111:0/1464995807 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3538019030 con 0x7f353c0b0160 2023-11-28T12:59:27.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.014+0000 7f3532ffd700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f35200682f0 0x7f352006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.015+0000 7f3532ffd700 1 -- 172.21.15.111:0/1464995807 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3538085260 con 0x7f353c0b0160 2023-11-28T12:59:27.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.015+0000 7f35418ab700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f35200682f0 0x7f352006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.015+0000 7f35418ab700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f35200682f0 0x7f352006a7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f352c006fd0 tx=0x7f352c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.016+0000 7f3550499b80 1 Processor -- start 2023-11-28T12:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.017+0000 7f3550499b80 1 -- start start 2023-11-28T12:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.017+0000 7f3550499b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165912810 0x55d1658e8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.017+0000 7f3550499b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d165912be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.017+0000 7f3550499b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165913120 0x55d16592e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.017+0000 7f3550499b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1657d2bc0 con 0x55d165910130 2023-11-28T12:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.017+0000 7f3550499b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d165841350 con 0x55d165912810 2023-11-28T12:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.017+0000 7f3550499b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d16583a880 con 0x55d165913120 2023-11-28T12:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.017+0000 7f35418ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d165912be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.017+0000 7f35418ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d165912be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33728/0 (socket says 172.21.15.111:33728) 2023-11-28T12:59:27.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.017+0000 7f35418ab700 1 -- 172.21.15.111:0/1268974507 learned_addr learned my addr 172.21.15.111:0/1268974507 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:27.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.018+0000 7f35420ac700 1 --2- 172.21.15.111:0/1268974507 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165912810 0x55d1658e8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.018+0000 7f35428ad700 1 --2- 172.21.15.111:0/1268974507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165913120 0x55d16592e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.018+0000 7f35418ab700 1 -- 172.21.15.111:0/1268974507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165913120 msgr2=0x55d16592e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:27.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.018+0000 7f35418ab700 1 --2- 172.21.15.111:0/1268974507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165913120 0x55d16592e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.018+0000 7f35418ab700 1 -- 172.21.15.111:0/1268974507 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165912810 msgr2=0x55d1658e8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:27.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.018+0000 7f35418ab700 1 --2- 172.21.15.111:0/1268974507 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165912810 0x55d1658e8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.018+0000 7f35418ab700 1 -- 172.21.15.111:0/1268974507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1656bc5e0 con 0x55d165910130 2023-11-28T12:59:27.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.018+0000 7f35420ac700 1 --2- 172.21.15.111:0/1268974507 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165912810 0x55d1658e8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:27.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.018+0000 7f35418ab700 1 --2- 172.21.15.111:0/1268974507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d165912be0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f352c00d9d0 tx=0x7f352c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1162efb8a8cd63 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:27.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.018+0000 7f351ffff700 1 -- 172.21.15.111:0/1268974507 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352c00b050 con 0x55d165910130 2023-11-28T12:59:27.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.018+0000 7f351ffff700 1 -- 172.21.15.111:0/1268974507 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352c017070 con 0x55d165910130 2023-11-28T12:59:27.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.018+0000 7f351ffff700 1 -- 172.21.15.111:0/1268974507 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352c012cd0 con 0x55d165910130 2023-11-28T12:59:27.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.019+0000 7f3550499b80 1 -- 172.21.15.111:0/1268974507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 msgr2=0x55d165912be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.019+0000 7f3550499b80 1 --2- 172.21.15.111:0/1268974507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d165912be0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f352c00d9d0 tx=0x7f352c010000 comp rx=0 tx=0).stop 2023-11-28T12:59:27.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.019+0000 7f3550499b80 1 -- 172.21.15.111:0/1268974507 shutdown_connections 2023-11-28T12:59:27.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.019+0000 7f3550499b80 1 --2- 172.21.15.111:0/1268974507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165913120 0x55d16592e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.019+0000 7f3550499b80 1 --2- 172.21.15.111:0/1268974507 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165912810 0x55d1658e8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.019+0000 7f3550499b80 1 --2- 172.21.15.111:0/1268974507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d165912be0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.019+0000 7f3550499b80 1 -- 172.21.15.111:0/1268974507 >> 172.21.15.111:0/1268974507 conn(0x55d1658e5d60 msgr2=0x55d1658e66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:27.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.019+0000 7f3550499b80 1 -- 172.21.15.111:0/1268974507 shutdown_connections 2023-11-28T12:59:27.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.019+0000 7f3550499b80 1 -- 172.21.15.111:0/1268974507 wait complete. 2023-11-28T12:59:27.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.019+0000 7f3550499b80 1 -- start start 2023-11-28T12:59:27.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.019+0000 7f3550499b80 1 Processor -- start 2023-11-28T12:59:27.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.020+0000 7f3550499b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d16590b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.020+0000 7f3550499b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165912810 0x55d165931cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.020+0000 7f3550499b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165913120 0x55d165933700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.020+0000 7f3550499b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1656be8d0 con 0x55d165910130 2023-11-28T12:59:27.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.020+0000 7f3550499b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d16590c310 con 0x55d165912810 2023-11-28T12:59:27.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.020+0000 7f3550499b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d165932410 con 0x55d165913120 2023-11-28T12:59:27.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.020+0000 7f3550499b80 10 client.0 _ll_register_callbacks cb 0x55d1656e0240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T12:59:27.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.020+0000 7f35420ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d16590b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.020+0000 7f35420ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d16590b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33732/0 (socket says 172.21.15.111:33732) 2023-11-28T12:59:27.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.020+0000 7f35420ac700 1 -- 172.21.15.111:0/1423706990 learned_addr learned my addr 172.21.15.111:0/1423706990 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:27.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f35418ab700 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165912810 0x55d165931cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f35428ad700 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165913120 0x55d165933700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f35420ac700 1 -- 172.21.15.111:0/1423706990 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165913120 msgr2=0x55d165933700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:27.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f35420ac700 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165913120 0x55d165933700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f35420ac700 1 -- 172.21.15.111:0/1423706990 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165912810 msgr2=0x55d165931cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:27.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f35420ac700 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165912810 0x55d165931cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f35420ac700 1 -- 172.21.15.111:0/1423706990 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f352c00d680 con 0x55d165910130 2023-11-28T12:59:27.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f35418ab700 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165912810 0x55d165931cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:27.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f35420ac700 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d16590b2c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f35380519f0 tx=0x7f3538054dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d154d3ae6adae327 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:27.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f351d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T12:59:27.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f351d7fa700 1 -- 172.21.15.111:0/1423706990 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3538005ea0 con 0x55d165910130 2023-11-28T12:59:27.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f351d7fa700 1 -- 172.21.15.111:0/1423706990 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3538068dd0 con 0x55d165910130 2023-11-28T12:59:27.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f3550499b80 1 -- 172.21.15.111:0/1423706990 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d165808140 con 0x55d165910130 2023-11-28T12:59:27.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f351d7fa700 1 -- 172.21.15.111:0/1423706990 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35380850d0 con 0x55d165910130 2023-11-28T12:59:27.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f3550499b80 10 client.4824 fetch_fsmap learned FSMap version 9 2023-11-28T12:59:27.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f3550499b80 1 -- 172.21.15.111:0/1423706990 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x55d165668d90 con 0x55d165910130 2023-11-28T12:59:27.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f351d7fa700 1 -- 172.21.15.111:0/1423706990 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3538061080 con 0x55d165910130 2023-11-28T12:59:27.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f351d7fa700 1 -- 172.21.15.111:0/1423706990 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 9) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3538083120 con 0x55d165910130 2023-11-28T12:59:27.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f3550499b80 10 client.4824 fetch_fsmap finished waiting for FSMap version 9 2023-11-28T12:59:27.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f3550499b80 10 client.4824 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T12:59:27.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f3550499b80 20 client.4824 populate_metadata read hostname 'smithi111' 2023-11-28T12:59:27.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f3550499b80 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x55d165935730 0x55d165914d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f3550499b80 4 client.4824 mds_command: new command op to 4305 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-28T12:59:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.021+0000 7f3550499b80 1 -- 172.21.15.111:0/1423706990 --> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d1656bef10 con 0x55d165935730 2023-11-28T12:59:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.022+0000 7f35418ab700 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x55d165935730 0x55d165914d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.022+0000 7f351d7fa700 10 client.4824 ms_handle_connect on v2:172.21.15.111:6835/3180164366 2023-11-28T12:59:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.022+0000 7f35418ab700 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x55d165935730 0x55d165914d90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60052f2a68a3cae0 server_cookie=c378ed656272a730 in_seq=0 out_seq=0 2023-11-28T12:59:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.022+0000 7f351d7fa700 1 -- 172.21.15.111:0/1423706990 <== mds.0 v2:172.21.15.111:6835/3180164366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55d1656bef10 con 0x55d165935730 2023-11-28T12:59:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.022+0000 7f351d7fa700 10 client.4824 handle_command_reply: tid=0 2023-11-28T12:59:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.022+0000 7f3550499b80 1 client.4824 shutdown 2023-11-28T12:59:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.023+0000 7f3550499b80 1 -- 172.21.15.111:0/1423706990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 msgr2=0x55d16590b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.023+0000 7f3550499b80 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d16590b2c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f35380519f0 tx=0x7f3538054dc0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.023+0000 7f3550499b80 1 -- 172.21.15.111:0/1423706990 shutdown_connections 2023-11-28T12:59:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.023+0000 7f3550499b80 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x55d165935730 0x55d165914d90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.023+0000 7f3550499b80 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165913120 0x55d165933700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.023+0000 7f351d7fa700 0 client.4824 ms_handle_reset on v2:172.21.15.111:6835/3180164366 2023-11-28T12:59:27.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.023+0000 7f3550499b80 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d16590b2c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.023+0000 7f3550499b80 1 --2- 172.21.15.111:0/1423706990 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165912810 0x55d165931cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.023+0000 7f3550499b80 1 -- 172.21.15.111:0/1423706990 >> 172.21.15.111:0/1423706990 conn(0x55d1658e5d60 msgr2=0x55d1658e8990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:27.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.023+0000 7f3550499b80 1 -- 172.21.15.111:0/1423706990 shutdown_connections 2023-11-28T12:59:27.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.023+0000 7f3550499b80 1 -- 172.21.15.111:0/1423706990 wait complete. 2023-11-28T12:59:27.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.023+0000 7f3550499b80 20 client.4824 trim_cache size 0 max 16384 2023-11-28T12:59:27.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.040+0000 7f3550499b80 1 Processor -- start 2023-11-28T12:59:27.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.040+0000 7f3550499b80 1 -- start start 2023-11-28T12:59:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f3550499b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d16590fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f3550499b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165912810 0x55d1658e2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f3550499b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165913120 0x55d1659342d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f3550499b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d165931b60 con 0x55d165910130 2023-11-28T12:59:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f3550499b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1658e3360 con 0x55d165913120 2023-11-28T12:59:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f3550499b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d165916730 con 0x55d165912810 2023-11-28T12:59:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f35420ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d16590fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f35420ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d16590fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33740/0 (socket says 172.21.15.111:33740) 2023-11-28T12:59:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f35420ac700 1 -- 172.21.15.111:0/2865210757 learned_addr learned my addr 172.21.15.111:0/2865210757 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f35428ad700 1 --2- 172.21.15.111:0/2865210757 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165913120 0x55d1659342d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f35418ab700 1 --2- 172.21.15.111:0/2865210757 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165912810 0x55d1658e2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f35420ac700 1 -- 172.21.15.111:0/2865210757 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165912810 msgr2=0x55d1658e2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f35420ac700 1 --2- 172.21.15.111:0/2865210757 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165912810 0x55d1658e2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f35420ac700 1 -- 172.21.15.111:0/2865210757 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165913120 msgr2=0x55d1659342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f35420ac700 1 --2- 172.21.15.111:0/2865210757 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165913120 0x55d1659342d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f35420ac700 1 -- 172.21.15.111:0/2865210757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d165801140 con 0x55d165910130 2023-11-28T12:59:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f35428ad700 1 --2- 172.21.15.111:0/2865210757 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165913120 0x55d1659342d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f35418ab700 1 --2- 172.21.15.111:0/2865210757 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165912810 0x55d1658e2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f35420ac700 1 --2- 172.21.15.111:0/2865210757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d16590fa50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3538055470 tx=0x7f353801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c97034e6a59755a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f3530dad700 1 -- 172.21.15.111:0/2865210757 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3538005ea0 con 0x55d165910130 2023-11-28T12:59:27.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f3530dad700 1 -- 172.21.15.111:0/2865210757 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d165808140 con 0x55d165910130 2023-11-28T12:59:27.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.041+0000 7f3530dad700 1 -- 172.21.15.111:0/2865210757 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3538068dd0 con 0x55d165910130 2023-11-28T12:59:27.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.042+0000 7f3550499b80 1 -- 172.21.15.111:0/2865210757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 msgr2=0x55d16590fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.042+0000 7f3550499b80 1 --2- 172.21.15.111:0/2865210757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d16590fa50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3538055470 tx=0x7f353801e000 comp rx=0 tx=0).stop 2023-11-28T12:59:27.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.042+0000 7f3550499b80 1 -- 172.21.15.111:0/2865210757 shutdown_connections 2023-11-28T12:59:27.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.042+0000 7f3550499b80 1 --2- 172.21.15.111:0/2865210757 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165913120 0x55d1659342d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.042+0000 7f3550499b80 1 --2- 172.21.15.111:0/2865210757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165910130 0x55d16590fa50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.042+0000 7f3550499b80 1 --2- 172.21.15.111:0/2865210757 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165912810 0x55d1658e2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.042+0000 7f3550499b80 1 -- 172.21.15.111:0/2865210757 >> 172.21.15.111:0/2865210757 conn(0x55d1658e5d60 msgr2=0x55d1658e9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:27.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.042+0000 7f3550499b80 1 -- 172.21.15.111:0/2865210757 shutdown_connections 2023-11-28T12:59:27.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.042+0000 7f3550499b80 1 -- 172.21.15.111:0/2865210757 wait complete. 2023-11-28T12:59:27.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.042+0000 7f3550499b80 1 -- start start 2023-11-28T12:59:27.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.043+0000 7f3550499b80 1 Processor -- start 2023-11-28T12:59:27.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.043+0000 7f3550499b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165910130 0x55d165946ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.043+0000 7f3550499b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165912810 0x55d1658e2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.043+0000 7f3550499b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165913120 0x55d1658e3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.043+0000 7f3550499b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1659932c0 con 0x55d165913120 2023-11-28T12:59:27.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.043+0000 7f3550499b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d165926690 con 0x55d165910130 2023-11-28T12:59:27.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.043+0000 7f3550499b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d1659268e0 con 0x55d165912810 2023-11-28T12:59:27.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.043+0000 7f3550499b80 10 client.0 _ll_register_callbacks cb 0x55d1656e0240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T12:59:27.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f35428ad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165913120 0x55d1658e3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f35428ad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165913120 0x55d1658e3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33750/0 (socket says 172.21.15.111:33750) 2023-11-28T12:59:27.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f35428ad700 1 -- 172.21.15.111:0/1892254010 learned_addr learned my addr 172.21.15.111:0/1892254010 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:27.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f35420ac700 1 --2- 172.21.15.111:0/1892254010 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165910130 0x55d165946ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f35418ab700 1 --2- 172.21.15.111:0/1892254010 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165912810 0x55d1658e2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f35428ad700 1 -- 172.21.15.111:0/1892254010 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165912810 msgr2=0x55d1658e2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:27.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f35428ad700 1 --2- 172.21.15.111:0/1892254010 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165912810 0x55d1658e2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f35428ad700 1 -- 172.21.15.111:0/1892254010 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165910130 msgr2=0x55d165946ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:27.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f35428ad700 1 --2- 172.21.15.111:0/1892254010 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165910130 0x55d165946ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f35428ad700 1 -- 172.21.15.111:0/1892254010 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d165668d90 con 0x55d165913120 2023-11-28T12:59:27.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f35428ad700 1 --2- 172.21.15.111:0/1892254010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165913120 0x55d1658e3360 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f353c06d8f0 tx=0x7f353400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f43fd75b45cc2f6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:27.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f351d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T12:59:27.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f351d7fa700 1 -- 172.21.15.111:0/1892254010 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3534021040 con 0x55d165913120 2023-11-28T12:59:27.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f351d7fa700 1 -- 172.21.15.111:0/1892254010 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3534012d20 con 0x55d165913120 2023-11-28T12:59:27.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f3550499b80 1 -- 172.21.15.111:0/1892254010 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d1657f8160 con 0x55d165913120 2023-11-28T12:59:27.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f351d7fa700 1 -- 172.21.15.111:0/1892254010 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3534004040 con 0x55d165913120 2023-11-28T12:59:27.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f3550499b80 10 client.4830 fetch_fsmap learned FSMap version 9 2023-11-28T12:59:27.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f3550499b80 1 -- 172.21.15.111:0/1892254010 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x55d16568f200 con 0x55d165913120 2023-11-28T12:59:27.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f351d7fa700 1 -- 172.21.15.111:0/1892254010 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d1657f8160 con 0x55d165913120 2023-11-28T12:59:27.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f351d7fa700 1 -- 172.21.15.111:0/1892254010 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 9) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f35340041a0 con 0x55d165913120 2023-11-28T12:59:27.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.044+0000 7f3550499b80 10 client.4830 fetch_fsmap finished waiting for FSMap version 9 2023-11-28T12:59:27.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 10 client.4830 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T12:59:27.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 20 client.4830 populate_metadata read hostname 'smithi111' 2023-11-28T12:59:27.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 1 --2- 172.21.15.111:0/1892254010 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x55d1659783e0 0x55d1659787b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 4 client.4830 mds_command: new command op to 4305 tid=0[{"prefix": "status"}] 2023-11-28T12:59:27.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 1 -- 172.21.15.111:0/1892254010 --> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d1657e2170 con 0x55d1659783e0 2023-11-28T12:59:27.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f35420ac700 1 --2- 172.21.15.111:0/1892254010 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x55d1659783e0 0x55d1659787b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f351d7fa700 10 client.4830 ms_handle_connect on v2:172.21.15.111:6835/3180164366 2023-11-28T12:59:27.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f35420ac700 1 --2- 172.21.15.111:0/1892254010 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x55d1659783e0 0x55d1659787b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21754699af079a7d server_cookie=bb7da7be7f2bc613 in_seq=0 out_seq=0 2023-11-28T12:59:27.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f351d7fa700 1 -- 172.21.15.111:0/1892254010 <== mds.0 v2:172.21.15.111:6835/3180164366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55d1657e2170 con 0x55d1659783e0 2023-11-28T12:59:27.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f351d7fa700 10 client.4830 handle_command_reply: tid=0 2023-11-28T12:59:27.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 1 client.4830 shutdown 2023-11-28T12:59:27.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 1 -- 172.21.15.111:0/1892254010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165913120 msgr2=0x55d1658e3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 1 --2- 172.21.15.111:0/1892254010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165913120 0x55d1658e3360 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f353c06d8f0 tx=0x7f353400e000 comp rx=0 tx=0).stop 2023-11-28T12:59:27.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 1 -- 172.21.15.111:0/1892254010 shutdown_connections 2023-11-28T12:59:27.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 1 --2- 172.21.15.111:0/1892254010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d165913120 0x55d1658e3360 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 1 --2- 172.21.15.111:0/1892254010 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d165910130 0x55d165946ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 1 --2- 172.21.15.111:0/1892254010 >> [v2:172.21.15.111:6835/3180164366,v1:172.21.15.111:6837/3180164366] conn(0x55d1659783e0 0x55d1659787b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 1 --2- 172.21.15.111:0/1892254010 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d165912810 0x55d1658e2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f351d7fa700 0 client.4830 ms_handle_reset on v2:172.21.15.111:6835/3180164366 2023-11-28T12:59:27.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.045+0000 7f3550499b80 1 -- 172.21.15.111:0/1892254010 >> 172.21.15.111:0/1892254010 conn(0x55d1658e5d60 msgr2=0x55d16590f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:27.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3550499b80 1 -- 172.21.15.111:0/1892254010 shutdown_connections 2023-11-28T12:59:27.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3550499b80 1 -- 172.21.15.111:0/1892254010 wait complete. 2023-11-28T12:59:27.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3550499b80 20 client.4830 trim_cache size 0 max 16384 2023-11-28T12:59:27.054 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T12:59:27.054 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "a1b2d00c-aeec-4b5d-93e3-41820c4189ba", 2023-11-28T12:59:27.054 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-11-28T12:59:27.054 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 4305, 2023-11-28T12:59:27.054 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-11-28T12:59:27.054 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-11-28T12:59:27.054 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-11-28T12:59:27.054 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 31.410461119000001, 2023-11-28T12:59:27.054 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 9, 2023-11-28T12:59:27.054 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 26, 2023-11-28T12:59:27.054 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 26, 2023-11-28T12:59:27.055 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 36.736420043999999 2023-11-28T12:59:27.055 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T12:59:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3544310700 1 -- 172.21.15.111:0/1464995807 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f35200682f0 msgr2=0x7f352006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3544310700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f35200682f0 0x7f352006a7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f352c006fd0 tx=0x7f352c008040 comp rx=0 tx=0).stop 2023-11-28T12:59:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3544310700 1 -- 172.21.15.111:0/1464995807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f353c0b0160 msgr2=0x7f353c06cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3544310700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f353c0b0160 0x7f353c06cb60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f353800c700 tx=0x7f353800cac0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3544310700 1 -- 172.21.15.111:0/1464995807 shutdown_connections 2023-11-28T12:59:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3544310700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f35200682f0 0x7f352006a7a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3544310700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f353c15a5d0 0x7f353c0ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3544310700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f353c0b0160 0x7f353c06cb60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3544310700 1 --2- 172.21.15.111:0/1464995807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f353c0cef60 0x7f353c0b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3544310700 1 -- 172.21.15.111:0/1464995807 >> 172.21.15.111:0/1464995807 conn(0x7f353c09a040 msgr2=0x7f353c157bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:27.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3544310700 1 -- 172.21.15.111:0/1464995807 shutdown_connections 2023-11-28T12:59:27.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.046+0000 7f3544310700 1 -- 172.21.15.111:0/1464995807 wait complete. 2023-11-28T12:59:27.058 INFO:teuthology.run_tasks:Running task kclient... 2023-11-28T12:59:27.071 INFO:tasks.kclient:Mounting kernel clients... 2023-11-28T12:59:27.072 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-11-28T12:59:27.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:27.072 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-11-28T12:59:27.089 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-11-28T12:59:27.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:27.089 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-28T12:59:27.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:27.165 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link delete ceph-brx 2023-11-28T12:59:27.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:27.274 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-11-28T12:59:27.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:27.292 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link delete ceph-brx 2023-11-28T12:59:27.362 INFO:teuthology.orchestra.run.smithi111.stderr:Cannot find device "ceph-brx" 2023-11-28T12:59:27.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T12:59:27.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:27.363 DEBUG:teuthology.orchestra.run.smithi149:> ip netns list 2023-11-28T12:59:27.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:27.385 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip link delete ceph-brx 2023-11-28T12:59:27.459 INFO:teuthology.orchestra.run.smithi149.stderr:Cannot find device "ceph-brx" 2023-11-28T12:59:27.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T12:59:27.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:27.461 DEBUG:teuthology.orchestra.run.smithi149:> ip netns list 2023-11-28T12:59:27.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:27.520 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip link delete ceph-brx 2023-11-28T12:59:27.591 INFO:teuthology.orchestra.run.smithi149.stderr:Cannot find device "ceph-brx" 2023-11-28T12:59:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T12:59:27.592 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-11-28T12:59:27.593 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T12:59:27.593 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T12:59:27.593 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T12:59:27.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.717+0000 7f732552e700 1 Processor -- start 2023-11-28T12:59:27.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.718+0000 7f732552e700 1 -- start start 2023-11-28T12:59:27.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.718+0000 7f732552e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f732006a980 0x7f73200c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.718+0000 7f732552e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7320159b10 con 0x7f73200ca430 2023-11-28T12:59:27.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.718+0000 7f732552e700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7320143250 con 0x7f7320142e80 2023-11-28T12:59:27.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.718+0000 7f732552e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73200c9380 con 0x7f732006a980 2023-11-28T12:59:27.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.718+0000 7f731f7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f73200ca430 0x7f73200cd0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34536/0 (socket says 172.21.15.111:34536) 2023-11-28T12:59:27.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731f7fe700 1 -- 172.21.15.111:0/3292624938 learned_addr learned my addr 172.21.15.111:0/3292624938 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:27.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731e7fc700 1 --2- 172.21.15.111:0/3292624938 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f732006a980 0x7f73200c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659560119 0 0) 0x7f7320159b10 con 0x7f73200ca430 2023-11-28T12:59:27.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73080035b0 con 0x7f73200ca430 2023-11-28T12:59:27.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1660093428 0 0) 0x7f73080035b0 con 0x7f73200ca430 2023-11-28T12:59:27.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f732006a980 msgr2=0x7f73200c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:27.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 --2- 172.21.15.111:0/3292624938 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f732006a980 0x7f73200c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 >> v1:172.21.15.149:6789/0 conn(0x7f7320142e80 legacy=0x7f732006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73200ca870 con 0x7f73200ca430 2023-11-28T12:59:27.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f73180019f0 con 0x7f73200ca430 2023-11-28T12:59:27.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 >> v1:172.21.15.111:6789/0 conn(0x7f73200ca430 legacy=0x7f73200cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 --2- 172.21.15.111:0/3292624938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7308003a30 0x7f7308003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 --2- 172.21.15.111:0/3292624938 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f732006a980 0x7f7308004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 --2- 172.21.15.111:0/3292624938 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7308004d90 0x7f7308005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73200c9380 con 0x7f7308003a30 2023-11-28T12:59:27.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730800bb50 con 0x7f7308004d90 2023-11-28T12:59:27.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730800bce0 con 0x7f732006a980 2023-11-28T12:59:27.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731effd700 1 --2- 172.21.15.111:0/3292624938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7308003a30 0x7f7308003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731e7fc700 1 --2- 172.21.15.111:0/3292624938 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f732006a980 0x7f7308004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731f7fe700 1 --2- 172.21.15.111:0/3292624938 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7308004d90 0x7f7308005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.719+0000 7f731effd700 1 -- 172.21.15.111:0/3292624938 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f732006a980 msgr2=0x7f7308004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:27.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f731effd700 1 --2- 172.21.15.111:0/3292624938 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f732006a980 0x7f7308004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f731effd700 1 -- 172.21.15.111:0/3292624938 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7308004d90 msgr2=0x7f7308005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:27.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f731effd700 1 --2- 172.21.15.111:0/3292624938 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7308004d90 0x7f7308005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f731effd700 1 -- 172.21.15.111:0/3292624938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73080035b0 con 0x7f7308003a30 2023-11-28T12:59:27.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f731effd700 1 --2- 172.21.15.111:0/3292624938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7308003a30 0x7f7308003e40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7314001800 tx=0x7f7308005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc489f7836a3943 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:27.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73140012f0 con 0x7f7308003a30 2023-11-28T12:59:27.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73140052f0 con 0x7f7308003a30 2023-11-28T12:59:27.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f731dffb700 1 -- 172.21.15.111:0/3292624938 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314006860 con 0x7f7308003a30 2023-11-28T12:59:27.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f732552e700 1 -- 172.21.15.111:0/3292624938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7308003a30 msgr2=0x7f7308003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f732552e700 1 --2- 172.21.15.111:0/3292624938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7308003a30 0x7f7308003e40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7314001800 tx=0x7f7308005880 comp rx=0 tx=0).stop 2023-11-28T12:59:27.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f731f7fe700 1 -- 172.21.15.111:0/3292624938 reap_dead start 2023-11-28T12:59:27.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f732552e700 1 -- 172.21.15.111:0/3292624938 shutdown_connections 2023-11-28T12:59:27.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f732552e700 1 -- 172.21.15.111:0/3292624938 >> 172.21.15.111:0/3292624938 conn(0x7f73200cb490 msgr2=0x7f73200cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:27.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f732552e700 1 -- 172.21.15.111:0/3292624938 shutdown_connections 2023-11-28T12:59:27.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.720+0000 7f732552e700 1 -- 172.21.15.111:0/3292624938 wait complete. 2023-11-28T12:59:27.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.722+0000 7f732552e700 1 Processor -- start 2023-11-28T12:59:27.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.722+0000 7f732552e700 1 -- start start 2023-11-28T12:59:27.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.722+0000 7f732552e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7308003a30 0x7f7320159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.722+0000 7f732552e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73200acf40 0x7f7320157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.722+0000 7f732552e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73200709f0 0x7f7320070e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.722+0000 7f732552e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73200713d0 con 0x7f73200acf40 2023-11-28T12:59:27.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.722+0000 7f732552e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73201583a0 con 0x7f7308003a30 2023-11-28T12:59:27.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.722+0000 7f732552e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732015a330 con 0x7f73200709f0 2023-11-28T12:59:27.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.722+0000 7f731e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73200acf40 0x7f7320157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f731e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73200acf40 0x7f7320157e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33768/0 (socket says 172.21.15.111:33768) 2023-11-28T12:59:27.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f731e7fc700 1 -- 172.21.15.111:0/118554315 learned_addr learned my addr 172.21.15.111:0/118554315 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:27.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f731f7fe700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73200709f0 0x7f7320070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f731effd700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7308003a30 0x7f7320159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f731e7fc700 1 -- 172.21.15.111:0/118554315 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73200709f0 msgr2=0x7f7320070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f731e7fc700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73200709f0 0x7f7320070e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f731e7fc700 1 -- 172.21.15.111:0/118554315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7308003a30 msgr2=0x7f7320159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f731e7fc700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7308003a30 0x7f7320159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f731e7fc700 1 -- 172.21.15.111:0/118554315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73200cc5f0 con 0x7f73200acf40 2023-11-28T12:59:27.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f731f7fe700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73200709f0 0x7f7320070e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:27.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f731effd700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7308003a30 0x7f7320159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:27.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f731e7fc700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73200acf40 0x7f7320157e60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f73100119c0 tx=0x7f7310011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:27.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f73077fe700 1 -- 172.21.15.111:0/118554315 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310004240 con 0x7f73200acf40 2023-11-28T12:59:27.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f73077fe700 1 -- 172.21.15.111:0/118554315 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73100043a0 con 0x7f73200acf40 2023-11-28T12:59:27.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f732552e700 1 -- 172.21.15.111:0/118554315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73200cab00 con 0x7f73200acf40 2023-11-28T12:59:27.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f73077fe700 1 -- 172.21.15.111:0/118554315 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731000f730 con 0x7f73200acf40 2023-11-28T12:59:27.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.723+0000 7f732552e700 1 -- 172.21.15.111:0/118554315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73200cc8d0 con 0x7f73200acf40 2023-11-28T12:59:27.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.724+0000 7f732552e700 1 -- 172.21.15.111:0/118554315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7320067880 con 0x7f73200acf40 2023-11-28T12:59:27.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.725+0000 7f73077fe700 1 -- 172.21.15.111:0/118554315 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7310025070 con 0x7f73200acf40 2023-11-28T12:59:27.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.725+0000 7f73077fe700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f730c0682f0 0x7f730c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:27.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.725+0000 7f73077fe700 1 -- 172.21.15.111:0/118554315 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f7310086360 con 0x7f73200acf40 2023-11-28T12:59:27.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.725+0000 7f731effd700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f730c0682f0 0x7f730c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:27.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.725+0000 7f731effd700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f730c0682f0 0x7f730c06a7a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7314000c00 tx=0x7f7314009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:27.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.729+0000 7f73077fe700 1 -- 172.21.15.111:0/118554315 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f73100528d0 con 0x7f73200acf40 2023-11-28T12:59:27.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.959+0000 7f732552e700 1 -- 172.21.15.111:0/118554315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f732015cf30 con 0x7f73200acf40 2023-11-28T12:59:27.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.959+0000 7f73077fe700 1 -- 172.21.15.111:0/118554315 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f7310019070 con 0x7f73200acf40 2023-11-28T12:59:27.960 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-28T12:59:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.961+0000 7f732552e700 1 -- 172.21.15.111:0/118554315 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f730c0682f0 msgr2=0x7f730c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.961+0000 7f732552e700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f730c0682f0 0x7f730c06a7a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7314000c00 tx=0x7f7314009030 comp rx=0 tx=0).stop 2023-11-28T12:59:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.961+0000 7f732552e700 1 -- 172.21.15.111:0/118554315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73200acf40 msgr2=0x7f7320157e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.961+0000 7f732552e700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73200acf40 0x7f7320157e60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f73100119c0 tx=0x7f7310011d80 comp rx=0 tx=0).stop 2023-11-28T12:59:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.961+0000 7f732552e700 1 -- 172.21.15.111:0/118554315 shutdown_connections 2023-11-28T12:59:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.961+0000 7f732552e700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73200709f0 0x7f7320070e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.961+0000 7f732552e700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f730c0682f0 0x7f730c06a7a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.961+0000 7f732552e700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7308003a30 0x7f7320159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.961+0000 7f732552e700 1 --2- 172.21.15.111:0/118554315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73200acf40 0x7f7320157e60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:27.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.961+0000 7f732552e700 1 -- 172.21.15.111:0/118554315 >> 172.21.15.111:0/118554315 conn(0x7f73200cb490 msgr2=0x7f7320158680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:27.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.961+0000 7f732552e700 1 -- 172.21.15.111:0/118554315 shutdown_connections 2023-11-28T12:59:27.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:27.961+0000 7f732552e700 1 -- 172.21.15.111:0/118554315 wait complete. 2023-11-28T12:59:27.973 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T12:59:27.973 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T12:59:27.974 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-11-28T12:59:27.974 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T12:59:27.974 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T12:59:27.974 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T12:59:27.974 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T12:59:27.974 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T12:59:27.974 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T12:59:27.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:27.974 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-11-28T12:59:27.991 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T12:59:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T12:59:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T12:59:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-11-28T12:59:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:27.992 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:27.992 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:27.992 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:27.992 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T12:59:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T12:59:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 42358sec preferred_lft 42358sec 2023-11-28T12:59:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::9195:85a0:8e3c:2da4/64 scope link noprefixroute 2023-11-28T12:59:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:27.993 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-28T12:59:27.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:27.993 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:27.993 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:27.993 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add name ceph-brx type bridge 2023-11-28T12:59:27.993 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr flush dev ceph-brx 2023-11-28T12:59:27.993 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set ceph-brx up 2023-11-28T12:59:27.993 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-28T12:59:27.993 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:28.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:28.107 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T12:59:28.132 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-11-28T12:59:28.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:28.133 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-11-28T12:59:28.192 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-11-28T12:59:28.193 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T12:59:28.193 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T12:59:28.193 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T12:59:28.193 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T12:59:28.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:28.193 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:28.193 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:28.193 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T12:59:28.193 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T12:59:28.194 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T12:59:28.194 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:28.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:28.298 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-11-28T12:59:28.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:28.315 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list-id 2023-11-28T12:59:28.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:28.372 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:28.373 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:28.373 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T12:59:28.373 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-11-28T12:59:28.373 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:28.465 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-11-28T12:59:28.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:28.466 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:28.466 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:28.466 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-11-28T12:59:28.466 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-28T12:59:28.467 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-11-28T12:59:28.467 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-11-28T12:59:28.467 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-11-28T12:59:28.467 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:28.529 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T12:59:28.531 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T12:59:28.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:28.674 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:28.674 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:28.674 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-11-28T12:59:28.674 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set dev brx.0 master ceph-brx 2023-11-28T12:59:28.675 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:28.794 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T12:59:28.795 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T12:59:28.821 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T12:59:28.821 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T12:59:28.821 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T12:59:28.878 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T12:59:28.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:28.879 DEBUG:teuthology.orchestra.run.smithi111:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T12:59:28.964 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T12:59:28.964 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T12:59:28.964 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T12:59:29.068 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T12:59:29.068 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T12:59:29.068 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T12:59:29.069 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T12:59:29.069 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T12:59:29.098 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T12:59:29.099 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T12:59:29.099 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T12:59:29.099 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T12:59:29.099 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T12:59:29.099 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790" 2023-11-28T12:59:29.099 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790:/ 2023-11-28T12:59:29.099 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=a1b2d00c-aeec-4b5d-93e3-41820c4189ba" will pass to kernel 2023-11-28T12:59:29.102 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T12:59:29.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:29.102 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T12:59:29.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:29.131 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T12:59:29.202 INFO:teuthology.orchestra.run.smithi111.stdout:client4839 2023-11-28T12:59:29.203 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:59:29.203 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:29.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:29.259 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T12:59:29.330 INFO:teuthology.orchestra.run.smithi111.stdout:client4839 2023-11-28T12:59:29.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:29.331 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T12:59:29.401 INFO:teuthology.orchestra.run.smithi111.stdout:client4839 2023-11-28T12:59:29.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:29.402 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a1b2d00c-aeec-4b5d-93e3-41820c4189ba.client4839/status') 2023-11-28T12:59:29.481 INFO:teuthology.orchestra.run.smithi111.stdout:instance: client.4839 (3)192.168.0.1:0/1266867580 2023-11-28T12:59:29.482 INFO:teuthology.orchestra.run.smithi111.stdout:blocklisted: false 2023-11-28T12:59:29.482 INFO:teuthology.orchestra.run.smithi111.stderr:0+1 records in 2023-11-28T12:59:29.483 INFO:teuthology.orchestra.run.smithi111.stderr:0+1 records out 2023-11-28T12:59:29.483 INFO:teuthology.orchestra.run.smithi111.stderr:69 bytes copied, 2.4552e-05 s, 2.8 MB/s 2023-11-28T12:59:29.483 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2023-11-28T12:59:29.483 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T12:59:29.483 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-28T12:59:29.483 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T12:59:29.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.644+0000 7fdefb4c9700 1 Processor -- start 2023-11-28T12:59:29.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.644+0000 7fdefb4c9700 1 -- start start 2023-11-28T12:59:29.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdefb4c9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdef4159520 0x7fdef40cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:29.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdefb4c9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdef40ae4a0 0x7fdef40cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:29.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdefb4c9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef40b1510 con 0x7fdef406c0a0 2023-11-28T12:59:29.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdefb4c9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef41598f0 con 0x7fdef40ae4a0 2023-11-28T12:59:29.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdefb4c9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef40b0f50 con 0x7fdef4159520 2023-11-28T12:59:29.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdef9a66700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdef40ae4a0 0x7fdef40cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:29.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdef8a64700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdef406c0a0 0x7fdef40af530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34558/0 (socket says 172.21.15.111:34558) 2023-11-28T12:59:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdef8a64700 1 -- 172.21.15.111:0/3371484440 learned_addr learned my addr 172.21.15.111:0/3371484440 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdef9265700 1 --2- 172.21.15.111:0/3371484440 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdef4159520 0x7fdef40cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdef9a66700 1 -- 172.21.15.111:0/3371484440 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdef4159520 msgr2=0x7fdef40cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdef9a66700 1 --2- 172.21.15.111:0/3371484440 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdef4159520 0x7fdef40cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdef9a66700 1 -- 172.21.15.111:0/3371484440 >> v1:172.21.15.111:6789/0 conn(0x7fdef406c0a0 legacy=0x7fdef40af530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdef9a66700 1 -- 172.21.15.111:0/3371484440 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef40b1230 con 0x7fdef40ae4a0 2023-11-28T12:59:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdeebfff700 1 -- 172.21.15.111:0/3371484440 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010855022 0 0) 0x7fdef40b1510 con 0x7fdef406c0a0 2023-11-28T12:59:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.645+0000 7fdef9a66700 1 --2- 172.21.15.111:0/3371484440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef40ae4a0 0x7fdef40cdce0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdeec009d90 tx=0x7fdeec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f56097d55c0a10 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.646+0000 7fdeebfff700 1 -- 172.21.15.111:0/3371484440 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec00e070 con 0x7fdef40ae4a0 2023-11-28T12:59:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.646+0000 7fdeebfff700 1 -- 172.21.15.111:0/3371484440 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec00bbd0 con 0x7fdef40ae4a0 2023-11-28T12:59:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.646+0000 7fdeebfff700 1 -- 172.21.15.111:0/3371484440 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec012430 con 0x7fdef40ae4a0 2023-11-28T12:59:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.646+0000 7fdefb4c9700 1 -- 172.21.15.111:0/3371484440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef40ae4a0 msgr2=0x7fdef40cdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:29.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.646+0000 7fdefb4c9700 1 --2- 172.21.15.111:0/3371484440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef40ae4a0 0x7fdef40cdce0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdeec009d90 tx=0x7fdeec00b330 comp rx=0 tx=0).stop 2023-11-28T12:59:29.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.646+0000 7fdefb4c9700 1 -- 172.21.15.111:0/3371484440 shutdown_connections 2023-11-28T12:59:29.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.646+0000 7fdefb4c9700 1 --2- 172.21.15.111:0/3371484440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef40ae4a0 0x7fdef40cdce0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:29.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.646+0000 7fdefb4c9700 1 --2- 172.21.15.111:0/3371484440 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdef4159520 0x7fdef40cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:29.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.646+0000 7fdefb4c9700 1 -- 172.21.15.111:0/3371484440 >> 172.21.15.111:0/3371484440 conn(0x7fdef406bca0 msgr2=0x7fdef40cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:29.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.646+0000 7fdefb4c9700 1 -- 172.21.15.111:0/3371484440 shutdown_connections 2023-11-28T12:59:29.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.646+0000 7fdefb4c9700 1 -- 172.21.15.111:0/3371484440 wait complete. 2023-11-28T12:59:29.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.647+0000 7fdefb4c9700 1 Processor -- start 2023-11-28T12:59:29.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.647+0000 7fdefb4c9700 1 -- start start 2023-11-28T12:59:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdefb4c9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdef406c0a0 0x7fdef41f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdefb4c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef40ae4a0 0x7fdef40c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdefb4c9700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdef4159520 0x7fdef406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdefb4c9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef40ccd20 con 0x7fdef40ae4a0 2023-11-28T12:59:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdefb4c9700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef41f1db0 con 0x7fdef4159520 2023-11-28T12:59:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdefb4c9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef41f6280 con 0x7fdef406c0a0 2023-11-28T12:59:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdef8a64700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef40ae4a0 0x7fdef40c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdef8a64700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef40ae4a0 0x7fdef40c8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33786/0 (socket says 172.21.15.111:33786) 2023-11-28T12:59:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdef8a64700 1 -- 172.21.15.111:0/2780830711 learned_addr learned my addr 172.21.15.111:0/2780830711 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdef9265700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdef406c0a0 0x7fdef41f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdef9a66700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdef4159520 0x7fdef406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdef8a64700 1 -- 172.21.15.111:0/2780830711 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdef406c0a0 msgr2=0x7fdef41f5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdef8a64700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdef406c0a0 0x7fdef41f5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdef8a64700 1 -- 172.21.15.111:0/2780830711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdef4159520 msgr2=0x7fdef406e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdef8a64700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdef4159520 0x7fdef406e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdef8a64700 1 -- 172.21.15.111:0/2780830711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeec009a70 con 0x7fdef40ae4a0 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdef9a66700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdef4159520 0x7fdef406e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.648+0000 7fdef9265700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdef406c0a0 0x7fdef41f5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.649+0000 7fdef8a64700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef40ae4a0 0x7fdef40c8380 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdee400c580 tx=0x7fdee400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.649+0000 7fdee9ffb700 1 -- 172.21.15.111:0/2780830711 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee4004c80 con 0x7fdef40ae4a0 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.649+0000 7fdee9ffb700 1 -- 172.21.15.111:0/2780830711 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee4004e00 con 0x7fdef40ae4a0 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.649+0000 7fdefb4c9700 1 -- 172.21.15.111:0/2780830711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef40acc90 con 0x7fdef40ae4a0 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.649+0000 7fdee9ffb700 1 -- 172.21.15.111:0/2780830711 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee4007310 con 0x7fdef40ae4a0 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.649+0000 7fdefb4c9700 1 -- 172.21.15.111:0/2780830711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef40ac9b0 con 0x7fdef40ae4a0 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.650+0000 7fdefb4c9700 1 -- 172.21.15.111:0/2780830711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef40acf70 con 0x7fdef40ae4a0 2023-11-28T12:59:29.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.650+0000 7fdee9ffb700 1 -- 172.21.15.111:0/2780830711 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdee401a020 con 0x7fdef40ae4a0 2023-11-28T12:59:29.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.651+0000 7fdee9ffb700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fded80682f0 0x7fded806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:29.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.651+0000 7fdee9ffb700 1 -- 172.21.15.111:0/2780830711 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fdee4085ac0 con 0x7fdef40ae4a0 2023-11-28T12:59:29.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.651+0000 7fdef9265700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fded80682f0 0x7fded806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:29.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.651+0000 7fdef9265700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fded80682f0 0x7fded806a7a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdef41ef930 tx=0x7fdef0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:29.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.655+0000 7fdee9ffb700 1 -- 172.21.15.111:0/2780830711 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdee4052030 con 0x7fdef40ae4a0 2023-11-28T12:59:29.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.882+0000 7fdefb4c9700 1 -- 172.21.15.111:0/2780830711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdef40cbc10 con 0x7fdef40ae4a0 2023-11-28T12:59:29.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.882+0000 7fdee9ffb700 1 -- 172.21.15.111:0/2780830711 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7fdee4055650 con 0x7fdef40ae4a0 2023-11-28T12:59:29.883 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-28T12:59:29.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.884+0000 7fdefb4c9700 1 -- 172.21.15.111:0/2780830711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fded80682f0 msgr2=0x7fded806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:29.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.884+0000 7fdefb4c9700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fded80682f0 0x7fded806a7a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdef41ef930 tx=0x7fdef0008040 comp rx=0 tx=0).stop 2023-11-28T12:59:29.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.884+0000 7fdefb4c9700 1 -- 172.21.15.111:0/2780830711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef40ae4a0 msgr2=0x7fdef40c8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:29.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.884+0000 7fdefb4c9700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef40ae4a0 0x7fdef40c8380 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdee400c580 tx=0x7fdee400c940 comp rx=0 tx=0).stop 2023-11-28T12:59:29.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.884+0000 7fdefb4c9700 1 -- 172.21.15.111:0/2780830711 shutdown_connections 2023-11-28T12:59:29.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.884+0000 7fdefb4c9700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fded80682f0 0x7fded806a7a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:29.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.884+0000 7fdefb4c9700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdef4159520 0x7fdef406e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:29.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.884+0000 7fdefb4c9700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdef406c0a0 0x7fdef41f5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:29.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.884+0000 7fdefb4c9700 1 --2- 172.21.15.111:0/2780830711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef40ae4a0 0x7fdef40c8380 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:29.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.884+0000 7fdefb4c9700 1 -- 172.21.15.111:0/2780830711 >> 172.21.15.111:0/2780830711 conn(0x7fdef406bca0 msgr2=0x7fdef40b1450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:29.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.884+0000 7fdefb4c9700 1 -- 172.21.15.111:0/2780830711 shutdown_connections 2023-11-28T12:59:29.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:29.884+0000 7fdefb4c9700 1 -- 172.21.15.111:0/2780830711 wait complete. 2023-11-28T12:59:29.896 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T12:59:29.896 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T12:59:29.896 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-11-28T12:59:29.896 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-11-28T12:59:29.896 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-28T12:59:29.896 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T12:59:29.897 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T12:59:29.897 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T12:59:29.897 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-11-28T12:59:29.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:29.897 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-11-28T12:59:29.914 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T12:59:29.914 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T12:59:29.914 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T12:59:29.914 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:29.914 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-11-28T12:59:29.914 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:29.914 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:29.914 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:29.914 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:29.915 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:29.915 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:29.915 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:29.915 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T12:59:29.915 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:29.915 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T12:59:29.915 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 42356sec preferred_lft 42356sec 2023-11-28T12:59:29.915 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::9195:85a0:8e3c:2da4/64 scope link noprefixroute 2023-11-28T12:59:29.915 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:29.915 INFO:teuthology.orchestra.run.smithi111.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T12:59:29.915 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 96:9a:a2:c9:67:8f brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:29.915 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T12:59:29.915 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:29.916 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::f893:44ff:fe17:3abf/64 scope link 2023-11-28T12:59:29.916 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:29.916 INFO:teuthology.orchestra.run.smithi111.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T12:59:29.916 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 96:9a:a2:c9:67:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T12:59:29.916 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::949a:a2ff:fec9:678f/64 scope link 2023-11-28T12:59:29.916 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:29.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:29.916 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T12:59:29.979 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-11-28T12:59:29.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:29.981 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-11-28T12:59:30.040 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-11-28T12:59:30.040 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T12:59:30.040 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T12:59:30.040 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T12:59:30.040 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T12:59:30.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:30.041 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:30.041 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:30.041 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T12:59:30.041 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T12:59:30.041 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T12:59:30.041 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:30.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:30.144 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-11-28T12:59:30.161 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T12:59:30.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:30.161 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list-id 2023-11-28T12:59:30.218 INFO:teuthology.orchestra.run.smithi111.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-11-28T12:59:30.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:30.218 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:30.218 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:30.218 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T12:59:30.219 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-11-28T12:59:30.219 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:30.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:30.311 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-28T12:59:30.348 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T12:59:30.348 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T12:59:30.348 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T12:59:30.348 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:30.349 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-11-28T12:59:30.349 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:30.349 INFO:teuthology.orchestra.run.smithi111.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T12:59:30.349 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 2e:4b:6f:53:6c:bb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-28T12:59:30.349 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-28T12:59:30.349 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:30.349 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::2c4b:6fff:fe53:6cbb/64 scope link 2023-11-28T12:59:30.349 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:30.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:30.365 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-28T12:59:30.451 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T12:59:30.451 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T12:59:30.451 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T12:59:30.452 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:30.452 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-11-28T12:59:30.452 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:30.452 INFO:teuthology.orchestra.run.smithi111.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T12:59:30.452 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 2e:4b:6f:53:6c:bb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-28T12:59:30.452 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-28T12:59:30.452 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:30.452 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::2c4b:6fff:fe53:6cbb/64 scope link 2023-11-28T12:59:30.452 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:30.465 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-11-28T12:59:30.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:30.465 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:30.465 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:30.465 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-11-28T12:59:30.465 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-28T12:59:30.465 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-11-28T12:59:30.465 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-11-28T12:59:30.465 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-11-28T12:59:30.465 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:30.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:30.671 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T12:59:30.671 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T12:59:30.671 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.1 up 2023-11-28T12:59:30.671 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set dev brx.1 master ceph-brx 2023-11-28T12:59:30.671 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T12:59:30.788 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T12:59:30.788 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-11-28T12:59:30.812 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-11-28T12:59:30.813 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T12:59:30.813 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-11-28T12:59:30.870 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T12:59:30.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:30.870 DEBUG:teuthology.orchestra.run.smithi111:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-11-28T12:59:30.955 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-11-28T12:59:30.955 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-11-28T12:59:30.955 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T12:59:30.964 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-11-28T12:59:30.964 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-11-28T12:59:30.964 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T12:59:30.964 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T12:59:30.964 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T12:59:30.996 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-11-28T12:59:30.996 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-11-28T12:59:30.996 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T12:59:30.996 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T12:59:30.996 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T12:59:30.996 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790" 2023-11-28T12:59:30.996 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790:/ 2023-11-28T12:59:30.997 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=a1b2d00c-aeec-4b5d-93e3-41820c4189ba" will pass to kernel 2023-11-28T12:59:31.000 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T12:59:31.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:31.001 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-11-28T12:59:31.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:31.033 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T12:59:31.104 INFO:teuthology.orchestra.run.smithi111.stdout:client4851 2023-11-28T12:59:31.104 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:59:31.104 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:31.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:31.162 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T12:59:31.238 INFO:teuthology.orchestra.run.smithi111.stdout:client4851 2023-11-28T12:59:31.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:31.239 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T12:59:31.310 INFO:teuthology.orchestra.run.smithi111.stdout:client4851 2023-11-28T12:59:31.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:31.310 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a1b2d00c-aeec-4b5d-93e3-41820c4189ba.client4851/status') 2023-11-28T12:59:31.390 INFO:teuthology.orchestra.run.smithi111.stdout:instance: client.4851 (3)192.168.0.2:0/4278863345 2023-11-28T12:59:31.391 INFO:teuthology.orchestra.run.smithi111.stdout:blocklisted: false 2023-11-28T12:59:31.391 INFO:teuthology.orchestra.run.smithi111.stderr:0+1 records in 2023-11-28T12:59:31.391 INFO:teuthology.orchestra.run.smithi111.stderr:0+1 records out 2023-11-28T12:59:31.391 INFO:teuthology.orchestra.run.smithi111.stderr:69 bytes copied, 2.2098e-05 s, 3.1 MB/s 2023-11-28T12:59:31.392 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2023-11-28T12:59:31.392 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T12:59:31.392 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-28T12:59:31.392 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T12:59:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.552+0000 7f401ea27700 1 Processor -- start 2023-11-28T12:59:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.552+0000 7f401ea27700 1 -- start start 2023-11-28T12:59:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.553+0000 7f401ea27700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f401815a6a0 0x7f401806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.553+0000 7f401ea27700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401815aa70 con 0x7f40180af160 2023-11-28T12:59:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.553+0000 7f401ea27700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40180ad280 con 0x7f40180ae720 2023-11-28T12:59:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.553+0000 7f401ea27700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40180ac950 con 0x7f401815a6a0 2023-11-28T12:59:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.553+0000 7f40177fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f40180af160 0x7f40180cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34576/0 (socket says 172.21.15.111:34576) 2023-11-28T12:59:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.553+0000 7f40177fe700 1 -- 172.21.15.111:0/1611002876 learned_addr learned my addr 172.21.15.111:0/1611002876 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.553+0000 7f4017fff700 1 --2- 172.21.15.111:0/1611002876 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f401815a6a0 0x7f401806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.553+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642209120 0 0) 0x7f4008001110 con 0x7f40180af160 2023-11-28T12:59:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.553+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40000035b0 con 0x7f40180af160 2023-11-28T12:59:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350256396 0 0) 0x7f40180ad280 con 0x7f40180ae720 2023-11-28T12:59:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4008001110 con 0x7f40180ae720 2023-11-28T12:59:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3920293199 0 0) 0x7f40000035b0 con 0x7f40180af160 2023-11-28T12:59:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f401815a6a0 msgr2=0x7f401806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 --2- 172.21.15.111:0/1611002876 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f401815a6a0 0x7f401806a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 >> v1:172.21.15.149:6789/0 conn(0x7f40180ae720 legacy=0x7f40180d0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40180acc30 con 0x7f40180af160 2023-11-28T12:59:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f4008001800 con 0x7f40180af160 2023-11-28T12:59:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 >> v1:172.21.15.111:6789/0 conn(0x7f40180af160 legacy=0x7f40180cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 --2- 172.21.15.111:0/1611002876 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f401815a6a0 0x7f4000003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 --2- 172.21.15.111:0/1611002876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4000004170 0x7f4000004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 --2- 172.21.15.111:0/1611002876 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4000004d90 0x7f4000005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40180ac950 con 0x7f4000004170 2023-11-28T12:59:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400000bba0 con 0x7f4000004d90 2023-11-28T12:59:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400000bd60 con 0x7f401815a6a0 2023-11-28T12:59:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f40177fe700 1 --2- 172.21.15.111:0/1611002876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4000004170 0x7f4000004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4017fff700 1 --2- 172.21.15.111:0/1611002876 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f401815a6a0 0x7f4000003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f401cfc4700 1 --2- 172.21.15.111:0/1611002876 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4000004d90 0x7f4000005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f40177fe700 1 -- 172.21.15.111:0/1611002876 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f401815a6a0 msgr2=0x7f4000003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f40177fe700 1 --2- 172.21.15.111:0/1611002876 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f401815a6a0 0x7f4000003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f40177fe700 1 -- 172.21.15.111:0/1611002876 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4000004d90 msgr2=0x7f4000005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f40177fe700 1 --2- 172.21.15.111:0/1611002876 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4000004d90 0x7f4000005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f40177fe700 1 -- 172.21.15.111:0/1611002876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40000035b0 con 0x7f4000004170 2023-11-28T12:59:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f401cfc4700 1 --2- 172.21.15.111:0/1611002876 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4000004d90 0x7f4000005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f40177fe700 1 --2- 172.21.15.111:0/1611002876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4000004170 0x7f4000004850 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f400800c110 tx=0x7f4000005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f42e9b18711cb3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:31.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400800cc50 con 0x7f4000004170 2023-11-28T12:59:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400800cdd0 con 0x7f4000004170 2023-11-28T12:59:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.554+0000 7f4016ffd700 1 -- 172.21.15.111:0/1611002876 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008005ba0 con 0x7f4000004170 2023-11-28T12:59:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.555+0000 7f401ea27700 1 -- 172.21.15.111:0/1611002876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4000004170 msgr2=0x7f4000004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.555+0000 7f401ea27700 1 --2- 172.21.15.111:0/1611002876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4000004170 0x7f4000004850 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f400800c110 tx=0x7f4000005880 comp rx=0 tx=0).stop 2023-11-28T12:59:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.555+0000 7f401cfc4700 1 -- 172.21.15.111:0/1611002876 reap_dead start 2023-11-28T12:59:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.555+0000 7f401ea27700 1 -- 172.21.15.111:0/1611002876 shutdown_connections 2023-11-28T12:59:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.555+0000 7f401ea27700 1 -- 172.21.15.111:0/1611002876 >> 172.21.15.111:0/1611002876 conn(0x7f4018098ee0 msgr2=0x7f401806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.555+0000 7f401ea27700 1 -- 172.21.15.111:0/1611002876 shutdown_connections 2023-11-28T12:59:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.555+0000 7f401ea27700 1 -- 172.21.15.111:0/1611002876 wait complete. 2023-11-28T12:59:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.556+0000 7f401ea27700 1 Processor -- start 2023-11-28T12:59:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.556+0000 7f401ea27700 1 -- start start 2023-11-28T12:59:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f401ea27700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4000004170 0x7f40180b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f401ea27700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f40180cc490 0x7f401815bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f401ea27700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40180cc900 0x7f401815c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f401ea27700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40180cb1a0 con 0x7f4000004170 2023-11-28T12:59:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f401ea27700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40180cb3f0 con 0x7f40180cc900 2023-11-28T12:59:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f401ea27700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40180cb6d0 con 0x7f40180cc490 2023-11-28T12:59:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f4017fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4000004170 0x7f40180b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f4017fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4000004170 0x7f40180b3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33808/0 (socket says 172.21.15.111:33808) 2023-11-28T12:59:31.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f4017fff700 1 -- 172.21.15.111:0/4102314823 learned_addr learned my addr 172.21.15.111:0/4102314823 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f401cfc4700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40180cc900 0x7f401815c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f40177fe700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f40180cc490 0x7f401815bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f4017fff700 1 -- 172.21.15.111:0/4102314823 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f40180cc490 msgr2=0x7f401815bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f4017fff700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f40180cc490 0x7f401815bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f4017fff700 1 -- 172.21.15.111:0/4102314823 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40180cc900 msgr2=0x7f401815c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.557+0000 7f4017fff700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40180cc900 0x7f401815c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.558+0000 7f4017fff700 1 -- 172.21.15.111:0/4102314823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40180d0950 con 0x7f4000004170 2023-11-28T12:59:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.558+0000 7f40177fe700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f40180cc490 0x7f401815bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.558+0000 7f401cfc4700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40180cc900 0x7f401815c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.558+0000 7f4017fff700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4000004170 0x7f40180b3880 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f400c011aa0 tx=0x7f400c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.558+0000 7f4014ff9700 1 -- 172.21.15.111:0/4102314823 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00f980 con 0x7f4000004170 2023-11-28T12:59:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.558+0000 7f4014ff9700 1 -- 172.21.15.111:0/4102314823 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400c00fae0 con 0x7f4000004170 2023-11-28T12:59:31.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.558+0000 7f401ea27700 1 -- 172.21.15.111:0/4102314823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40180acf10 con 0x7f4000004170 2023-11-28T12:59:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.558+0000 7f4014ff9700 1 -- 172.21.15.111:0/4102314823 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00fe90 con 0x7f4000004170 2023-11-28T12:59:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.558+0000 7f401ea27700 1 -- 172.21.15.111:0/4102314823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40180992e0 con 0x7f4000004170 2023-11-28T12:59:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.559+0000 7f401ea27700 1 -- 172.21.15.111:0/4102314823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4018067170 con 0x7f4000004170 2023-11-28T12:59:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.559+0000 7f4014ff9700 1 -- 172.21.15.111:0/4102314823 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f400c014440 con 0x7f4000004170 2023-11-28T12:59:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.560+0000 7f4014ff9700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f40040682f0 0x7f400406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.560+0000 7f4014ff9700 1 -- 172.21.15.111:0/4102314823 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f400c086780 con 0x7f4000004170 2023-11-28T12:59:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.560+0000 7f40177fe700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f40040682f0 0x7f400406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.560+0000 7f40177fe700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f40040682f0 0x7f400406a7a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f40180cceb0 tx=0x7f4008007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:31.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.564+0000 7f4014ff9700 1 -- 172.21.15.111:0/4102314823 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f400c052d50 con 0x7f4000004170 2023-11-28T12:59:31.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.793+0000 7f401ea27700 1 -- 172.21.15.111:0/4102314823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f401806e730 con 0x7f4000004170 2023-11-28T12:59:31.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.793+0000 7f4014ff9700 1 -- 172.21.15.111:0/4102314823 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f400c056370 con 0x7f4000004170 2023-11-28T12:59:31.794 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-28T12:59:31.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.795+0000 7f401ea27700 1 -- 172.21.15.111:0/4102314823 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f40040682f0 msgr2=0x7f400406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:31.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.795+0000 7f401ea27700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f40040682f0 0x7f400406a7a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f40180cceb0 tx=0x7f4008007000 comp rx=0 tx=0).stop 2023-11-28T12:59:31.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.795+0000 7f401ea27700 1 -- 172.21.15.111:0/4102314823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4000004170 msgr2=0x7f40180b3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:31.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.795+0000 7f401ea27700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4000004170 0x7f40180b3880 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f400c011aa0 tx=0x7f400c011e60 comp rx=0 tx=0).stop 2023-11-28T12:59:31.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.795+0000 7f401ea27700 1 -- 172.21.15.111:0/4102314823 shutdown_connections 2023-11-28T12:59:31.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.795+0000 7f401ea27700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f40040682f0 0x7f400406a7a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:31.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.795+0000 7f401ea27700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40180cc900 0x7f401815c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:31.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.795+0000 7f401ea27700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4000004170 0x7f40180b3880 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:31.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.795+0000 7f401ea27700 1 --2- 172.21.15.111:0/4102314823 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f40180cc490 0x7f401815bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:31.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.795+0000 7f401ea27700 1 -- 172.21.15.111:0/4102314823 >> 172.21.15.111:0/4102314823 conn(0x7f4018098ee0 msgr2=0x7f4018158b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:31.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.796+0000 7f401ea27700 1 -- 172.21.15.111:0/4102314823 shutdown_connections 2023-11-28T12:59:31.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:31.796+0000 7f401ea27700 1 -- 172.21.15.111:0/4102314823 wait complete. 2023-11-28T12:59:31.808 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T12:59:31.808 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T12:59:31.808 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi149.front.sepia.ceph.com 2023-11-28T12:59:31.808 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-28T12:59:31.808 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-28T12:59:31.808 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T12:59:31.808 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T12:59:31.808 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T12:59:31.808 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-28T12:59:31.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:31.809 DEBUG:teuthology.orchestra.run.smithi149:> ip addr 2023-11-28T12:59:31.830 INFO:teuthology.orchestra.run.smithi149.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T12:59:31.830 INFO:teuthology.orchestra.run.smithi149.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T12:59:31.830 INFO:teuthology.orchestra.run.smithi149.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T12:59:31.831 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:31.831 INFO:teuthology.orchestra.run.smithi149.stdout: inet6 ::1/128 scope host 2023-11-28T12:59:31.831 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:31.831 INFO:teuthology.orchestra.run.smithi149.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:31.831 INFO:teuthology.orchestra.run.smithi149.stdout: link/ether 0c:c4:7a:da:6b:a4 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:31.831 INFO:teuthology.orchestra.run.smithi149.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:31.831 INFO:teuthology.orchestra.run.smithi149.stdout: link/ether 0c:c4:7a:da:6b:a5 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:31.831 INFO:teuthology.orchestra.run.smithi149.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:31.831 INFO:teuthology.orchestra.run.smithi149.stdout: link/ether 0c:c4:7a:88:7d:56 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:31.831 INFO:teuthology.orchestra.run.smithi149.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T12:59:31.831 INFO:teuthology.orchestra.run.smithi149.stdout: link/ether 0c:c4:7a:88:7d:57 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:31.831 INFO:teuthology.orchestra.run.smithi149.stdout: inet 172.21.15.149/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T12:59:31.832 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft 42342sec preferred_lft 42342sec 2023-11-28T12:59:31.832 INFO:teuthology.orchestra.run.smithi149.stdout: inet6 fe80::709:bc8:9b22:1807/64 scope link noprefixroute 2023-11-28T12:59:31.832 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:31.832 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-28T12:59:31.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:31.833 DEBUG:teuthology.orchestra.run.smithi149:> (cd / && exec bash -c ' 2023-11-28T12:59:31.833 DEBUG:teuthology.orchestra.run.smithi149:> set -e 2023-11-28T12:59:31.833 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip link add name ceph-brx type bridge 2023-11-28T12:59:31.833 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip addr flush dev ceph-brx 2023-11-28T12:59:31.833 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip link set ceph-brx up 2023-11-28T12:59:31.833 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-28T12:59:31.834 DEBUG:teuthology.orchestra.run.smithi149:> ') 2023-11-28T12:59:32.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:32.004 DEBUG:teuthology.orchestra.run.smithi149:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T12:59:32.032 INFO:teuthology.orchestra.run.smithi149.stdout:1 2023-11-28T12:59:32.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:32.034 DEBUG:teuthology.orchestra.run.smithi149:> route 2023-11-28T12:59:32.110 INFO:teuthology.orchestra.run.smithi149.stdout:Kernel IP routing table 2023-11-28T12:59:32.110 INFO:teuthology.orchestra.run.smithi149.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T12:59:32.110 INFO:teuthology.orchestra.run.smithi149.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T12:59:32.110 INFO:teuthology.orchestra.run.smithi149.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T12:59:32.111 INFO:teuthology.orchestra.run.smithi149.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T12:59:32.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:32.111 DEBUG:teuthology.orchestra.run.smithi149:> (cd / && exec bash -c ' 2023-11-28T12:59:32.111 DEBUG:teuthology.orchestra.run.smithi149:> set -e 2023-11-28T12:59:32.111 DEBUG:teuthology.orchestra.run.smithi149:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T12:59:32.111 DEBUG:teuthology.orchestra.run.smithi149:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T12:59:32.111 DEBUG:teuthology.orchestra.run.smithi149:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T12:59:32.111 DEBUG:teuthology.orchestra.run.smithi149:> ') 2023-11-28T12:59:32.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:32.358 DEBUG:teuthology.orchestra.run.smithi149:> ip netns list 2023-11-28T12:59:32.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:32.375 DEBUG:teuthology.orchestra.run.smithi149:> ip netns list-id 2023-11-28T12:59:32.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:32.432 DEBUG:teuthology.orchestra.run.smithi149:> (cd / && exec bash -c ' 2023-11-28T12:59:32.432 DEBUG:teuthology.orchestra.run.smithi149:> set -e 2023-11-28T12:59:32.433 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T12:59:32.433 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-11-28T12:59:32.433 DEBUG:teuthology.orchestra.run.smithi149:> ') 2023-11-28T12:59:32.527 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-11-28T12:59:32.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:32.527 DEBUG:teuthology.orchestra.run.smithi149:> (cd / && exec bash -c ' 2023-11-28T12:59:32.527 DEBUG:teuthology.orchestra.run.smithi149:> set -e 2023-11-28T12:59:32.528 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-11-28T12:59:32.528 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-28T12:59:32.528 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-11-28T12:59:32.528 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-11-28T12:59:32.528 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-11-28T12:59:32.528 DEBUG:teuthology.orchestra.run.smithi149:> ') 2023-11-28T12:59:32.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:32.736 DEBUG:teuthology.orchestra.run.smithi149:> (cd / && exec bash -c ' 2023-11-28T12:59:32.736 DEBUG:teuthology.orchestra.run.smithi149:> set -e 2023-11-28T12:59:32.737 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip link set brx.0 up 2023-11-28T12:59:32.737 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip link set dev brx.0 master ceph-brx 2023-11-28T12:59:32.737 DEBUG:teuthology.orchestra.run.smithi149:> ') 2023-11-28T12:59:32.842 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T12:59:32.843 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-28T12:59:32.861 INFO:teuthology.orchestra.run.smithi149.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-28T12:59:32.861 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T12:59:32.862 DEBUG:teuthology.orchestra.run.smithi149:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-28T12:59:32.918 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T12:59:32.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:32.918 DEBUG:teuthology.orchestra.run.smithi149:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-28T12:59:33.006 INFO:teuthology.orchestra.run.smithi149.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-28T12:59:33.007 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-28T12:59:33.007 INFO:teuthology.orchestra.run.smithi149.stdout:invalid new device string format 2023-11-28T12:59:33.100 INFO:teuthology.orchestra.run.smithi149.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-28T12:59:33.100 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-28T12:59:33.101 INFO:teuthology.orchestra.run.smithi149.stdout:invalid new device string format 2023-11-28T12:59:33.101 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T12:59:33.101 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T12:59:33.126 INFO:teuthology.orchestra.run.smithi149.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-11-28T12:59:33.127 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-11-28T12:59:33.127 INFO:teuthology.orchestra.run.smithi149.stdout:invalid new device string format 2023-11-28T12:59:33.127 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T12:59:33.127 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T12:59:33.127 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: resolved to: "172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790" 2023-11-28T12:59:33.127 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790:/ 2023-11-28T12:59:33.127 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=a1b2d00c-aeec-4b5d-93e3-41820c4189ba" will pass to kernel 2023-11-28T12:59:33.131 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T12:59:33.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:33.131 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-28T12:59:33.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:33.162 DEBUG:teuthology.orchestra.run.smithi149:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T12:59:33.233 INFO:teuthology.orchestra.run.smithi149.stdout:client4705 2023-11-28T12:59:33.233 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:59:33.233 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:33.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:33.292 DEBUG:teuthology.orchestra.run.smithi149:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T12:59:33.365 INFO:teuthology.orchestra.run.smithi149.stdout:client4705 2023-11-28T12:59:33.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:33.365 DEBUG:teuthology.orchestra.run.smithi149:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T12:59:33.439 INFO:teuthology.orchestra.run.smithi149.stdout:client4705 2023-11-28T12:59:33.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:33.440 DEBUG:teuthology.orchestra.run.smithi149:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a1b2d00c-aeec-4b5d-93e3-41820c4189ba.client4705/status') 2023-11-28T12:59:33.520 INFO:teuthology.orchestra.run.smithi149.stdout:instance: client.4705 (3)192.168.0.1:0/916682850 2023-11-28T12:59:33.520 INFO:teuthology.orchestra.run.smithi149.stdout:blocklisted: false 2023-11-28T12:59:33.520 INFO:teuthology.orchestra.run.smithi149.stderr:0+1 records in 2023-11-28T12:59:33.520 INFO:teuthology.orchestra.run.smithi149.stderr:0+1 records out 2023-11-28T12:59:33.520 INFO:teuthology.orchestra.run.smithi149.stderr:68 bytes copied, 2.5182e-05 s, 2.7 MB/s 2023-11-28T12:59:33.522 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2023-11-28T12:59:33.522 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T12:59:33.522 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-28T12:59:33.522 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T12:59:33.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.655+0000 7ff9e6f6b700 1 Processor -- start 2023-11-28T12:59:33.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.655+0000 7ff9e6f6b700 1 -- start start 2023-11-28T12:59:33.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e6f6b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff9e015c7b0 0x7ff9e00cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:33.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e6f6b700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff9e006b380 0x7ff9e006b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:33.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e6f6b700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e00cb810 con 0x7ff9e00d0490 2023-11-28T12:59:33.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e6f6b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e00cc650 con 0x7ff9e015c7b0 2023-11-28T12:59:33.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e6f6b700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e00cba40 con 0x7ff9e006b380 2023-11-28T12:59:33.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e4d07700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff9e015c7b0 0x7ff9e00cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:33.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e4d07700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff9e015c7b0 0x7ff9e00cb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33816/0 (socket says 172.21.15.111:33816) 2023-11-28T12:59:33.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e4d07700 1 -- 172.21.15.111:0/2890905290 learned_addr learned my addr 172.21.15.111:0/2890905290 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:33.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e5508700 1 --2- 172.21.15.111:0/2890905290 >> v2:172.21.15.149:3300/0 conn(0x7ff9e006b380 0x7ff9e006b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:33.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e4d07700 1 -- 172.21.15.111:0/2890905290 >> v2:172.21.15.149:3300/0 conn(0x7ff9e006b380 msgr2=0x7ff9e006b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:33.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e4d07700 1 --2- 172.21.15.111:0/2890905290 >> v2:172.21.15.149:3300/0 conn(0x7ff9e006b380 0x7ff9e006b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:33.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e4d07700 1 -- 172.21.15.111:0/2890905290 >> v1:172.21.15.149:6789/0 conn(0x7ff9e00d0490 legacy=0x7ff9e00ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:33.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e4d07700 1 -- 172.21.15.111:0/2890905290 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e006bd20 con 0x7ff9e015c7b0 2023-11-28T12:59:33.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e5508700 1 --2- 172.21.15.111:0/2890905290 >> v2:172.21.15.149:3300/0 conn(0x7ff9e006b380 0x7ff9e006b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:33.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9e4d07700 1 --2- 172.21.15.111:0/2890905290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e015c7b0 0x7ff9e00cb2d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00cb260 tx=0x7ff9d4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=138841115eaf193f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:33.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9df7fe700 1 -- 172.21.15.111:0/2890905290 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4005e30 con 0x7ff9e015c7b0 2023-11-28T12:59:33.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9df7fe700 1 -- 172.21.15.111:0/2890905290 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d4004030 con 0x7ff9e015c7b0 2023-11-28T12:59:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.656+0000 7ff9df7fe700 1 -- 172.21.15.111:0/2890905290 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d40043e0 con 0x7ff9e015c7b0 2023-11-28T12:59:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.657+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/2890905290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e015c7b0 msgr2=0x7ff9e00cb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.657+0000 7ff9e6f6b700 1 --2- 172.21.15.111:0/2890905290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e015c7b0 0x7ff9e00cb2d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00cb260 tx=0x7ff9d4009bd0 comp rx=0 tx=0).stop 2023-11-28T12:59:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.657+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/2890905290 shutdown_connections 2023-11-28T12:59:33.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.657+0000 7ff9e6f6b700 1 --2- 172.21.15.111:0/2890905290 >> v2:172.21.15.149:3300/0 conn(0x7ff9e006b380 0x7ff9e006b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:33.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.657+0000 7ff9e6f6b700 1 --2- 172.21.15.111:0/2890905290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e015c7b0 0x7ff9e00cb2d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:33.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.657+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/2890905290 >> 172.21.15.111:0/2890905290 conn(0x7ff9e009a040 msgr2=0x7ff9e00d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:33.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.657+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/2890905290 shutdown_connections 2023-11-28T12:59:33.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.657+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/2890905290 wait complete. 2023-11-28T12:59:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.658+0000 7ff9e6f6b700 1 Processor -- start 2023-11-28T12:59:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.658+0000 7ff9e6f6b700 1 -- start start 2023-11-28T12:59:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.659+0000 7ff9e6f6b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9e006b380 0x7ff9e0158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.659+0000 7ff9e6f6b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9e00d0490 0x7ff9e00c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.659+0000 7ff9e6f6b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e015c7b0 0x7ff9e0158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.659+0000 7ff9e6f6b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00c6f20 con 0x7ff9e015c7b0 2023-11-28T12:59:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.659+0000 7ff9e6f6b700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00b4ff0 con 0x7ff9e00d0490 2023-11-28T12:59:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.659+0000 7ff9e6f6b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e00b5960 con 0x7ff9e006b380 2023-11-28T12:59:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.659+0000 7ff9e5508700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e015c7b0 0x7ff9e0158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.659+0000 7ff9e5508700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e015c7b0 0x7ff9e0158640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33826/0 (socket says 172.21.15.111:33826) 2023-11-28T12:59:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.659+0000 7ff9e5508700 1 -- 172.21.15.111:0/242850990 learned_addr learned my addr 172.21.15.111:0/242850990 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.659+0000 7ff9dffff700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9e00d0490 0x7ff9e00c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.659+0000 7ff9e4d07700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9e006b380 0x7ff9e0158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.660+0000 7ff9e5508700 1 -- 172.21.15.111:0/242850990 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9e006b380 msgr2=0x7ff9e0158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.660+0000 7ff9e5508700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9e006b380 0x7ff9e0158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.660+0000 7ff9e5508700 1 -- 172.21.15.111:0/242850990 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9e00d0490 msgr2=0x7ff9e00c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.660+0000 7ff9e5508700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9e00d0490 0x7ff9e00c73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.660+0000 7ff9e5508700 1 -- 172.21.15.111:0/242850990 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d4009500 con 0x7ff9e015c7b0 2023-11-28T12:59:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.660+0000 7ff9e5508700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e015c7b0 0x7ff9e0158640 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8007e20 tx=0x7ff9d800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.660+0000 7ff9dd7fa700 1 -- 172.21.15.111:0/242850990 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d800dd20 con 0x7ff9e015c7b0 2023-11-28T12:59:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.660+0000 7ff9dd7fa700 1 -- 172.21.15.111:0/242850990 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d800f040 con 0x7ff9e015c7b0 2023-11-28T12:59:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.660+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/242850990 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e00cc000 con 0x7ff9e015c7b0 2023-11-28T12:59:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.660+0000 7ff9dd7fa700 1 -- 172.21.15.111:0/242850990 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8003c60 con 0x7ff9e015c7b0 2023-11-28T12:59:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.660+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/242850990 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e00cbd20 con 0x7ff9e015c7b0 2023-11-28T12:59:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.661+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/242850990 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e00cc2e0 con 0x7ff9e015c7b0 2023-11-28T12:59:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.661+0000 7ff9dd7fa700 1 -- 172.21.15.111:0/242850990 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff9d801c030 con 0x7ff9e015c7b0 2023-11-28T12:59:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.662+0000 7ff9dd7fa700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9c40682f0 0x7ff9c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.662+0000 7ff9e4d07700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9c40682f0 0x7ff9c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.662+0000 7ff9dd7fa700 1 -- 172.21.15.111:0/242850990 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff9d8085e60 con 0x7ff9e015c7b0 2023-11-28T12:59:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.662+0000 7ff9e4d07700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9c40682f0 0x7ff9c406a7a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00cb260 tx=0x7ff9d4008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.666+0000 7ff9dd7fa700 1 -- 172.21.15.111:0/242850990 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff9d80523d0 con 0x7ff9e015c7b0 2023-11-28T12:59:33.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.895+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/242850990 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff9e0067170 con 0x7ff9e015c7b0 2023-11-28T12:59:33.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.895+0000 7ff9dd7fa700 1 -- 172.21.15.111:0/242850990 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7ff9d801a070 con 0x7ff9e015c7b0 2023-11-28T12:59:33.896 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-28T12:59:33.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.897+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/242850990 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9c40682f0 msgr2=0x7ff9c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:33.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.897+0000 7ff9e6f6b700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9c40682f0 0x7ff9c406a7a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00cb260 tx=0x7ff9d4008070 comp rx=0 tx=0).stop 2023-11-28T12:59:33.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.897+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/242850990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e015c7b0 msgr2=0x7ff9e0158640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:33.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.897+0000 7ff9e6f6b700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e015c7b0 0x7ff9e0158640 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8007e20 tx=0x7ff9d800d3b0 comp rx=0 tx=0).stop 2023-11-28T12:59:33.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.897+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/242850990 shutdown_connections 2023-11-28T12:59:33.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.897+0000 7ff9e6f6b700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9e015c7b0 0x7ff9e0158640 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:33.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.897+0000 7ff9e6f6b700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9e006b380 0x7ff9e0158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:33.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.897+0000 7ff9e6f6b700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9c40682f0 0x7ff9c406a7a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:33.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.897+0000 7ff9e6f6b700 1 --2- 172.21.15.111:0/242850990 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9e00d0490 0x7ff9e00c73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:33.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.897+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/242850990 >> 172.21.15.111:0/242850990 conn(0x7ff9e009a040 msgr2=0x7ff9e0070650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:33.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.897+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/242850990 shutdown_connections 2023-11-28T12:59:33.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:33.897+0000 7ff9e6f6b700 1 -- 172.21.15.111:0/242850990 wait complete. 2023-11-28T12:59:33.909 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T12:59:33.910 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T12:59:33.910 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi149.front.sepia.ceph.com 2023-11-28T12:59:33.910 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-11-28T12:59:33.910 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-28T12:59:33.910 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T12:59:33.911 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T12:59:33.911 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T12:59:33.911 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-11-28T12:59:33.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:33.911 DEBUG:teuthology.orchestra.run.smithi149:> ip addr 2023-11-28T12:59:33.929 INFO:teuthology.orchestra.run.smithi149.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T12:59:33.930 INFO:teuthology.orchestra.run.smithi149.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T12:59:33.930 INFO:teuthology.orchestra.run.smithi149.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T12:59:33.930 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:33.930 INFO:teuthology.orchestra.run.smithi149.stdout: inet6 ::1/128 scope host 2023-11-28T12:59:33.930 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:33.930 INFO:teuthology.orchestra.run.smithi149.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:33.931 INFO:teuthology.orchestra.run.smithi149.stdout: link/ether 0c:c4:7a:da:6b:a4 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:33.931 INFO:teuthology.orchestra.run.smithi149.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:33.931 INFO:teuthology.orchestra.run.smithi149.stdout: link/ether 0c:c4:7a:da:6b:a5 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:33.931 INFO:teuthology.orchestra.run.smithi149.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T12:59:33.931 INFO:teuthology.orchestra.run.smithi149.stdout: link/ether 0c:c4:7a:88:7d:56 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:33.931 INFO:teuthology.orchestra.run.smithi149.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T12:59:33.931 INFO:teuthology.orchestra.run.smithi149.stdout: link/ether 0c:c4:7a:88:7d:57 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:33.931 INFO:teuthology.orchestra.run.smithi149.stdout: inet 172.21.15.149/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T12:59:33.932 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft 42340sec preferred_lft 42340sec 2023-11-28T12:59:33.932 INFO:teuthology.orchestra.run.smithi149.stdout: inet6 fe80::709:bc8:9b22:1807/64 scope link noprefixroute 2023-11-28T12:59:33.932 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:33.932 INFO:teuthology.orchestra.run.smithi149.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T12:59:33.932 INFO:teuthology.orchestra.run.smithi149.stdout: link/ether 1a:d5:7f:79:3a:80 brd ff:ff:ff:ff:ff:ff 2023-11-28T12:59:33.932 INFO:teuthology.orchestra.run.smithi149.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T12:59:33.932 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:33.933 INFO:teuthology.orchestra.run.smithi149.stdout: inet6 fe80::e00c:7cff:fead:df51/64 scope link 2023-11-28T12:59:33.933 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:33.933 INFO:teuthology.orchestra.run.smithi149.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T12:59:33.933 INFO:teuthology.orchestra.run.smithi149.stdout: link/ether 1a:d5:7f:79:3a:80 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T12:59:33.933 INFO:teuthology.orchestra.run.smithi149.stdout: inet6 fe80::18d5:7fff:fe79:3a80/64 scope link tentative 2023-11-28T12:59:33.933 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:33.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:33.934 DEBUG:teuthology.orchestra.run.smithi149:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T12:59:33.995 INFO:teuthology.orchestra.run.smithi149.stdout:1 2023-11-28T12:59:33.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:33.997 DEBUG:teuthology.orchestra.run.smithi149:> route 2023-11-28T12:59:34.055 INFO:teuthology.orchestra.run.smithi149.stdout:Kernel IP routing table 2023-11-28T12:59:34.055 INFO:teuthology.orchestra.run.smithi149.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T12:59:34.055 INFO:teuthology.orchestra.run.smithi149.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T12:59:34.056 INFO:teuthology.orchestra.run.smithi149.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T12:59:34.056 INFO:teuthology.orchestra.run.smithi149.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T12:59:34.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:34.056 DEBUG:teuthology.orchestra.run.smithi149:> (cd / && exec bash -c ' 2023-11-28T12:59:34.056 DEBUG:teuthology.orchestra.run.smithi149:> set -e 2023-11-28T12:59:34.056 DEBUG:teuthology.orchestra.run.smithi149:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T12:59:34.056 DEBUG:teuthology.orchestra.run.smithi149:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T12:59:34.056 DEBUG:teuthology.orchestra.run.smithi149:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T12:59:34.056 DEBUG:teuthology.orchestra.run.smithi149:> ') 2023-11-28T12:59:34.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:34.160 DEBUG:teuthology.orchestra.run.smithi149:> ip netns list 2023-11-28T12:59:34.176 INFO:teuthology.orchestra.run.smithi149.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-28T12:59:34.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:34.177 DEBUG:teuthology.orchestra.run.smithi149:> ip netns list-id 2023-11-28T12:59:34.233 INFO:teuthology.orchestra.run.smithi149.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-11-28T12:59:34.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:34.234 DEBUG:teuthology.orchestra.run.smithi149:> (cd / && exec bash -c ' 2023-11-28T12:59:34.234 DEBUG:teuthology.orchestra.run.smithi149:> set -e 2023-11-28T12:59:34.234 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T12:59:34.234 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-11-28T12:59:34.234 DEBUG:teuthology.orchestra.run.smithi149:> ') 2023-11-28T12:59:34.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:34.328 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-28T12:59:34.371 INFO:teuthology.orchestra.run.smithi149.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T12:59:34.371 INFO:teuthology.orchestra.run.smithi149.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T12:59:34.372 INFO:teuthology.orchestra.run.smithi149.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T12:59:34.372 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:34.372 INFO:teuthology.orchestra.run.smithi149.stdout: inet6 ::1/128 scope host 2023-11-28T12:59:34.372 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:34.372 INFO:teuthology.orchestra.run.smithi149.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T12:59:34.372 INFO:teuthology.orchestra.run.smithi149.stdout: link/ether 6a:8d:96:c9:2b:1e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-28T12:59:34.372 INFO:teuthology.orchestra.run.smithi149.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-28T12:59:34.372 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:34.372 INFO:teuthology.orchestra.run.smithi149.stdout: inet6 fe80::688d:96ff:fec9:2b1e/64 scope link 2023-11-28T12:59:34.372 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:34.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:34.384 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-28T12:59:34.461 INFO:teuthology.orchestra.run.smithi149.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T12:59:34.461 INFO:teuthology.orchestra.run.smithi149.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T12:59:34.462 INFO:teuthology.orchestra.run.smithi149.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T12:59:34.462 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:34.462 INFO:teuthology.orchestra.run.smithi149.stdout: inet6 ::1/128 scope host 2023-11-28T12:59:34.462 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:34.462 INFO:teuthology.orchestra.run.smithi149.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T12:59:34.462 INFO:teuthology.orchestra.run.smithi149.stdout: link/ether 6a:8d:96:c9:2b:1e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-28T12:59:34.462 INFO:teuthology.orchestra.run.smithi149.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-28T12:59:34.462 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:34.462 INFO:teuthology.orchestra.run.smithi149.stdout: inet6 fe80::688d:96ff:fec9:2b1e/64 scope link 2023-11-28T12:59:34.462 INFO:teuthology.orchestra.run.smithi149.stdout: valid_lft forever preferred_lft forever 2023-11-28T12:59:34.475 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-11-28T12:59:34.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:34.475 DEBUG:teuthology.orchestra.run.smithi149:> (cd / && exec bash -c ' 2023-11-28T12:59:34.475 DEBUG:teuthology.orchestra.run.smithi149:> set -e 2023-11-28T12:59:34.475 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-11-28T12:59:34.475 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-28T12:59:34.475 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-11-28T12:59:34.475 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-11-28T12:59:34.476 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-11-28T12:59:34.476 DEBUG:teuthology.orchestra.run.smithi149:> ') 2023-11-28T12:59:34.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:34.732 DEBUG:teuthology.orchestra.run.smithi149:> (cd / && exec bash -c ' 2023-11-28T12:59:34.732 DEBUG:teuthology.orchestra.run.smithi149:> set -e 2023-11-28T12:59:34.733 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip link set brx.1 up 2023-11-28T12:59:34.733 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip link set dev brx.1 master ceph-brx 2023-11-28T12:59:34.733 DEBUG:teuthology.orchestra.run.smithi149:> ') 2023-11-28T12:59:34.844 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T12:59:34.844 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-11-28T12:59:34.867 INFO:teuthology.orchestra.run.smithi149.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-11-28T12:59:34.867 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T12:59:34.867 DEBUG:teuthology.orchestra.run.smithi149:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-11-28T12:59:34.928 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T12:59:34.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:34.928 DEBUG:teuthology.orchestra.run.smithi149:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-11-28T12:59:35.013 INFO:teuthology.orchestra.run.smithi149.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-11-28T12:59:35.013 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-11-28T12:59:35.013 INFO:teuthology.orchestra.run.smithi149.stdout:invalid new device string format 2023-11-28T12:59:35.021 INFO:teuthology.orchestra.run.smithi149.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-11-28T12:59:35.021 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-11-28T12:59:35.022 INFO:teuthology.orchestra.run.smithi149.stdout:invalid new device string format 2023-11-28T12:59:35.022 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T12:59:35.022 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T12:59:35.096 INFO:teuthology.orchestra.run.smithi149.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-11-28T12:59:35.096 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-11-28T12:59:35.096 INFO:teuthology.orchestra.run.smithi149.stdout:invalid new device string format 2023-11-28T12:59:35.096 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T12:59:35.096 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T12:59:35.096 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: resolved to: "172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790" 2023-11-28T12:59:35.096 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790:/ 2023-11-28T12:59:35.096 INFO:teuthology.orchestra.run.smithi149.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=a1b2d00c-aeec-4b5d-93e3-41820c4189ba" will pass to kernel 2023-11-28T12:59:35.098 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T12:59:35.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:35.099 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-11-28T12:59:35.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:35.125 DEBUG:teuthology.orchestra.run.smithi149:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T12:59:35.197 INFO:teuthology.orchestra.run.smithi149.stdout:client4866 2023-11-28T12:59:35.197 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:59:35.197 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:35.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:35.256 DEBUG:teuthology.orchestra.run.smithi149:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T12:59:35.328 INFO:teuthology.orchestra.run.smithi149.stdout:client4866 2023-11-28T12:59:35.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:35.328 DEBUG:teuthology.orchestra.run.smithi149:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T12:59:35.401 INFO:teuthology.orchestra.run.smithi149.stdout:client4866 2023-11-28T12:59:35.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:35.401 DEBUG:teuthology.orchestra.run.smithi149:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a1b2d00c-aeec-4b5d-93e3-41820c4189ba.client4866/status') 2023-11-28T12:59:35.481 INFO:teuthology.orchestra.run.smithi149.stdout:instance: client.4866 (3)172.21.15.149:0/894596966 2023-11-28T12:59:35.482 INFO:teuthology.orchestra.run.smithi149.stdout:blocklisted: false 2023-11-28T12:59:35.482 INFO:teuthology.orchestra.run.smithi149.stderr:0+1 records in 2023-11-28T12:59:35.482 INFO:teuthology.orchestra.run.smithi149.stderr:0+1 records out 2023-11-28T12:59:35.482 INFO:teuthology.orchestra.run.smithi149.stderr:70 bytes copied, 2.3716e-05 s, 3.0 MB/s 2023-11-28T12:59:35.483 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-11-28T12:59:35.504 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:35.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.636+0000 7ff624c7b700 1 Processor -- start 2023-11-28T12:59:35.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.636+0000 7ff624c7b700 1 -- start start 2023-11-28T12:59:35.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.637+0000 7ff624c7b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff6200b0d70 0x7ff6201604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:35.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.637+0000 7ff624c7b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff6200b1140 0x7ff6200ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:35.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.637+0000 7ff624c7b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62015cd90 con 0x7ff6200b15b0 2023-11-28T12:59:35.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.637+0000 7ff624c7b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6201613e0 con 0x7ff6200b1140 2023-11-28T12:59:35.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.637+0000 7ff624c7b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff620160b40 con 0x7ff6200b0d70 2023-11-28T12:59:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.637+0000 7ff61dd9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff6200b1140 0x7ff6200ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.637+0000 7ff61ed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff6200b15b0 0x7ff6200cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34612/0 (socket says 172.21.15.111:34612) 2023-11-28T12:59:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.637+0000 7ff61ed9d700 1 -- 172.21.15.111:0/3161628599 learned_addr learned my addr 172.21.15.111:0/3161628599 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.637+0000 7ff61e59c700 1 --2- 172.21.15.111:0/3161628599 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff6200b0d70 0x7ff6201604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.637+0000 7ff61dd9b700 1 -- 172.21.15.111:0/3161628599 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff6200b0d70 msgr2=0x7ff6201604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.637+0000 7ff61dd9b700 1 --2- 172.21.15.111:0/3161628599 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff6200b0d70 0x7ff6201604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.637+0000 7ff61dd9b700 1 -- 172.21.15.111:0/3161628599 >> v1:172.21.15.111:6789/0 conn(0x7ff6200b15b0 legacy=0x7ff6200cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff61dd9b700 1 -- 172.21.15.111:0/3161628599 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff620160e20 con 0x7ff6200b1140 2023-11-28T12:59:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.637+0000 7ff61d59a700 1 -- 172.21.15.111:0/3161628599 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932155788 0 0) 0x7ff618001490 con 0x7ff6200b15b0 2023-11-28T12:59:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff61e59c700 1 --2- 172.21.15.111:0/3161628599 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff6200b0d70 0x7ff6201604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff61dd9b700 1 --2- 172.21.15.111:0/3161628599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6200b1140 0x7ff6200ce6c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff610009840 tx=0x7ff610009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a2bbf025382902 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff61d59a700 1 -- 172.21.15.111:0/3161628599 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610011830 con 0x7ff6200b1140 2023-11-28T12:59:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff61d59a700 1 -- 172.21.15.111:0/3161628599 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff610011990 con 0x7ff6200b1140 2023-11-28T12:59:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff61d59a700 1 -- 172.21.15.111:0/3161628599 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610011d20 con 0x7ff6200b1140 2023-11-28T12:59:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff624c7b700 1 -- 172.21.15.111:0/3161628599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6200b1140 msgr2=0x7ff6200ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff624c7b700 1 --2- 172.21.15.111:0/3161628599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6200b1140 0x7ff6200ce6c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff610009840 tx=0x7ff610009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff624c7b700 1 -- 172.21.15.111:0/3161628599 shutdown_connections 2023-11-28T12:59:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff624c7b700 1 --2- 172.21.15.111:0/3161628599 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff6200b0d70 0x7ff6201604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff624c7b700 1 --2- 172.21.15.111:0/3161628599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6200b1140 0x7ff6200ce6c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff624c7b700 1 -- 172.21.15.111:0/3161628599 >> 172.21.15.111:0/3161628599 conn(0x7ff620098ee0 msgr2=0x7ff62006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff624c7b700 1 -- 172.21.15.111:0/3161628599 shutdown_connections 2023-11-28T12:59:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.638+0000 7ff624c7b700 1 -- 172.21.15.111:0/3161628599 wait complete. 2023-11-28T12:59:35.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.640+0000 7ff624c7b700 1 Processor -- start 2023-11-28T12:59:35.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.640+0000 7ff624c7b700 1 -- start start 2023-11-28T12:59:35.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.641+0000 7ff624c7b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff6200b0d70 0x7ff62014ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.641+0000 7ff624c7b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6200b1140 0x7ff62015f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.641+0000 7ff624c7b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff6200b15b0 0x7ff62014d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.641+0000 7ff624c7b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62015e3d0 con 0x7ff6200b1140 2023-11-28T12:59:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.641+0000 7ff624c7b700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62015e6b0 con 0x7ff6200b15b0 2023-11-28T12:59:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.641+0000 7ff624c7b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62015e990 con 0x7ff6200b0d70 2023-11-28T12:59:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.641+0000 7ff61dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6200b1140 0x7ff62015f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.641+0000 7ff61dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6200b1140 0x7ff62015f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33838/0 (socket says 172.21.15.111:33838) 2023-11-28T12:59:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.641+0000 7ff61dd9b700 1 -- 172.21.15.111:0/2450628124 learned_addr learned my addr 172.21.15.111:0/2450628124 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.641+0000 7ff61e59c700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff6200b0d70 0x7ff62014ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.641+0000 7ff61ed9d700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff6200b15b0 0x7ff62014d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:35.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.642+0000 7ff61dd9b700 1 -- 172.21.15.111:0/2450628124 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff6200b0d70 msgr2=0x7ff62014ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.642+0000 7ff61dd9b700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff6200b0d70 0x7ff62014ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.642+0000 7ff61dd9b700 1 -- 172.21.15.111:0/2450628124 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff6200b15b0 msgr2=0x7ff62014d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.642+0000 7ff61dd9b700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff6200b15b0 0x7ff62014d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.642+0000 7ff61dd9b700 1 -- 172.21.15.111:0/2450628124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6100094a0 con 0x7ff6200b1140 2023-11-28T12:59:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.642+0000 7ff61e59c700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff6200b0d70 0x7ff62014ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.642+0000 7ff61ed9d700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff6200b15b0 0x7ff62014d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.642+0000 7ff61dd9b700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6200b1140 0x7ff62015f470 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff610000c00 tx=0x7ff610004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.642+0000 7ff60effd700 1 -- 172.21.15.111:0/2450628124 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6100045f0 con 0x7ff6200b1140 2023-11-28T12:59:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.642+0000 7ff60effd700 1 -- 172.21.15.111:0/2450628124 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61001f070 con 0x7ff6200b1140 2023-11-28T12:59:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.642+0000 7ff624c7b700 1 -- 172.21.15.111:0/2450628124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6200af340 con 0x7ff6200b1140 2023-11-28T12:59:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.642+0000 7ff60effd700 1 -- 172.21.15.111:0/2450628124 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610003ae0 con 0x7ff6200b1140 2023-11-28T12:59:35.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.642+0000 7ff624c7b700 1 -- 172.21.15.111:0/2450628124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff620161100 con 0x7ff6200b1140 2023-11-28T12:59:35.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.643+0000 7ff624c7b700 1 -- 172.21.15.111:0/2450628124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6200cec00 con 0x7ff6200b1140 2023-11-28T12:59:35.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.643+0000 7ff60effd700 1 -- 172.21.15.111:0/2450628124 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff61000f050 con 0x7ff6200b1140 2023-11-28T12:59:35.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.644+0000 7ff60effd700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff6000682f0 0x7ff60006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:35.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.644+0000 7ff60effd700 1 -- 172.21.15.111:0/2450628124 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff610021070 con 0x7ff6200b1140 2023-11-28T12:59:35.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.644+0000 7ff61e59c700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff6000682f0 0x7ff60006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:35.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.644+0000 7ff61e59c700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff6000682f0 0x7ff60006a7a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff61400aaf0 tx=0x7ff614008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:35.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.648+0000 7ff60effd700 1 -- 172.21.15.111:0/2450628124 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff610053fe0 con 0x7ff6200b1140 2023-11-28T12:59:35.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.835+0000 7ff624c7b700 1 -- 172.21.15.111:0/2450628124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6200cdfc0 con 0x7ff6200b1140 2023-11-28T12:59:35.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.836+0000 7ff60effd700 1 -- 172.21.15.111:0/2450628124 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19796 (secure 0 0 0) 0x7ff61002b090 con 0x7ff6200b1140 2023-11-28T12:59:35.836 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:35.837 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:00.053943+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:35.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.838+0000 7ff624c7b700 1 -- 172.21.15.111:0/2450628124 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff6000682f0 msgr2=0x7ff60006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:35.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.838+0000 7ff624c7b700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff6000682f0 0x7ff60006a7a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff61400aaf0 tx=0x7ff614008040 comp rx=0 tx=0).stop 2023-11-28T12:59:35.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.838+0000 7ff624c7b700 1 -- 172.21.15.111:0/2450628124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6200b1140 msgr2=0x7ff62015f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:35.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.838+0000 7ff624c7b700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6200b1140 0x7ff62015f470 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff610000c00 tx=0x7ff610004750 comp rx=0 tx=0).stop 2023-11-28T12:59:35.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.838+0000 7ff624c7b700 1 -- 172.21.15.111:0/2450628124 shutdown_connections 2023-11-28T12:59:35.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.838+0000 7ff624c7b700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff6000682f0 0x7ff60006a7a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:35.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.838+0000 7ff624c7b700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff6200b15b0 0x7ff62014d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:35.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.838+0000 7ff624c7b700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff6200b0d70 0x7ff62014ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:35.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.838+0000 7ff624c7b700 1 --2- 172.21.15.111:0/2450628124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6200b1140 0x7ff62015f470 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:35.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.838+0000 7ff624c7b700 1 -- 172.21.15.111:0/2450628124 >> 172.21.15.111:0/2450628124 conn(0x7ff620098ee0 msgr2=0x7ff62015c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:35.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.838+0000 7ff624c7b700 1 -- 172.21.15.111:0/2450628124 shutdown_connections 2023-11-28T12:59:35.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.838+0000 7ff624c7b700 1 -- 172.21.15.111:0/2450628124 wait complete. 2023-11-28T12:59:35.851 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T12:58:53.275206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T12:58:54.588499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T12:58:57.666033+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T12:59:35.852 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:59:35.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.975+0000 7f379827e700 1 Processor -- start 2023-11-28T12:59:35.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.975+0000 7f379827e700 1 -- start start 2023-11-28T12:59:35.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.975+0000 7f379827e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f379015c5e0 0x7f379006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:35.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.975+0000 7f379827e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379015c9b0 con 0x7f379006b720 2023-11-28T12:59:35.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.975+0000 7f379827e700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3790160e60 con 0x7f379006e970 2023-11-28T12:59:35.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.975+0000 7f379827e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37900cce90 con 0x7f379015c5e0 2023-11-28T12:59:35.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.976+0000 7f379601a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f379015c5e0 0x7f379006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:35.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.976+0000 7f3795819700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f379006b720 0x7f379006a7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34622/0 (socket says 172.21.15.111:34622) 2023-11-28T12:59:35.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.976+0000 7f3795819700 1 -- 172.21.15.111:0/3179029292 learned_addr learned my addr 172.21.15.111:0/3179029292 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:35.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.976+0000 7f379601a700 1 -- 172.21.15.111:0/3179029292 >> v1:172.21.15.149:6789/0 conn(0x7f379006e970 legacy=0x7f379006ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:35.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.976+0000 7f379601a700 1 -- 172.21.15.111:0/3179029292 >> v1:172.21.15.111:6789/0 conn(0x7f379006b720 legacy=0x7f379006a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:35.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.976+0000 7f379601a700 1 -- 172.21.15.111:0/3179029292 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37900cd0a0 con 0x7f379015c5e0 2023-11-28T12:59:35.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.976+0000 7f3795018700 1 -- 172.21.15.111:0/3179029292 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85658067 0 0) 0x7f379015c9b0 con 0x7f379006b720 2023-11-28T12:59:35.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.976+0000 7f379601a700 1 --2- 172.21.15.111:0/3179029292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f379015c5e0 0x7f379006a270 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f378c009840 tx=0x7f378c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb28bf3293a6dfa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:35.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.976+0000 7f3795018700 1 -- 172.21.15.111:0/3179029292 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c004230 con 0x7f379015c5e0 2023-11-28T12:59:35.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.976+0000 7f3795018700 1 -- 172.21.15.111:0/3179029292 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c004390 con 0x7f379015c5e0 2023-11-28T12:59:35.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.976+0000 7f3795018700 1 -- 172.21.15.111:0/3179029292 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c0036a0 con 0x7f379015c5e0 2023-11-28T12:59:35.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.977+0000 7f379827e700 1 -- 172.21.15.111:0/3179029292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f379015c5e0 msgr2=0x7f379006a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:35.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.977+0000 7f379827e700 1 --2- 172.21.15.111:0/3179029292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f379015c5e0 0x7f379006a270 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f378c009840 tx=0x7f378c009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:35.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.977+0000 7f379827e700 1 -- 172.21.15.111:0/3179029292 shutdown_connections 2023-11-28T12:59:35.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.977+0000 7f379827e700 1 --2- 172.21.15.111:0/3179029292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f379015c5e0 0x7f379006a270 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:35.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.977+0000 7f379827e700 1 -- 172.21.15.111:0/3179029292 >> 172.21.15.111:0/3179029292 conn(0x7f37900b0db0 msgr2=0x7f37900b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:35.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.977+0000 7f379827e700 1 -- 172.21.15.111:0/3179029292 shutdown_connections 2023-11-28T12:59:35.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.977+0000 7f379827e700 1 -- 172.21.15.111:0/3179029292 wait complete. 2023-11-28T12:59:35.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.978+0000 7f379827e700 1 Processor -- start 2023-11-28T12:59:35.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.978+0000 7f379827e700 1 -- start start 2023-11-28T12:59:35.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f379827e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f379006b720 0x7f37900c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:35.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f379827e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f379006e970 0x7f379015f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:35.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f379827e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f379015c5e0 0x7f37900af770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:35.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f379827e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37900b4ef0 con 0x7f379006e970 2023-11-28T12:59:35.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f379827e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37901f1e50 con 0x7f379015c5e0 2023-11-28T12:59:35.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f379827e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37900c5660 con 0x7f379006b720 2023-11-28T12:59:35.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f3795819700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f379006e970 0x7f379015f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:35.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f3795819700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f379006e970 0x7f379015f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33852/0 (socket says 172.21.15.111:33852) 2023-11-28T12:59:35.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f3795819700 1 -- 172.21.15.111:0/771855475 learned_addr learned my addr 172.21.15.111:0/771855475 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:35.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f3795819700 1 -- 172.21.15.111:0/771855475 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f379006b720 msgr2=0x7f37900c4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:35.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f3795819700 1 --2- 172.21.15.111:0/771855475 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f379006b720 0x7f37900c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:35.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f3795819700 1 -- 172.21.15.111:0/771855475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f379015c5e0 msgr2=0x7f37900af770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:35.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f3795819700 1 --2- 172.21.15.111:0/771855475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f379015c5e0 0x7f37900af770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:35.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f3795819700 1 -- 172.21.15.111:0/771855475 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c0094a0 con 0x7f379006e970 2023-11-28T12:59:35.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f3795819700 1 --2- 172.21.15.111:0/771855475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f379006e970 0x7f379015f720 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f378000c450 tx=0x7f378000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:35.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f3786ffd700 1 -- 172.21.15.111:0/771855475 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780004c80 con 0x7f379006e970 2023-11-28T12:59:35.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f3786ffd700 1 -- 172.21.15.111:0/771855475 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3780004e00 con 0x7f379006e970 2023-11-28T12:59:35.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f379827e700 1 -- 172.21.15.111:0/771855475 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37900cd660 con 0x7f379006e970 2023-11-28T12:59:35.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f3786ffd700 1 -- 172.21.15.111:0/771855475 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780007310 con 0x7f379006e970 2023-11-28T12:59:35.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.979+0000 7f379827e700 1 -- 172.21.15.111:0/771855475 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37900cd380 con 0x7f379006e970 2023-11-28T12:59:35.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.980+0000 7f379827e700 1 -- 172.21.15.111:0/771855475 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3790160af0 con 0x7f379006e970 2023-11-28T12:59:35.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.981+0000 7f3786ffd700 1 -- 172.21.15.111:0/771855475 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f378001a020 con 0x7f379006e970 2023-11-28T12:59:35.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.981+0000 7f3786ffd700 1 --2- 172.21.15.111:0/771855475 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f37740682f0 0x7f377406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:35.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.981+0000 7f3786ffd700 1 -- 172.21.15.111:0/771855475 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3780085a40 con 0x7f379006e970 2023-11-28T12:59:35.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.981+0000 7f379601a700 1 --2- 172.21.15.111:0/771855475 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f37740682f0 0x7f377406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:35.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.982+0000 7f379601a700 1 --2- 172.21.15.111:0/771855475 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f37740682f0 0x7f377406a7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f37900b0180 tx=0x7f378c010000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:35.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:35.985+0000 7f3786ffd700 1 -- 172.21.15.111:0/771855475 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3780051fb0 con 0x7f379006e970 2023-11-28T12:59:36.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.167+0000 7f379827e700 1 -- 172.21.15.111:0/771855475 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3790067d30 con 0x7f379006e970 2023-11-28T12:59:36.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.167+0000 7f3786ffd700 1 -- 172.21.15.111:0/771855475 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f37800555d0 con 0x7f379006e970 2023-11-28T12:59:36.168 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:59:36.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.169+0000 7f379827e700 1 -- 172.21.15.111:0/771855475 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f37740682f0 msgr2=0x7f377406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.169+0000 7f379827e700 1 --2- 172.21.15.111:0/771855475 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f37740682f0 0x7f377406a7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f37900b0180 tx=0x7f378c010000 comp rx=0 tx=0).stop 2023-11-28T12:59:36.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.169+0000 7f379827e700 1 -- 172.21.15.111:0/771855475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f379006e970 msgr2=0x7f379015f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.169+0000 7f379827e700 1 --2- 172.21.15.111:0/771855475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f379006e970 0x7f379015f720 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f378000c450 tx=0x7f378000c810 comp rx=0 tx=0).stop 2023-11-28T12:59:36.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.169+0000 7f379827e700 1 -- 172.21.15.111:0/771855475 shutdown_connections 2023-11-28T12:59:36.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.169+0000 7f379827e700 1 --2- 172.21.15.111:0/771855475 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f37740682f0 0x7f377406a7a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.169+0000 7f379827e700 1 --2- 172.21.15.111:0/771855475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f379015c5e0 0x7f37900af770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.169+0000 7f379827e700 1 --2- 172.21.15.111:0/771855475 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f379006b720 0x7f37900c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.169+0000 7f379827e700 1 --2- 172.21.15.111:0/771855475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f379006e970 0x7f379015f720 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.169+0000 7f379827e700 1 -- 172.21.15.111:0/771855475 >> 172.21.15.111:0/771855475 conn(0x7f37900b0db0 msgr2=0x7f379006edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:36.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.169+0000 7f379827e700 1 -- 172.21.15.111:0/771855475 shutdown_connections 2023-11-28T12:59:36.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.169+0000 7f379827e700 1 -- 172.21.15.111:0/771855475 wait complete. 2023-11-28T12:59:36.181 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:59:36.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.304+0000 7f9cc4b25700 1 Processor -- start 2023-11-28T12:59:36.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.304+0000 7f9cc4b25700 1 -- start start 2023-11-28T12:59:36.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cc4b25700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cc00cdd90 0x7f9cc00b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cc4b25700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9cc00aecc0 0x7f9cc00ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cc4b25700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc00ad420 con 0x7f9cc0159960 2023-11-28T12:59:36.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cc4b25700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc0159d30 con 0x7f9cc00aecc0 2023-11-28T12:59:36.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cc4b25700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc00ac810 con 0x7f9cc00cdd90 2023-11-28T12:59:36.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbe59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9cc0159960 0x7f9cc00cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34636/0 (socket says 172.21.15.111:34636) 2023-11-28T12:59:36.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbe59c700 1 -- 172.21.15.111:0/3308904545 learned_addr learned my addr 172.21.15.111:0/3308904545 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:36.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591103901 0 0) 0x7f9cc00ad420 con 0x7f9cc0159960 2023-11-28T12:59:36.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbed9d700 1 --2- 172.21.15.111:0/3308904545 >> v2:172.21.15.149:3300/0 conn(0x7f9cc00aecc0 0x7f9cc00ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca80035b0 con 0x7f9cc0159960 2023-11-28T12:59:36.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbdd9b700 1 --2- 172.21.15.111:0/3308904545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cc00cdd90 0x7f9cc00b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1326421908 0 0) 0x7f9ca80035b0 con 0x7f9cc0159960 2023-11-28T12:59:36.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cc00cdd90 msgr2=0x7f9cc00b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:36.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 --2- 172.21.15.111:0/3308904545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cc00cdd90 0x7f9cc00b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 >> v2:172.21.15.149:3300/0 conn(0x7f9cc00aecc0 msgr2=0x7f9cc00ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:36.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 --2- 172.21.15.111:0/3308904545 >> v2:172.21.15.149:3300/0 conn(0x7f9cc00aecc0 0x7f9cc00ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc00acaf0 con 0x7f9cc0159960 2023-11-28T12:59:36.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f9cb40017f0 con 0x7f9cc0159960 2023-11-28T12:59:36.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 >> v1:172.21.15.111:6789/0 conn(0x7f9cc0159960 legacy=0x7f9cc00cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 --2- 172.21.15.111:0/3308904545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cc00cdd90 0x7f9ca8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 --2- 172.21.15.111:0/3308904545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ca8003fd0 0x7f9ca8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 --2- 172.21.15.111:0/3308904545 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ca8004d90 0x7f9ca8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc0159d30 con 0x7f9ca8003fd0 2023-11-28T12:59:36.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc00ac810 con 0x7f9ca8004d90 2023-11-28T12:59:36.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca800bb40 con 0x7f9cc00cdd90 2023-11-28T12:59:36.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.306+0000 7f9cbdd9b700 1 --2- 172.21.15.111:0/3308904545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ca8003fd0 0x7f9ca8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.307+0000 7f9cbe59c700 1 --2- 172.21.15.111:0/3308904545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cc00cdd90 0x7f9ca8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.307+0000 7f9cbed9d700 1 --2- 172.21.15.111:0/3308904545 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ca8004d90 0x7f9ca8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.307+0000 7f9cbdd9b700 1 -- 172.21.15.111:0/3308904545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cc00cdd90 msgr2=0x7f9ca8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:36.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.307+0000 7f9cbdd9b700 1 --2- 172.21.15.111:0/3308904545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cc00cdd90 0x7f9ca8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.307+0000 7f9cbdd9b700 1 -- 172.21.15.111:0/3308904545 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ca8004d90 msgr2=0x7f9ca8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:36.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.307+0000 7f9cbdd9b700 1 --2- 172.21.15.111:0/3308904545 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ca8004d90 0x7f9ca8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.307+0000 7f9cbdd9b700 1 -- 172.21.15.111:0/3308904545 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca80035b0 con 0x7f9ca8003fd0 2023-11-28T12:59:36.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.307+0000 7f9cbdd9b700 1 --2- 172.21.15.111:0/3308904545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ca8003fd0 0x7f9ca8004850 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00ce180 tx=0x7f9ca8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6b0ac05d74ff51 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:36.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.307+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb000c070 con 0x7f9ca8003fd0 2023-11-28T12:59:36.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.307+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb0013610 con 0x7f9ca8003fd0 2023-11-28T12:59:36.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.307+0000 7f9cbd59a700 1 -- 172.21.15.111:0/3308904545 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb00139a0 con 0x7f9ca8003fd0 2023-11-28T12:59:36.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.308+0000 7f9cc4b25700 1 -- 172.21.15.111:0/3308904545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ca8003fd0 msgr2=0x7f9ca8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.308+0000 7f9cc4b25700 1 --2- 172.21.15.111:0/3308904545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ca8003fd0 0x7f9ca8004850 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00ce180 tx=0x7f9ca8005880 comp rx=0 tx=0).stop 2023-11-28T12:59:36.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.308+0000 7f9cbed9d700 1 -- 172.21.15.111:0/3308904545 reap_dead start 2023-11-28T12:59:36.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.308+0000 7f9cc4b25700 1 -- 172.21.15.111:0/3308904545 shutdown_connections 2023-11-28T12:59:36.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.308+0000 7f9cc4b25700 1 -- 172.21.15.111:0/3308904545 >> 172.21.15.111:0/3308904545 conn(0x7f9cc0098ee0 msgr2=0x7f9cc006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:36.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.308+0000 7f9cc4b25700 1 -- 172.21.15.111:0/3308904545 shutdown_connections 2023-11-28T12:59:36.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.308+0000 7f9cc4b25700 1 -- 172.21.15.111:0/3308904545 wait complete. 2023-11-28T12:59:36.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.309+0000 7f9cc4b25700 1 Processor -- start 2023-11-28T12:59:36.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.309+0000 7f9cc4b25700 1 -- start start 2023-11-28T12:59:36.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cc4b25700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ca8003fd0 0x7f9cc0154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cc4b25700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cc00b3ea0 0x7f9cc00b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cc4b25700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cc006eb40 0x7f9cc006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cc4b25700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc0154770 con 0x7f9cc006eb40 2023-11-28T12:59:36.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cc4b25700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc006f4f0 con 0x7f9ca8003fd0 2023-11-28T12:59:36.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cc4b25700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc00b2970 con 0x7f9cc00b3ea0 2023-11-28T12:59:36.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cbed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cc006eb40 0x7f9cc006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cbed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cc006eb40 0x7f9cc006efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33874/0 (socket says 172.21.15.111:33874) 2023-11-28T12:59:36.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cbed9d700 1 -- 172.21.15.111:0/516597188 learned_addr learned my addr 172.21.15.111:0/516597188 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:36.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cbe59c700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ca8003fd0 0x7f9cc0154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cbdd9b700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cc00b3ea0 0x7f9cc00b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cbed9d700 1 -- 172.21.15.111:0/516597188 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cc00b3ea0 msgr2=0x7f9cc00b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cbed9d700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cc00b3ea0 0x7f9cc00b4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cbed9d700 1 -- 172.21.15.111:0/516597188 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ca8003fd0 msgr2=0x7f9cc0154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cbed9d700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ca8003fd0 0x7f9cc0154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cbed9d700 1 -- 172.21.15.111:0/516597188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc00ad0b0 con 0x7f9cc006eb40 2023-11-28T12:59:36.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cbed9d700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cc006eb40 0x7f9cc006efb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0099590 tx=0x7f9cb8011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:36.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9caeffd700 1 -- 172.21.15.111:0/516597188 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb800e040 con 0x7f9cc006eb40 2023-11-28T12:59:36.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9caeffd700 1 -- 172.21.15.111:0/516597188 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb800f040 con 0x7f9cc006eb40 2023-11-28T12:59:36.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.310+0000 7f9cc4b25700 1 -- 172.21.15.111:0/516597188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc00acdd0 con 0x7f9cc006eb40 2023-11-28T12:59:36.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.311+0000 7f9caeffd700 1 -- 172.21.15.111:0/516597188 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb800fc40 con 0x7f9cc006eb40 2023-11-28T12:59:36.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.311+0000 7f9cc4b25700 1 -- 172.21.15.111:0/516597188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc00b1fd0 con 0x7f9cc006eb40 2023-11-28T12:59:36.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.311+0000 7f9cc4b25700 1 -- 172.21.15.111:0/516597188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc0067170 con 0x7f9cc006eb40 2023-11-28T12:59:36.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.312+0000 7f9caeffd700 1 -- 172.21.15.111:0/516597188 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9cb800fda0 con 0x7f9cc006eb40 2023-11-28T12:59:36.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.312+0000 7f9caeffd700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9ca00682f0 0x7f9ca006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.312+0000 7f9caeffd700 1 -- 172.21.15.111:0/516597188 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9cb8086530 con 0x7f9cc006eb40 2023-11-28T12:59:36.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.312+0000 7f9cbe59c700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9ca00682f0 0x7f9ca006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.313+0000 7f9cbe59c700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9ca00682f0 0x7f9ca006a7a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4000c00 tx=0x7f9cb4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:36.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.316+0000 7f9caeffd700 1 -- 172.21.15.111:0/516597188 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9cb8052aa0 con 0x7f9cc006eb40 2023-11-28T12:59:36.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.500+0000 7f9cc4b25700 1 -- 172.21.15.111:0/516597188 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9cc006d9a0 con 0x7f9cc006eb40 2023-11-28T12:59:36.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.500+0000 7f9caeffd700 1 -- 172.21.15.111:0/516597188 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9cb80560c0 con 0x7f9cc006eb40 2023-11-28T12:59:36.501 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:59:36.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.503+0000 7f9cc4b25700 1 -- 172.21.15.111:0/516597188 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9ca00682f0 msgr2=0x7f9ca006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.503+0000 7f9cc4b25700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9ca00682f0 0x7f9ca006a7a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4000c00 tx=0x7f9cb4009040 comp rx=0 tx=0).stop 2023-11-28T12:59:36.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.503+0000 7f9cc4b25700 1 -- 172.21.15.111:0/516597188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cc006eb40 msgr2=0x7f9cc006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.503+0000 7f9cc4b25700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cc006eb40 0x7f9cc006efb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0099590 tx=0x7f9cb8011ed0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.503+0000 7f9cc4b25700 1 -- 172.21.15.111:0/516597188 shutdown_connections 2023-11-28T12:59:36.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.503+0000 7f9cc4b25700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9cc006eb40 0x7f9cc006efb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.503+0000 7f9cc4b25700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9ca00682f0 0x7f9ca006a7a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.503+0000 7f9cc4b25700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ca8003fd0 0x7f9cc0154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.503+0000 7f9cc4b25700 1 --2- 172.21.15.111:0/516597188 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9cc00b3ea0 0x7f9cc00b4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.503+0000 7f9cc4b25700 1 -- 172.21.15.111:0/516597188 >> 172.21.15.111:0/516597188 conn(0x7f9cc0098ee0 msgr2=0x7f9cc006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:36.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.503+0000 7f9cc4b25700 1 -- 172.21.15.111:0/516597188 shutdown_connections 2023-11-28T12:59:36.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.503+0000 7f9cc4b25700 1 -- 172.21.15.111:0/516597188 wait complete. 2023-11-28T12:59:36.515 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T12:59:36.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.637+0000 7fc1d3e16700 1 Processor -- start 2023-11-28T12:59:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.637+0000 7fc1d3e16700 1 -- start start 2023-11-28T12:59:36.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.638+0000 7fc1d3e16700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc1cc15d6e0 0x7fc1cc164360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.638+0000 7fc1d3e16700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1cc15de70 con 0x7fc1cc159310 2023-11-28T12:59:36.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.638+0000 7fc1d3e16700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1cc0ced10 con 0x7fc1cc1596e0 2023-11-28T12:59:36.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.638+0000 7fc1d3e16700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0ceff0 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.638+0000 7fc1d1bb2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc1cc159310 0x7fc1cc0ce1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34656/0 (socket says 172.21.15.111:34656) 2023-11-28T12:59:36.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.638+0000 7fc1d1bb2700 1 -- 172.21.15.111:0/1438098612 learned_addr learned my addr 172.21.15.111:0/1438098612 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:36.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.638+0000 7fc1d23b3700 1 --2- 172.21.15.111:0/1438098612 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc1cc15d6e0 0x7fc1cc164360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.638+0000 7fc1d0bb0700 1 -- 172.21.15.111:0/1438098612 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562535044 0 0) 0x7fc1cc15de70 con 0x7fc1cc159310 2023-11-28T12:59:36.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.638+0000 7fc1d0bb0700 1 -- 172.21.15.111:0/1438098612 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b80035b0 con 0x7fc1cc159310 2023-11-28T12:59:36.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.639+0000 7fc1d0bb0700 1 -- 172.21.15.111:0/1438098612 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825660160 0 0) 0x7fc1cc0ced10 con 0x7fc1cc1596e0 2023-11-28T12:59:36.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.639+0000 7fc1d0bb0700 1 -- 172.21.15.111:0/1438098612 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1cc15de70 con 0x7fc1cc1596e0 2023-11-28T12:59:36.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.639+0000 7fc1d23b3700 1 -- 172.21.15.111:0/1438098612 >> v1:172.21.15.149:6789/0 conn(0x7fc1cc1596e0 legacy=0x7fc1cc15cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.639+0000 7fc1d23b3700 1 -- 172.21.15.111:0/1438098612 >> v1:172.21.15.111:6789/0 conn(0x7fc1cc159310 legacy=0x7fc1cc0ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.639+0000 7fc1d0bb0700 1 -- 172.21.15.111:0/1438098612 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 191665658 0 0) 0x7fc1b80035b0 con 0x7fc1cc159310 2023-11-28T12:59:36.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.639+0000 7fc1d23b3700 1 -- 172.21.15.111:0/1438098612 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc0cf2d0 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.639+0000 7fc1d23b3700 1 --2- 172.21.15.111:0/1438098612 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc1cc15d6e0 0x7fc1cc164360 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009dc0 tx=0x7fc1c400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5de152e370cb947 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:36.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.639+0000 7fc1d0bb0700 1 -- 172.21.15.111:0/1438098612 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c400e070 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.639+0000 7fc1d0bb0700 1 -- 172.21.15.111:0/1438098612 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c4003a20 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.639+0000 7fc1d0bb0700 1 -- 172.21.15.111:0/1438098612 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc001670 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.640+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1438098612 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc1cc15d6e0 msgr2=0x7fc1cc164360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.640+0000 7fc1d3e16700 1 --2- 172.21.15.111:0/1438098612 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc1cc15d6e0 0x7fc1cc164360 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009dc0 tx=0x7fc1c400b330 comp rx=0 tx=0).stop 2023-11-28T12:59:36.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.640+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1438098612 shutdown_connections 2023-11-28T12:59:36.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.640+0000 7fc1d3e16700 1 --2- 172.21.15.111:0/1438098612 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc1cc15d6e0 0x7fc1cc164360 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.640+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1438098612 >> 172.21.15.111:0/1438098612 conn(0x7fc1cc06db10 msgr2=0x7fc1cc06df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:36.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.640+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1438098612 shutdown_connections 2023-11-28T12:59:36.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.640+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1438098612 wait complete. 2023-11-28T12:59:36.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.641+0000 7fc1d3e16700 1 Processor -- start 2023-11-28T12:59:36.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.641+0000 7fc1d3e16700 1 -- start start 2023-11-28T12:59:36.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d3e16700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc1cc159310 0x7fc1cc1f6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d3e16700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc1cc1596e0 0x7fc1cc1541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d3e16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc15d6e0 0x7fc1cc06a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d3e16700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0c9a60 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d3e16700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc1f66d0 con 0x7fc1cc159310 2023-11-28T12:59:36.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d3e16700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0c8e50 con 0x7fc1cc1596e0 2023-11-28T12:59:36.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d23b3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc15d6e0 0x7fc1cc06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d23b3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc15d6e0 0x7fc1cc06a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33888/0 (socket says 172.21.15.111:33888) 2023-11-28T12:59:36.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d23b3700 1 -- 172.21.15.111:0/1625283343 learned_addr learned my addr 172.21.15.111:0/1625283343 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:36.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d1bb2700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc1cc159310 0x7fc1cc1f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d13b1700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc1cc1596e0 0x7fc1cc1541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d23b3700 1 -- 172.21.15.111:0/1625283343 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc1cc1596e0 msgr2=0x7fc1cc1541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d23b3700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc1cc1596e0 0x7fc1cc1541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d23b3700 1 -- 172.21.15.111:0/1625283343 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc1cc159310 msgr2=0x7fc1cc1f6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d23b3700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc1cc159310 0x7fc1cc1f6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d23b3700 1 -- 172.21.15.111:0/1625283343 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c4009a70 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d13b1700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc1cc1596e0 0x7fc1cc1541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:36.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d23b3700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc15d6e0 0x7fc1cc06a270 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009d90 tx=0x7fc1c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:36.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1c27fc700 1 -- 172.21.15.111:0/1625283343 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c40046f0 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1c27fc700 1 -- 172.21.15.111:0/1625283343 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c4004850 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1625283343 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc0cb3d0 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1c27fc700 1 -- 172.21.15.111:0/1625283343 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c401d6b0 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.642+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1625283343 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc0cb0f0 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.643+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1625283343 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1cc0ce8b0 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.644+0000 7fc1c27fc700 1 -- 172.21.15.111:0/1625283343 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc1c4023030 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.644+0000 7fc1c27fc700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc1b00682f0 0x7fc1b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.644+0000 7fc1c27fc700 1 -- 172.21.15.111:0/1625283343 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc1c4087bb0 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.644+0000 7fc1d1bb2700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc1b00682f0 0x7fc1b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.644+0000 7fc1d1bb2700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc1b00682f0 0x7fc1b006a7a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc000f60 tx=0x7fc1bc0014b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:36.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.648+0000 7fc1c27fc700 1 -- 172.21.15.111:0/1625283343 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc1c408e080 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.831+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1625283343 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc1cc067170 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.831+0000 7fc1c27fc700 1 -- 172.21.15.111:0/1625283343 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc1c40540c0 con 0x7fc1cc15d6e0 2023-11-28T12:59:36.832 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:36.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.833+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1625283343 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc1b00682f0 msgr2=0x7fc1b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.833+0000 7fc1d3e16700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc1b00682f0 0x7fc1b006a7a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc000f60 tx=0x7fc1bc0014b0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.833+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1625283343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc15d6e0 msgr2=0x7fc1cc06a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.833+0000 7fc1d3e16700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc15d6e0 0x7fc1cc06a270 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009d90 tx=0x7fc1c4004ad0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.833+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1625283343 shutdown_connections 2023-11-28T12:59:36.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.833+0000 7fc1d3e16700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc15d6e0 0x7fc1cc06a270 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.833+0000 7fc1d3e16700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc1b00682f0 0x7fc1b006a7a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.833+0000 7fc1d3e16700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc1cc159310 0x7fc1cc1f6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.833+0000 7fc1d3e16700 1 --2- 172.21.15.111:0/1625283343 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc1cc1596e0 0x7fc1cc1541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.833+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1625283343 >> 172.21.15.111:0/1625283343 conn(0x7fc1cc06db10 msgr2=0x7fc1cc06bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:36.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.833+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1625283343 shutdown_connections 2023-11-28T12:59:36.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.833+0000 7fc1d3e16700 1 -- 172.21.15.111:0/1625283343 wait complete. 2023-11-28T12:59:36.846 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T12:59:36.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.968+0000 7ffa4af6d700 1 Processor -- start 2023-11-28T12:59:36.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.968+0000 7ffa4af6d700 1 -- start start 2023-11-28T12:59:36.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.969+0000 7ffa4af6d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffa4415a5d0 0x7ffa440ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.969+0000 7ffa4af6d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa440cb2d0 0x7ffa440ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.969+0000 7ffa4af6d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa440ae8b0 con 0x7ffa440b0160 2023-11-28T12:59:36.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.969+0000 7ffa4af6d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa440cc1b0 con 0x7ffa4415a5d0 2023-11-28T12:59:36.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.969+0000 7ffa4af6d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa440cc390 con 0x7ffa440cb2d0 2023-11-28T12:59:36.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.969+0000 7ffa48d09700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffa4415a5d0 0x7ffa440ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.969+0000 7ffa43fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ffa440b0160 0x7ffa4415e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34670/0 (socket says 172.21.15.111:34670) 2023-11-28T12:59:36.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.969+0000 7ffa43fff700 1 -- 172.21.15.111:0/3620725993 learned_addr learned my addr 172.21.15.111:0/3620725993 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:36.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.970+0000 7ffa4950a700 1 --2- 172.21.15.111:0/3620725993 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa440cb2d0 0x7ffa440ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.970+0000 7ffa48d09700 1 -- 172.21.15.111:0/3620725993 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa440cb2d0 msgr2=0x7ffa440ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:36.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.970+0000 7ffa48d09700 1 --2- 172.21.15.111:0/3620725993 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa440cb2d0 0x7ffa440ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.970+0000 7ffa48d09700 1 -- 172.21.15.111:0/3620725993 >> v1:172.21.15.111:6789/0 conn(0x7ffa440b0160 legacy=0x7ffa4415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.970+0000 7ffa48d09700 1 -- 172.21.15.111:0/3620725993 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa440cc670 con 0x7ffa4415a5d0 2023-11-28T12:59:36.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.970+0000 7ffa437fe700 1 -- 172.21.15.111:0/3620725993 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954212144 0 0) 0x7ffa440ae8b0 con 0x7ffa440b0160 2023-11-28T12:59:36.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.970+0000 7ffa48d09700 1 --2- 172.21.15.111:0/3620725993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa4415a5d0 0x7ffa440ce850 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009820 tx=0x7ffa38009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194a429acf8ced66 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:36.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.970+0000 7ffa437fe700 1 -- 172.21.15.111:0/3620725993 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa38005e30 con 0x7ffa4415a5d0 2023-11-28T12:59:36.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.970+0000 7ffa437fe700 1 -- 172.21.15.111:0/3620725993 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa38006030 con 0x7ffa4415a5d0 2023-11-28T12:59:36.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.970+0000 7ffa437fe700 1 -- 172.21.15.111:0/3620725993 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa380063e0 con 0x7ffa4415a5d0 2023-11-28T12:59:36.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.970+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3620725993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa4415a5d0 msgr2=0x7ffa440ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.970+0000 7ffa4af6d700 1 --2- 172.21.15.111:0/3620725993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa4415a5d0 0x7ffa440ce850 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009820 tx=0x7ffa38009b30 comp rx=0 tx=0).stop 2023-11-28T12:59:36.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.971+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3620725993 shutdown_connections 2023-11-28T12:59:36.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.971+0000 7ffa4af6d700 1 --2- 172.21.15.111:0/3620725993 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa440cb2d0 0x7ffa440ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.971+0000 7ffa4af6d700 1 --2- 172.21.15.111:0/3620725993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa4415a5d0 0x7ffa440ce850 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.971+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3620725993 >> 172.21.15.111:0/3620725993 conn(0x7ffa4409a040 msgr2=0x7ffa440d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:36.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.971+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3620725993 shutdown_connections 2023-11-28T12:59:36.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.971+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3620725993 wait complete. 2023-11-28T12:59:36.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.972+0000 7ffa4af6d700 1 Processor -- start 2023-11-28T12:59:36.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.972+0000 7ffa4af6d700 1 -- start start 2023-11-28T12:59:36.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa4af6d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa440b0160 0x7ffa440ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa4af6d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa440cb2d0 0x7ffa440b5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa4af6d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa4415a5d0 0x7ffa440b56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa4af6d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa440c6bf0 con 0x7ffa440b0160 2023-11-28T12:59:36.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa4af6d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa440b46f0 con 0x7ffa4415a5d0 2023-11-28T12:59:36.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa4af6d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa440c6910 con 0x7ffa440cb2d0 2023-11-28T12:59:36.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa48d09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa440b0160 0x7ffa440ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa48d09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa440b0160 0x7ffa440ca190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33898/0 (socket says 172.21.15.111:33898) 2023-11-28T12:59:36.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa48d09700 1 -- 172.21.15.111:0/3121223075 learned_addr learned my addr 172.21.15.111:0/3121223075 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:36.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa43fff700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa440cb2d0 0x7ffa440b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa48d09700 1 -- 172.21.15.111:0/3121223075 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa440cb2d0 msgr2=0x7ffa440b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa48d09700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa440cb2d0 0x7ffa440b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa4950a700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa4415a5d0 0x7ffa440b56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa48d09700 1 -- 172.21.15.111:0/3121223075 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa4415a5d0 msgr2=0x7ffa440b56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:36.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa48d09700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa4415a5d0 0x7ffa440b56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa48d09700 1 -- 172.21.15.111:0/3121223075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa38009430 con 0x7ffa440b0160 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa43fff700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa440cb2d0 0x7ffa440b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa48d09700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa440b0160 0x7ffa440ca190 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009b70 tx=0x7ffa38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa417fa700 1 -- 172.21.15.111:0/3121223075 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa380069f0 con 0x7ffa440b0160 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa417fa700 1 -- 172.21.15.111:0/3121223075 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa38020070 con 0x7ffa440b0160 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3121223075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa440ccc30 con 0x7ffa440b0160 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa417fa700 1 -- 172.21.15.111:0/3121223075 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa38004280 con 0x7ffa440b0160 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.973+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3121223075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa440cc950 con 0x7ffa440b0160 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.974+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3121223075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa440ccf10 con 0x7ffa440b0160 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.975+0000 7ffa417fa700 1 -- 172.21.15.111:0/3121223075 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffa3800f050 con 0x7ffa440b0160 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.975+0000 7ffa417fa700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffa280682f0 0x7ffa2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.975+0000 7ffa417fa700 1 -- 172.21.15.111:0/3121223075 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ffa38087710 con 0x7ffa440b0160 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.975+0000 7ffa43fff700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffa280682f0 0x7ffa2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:36.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.976+0000 7ffa43fff700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffa280682f0 0x7ffa2806a7a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffa30007fd0 tx=0x7ffa30009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:36.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:36.979+0000 7ffa417fa700 1 -- 172.21.15.111:0/3121223075 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffa38053c80 con 0x7ffa440b0160 2023-11-28T12:59:37.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.161+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3121223075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffa44067170 con 0x7ffa440b0160 2023-11-28T12:59:37.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.161+0000 7ffa417fa700 1 -- 172.21.15.111:0/3121223075 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffa380572a0 con 0x7ffa440b0160 2023-11-28T12:59:37.162 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T12:59:37.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.163+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3121223075 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffa280682f0 msgr2=0x7ffa2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.163+0000 7ffa4af6d700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffa280682f0 0x7ffa2806a7a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffa30007fd0 tx=0x7ffa30009040 comp rx=0 tx=0).stop 2023-11-28T12:59:37.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.163+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3121223075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa440b0160 msgr2=0x7ffa440ca190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.163+0000 7ffa4af6d700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa440b0160 0x7ffa440ca190 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009b70 tx=0x7ffa38004750 comp rx=0 tx=0).stop 2023-11-28T12:59:37.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.163+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3121223075 shutdown_connections 2023-11-28T12:59:37.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.163+0000 7ffa4af6d700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffa280682f0 0x7ffa2806a7a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.163+0000 7ffa4af6d700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa4415a5d0 0x7ffa440b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.163+0000 7ffa4af6d700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa440b0160 0x7ffa440ca190 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.163+0000 7ffa4af6d700 1 --2- 172.21.15.111:0/3121223075 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa440cb2d0 0x7ffa440b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.163+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3121223075 >> 172.21.15.111:0/3121223075 conn(0x7ffa4409a040 msgr2=0x7ffa4406ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:37.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.163+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3121223075 shutdown_connections 2023-11-28T12:59:37.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.163+0000 7ffa4af6d700 1 -- 172.21.15.111:0/3121223075 wait complete. 2023-11-28T12:59:37.175 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T12:59:37.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.297+0000 7fdf34cf2700 1 Processor -- start 2023-11-28T12:59:37.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.297+0000 7fdf34cf2700 1 -- start start 2023-11-28T12:59:37.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.298+0000 7fdf34cf2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf300af5a0 0x7fdf300b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.298+0000 7fdf34cf2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdf300af9f0 0x7fdf300b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.298+0000 7fdf34cf2700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3006ac80 con 0x7fdf3015fed0 2023-11-28T12:59:37.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.298+0000 7fdf34cf2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf300cee00 con 0x7fdf300af9f0 2023-11-28T12:59:37.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.298+0000 7fdf34cf2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf300cf9a0 con 0x7fdf300af5a0 2023-11-28T12:59:37.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.299+0000 7fdf2ed9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdf300af9f0 0x7fdf300b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.299+0000 7fdf2e59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdf3015fed0 0x7fdf300cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34684/0 (socket says 172.21.15.111:34684) 2023-11-28T12:59:37.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.299+0000 7fdf2e59c700 1 -- 172.21.15.111:0/3813806319 learned_addr learned my addr 172.21.15.111:0/3813806319 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:37.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.299+0000 7fdf2dd9b700 1 --2- 172.21.15.111:0/3813806319 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf300af5a0 0x7fdf300b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.299+0000 7fdf2ed9d700 1 -- 172.21.15.111:0/3813806319 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf300af5a0 msgr2=0x7fdf300b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:37.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.299+0000 7fdf2ed9d700 1 --2- 172.21.15.111:0/3813806319 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf300af5a0 0x7fdf300b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.299+0000 7fdf2ed9d700 1 -- 172.21.15.111:0/3813806319 >> v1:172.21.15.111:6789/0 conn(0x7fdf3015fed0 legacy=0x7fdf300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.299+0000 7fdf2ed9d700 1 -- 172.21.15.111:0/3813806319 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3006aed0 con 0x7fdf300af9f0 2023-11-28T12:59:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.299+0000 7fdf2dd9b700 1 --2- 172.21.15.111:0/3813806319 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf300af5a0 0x7fdf300b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.299+0000 7fdf2ed9d700 1 --2- 172.21.15.111:0/3813806319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf300af9f0 0x7fdf300b09f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009d90 tx=0x7fdf2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5a745087ac861b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.299+0000 7fdf2d59a700 1 -- 172.21.15.111:0/3813806319 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2800e070 con 0x7fdf300af9f0 2023-11-28T12:59:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.299+0000 7fdf2d59a700 1 -- 172.21.15.111:0/3813806319 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2800bbd0 con 0x7fdf300af9f0 2023-11-28T12:59:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.299+0000 7fdf2d59a700 1 -- 172.21.15.111:0/3813806319 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28012430 con 0x7fdf300af9f0 2023-11-28T12:59:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.300+0000 7fdf34cf2700 1 -- 172.21.15.111:0/3813806319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf300af9f0 msgr2=0x7fdf300b09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.300+0000 7fdf34cf2700 1 --2- 172.21.15.111:0/3813806319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf300af9f0 0x7fdf300b09f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009d90 tx=0x7fdf2800b330 comp rx=0 tx=0).stop 2023-11-28T12:59:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.300+0000 7fdf34cf2700 1 -- 172.21.15.111:0/3813806319 shutdown_connections 2023-11-28T12:59:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.300+0000 7fdf34cf2700 1 --2- 172.21.15.111:0/3813806319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf300af9f0 0x7fdf300b09f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.300+0000 7fdf34cf2700 1 --2- 172.21.15.111:0/3813806319 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf300af5a0 0x7fdf300b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.300+0000 7fdf34cf2700 1 -- 172.21.15.111:0/3813806319 >> 172.21.15.111:0/3813806319 conn(0x7fdf3006bca0 msgr2=0x7fdf300cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:37.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.300+0000 7fdf34cf2700 1 -- 172.21.15.111:0/3813806319 shutdown_connections 2023-11-28T12:59:37.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.300+0000 7fdf34cf2700 1 -- 172.21.15.111:0/3813806319 wait complete. 2023-11-28T12:59:37.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.301+0000 7fdf34cf2700 1 Processor -- start 2023-11-28T12:59:37.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.301+0000 7fdf34cf2700 1 -- start start 2023-11-28T12:59:37.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf34cf2700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf300af5a0 0x7fdf301e87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf34cf2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf300af9f0 0x7fdf300c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf34cf2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf3015fed0 0x7fdf300c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf34cf2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf300c7e60 con 0x7fdf3015fed0 2023-11-28T12:59:37.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf34cf2700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf300c7500 con 0x7fdf300af5a0 2023-11-28T12:59:37.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf34cf2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf300b5270 con 0x7fdf300af9f0 2023-11-28T12:59:37.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf2ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf3015fed0 0x7fdf300c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf2ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf3015fed0 0x7fdf300c3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33916/0 (socket says 172.21.15.111:33916) 2023-11-28T12:59:37.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf2ed9d700 1 -- 172.21.15.111:0/886064851 learned_addr learned my addr 172.21.15.111:0/886064851 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:37.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf2dd9b700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf300af9f0 0x7fdf300c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf2e59c700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf300af5a0 0x7fdf301e87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf2ed9d700 1 -- 172.21.15.111:0/886064851 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf300af9f0 msgr2=0x7fdf300c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf2ed9d700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf300af9f0 0x7fdf300c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf2ed9d700 1 -- 172.21.15.111:0/886064851 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf300af5a0 msgr2=0x7fdf301e87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf2ed9d700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf300af5a0 0x7fdf301e87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf2ed9d700 1 -- 172.21.15.111:0/886064851 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf28009a70 con 0x7fdf3015fed0 2023-11-28T12:59:37.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf2dd9b700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf300af9f0 0x7fdf300c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:37.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf2ed9d700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf3015fed0 0x7fdf300c3c20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdf28014040 tx=0x7fdf28004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:37.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.302+0000 7fdf1effd700 1 -- 172.21.15.111:0/886064851 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf280098f0 con 0x7fdf3015fed0 2023-11-28T12:59:37.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.303+0000 7fdf1effd700 1 -- 172.21.15.111:0/886064851 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf28020070 con 0x7fdf3015fed0 2023-11-28T12:59:37.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.303+0000 7fdf34cf2700 1 -- 172.21.15.111:0/886064851 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf301602a0 con 0x7fdf3015fed0 2023-11-28T12:59:37.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.303+0000 7fdf1effd700 1 -- 172.21.15.111:0/886064851 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28004280 con 0x7fdf3015fed0 2023-11-28T12:59:37.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.303+0000 7fdf34cf2700 1 -- 172.21.15.111:0/886064851 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf300cea90 con 0x7fdf3015fed0 2023-11-28T12:59:37.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.303+0000 7fdf34cf2700 1 -- 172.21.15.111:0/886064851 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf300ce6b0 con 0x7fdf3015fed0 2023-11-28T12:59:37.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.304+0000 7fdf1effd700 1 -- 172.21.15.111:0/886064851 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdf280043e0 con 0x7fdf3015fed0 2023-11-28T12:59:37.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.304+0000 7fdf1effd700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdf100682f0 0x7fdf1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.304+0000 7fdf1effd700 1 -- 172.21.15.111:0/886064851 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fdf28022070 con 0x7fdf3015fed0 2023-11-28T12:59:37.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.304+0000 7fdf2e59c700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdf100682f0 0x7fdf1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.305+0000 7fdf2e59c700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdf100682f0 0x7fdf1006a7a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdf24005910 tx=0x7fdf24009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:37.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.309+0000 7fdf1effd700 1 -- 172.21.15.111:0/886064851 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdf28054ab0 con 0x7fdf3015fed0 2023-11-28T12:59:37.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.491+0000 7fdf34cf2700 1 -- 172.21.15.111:0/886064851 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fdf300c7fa0 con 0x7fdf3015fed0 2023-11-28T12:59:37.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.491+0000 7fdf1effd700 1 -- 172.21.15.111:0/886064851 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fdf300c7fa0 con 0x7fdf3015fed0 2023-11-28T12:59:37.492 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:37.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.494+0000 7fdf34cf2700 1 -- 172.21.15.111:0/886064851 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdf100682f0 msgr2=0x7fdf1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.494+0000 7fdf34cf2700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdf100682f0 0x7fdf1006a7a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdf24005910 tx=0x7fdf24009040 comp rx=0 tx=0).stop 2023-11-28T12:59:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.494+0000 7fdf34cf2700 1 -- 172.21.15.111:0/886064851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf3015fed0 msgr2=0x7fdf300c3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.494+0000 7fdf34cf2700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf3015fed0 0x7fdf300c3c20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdf28014040 tx=0x7fdf28004ad0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.494+0000 7fdf34cf2700 1 -- 172.21.15.111:0/886064851 shutdown_connections 2023-11-28T12:59:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.494+0000 7fdf34cf2700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf3015fed0 0x7fdf300c3c20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.494+0000 7fdf34cf2700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdf100682f0 0x7fdf1006a7a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.494+0000 7fdf34cf2700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf300af5a0 0x7fdf301e87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.494+0000 7fdf34cf2700 1 --2- 172.21.15.111:0/886064851 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf300af9f0 0x7fdf300c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.494+0000 7fdf34cf2700 1 -- 172.21.15.111:0/886064851 >> 172.21.15.111:0/886064851 conn(0x7fdf3006bca0 msgr2=0x7fdf3006f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.494+0000 7fdf34cf2700 1 -- 172.21.15.111:0/886064851 shutdown_connections 2023-11-28T12:59:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.494+0000 7fdf34cf2700 1 -- 172.21.15.111:0/886064851 wait complete. 2023-11-28T12:59:37.506 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:37.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.626+0000 7fb952359700 1 Processor -- start 2023-11-28T12:59:37.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.626+0000 7fb952359700 1 -- start start 2023-11-28T12:59:37.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.627+0000 7fb952359700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb94c15a5d0 0x7fb94c0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.627+0000 7fb952359700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb94c0cb2d0 0x7fb94c0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.627+0000 7fb952359700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c0ae8b0 con 0x7fb94c0b0160 2023-11-28T12:59:37.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.627+0000 7fb952359700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb94c0cc1b0 con 0x7fb94c15a5d0 2023-11-28T12:59:37.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.627+0000 7fb952359700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0cc390 con 0x7fb94c0cb2d0 2023-11-28T12:59:37.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.627+0000 7fb94b7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb94c0b0160 0x7fb94c15e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34698/0 (socket says 172.21.15.111:34698) 2023-11-28T12:59:37.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.627+0000 7fb94b7fe700 1 -- 172.21.15.111:0/2906193091 learned_addr learned my addr 172.21.15.111:0/2906193091 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:37.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.627+0000 7fb9508f6700 1 --2- 172.21.15.111:0/2906193091 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb94c0cb2d0 0x7fb94c0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.627+0000 7fb94bfff700 1 --2- 172.21.15.111:0/2906193091 >> v2:172.21.15.149:3300/0 conn(0x7fb94c15a5d0 0x7fb94c0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809842805 0 0) 0x7fb94c0ae8b0 con 0x7fb94c0b0160 2023-11-28T12:59:37.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9340035b0 con 0x7fb94c0b0160 2023-11-28T12:59:37.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 610232142 0 0) 0x7fb9340035b0 con 0x7fb94c0b0160 2023-11-28T12:59:37.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb94c0cb2d0 msgr2=0x7fb94c0ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:37.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 --2- 172.21.15.111:0/2906193091 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb94c0cb2d0 0x7fb94c0ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 >> v2:172.21.15.149:3300/0 conn(0x7fb94c15a5d0 msgr2=0x7fb94c0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:37.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 --2- 172.21.15.111:0/2906193091 >> v2:172.21.15.149:3300/0 conn(0x7fb94c15a5d0 0x7fb94c0ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c0cc670 con 0x7fb94c0b0160 2023-11-28T12:59:37.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fb93c001670 con 0x7fb94c0b0160 2023-11-28T12:59:37.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 >> v1:172.21.15.111:6789/0 conn(0x7fb94c0b0160 legacy=0x7fb94c15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 --2- 172.21.15.111:0/2906193091 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb94c0cb2d0 0x7fb934003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 --2- 172.21.15.111:0/2906193091 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb934003f80 0x7fb934004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 --2- 172.21.15.111:0/2906193091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb934004d90 0x7fb934005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0cc1b0 con 0x7fb934004d90 2023-11-28T12:59:37.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0cc390 con 0x7fb934003f80 2023-11-28T12:59:37.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb93400ba10 con 0x7fb94c0cb2d0 2023-11-28T12:59:37.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb9508f6700 1 --2- 172.21.15.111:0/2906193091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb934004d90 0x7fb934005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94bfff700 1 --2- 172.21.15.111:0/2906193091 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb94c0cb2d0 0x7fb934003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb94b7fe700 1 --2- 172.21.15.111:0/2906193091 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb934003f80 0x7fb934004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb9508f6700 1 -- 172.21.15.111:0/2906193091 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb94c0cb2d0 msgr2=0x7fb934003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:37.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb9508f6700 1 --2- 172.21.15.111:0/2906193091 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb94c0cb2d0 0x7fb934003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb9508f6700 1 -- 172.21.15.111:0/2906193091 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb934003f80 msgr2=0x7fb934004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:37.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb9508f6700 1 --2- 172.21.15.111:0/2906193091 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb934003f80 0x7fb934004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb9508f6700 1 -- 172.21.15.111:0/2906193091 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9340035b0 con 0x7fb934004d90 2023-11-28T12:59:37.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.628+0000 7fb9508f6700 1 --2- 172.21.15.111:0/2906193091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb934004d90 0x7fb934005200 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb944009fc0 tx=0x7fb934005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7ecca6b360b03f6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:37.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.629+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94400ec30 con 0x7fb934004d90 2023-11-28T12:59:37.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.629+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94400f040 con 0x7fb934004d90 2023-11-28T12:59:37.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.629+0000 7fb94affd700 1 -- 172.21.15.111:0/2906193091 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944014480 con 0x7fb934004d90 2023-11-28T12:59:37.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.629+0000 7fb952359700 1 -- 172.21.15.111:0/2906193091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb934004d90 msgr2=0x7fb934005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.629+0000 7fb952359700 1 --2- 172.21.15.111:0/2906193091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb934004d90 0x7fb934005200 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb944009fc0 tx=0x7fb934005880 comp rx=0 tx=0).stop 2023-11-28T12:59:37.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.629+0000 7fb9508f6700 1 -- 172.21.15.111:0/2906193091 reap_dead start 2023-11-28T12:59:37.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.629+0000 7fb952359700 1 -- 172.21.15.111:0/2906193091 shutdown_connections 2023-11-28T12:59:37.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.629+0000 7fb952359700 1 -- 172.21.15.111:0/2906193091 >> 172.21.15.111:0/2906193091 conn(0x7fb94c09a040 msgr2=0x7fb94c0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:37.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.629+0000 7fb952359700 1 -- 172.21.15.111:0/2906193091 shutdown_connections 2023-11-28T12:59:37.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.629+0000 7fb952359700 1 -- 172.21.15.111:0/2906193091 wait complete. 2023-11-28T12:59:37.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.629+0000 7fb952359700 1 Processor -- start 2023-11-28T12:59:37.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.630+0000 7fb952359700 1 -- start start 2023-11-28T12:59:37.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.630+0000 7fb952359700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb94c0c6730 0x7fb94c0c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.630+0000 7fb952359700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb94c0c70a0 0x7fb94c153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.630+0000 7fb952359700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb94c0c6b00 0x7fb94c154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.630+0000 7fb952359700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c156ff0 con 0x7fb94c0c6b00 2023-11-28T12:59:37.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.630+0000 7fb952359700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c156370 con 0x7fb94c0c6730 2023-11-28T12:59:37.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.630+0000 7fb952359700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c156650 con 0x7fb94c0c70a0 2023-11-28T12:59:37.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.630+0000 7fb9508f6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb94c0c6b00 0x7fb94c154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.630+0000 7fb9508f6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb94c0c6b00 0x7fb94c154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33934/0 (socket says 172.21.15.111:33934) 2023-11-28T12:59:37.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.630+0000 7fb9508f6700 1 -- 172.21.15.111:0/756778805 learned_addr learned my addr 172.21.15.111:0/756778805 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:37.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.631+0000 7fb94bfff700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb94c0c6730 0x7fb94c0c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.631+0000 7fb94b7fe700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb94c0c70a0 0x7fb94c153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.631+0000 7fb9508f6700 1 -- 172.21.15.111:0/756778805 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb94c0c70a0 msgr2=0x7fb94c153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.631+0000 7fb9508f6700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb94c0c70a0 0x7fb94c153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.631+0000 7fb9508f6700 1 -- 172.21.15.111:0/756778805 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb94c0c6730 msgr2=0x7fb94c0c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.631+0000 7fb9508f6700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb94c0c6730 0x7fb94c0c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.631+0000 7fb9508f6700 1 -- 172.21.15.111:0/756778805 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c0ccc30 con 0x7fb94c0c6b00 2023-11-28T12:59:37.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.631+0000 7fb9508f6700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb94c0c6b00 0x7fb94c154300 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb944016000 tx=0x7fb944009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:37.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.631+0000 7fb948ff9700 1 -- 172.21.15.111:0/756778805 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944007000 con 0x7fb94c0c6b00 2023-11-28T12:59:37.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.631+0000 7fb948ff9700 1 -- 172.21.15.111:0/756778805 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb944022070 con 0x7fb94c0c6b00 2023-11-28T12:59:37.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.631+0000 7fb952359700 1 -- 172.21.15.111:0/756778805 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94c0cc950 con 0x7fb94c0c6b00 2023-11-28T12:59:37.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.631+0000 7fb948ff9700 1 -- 172.21.15.111:0/756778805 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944029480 con 0x7fb94c0c6b00 2023-11-28T12:59:37.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.631+0000 7fb952359700 1 -- 172.21.15.111:0/756778805 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c0ccf10 con 0x7fb94c0c6b00 2023-11-28T12:59:37.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.632+0000 7fb952359700 1 -- 172.21.15.111:0/756778805 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94c067170 con 0x7fb94c0c6b00 2023-11-28T12:59:37.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.632+0000 7fb948ff9700 1 -- 172.21.15.111:0/756778805 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb94401e430 con 0x7fb94c0c6b00 2023-11-28T12:59:37.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.633+0000 7fb948ff9700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb9380682f0 0x7fb93806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.633+0000 7fb948ff9700 1 -- 172.21.15.111:0/756778805 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb944024070 con 0x7fb94c0c6b00 2023-11-28T12:59:37.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.633+0000 7fb94bfff700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb9380682f0 0x7fb93806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.633+0000 7fb94bfff700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb9380682f0 0x7fb93806a7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb940009c00 tx=0x7fb940008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:37.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.637+0000 7fb948ff9700 1 -- 172.21.15.111:0/756778805 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb944054390 con 0x7fb94c0c6b00 2023-11-28T12:59:37.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.818+0000 7fb952359700 1 -- 172.21.15.111:0/756778805 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb94c155a70 con 0x7fb94c0c6b00 2023-11-28T12:59:37.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.820+0000 7fb948ff9700 1 -- 172.21.15.111:0/756778805 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19796 (secure 0 0 0) 0x7fb944018020 con 0x7fb94c0c6b00 2023-11-28T12:59:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:37.822 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:00.053943+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:37.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.822+0000 7fb952359700 1 -- 172.21.15.111:0/756778805 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb9380682f0 msgr2=0x7fb93806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.822+0000 7fb952359700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb9380682f0 0x7fb93806a7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb940009c00 tx=0x7fb940008040 comp rx=0 tx=0).stop 2023-11-28T12:59:37.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.822+0000 7fb952359700 1 -- 172.21.15.111:0/756778805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb94c0c6b00 msgr2=0x7fb94c154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.822+0000 7fb952359700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb94c0c6b00 0x7fb94c154300 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb944016000 tx=0x7fb944009750 comp rx=0 tx=0).stop 2023-11-28T12:59:37.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.822+0000 7fb952359700 1 -- 172.21.15.111:0/756778805 shutdown_connections 2023-11-28T12:59:37.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.822+0000 7fb952359700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb94c0c6b00 0x7fb94c154300 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.822+0000 7fb952359700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb9380682f0 0x7fb93806a7a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.822+0000 7fb952359700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb94c0c6730 0x7fb94c0c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.822+0000 7fb952359700 1 --2- 172.21.15.111:0/756778805 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb94c0c70a0 0x7fb94c153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.822+0000 7fb952359700 1 -- 172.21.15.111:0/756778805 >> 172.21.15.111:0/756778805 conn(0x7fb94c09a040 msgr2=0x7fb94c154a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:37.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.822+0000 7fb952359700 1 -- 172.21.15.111:0/756778805 shutdown_connections 2023-11-28T12:59:37.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.822+0000 7fb952359700 1 -- 172.21.15.111:0/756778805 wait complete. 2023-11-28T12:59:37.836 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T12:58:53.275206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T12:58:54.588499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T12:58:57.666033+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T12:59:37.836 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:59:37.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.958+0000 7f5a8e87d700 1 Processor -- start 2023-11-28T12:59:37.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.958+0000 7f5a8e87d700 1 -- start start 2023-11-28T12:59:37.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.959+0000 7f5a8e87d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5a880b0d70 0x7f5a881604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.959+0000 7f5a8e87d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8815cd90 con 0x7f5a880b1140 2023-11-28T12:59:37.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.959+0000 7f5a8e87d700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a881613e0 con 0x7f5a880cc550 2023-11-28T12:59:37.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.959+0000 7f5a8e87d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a880cedd0 con 0x7f5a880b0d70 2023-11-28T12:59:37.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.959+0000 7f5a87fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5a880b0d70 0x7f5a881604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.959+0000 7f5a877fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5a880b1140 0x7f5a880ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34716/0 (socket says 172.21.15.111:34716) 2023-11-28T12:59:37.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.959+0000 7f5a877fe700 1 -- 172.21.15.111:0/2873890965 learned_addr learned my addr 172.21.15.111:0/2873890965 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:37.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.959+0000 7f5a87fff700 1 -- 172.21.15.111:0/2873890965 >> v1:172.21.15.149:6789/0 conn(0x7f5a880cc550 legacy=0x7f5a880af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:37.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.959+0000 7f5a87fff700 1 -- 172.21.15.111:0/2873890965 >> v1:172.21.15.111:6789/0 conn(0x7f5a880b1140 legacy=0x7f5a880ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.959+0000 7f5a86ffd700 1 -- 172.21.15.111:0/2873890965 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212829620 0 0) 0x7f5a78001110 con 0x7f5a880b1140 2023-11-28T12:59:37.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.959+0000 7f5a87fff700 1 -- 172.21.15.111:0/2873890965 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a88160b40 con 0x7f5a880b0d70 2023-11-28T12:59:37.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.959+0000 7f5a87fff700 1 --2- 172.21.15.111:0/2873890965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a880b0d70 0x7f5a881604b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009820 tx=0x7f5a7c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c510c56d8be374c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:37.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.960+0000 7f5a86ffd700 1 -- 172.21.15.111:0/2873890965 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c005e30 con 0x7f5a880b0d70 2023-11-28T12:59:37.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.960+0000 7f5a86ffd700 1 -- 172.21.15.111:0/2873890965 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c004030 con 0x7f5a880b0d70 2023-11-28T12:59:37.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.960+0000 7f5a86ffd700 1 -- 172.21.15.111:0/2873890965 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c0043c0 con 0x7f5a880b0d70 2023-11-28T12:59:37.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.960+0000 7f5a8e87d700 1 -- 172.21.15.111:0/2873890965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a880b0d70 msgr2=0x7f5a881604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.960+0000 7f5a8e87d700 1 --2- 172.21.15.111:0/2873890965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a880b0d70 0x7f5a881604b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009820 tx=0x7f5a7c009b30 comp rx=0 tx=0).stop 2023-11-28T12:59:37.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.960+0000 7f5a8e87d700 1 -- 172.21.15.111:0/2873890965 shutdown_connections 2023-11-28T12:59:37.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.960+0000 7f5a8e87d700 1 --2- 172.21.15.111:0/2873890965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a880b0d70 0x7f5a881604b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.960+0000 7f5a8e87d700 1 -- 172.21.15.111:0/2873890965 >> 172.21.15.111:0/2873890965 conn(0x7f5a88098ee0 msgr2=0x7f5a8806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:37.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.960+0000 7f5a8e87d700 1 -- 172.21.15.111:0/2873890965 shutdown_connections 2023-11-28T12:59:37.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.960+0000 7f5a8e87d700 1 -- 172.21.15.111:0/2873890965 wait complete. 2023-11-28T12:59:37.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.961+0000 7f5a8e87d700 1 Processor -- start 2023-11-28T12:59:37.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.961+0000 7f5a8e87d700 1 -- start start 2023-11-28T12:59:37.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.962+0000 7f5a8e87d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5a880b0d70 0x7f5a880b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.962+0000 7f5a8e87d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5a880b1140 0x7f5a8815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.962+0000 7f5a8e87d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a880cc550 0x7f5a8815d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.962+0000 7f5a8e87d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a880c41f0 con 0x7f5a880cc550 2023-11-28T12:59:37.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.962+0000 7f5a8e87d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a880c3be0 con 0x7f5a880b0d70 2023-11-28T12:59:37.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.962+0000 7f5a8e87d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a881e8040 con 0x7f5a880b1140 2023-11-28T12:59:37.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.962+0000 7f5a8ce1a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a880cc550 0x7f5a8815d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.962+0000 7f5a8ce1a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a880cc550 0x7f5a8815d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33946/0 (socket says 172.21.15.111:33946) 2023-11-28T12:59:37.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.962+0000 7f5a8ce1a700 1 -- 172.21.15.111:0/1262938969 learned_addr learned my addr 172.21.15.111:0/1262938969 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:37.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.962+0000 7f5a87fff700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5a880b0d70 0x7f5a880b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.963+0000 7f5a877fe700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5a880b1140 0x7f5a8815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.963+0000 7f5a8ce1a700 1 -- 172.21.15.111:0/1262938969 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5a880b1140 msgr2=0x7f5a8815cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.963+0000 7f5a8ce1a700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5a880b1140 0x7f5a8815cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.963+0000 7f5a8ce1a700 1 -- 172.21.15.111:0/1262938969 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5a880b0d70 msgr2=0x7f5a880b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:37.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.963+0000 7f5a8ce1a700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5a880b0d70 0x7f5a880b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:37.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.963+0000 7f5a8ce1a700 1 -- 172.21.15.111:0/1262938969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7c009430 con 0x7f5a880cc550 2023-11-28T12:59:37.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.963+0000 7f5a87fff700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5a880b0d70 0x7f5a880b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:37.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.963+0000 7f5a8ce1a700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a880cc550 0x7f5a8815d470 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5a8000b970 tx=0x7f5a8000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:37.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.963+0000 7f5a84ff9700 1 -- 172.21.15.111:0/1262938969 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8000c9b0 con 0x7f5a880cc550 2023-11-28T12:59:37.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.963+0000 7f5a84ff9700 1 -- 172.21.15.111:0/1262938969 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8000cb10 con 0x7f5a880cc550 2023-11-28T12:59:37.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.963+0000 7f5a8e87d700 1 -- 172.21.15.111:0/1262938969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a88161100 con 0x7f5a880cc550 2023-11-28T12:59:37.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.963+0000 7f5a84ff9700 1 -- 172.21.15.111:0/1262938969 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80014440 con 0x7f5a880cc550 2023-11-28T12:59:37.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.963+0000 7f5a8e87d700 1 -- 172.21.15.111:0/1262938969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a88160e20 con 0x7f5a880cc550 2023-11-28T12:59:37.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.964+0000 7f5a8e87d700 1 -- 172.21.15.111:0/1262938969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a880af340 con 0x7f5a880cc550 2023-11-28T12:59:37.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.964+0000 7f5a84ff9700 1 -- 172.21.15.111:0/1262938969 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5a8001b030 con 0x7f5a880cc550 2023-11-28T12:59:37.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.965+0000 7f5a84ff9700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5a740682f0 0x7f5a7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:37.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.965+0000 7f5a84ff9700 1 -- 172.21.15.111:0/1262938969 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f5a80085b20 con 0x7f5a880cc550 2023-11-28T12:59:37.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.965+0000 7f5a87fff700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5a740682f0 0x7f5a7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:37.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.965+0000 7f5a87fff700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5a740682f0 0x7f5a7406a7a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c014040 tx=0x7f5a7c008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:37.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:37.969+0000 7f5a84ff9700 1 -- 172.21.15.111:0/1262938969 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5a80052030 con 0x7f5a880cc550 2023-11-28T12:59:38.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.151+0000 7f5a8e87d700 1 -- 172.21.15.111:0/1262938969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5a88067170 con 0x7f5a880cc550 2023-11-28T12:59:38.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.151+0000 7f5a84ff9700 1 -- 172.21.15.111:0/1262938969 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5a80055650 con 0x7f5a880cc550 2023-11-28T12:59:38.153 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:59:38.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.153+0000 7f5a8e87d700 1 -- 172.21.15.111:0/1262938969 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5a740682f0 msgr2=0x7f5a7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.153+0000 7f5a8e87d700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5a740682f0 0x7f5a7406a7a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c014040 tx=0x7f5a7c008070 comp rx=0 tx=0).stop 2023-11-28T12:59:38.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.153+0000 7f5a8e87d700 1 -- 172.21.15.111:0/1262938969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a880cc550 msgr2=0x7f5a8815d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.153+0000 7f5a8e87d700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a880cc550 0x7f5a8815d470 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5a8000b970 tx=0x7f5a8000bd30 comp rx=0 tx=0).stop 2023-11-28T12:59:38.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.153+0000 7f5a8e87d700 1 -- 172.21.15.111:0/1262938969 shutdown_connections 2023-11-28T12:59:38.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.153+0000 7f5a8e87d700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a880cc550 0x7f5a8815d470 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.153+0000 7f5a8e87d700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5a740682f0 0x7f5a7406a7a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.153+0000 7f5a8e87d700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5a880b0d70 0x7f5a880b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.153+0000 7f5a8e87d700 1 --2- 172.21.15.111:0/1262938969 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5a880b1140 0x7f5a8815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.153+0000 7f5a8e87d700 1 -- 172.21.15.111:0/1262938969 >> 172.21.15.111:0/1262938969 conn(0x7f5a88098ee0 msgr2=0x7f5a880af860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:38.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.153+0000 7f5a8e87d700 1 -- 172.21.15.111:0/1262938969 shutdown_connections 2023-11-28T12:59:38.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.153+0000 7f5a8e87d700 1 -- 172.21.15.111:0/1262938969 wait complete. 2023-11-28T12:59:38.167 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:59:38.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.288+0000 7f3135a28700 1 Processor -- start 2023-11-28T12:59:38.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.288+0000 7f3135a28700 1 -- start start 2023-11-28T12:59:38.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.289+0000 7f3135a28700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f31300c9300 0x7f31300ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.289+0000 7f3135a28700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f313006dd00 0x7f3130158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.289+0000 7f3135a28700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f313006a270 0x7f3130070910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.289+0000 7f3135a28700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f313006e150 con 0x7f313006dd00 2023-11-28T12:59:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.289+0000 7f3135a28700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f313006d870 con 0x7f313006a270 2023-11-28T12:59:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.289+0000 7f3135a28700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313006cc60 con 0x7f31300c9300 2023-11-28T12:59:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.289+0000 7f312e7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f313006dd00 0x7f3130158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.289+0000 7f312e7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f313006dd00 0x7f3130158cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33950/0 (socket says 172.21.15.111:33950) 2023-11-28T12:59:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.289+0000 7f312e7fc700 1 -- 172.21.15.111:0/1450388610 learned_addr learned my addr 172.21.15.111:0/1450388610 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.290+0000 7f312f7fe700 1 --2- 172.21.15.111:0/1450388610 >> v2:172.21.15.149:3300/0 conn(0x7f313006a270 0x7f3130070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.290+0000 7f312effd700 1 --2- 172.21.15.111:0/1450388610 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f31300c9300 0x7f31300ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.290+0000 7f312e7fc700 1 -- 172.21.15.111:0/1450388610 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f31300c9300 msgr2=0x7f31300ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.290+0000 7f312e7fc700 1 --2- 172.21.15.111:0/1450388610 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f31300c9300 0x7f31300ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.290+0000 7f312e7fc700 1 -- 172.21.15.111:0/1450388610 >> v2:172.21.15.149:3300/0 conn(0x7f313006a270 msgr2=0x7f3130070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.290+0000 7f312e7fc700 1 --2- 172.21.15.111:0/1450388610 >> v2:172.21.15.149:3300/0 conn(0x7f313006a270 0x7f3130070910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.290+0000 7f312e7fc700 1 -- 172.21.15.111:0/1450388610 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313006cf40 con 0x7f313006dd00 2023-11-28T12:59:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.290+0000 7f312e7fc700 1 --2- 172.21.15.111:0/1450388610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313006dd00 0x7f3130158cb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f31300b4b80 tx=0x7f312000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94f27bd123b057f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.290+0000 7f312dffb700 1 -- 172.21.15.111:0/1450388610 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312000dd30 con 0x7f313006dd00 2023-11-28T12:59:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.290+0000 7f312dffb700 1 -- 172.21.15.111:0/1450388610 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312000de90 con 0x7f313006dd00 2023-11-28T12:59:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.290+0000 7f312dffb700 1 -- 172.21.15.111:0/1450388610 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120010690 con 0x7f313006dd00 2023-11-28T12:59:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.291+0000 7f3135a28700 1 -- 172.21.15.111:0/1450388610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313006dd00 msgr2=0x7f3130158cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.291+0000 7f3135a28700 1 --2- 172.21.15.111:0/1450388610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313006dd00 0x7f3130158cb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f31300b4b80 tx=0x7f312000be70 comp rx=0 tx=0).stop 2023-11-28T12:59:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.291+0000 7f3135a28700 1 -- 172.21.15.111:0/1450388610 shutdown_connections 2023-11-28T12:59:38.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.291+0000 7f3135a28700 1 --2- 172.21.15.111:0/1450388610 >> v2:172.21.15.149:3300/0 conn(0x7f313006a270 0x7f3130070910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.291+0000 7f3135a28700 1 --2- 172.21.15.111:0/1450388610 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f31300c9300 0x7f31300ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.291+0000 7f3135a28700 1 --2- 172.21.15.111:0/1450388610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313006dd00 0x7f3130158cb0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.291+0000 7f3135a28700 1 -- 172.21.15.111:0/1450388610 >> 172.21.15.111:0/1450388610 conn(0x7f31300cb490 msgr2=0x7f31300cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:38.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.291+0000 7f3135a28700 1 -- 172.21.15.111:0/1450388610 shutdown_connections 2023-11-28T12:59:38.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.291+0000 7f3135a28700 1 -- 172.21.15.111:0/1450388610 wait complete. 2023-11-28T12:59:38.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.292+0000 7f3135a28700 1 Processor -- start 2023-11-28T12:59:38.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.292+0000 7f3135a28700 1 -- start start 2023-11-28T12:59:38.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f3135a28700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f313006a270 0x7f31300c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f3135a28700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f313006dd00 0x7f31301562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f3135a28700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31300c9300 0x7f31300b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f3135a28700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31300c3100 con 0x7f31300c9300 2023-11-28T12:59:38.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f3135a28700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31301567f0 con 0x7f313006a270 2023-11-28T12:59:38.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f3135a28700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31300c3e90 con 0x7f313006dd00 2023-11-28T12:59:38.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f312f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31300c9300 0x7f31300b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f312f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31300c9300 0x7f31300b2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33960/0 (socket says 172.21.15.111:33960) 2023-11-28T12:59:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f312f7fe700 1 -- 172.21.15.111:0/2981956883 learned_addr learned my addr 172.21.15.111:0/2981956883 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f312effd700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f313006a270 0x7f31300c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f312e7fc700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f313006dd00 0x7f31301562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f312f7fe700 1 -- 172.21.15.111:0/2981956883 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f313006dd00 msgr2=0x7f31301562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f312f7fe700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f313006dd00 0x7f31301562b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f312f7fe700 1 -- 172.21.15.111:0/2981956883 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f313006a270 msgr2=0x7f31300c2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f312f7fe700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f313006a270 0x7f31300c2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f312f7fe700 1 -- 172.21.15.111:0/2981956883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312000b7f0 con 0x7f31300c9300 2023-11-28T12:59:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f312f7fe700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31300c9300 0x7f31300b2ca0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3128007e60 tx=0x7f312800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f3117fff700 1 -- 172.21.15.111:0/2981956883 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312800ed30 con 0x7f31300c9300 2023-11-28T12:59:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f3117fff700 1 -- 172.21.15.111:0/2981956883 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3128014420 con 0x7f31300c9300 2023-11-28T12:59:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f3135a28700 1 -- 172.21.15.111:0/2981956883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313006d500 con 0x7f31300c9300 2023-11-28T12:59:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.293+0000 7f3117fff700 1 -- 172.21.15.111:0/2981956883 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31280147d0 con 0x7f31300c9300 2023-11-28T12:59:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.294+0000 7f3135a28700 1 -- 172.21.15.111:0/2981956883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313006d220 con 0x7f31300c9300 2023-11-28T12:59:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.294+0000 7f3135a28700 1 -- 172.21.15.111:0/2981956883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3130071b20 con 0x7f31300c9300 2023-11-28T12:59:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.295+0000 7f3117fff700 1 -- 172.21.15.111:0/2981956883 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3128012050 con 0x7f31300c9300 2023-11-28T12:59:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.295+0000 7f3117fff700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f311c0682f0 0x7f311c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.295+0000 7f3117fff700 1 -- 172.21.15.111:0/2981956883 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3128086550 con 0x7f31300c9300 2023-11-28T12:59:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.295+0000 7f312effd700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f311c0682f0 0x7f311c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.296+0000 7f312effd700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f311c0682f0 0x7f311c06a7a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3124006b20 tx=0x7f3124008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:38.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.300+0000 7f3117fff700 1 -- 172.21.15.111:0/2981956883 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f31280529d0 con 0x7f31300c9300 2023-11-28T12:59:38.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.481+0000 7f3135a28700 1 -- 172.21.15.111:0/2981956883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3130067880 con 0x7f31300c9300 2023-11-28T12:59:38.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.482+0000 7f3117fff700 1 -- 172.21.15.111:0/2981956883 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3128055ff0 con 0x7f31300c9300 2023-11-28T12:59:38.484 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:59:38.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.484+0000 7f3135a28700 1 -- 172.21.15.111:0/2981956883 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f311c0682f0 msgr2=0x7f311c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.484+0000 7f3135a28700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f311c0682f0 0x7f311c06a7a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3124006b20 tx=0x7f3124008040 comp rx=0 tx=0).stop 2023-11-28T12:59:38.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.484+0000 7f3135a28700 1 -- 172.21.15.111:0/2981956883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31300c9300 msgr2=0x7f31300b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.484+0000 7f3135a28700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31300c9300 0x7f31300b2ca0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3128007e60 tx=0x7f312800e3c0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.484+0000 7f3135a28700 1 -- 172.21.15.111:0/2981956883 shutdown_connections 2023-11-28T12:59:38.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.484+0000 7f3135a28700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31300c9300 0x7f31300b2ca0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.484+0000 7f3135a28700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f311c0682f0 0x7f311c06a7a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.484+0000 7f3135a28700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f313006a270 0x7f31300c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.484+0000 7f3135a28700 1 --2- 172.21.15.111:0/2981956883 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f313006dd00 0x7f31301562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.484+0000 7f3135a28700 1 -- 172.21.15.111:0/2981956883 >> 172.21.15.111:0/2981956883 conn(0x7f31300cb490 msgr2=0x7f31300cc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:38.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.484+0000 7f3135a28700 1 -- 172.21.15.111:0/2981956883 shutdown_connections 2023-11-28T12:59:38.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.484+0000 7f3135a28700 1 -- 172.21.15.111:0/2981956883 wait complete. 2023-11-28T12:59:38.498 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T12:59:38.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.619+0000 7fbee81dc700 1 Processor -- start 2023-11-28T12:59:38.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.619+0000 7fbee81dc700 1 -- start start 2023-11-28T12:59:38.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee81dc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee0159960 0x7fbee00b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee81dc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbee00aecc0 0x7fbee00cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee81dc700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee00ad420 con 0x7fbee00cdbc0 2023-11-28T12:59:38.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee81dc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbee00ce040 con 0x7fbee00aecc0 2023-11-28T12:59:38.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee81dc700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0159d30 con 0x7fbee0159960 2023-11-28T12:59:38.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee5777700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbee00aecc0 0x7fbee00cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee5777700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbee00aecc0 0x7fbee00cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33964/0 (socket says 172.21.15.111:33964) 2023-11-28T12:59:38.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee5777700 1 -- 172.21.15.111:0/422292023 learned_addr learned my addr 172.21.15.111:0/422292023 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:38.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee5f78700 1 --2- 172.21.15.111:0/422292023 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee0159960 0x7fbee00b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee5777700 1 -- 172.21.15.111:0/422292023 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee0159960 msgr2=0x7fbee00b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:38.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee5777700 1 --2- 172.21.15.111:0/422292023 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee0159960 0x7fbee00b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee5777700 1 -- 172.21.15.111:0/422292023 >> v1:172.21.15.149:6789/0 conn(0x7fbee00cdbc0 legacy=0x7fbee00afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee5777700 1 -- 172.21.15.111:0/422292023 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee00ac810 con 0x7fbee00aecc0 2023-11-28T12:59:38.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee5777700 1 --2- 172.21.15.111:0/422292023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee00aecc0 0x7fbee00cd680 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbed00098f0 tx=0x7fbed0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f633d67c31c1842b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:38.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.620+0000 7fbee4f76700 1 -- 172.21.15.111:0/422292023 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed000b8a0 con 0x7fbee00aecc0 2023-11-28T12:59:38.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.621+0000 7fbee4f76700 1 -- 172.21.15.111:0/422292023 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed000ba00 con 0x7fbee00aecc0 2023-11-28T12:59:38.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.621+0000 7fbee4f76700 1 -- 172.21.15.111:0/422292023 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed000bd90 con 0x7fbee00aecc0 2023-11-28T12:59:38.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.621+0000 7fbee81dc700 1 -- 172.21.15.111:0/422292023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee00aecc0 msgr2=0x7fbee00cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.621+0000 7fbee81dc700 1 --2- 172.21.15.111:0/422292023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee00aecc0 0x7fbee00cd680 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbed00098f0 tx=0x7fbed0009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:38.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.621+0000 7fbee81dc700 1 -- 172.21.15.111:0/422292023 shutdown_connections 2023-11-28T12:59:38.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.621+0000 7fbee81dc700 1 --2- 172.21.15.111:0/422292023 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee0159960 0x7fbee00b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.621+0000 7fbee81dc700 1 --2- 172.21.15.111:0/422292023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee00aecc0 0x7fbee00cd680 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.621+0000 7fbee81dc700 1 -- 172.21.15.111:0/422292023 >> 172.21.15.111:0/422292023 conn(0x7fbee0098ee0 msgr2=0x7fbee006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:38.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.621+0000 7fbee81dc700 1 -- 172.21.15.111:0/422292023 shutdown_connections 2023-11-28T12:59:38.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.621+0000 7fbee81dc700 1 -- 172.21.15.111:0/422292023 wait complete. 2023-11-28T12:59:38.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.622+0000 7fbee81dc700 1 Processor -- start 2023-11-28T12:59:38.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.622+0000 7fbee81dc700 1 -- start start 2023-11-28T12:59:38.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee81dc700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee00aecc0 0x7fbee0154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee81dc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee00cdbc0 0x7fbee0153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee81dc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee0159960 0x7fbee0150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee81dc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee01505a0 con 0x7fbee00cdbc0 2023-11-28T12:59:38.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee81dc700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0153990 con 0x7fbee00aecc0 2023-11-28T12:59:38.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee81dc700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0153c70 con 0x7fbee0159960 2023-11-28T12:59:38.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee5777700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee00cdbc0 0x7fbee0153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee5777700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee00cdbc0 0x7fbee0153290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33970/0 (socket says 172.21.15.111:33970) 2023-11-28T12:59:38.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee5777700 1 -- 172.21.15.111:0/1739612356 learned_addr learned my addr 172.21.15.111:0/1739612356 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:38.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee5f78700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee00aecc0 0x7fbee0154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee6779700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee0159960 0x7fbee0150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee5777700 1 -- 172.21.15.111:0/1739612356 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee0159960 msgr2=0x7fbee0150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee5777700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee0159960 0x7fbee0150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee5777700 1 -- 172.21.15.111:0/1739612356 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee00aecc0 msgr2=0x7fbee0154a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee5777700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee00aecc0 0x7fbee0154a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.623+0000 7fbee5777700 1 -- 172.21.15.111:0/1739612356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed0009550 con 0x7fbee00cdbc0 2023-11-28T12:59:38.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.624+0000 7fbee5777700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee00cdbc0 0x7fbee0153290 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009c40 tx=0x7fbed0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:38.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.624+0000 7fbed67fc700 1 -- 172.21.15.111:0/1739612356 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed0004300 con 0x7fbee00cdbc0 2023-11-28T12:59:38.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.624+0000 7fbed67fc700 1 -- 172.21.15.111:0/1739612356 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed001f070 con 0x7fbee00cdbc0 2023-11-28T12:59:38.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.624+0000 7fbee81dc700 1 -- 172.21.15.111:0/1739612356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee00acdd0 con 0x7fbee00cdbc0 2023-11-28T12:59:38.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.624+0000 7fbed67fc700 1 -- 172.21.15.111:0/1739612356 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed00036a0 con 0x7fbee00cdbc0 2023-11-28T12:59:38.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.624+0000 7fbee81dc700 1 -- 172.21.15.111:0/1739612356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee00acaf0 con 0x7fbee00cdbc0 2023-11-28T12:59:38.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.625+0000 7fbee81dc700 1 -- 172.21.15.111:0/1739612356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee00ad0b0 con 0x7fbee00cdbc0 2023-11-28T12:59:38.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.625+0000 7fbed67fc700 1 -- 172.21.15.111:0/1739612356 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbed002a020 con 0x7fbee00cdbc0 2023-11-28T12:59:38.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.625+0000 7fbed67fc700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbec40682f0 0x7fbec406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.625+0000 7fbed67fc700 1 -- 172.21.15.111:0/1739612356 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fbed0087460 con 0x7fbee00cdbc0 2023-11-28T12:59:38.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.625+0000 7fbee5f78700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbec40682f0 0x7fbec406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.626+0000 7fbee5f78700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbec40682f0 0x7fbec406a7a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbedc006fd0 tx=0x7fbedc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:38.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.630+0000 7fbed67fc700 1 -- 172.21.15.111:0/1739612356 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbed00539d0 con 0x7fbee00cdbc0 2023-11-28T12:59:38.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.812+0000 7fbee81dc700 1 -- 172.21.15.111:0/1739612356 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbee006c4f0 con 0x7fbee00cdbc0 2023-11-28T12:59:38.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.813+0000 7fbed67fc700 1 -- 172.21.15.111:0/1739612356 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbed0056ff0 con 0x7fbee00cdbc0 2023-11-28T12:59:38.817 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:38.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.815+0000 7fbee81dc700 1 -- 172.21.15.111:0/1739612356 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbec40682f0 msgr2=0x7fbec406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.815+0000 7fbee81dc700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbec40682f0 0x7fbec406a7a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbedc006fd0 tx=0x7fbedc008040 comp rx=0 tx=0).stop 2023-11-28T12:59:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.815+0000 7fbee81dc700 1 -- 172.21.15.111:0/1739612356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee00cdbc0 msgr2=0x7fbee0153290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.815+0000 7fbee81dc700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee00cdbc0 0x7fbee0153290 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009c40 tx=0x7fbed0004750 comp rx=0 tx=0).stop 2023-11-28T12:59:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.815+0000 7fbee81dc700 1 -- 172.21.15.111:0/1739612356 shutdown_connections 2023-11-28T12:59:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.815+0000 7fbee81dc700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee0159960 0x7fbee0150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.815+0000 7fbee81dc700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbec40682f0 0x7fbec406a7a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.815+0000 7fbee81dc700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee00aecc0 0x7fbee0154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.815+0000 7fbee81dc700 1 --2- 172.21.15.111:0/1739612356 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee00cdbc0 0x7fbee0153290 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.815+0000 7fbee81dc700 1 -- 172.21.15.111:0/1739612356 >> 172.21.15.111:0/1739612356 conn(0x7fbee0098ee0 msgr2=0x7fbee0158b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.815+0000 7fbee81dc700 1 -- 172.21.15.111:0/1739612356 shutdown_connections 2023-11-28T12:59:38.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.815+0000 7fbee81dc700 1 -- 172.21.15.111:0/1739612356 wait complete. 2023-11-28T12:59:38.828 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T12:59:38.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.951+0000 7f919c964700 1 Processor -- start 2023-11-28T12:59:38.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.952+0000 7f919c964700 1 -- start start 2023-11-28T12:59:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.952+0000 7f919c964700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9198072300 0x7f91980712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919c964700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91980717f0 con 0x7f919815cc50 2023-11-28T12:59:38.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919c964700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919806fb90 con 0x7f91980cceb0 2023-11-28T12:59:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919c964700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91980726d0 con 0x7f9198072300 2023-11-28T12:59:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f9195d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f919815cc50 0x7f919806b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34756/0 (socket says 172.21.15.111:34756) 2023-11-28T12:59:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f9195d9b700 1 -- 172.21.15.111:0/1430840114 learned_addr learned my addr 172.21.15.111:0/1430840114 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623844059 0 0) 0x7f91980717f0 con 0x7f919815cc50 2023-11-28T12:59:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919659c700 1 --2- 172.21.15.111:0/1430840114 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9198072300 0x7f91980712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91840035b0 con 0x7f919815cc50 2023-11-28T12:59:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2015577472 0 0) 0x7f91840035b0 con 0x7f919815cc50 2023-11-28T12:59:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9198072300 msgr2=0x7f91980712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:38.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 --2- 172.21.15.111:0/1430840114 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9198072300 0x7f91980712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 >> v1:172.21.15.149:6789/0 conn(0x7f91980cceb0 legacy=0x7f919806a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919815d0f0 con 0x7f919815cc50 2023-11-28T12:59:38.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f9188001670 con 0x7f919815cc50 2023-11-28T12:59:38.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 >> v1:172.21.15.111:6789/0 conn(0x7f919815cc50 legacy=0x7f919806b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 --2- 172.21.15.111:0/1430840114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184003a30 0x7f9184003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 --2- 172.21.15.111:0/1430840114 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9184004850 0x7f9184004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 --2- 172.21.15.111:0/1430840114 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9198072300 0x7f9184005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91980726d0 con 0x7f9184003a30 2023-11-28T12:59:38.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918400bb10 con 0x7f9184004850 2023-11-28T12:59:38.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918400bcd0 con 0x7f9198072300 2023-11-28T12:59:38.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.953+0000 7f919659c700 1 --2- 172.21.15.111:0/1430840114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184003a30 0x7f9184003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f9195d9b700 1 --2- 172.21.15.111:0/1430840114 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9184004850 0x7f9184004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f9196d9d700 1 --2- 172.21.15.111:0/1430840114 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9198072300 0x7f9184005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f919659c700 1 -- 172.21.15.111:0/1430840114 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9198072300 msgr2=0x7f9184005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:38.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f919659c700 1 --2- 172.21.15.111:0/1430840114 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9198072300 0x7f9184005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f919659c700 1 -- 172.21.15.111:0/1430840114 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9184004850 msgr2=0x7f9184004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:38.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f919659c700 1 --2- 172.21.15.111:0/1430840114 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9184004850 0x7f9184004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f919659c700 1 -- 172.21.15.111:0/1430840114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91840035b0 con 0x7f9184003a30 2023-11-28T12:59:38.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f9195d9b700 1 --2- 172.21.15.111:0/1430840114 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9184004850 0x7f9184004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:38.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f919659c700 1 --2- 172.21.15.111:0/1430840114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184003a30 0x7f9184003e40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9180009b60 tx=0x7f9184005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50a43cb15ac472a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:38.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918000e6c0 con 0x7f9184003a30 2023-11-28T12:59:38.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918000e820 con 0x7f9184003a30 2023-11-28T12:59:38.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f919559a700 1 -- 172.21.15.111:0/1430840114 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918000eb90 con 0x7f9184003a30 2023-11-28T12:59:38.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f919c964700 1 -- 172.21.15.111:0/1430840114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184003a30 msgr2=0x7f9184003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f919c964700 1 --2- 172.21.15.111:0/1430840114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184003a30 0x7f9184003e40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9180009b60 tx=0x7f9184005880 comp rx=0 tx=0).stop 2023-11-28T12:59:38.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.954+0000 7f9196d9d700 1 -- 172.21.15.111:0/1430840114 reap_dead start 2023-11-28T12:59:38.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.955+0000 7f919c964700 1 -- 172.21.15.111:0/1430840114 shutdown_connections 2023-11-28T12:59:38.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.955+0000 7f919c964700 1 -- 172.21.15.111:0/1430840114 >> 172.21.15.111:0/1430840114 conn(0x7f91980cf2e0 msgr2=0x7f91980cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:38.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.955+0000 7f919c964700 1 -- 172.21.15.111:0/1430840114 shutdown_connections 2023-11-28T12:59:38.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.955+0000 7f919c964700 1 -- 172.21.15.111:0/1430840114 wait complete. 2023-11-28T12:59:38.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.956+0000 7f919c964700 1 Processor -- start 2023-11-28T12:59:38.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.956+0000 7f919c964700 1 -- start start 2023-11-28T12:59:38.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919c964700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184003a30 0x7f9198158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919c964700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91980ab8d0 0x7f9198159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919c964700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91980aefb0 0x7f91980af420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919c964700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919814f1e0 con 0x7f9184003a30 2023-11-28T12:59:38.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919c964700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919815a250 con 0x7f91980ab8d0 2023-11-28T12:59:38.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919c964700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91980b07e0 con 0x7f91980aefb0 2023-11-28T12:59:38.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184003a30 0x7f9198158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184003a30 0x7f9198158050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33990/0 (socket says 172.21.15.111:33990) 2023-11-28T12:59:38.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919659c700 1 -- 172.21.15.111:0/3849353388 learned_addr learned my addr 172.21.15.111:0/3849353388 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:38.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f9196d9d700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91980aefb0 0x7f91980af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f9195d9b700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91980ab8d0 0x7f9198159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919659c700 1 -- 172.21.15.111:0/3849353388 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91980aefb0 msgr2=0x7f91980af420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919659c700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91980aefb0 0x7f91980af420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919659c700 1 -- 172.21.15.111:0/3849353388 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91980ab8d0 msgr2=0x7f9198159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:38.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919659c700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91980ab8d0 0x7f9198159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:38.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919659c700 1 -- 172.21.15.111:0/3849353388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919815d6b0 con 0x7f9184003a30 2023-11-28T12:59:38.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f9195d9b700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91980ab8d0 0x7f9198159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:38.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919659c700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184003a30 0x7f9198158050 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9180004270 tx=0x7f91800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:38.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f918effd700 1 -- 172.21.15.111:0/3849353388 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9180021070 con 0x7f9184003a30 2023-11-28T12:59:38.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f918effd700 1 -- 172.21.15.111:0/3849353388 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91800045b0 con 0x7f9184003a30 2023-11-28T12:59:38.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919c964700 1 -- 172.21.15.111:0/3849353388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919815d3d0 con 0x7f9184003a30 2023-11-28T12:59:38.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f918effd700 1 -- 172.21.15.111:0/3849353388 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918000e6c0 con 0x7f9184003a30 2023-11-28T12:59:38.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.957+0000 7f919c964700 1 -- 172.21.15.111:0/3849353388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919815d990 con 0x7f9184003a30 2023-11-28T12:59:38.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.958+0000 7f919c964700 1 -- 172.21.15.111:0/3849353388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9198067170 con 0x7f9184003a30 2023-11-28T12:59:38.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.959+0000 7f918effd700 1 -- 172.21.15.111:0/3849353388 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f918002e030 con 0x7f9184003a30 2023-11-28T12:59:38.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.959+0000 7f918effd700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f91780682f0 0x7f917806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:38.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.959+0000 7f9195d9b700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f91780682f0 0x7f917806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:38.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.959+0000 7f918effd700 1 -- 172.21.15.111:0/3849353388 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f91800871e0 con 0x7f9184003a30 2023-11-28T12:59:38.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.960+0000 7f9195d9b700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f91780682f0 0x7f917806a7a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9188001c90 tx=0x7f9188001170 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:38.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:38.963+0000 7f918effd700 1 -- 172.21.15.111:0/3849353388 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9180053750 con 0x7f9184003a30 2023-11-28T12:59:39.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.146+0000 7f919c964700 1 -- 172.21.15.111:0/3849353388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9198157b00 con 0x7f9184003a30 2023-11-28T12:59:39.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.146+0000 7f918effd700 1 -- 172.21.15.111:0/3849353388 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9180056d70 con 0x7f9184003a30 2023-11-28T12:59:39.148 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T12:59:39.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.148+0000 7f919c964700 1 -- 172.21.15.111:0/3849353388 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f91780682f0 msgr2=0x7f917806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.148+0000 7f919c964700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f91780682f0 0x7f917806a7a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9188001c90 tx=0x7f9188001170 comp rx=0 tx=0).stop 2023-11-28T12:59:39.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.148+0000 7f919c964700 1 -- 172.21.15.111:0/3849353388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184003a30 msgr2=0x7f9198158050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.148+0000 7f919c964700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184003a30 0x7f9198158050 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9180004270 tx=0x7f91800047c0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.148+0000 7f919c964700 1 -- 172.21.15.111:0/3849353388 shutdown_connections 2023-11-28T12:59:39.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.148+0000 7f919c964700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91980aefb0 0x7f91980af420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.148+0000 7f919c964700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9184003a30 0x7f9198158050 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.148+0000 7f919c964700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f91780682f0 0x7f917806a7a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.148+0000 7f919c964700 1 --2- 172.21.15.111:0/3849353388 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91980ab8d0 0x7f9198159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.148+0000 7f919c964700 1 -- 172.21.15.111:0/3849353388 >> 172.21.15.111:0/3849353388 conn(0x7f91980cf2e0 msgr2=0x7f91980c9910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:39.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.148+0000 7f919c964700 1 -- 172.21.15.111:0/3849353388 shutdown_connections 2023-11-28T12:59:39.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.148+0000 7f919c964700 1 -- 172.21.15.111:0/3849353388 wait complete. 2023-11-28T12:59:39.162 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T12:59:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.283+0000 7f911e81d700 1 Processor -- start 2023-11-28T12:59:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.283+0000 7f911e81d700 1 -- start start 2023-11-28T12:59:39.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.284+0000 7f911e81d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f91180b15c0 0x7f91180ac670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.284+0000 7f911e81d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91180b0250 0x7f91180b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.284+0000 7f911e81d700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91180af650 con 0x7f911815a9e0 2023-11-28T12:59:39.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.284+0000 7f911e81d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91180b0c10 con 0x7f91180b15c0 2023-11-28T12:59:39.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.284+0000 7f911e81d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91180cb6d0 con 0x7f91180b0250 2023-11-28T12:59:39.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.284+0000 7f91177fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f91180b15c0 0x7f91180ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.284+0000 7f91177fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f91180b15c0 0x7f91180ac670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33998/0 (socket says 172.21.15.111:33998) 2023-11-28T12:59:39.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.284+0000 7f91177fe700 1 -- 172.21.15.111:0/1285577292 learned_addr learned my addr 172.21.15.111:0/1285577292 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:39.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.284+0000 7f911cdba700 1 --2- 172.21.15.111:0/1285577292 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91180b0250 0x7f91180b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.284+0000 7f91177fe700 1 -- 172.21.15.111:0/1285577292 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91180b0250 msgr2=0x7f91180b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:39.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.284+0000 7f91177fe700 1 --2- 172.21.15.111:0/1285577292 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91180b0250 0x7f91180b06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.284+0000 7f91177fe700 1 -- 172.21.15.111:0/1285577292 >> v1:172.21.15.149:6789/0 conn(0x7f911815a9e0 legacy=0x7f91180caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:39.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.284+0000 7f91177fe700 1 -- 172.21.15.111:0/1285577292 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91180cb9b0 con 0x7f91180b15c0 2023-11-28T12:59:39.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.285+0000 7f91177fe700 1 --2- 172.21.15.111:0/1285577292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91180b15c0 0x7f91180ac670 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f91000098f0 tx=0x7f9100009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583ffcf3c0f15ba7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:39.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.285+0000 7f9116ffd700 1 -- 172.21.15.111:0/1285577292 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9100004230 con 0x7f91180b15c0 2023-11-28T12:59:39.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.285+0000 7f9116ffd700 1 -- 172.21.15.111:0/1285577292 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9100004390 con 0x7f91180b15c0 2023-11-28T12:59:39.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.285+0000 7f9116ffd700 1 -- 172.21.15.111:0/1285577292 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91000036a0 con 0x7f91180b15c0 2023-11-28T12:59:39.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.285+0000 7f911e81d700 1 -- 172.21.15.111:0/1285577292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91180b15c0 msgr2=0x7f91180ac670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.285+0000 7f911e81d700 1 --2- 172.21.15.111:0/1285577292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91180b15c0 0x7f91180ac670 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f91000098f0 tx=0x7f9100009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:39.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.285+0000 7f911e81d700 1 -- 172.21.15.111:0/1285577292 shutdown_connections 2023-11-28T12:59:39.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.285+0000 7f911e81d700 1 --2- 172.21.15.111:0/1285577292 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91180b0250 0x7f91180b06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.285+0000 7f911e81d700 1 --2- 172.21.15.111:0/1285577292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91180b15c0 0x7f91180ac670 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.285+0000 7f911e81d700 1 -- 172.21.15.111:0/1285577292 >> 172.21.15.111:0/1285577292 conn(0x7f9118098ee0 msgr2=0x7f911806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:39.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.285+0000 7f911e81d700 1 -- 172.21.15.111:0/1285577292 shutdown_connections 2023-11-28T12:59:39.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.285+0000 7f911e81d700 1 -- 172.21.15.111:0/1285577292 wait complete. 2023-11-28T12:59:39.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.287+0000 7f911e81d700 1 Processor -- start 2023-11-28T12:59:39.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.287+0000 7f911e81d700 1 -- start start 2023-11-28T12:59:39.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.287+0000 7f911e81d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91180b0250 0x7f911814d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.287+0000 7f911e81d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91180b15c0 0x7f91180c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.287+0000 7f911e81d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f911815a9e0 0x7f91180c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.287+0000 7f911e81d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911814d670 con 0x7f911815a9e0 2023-11-28T12:59:39.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.287+0000 7f911e81d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9118154330 con 0x7f91180b15c0 2023-11-28T12:59:39.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.287+0000 7f911e81d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911815d650 con 0x7f91180b0250 2023-11-28T12:59:39.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f911cdba700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f911815a9e0 0x7f91180c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f911cdba700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f911815a9e0 0x7f91180c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34006/0 (socket says 172.21.15.111:34006) 2023-11-28T12:59:39.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f911cdba700 1 -- 172.21.15.111:0/2898681592 learned_addr learned my addr 172.21.15.111:0/2898681592 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:39.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f91177fe700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91180b15c0 0x7f91180c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f911cdba700 1 -- 172.21.15.111:0/2898681592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91180b0250 msgr2=0x7f911814d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f9117fff700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91180b0250 0x7f911814d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f911cdba700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91180b0250 0x7f911814d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f911cdba700 1 -- 172.21.15.111:0/2898681592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91180b15c0 msgr2=0x7f91180c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f911cdba700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91180b15c0 0x7f91180c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f911cdba700 1 -- 172.21.15.111:0/2898681592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9100009550 con 0x7f911815a9e0 2023-11-28T12:59:39.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f911cdba700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f911815a9e0 0x7f91180c90d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f911000cc30 tx=0x7f911000a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:39.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f9114ff9700 1 -- 172.21.15.111:0/2898681592 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911000ab70 con 0x7f911815a9e0 2023-11-28T12:59:39.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f9114ff9700 1 -- 172.21.15.111:0/2898681592 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911000acd0 con 0x7f911815a9e0 2023-11-28T12:59:39.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f911e81d700 1 -- 172.21.15.111:0/2898681592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91180cbf70 con 0x7f911815a9e0 2023-11-28T12:59:39.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f9114ff9700 1 -- 172.21.15.111:0/2898681592 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9110003c40 con 0x7f911815a9e0 2023-11-28T12:59:39.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.288+0000 7f911e81d700 1 -- 172.21.15.111:0/2898681592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91180cbc90 con 0x7f911815a9e0 2023-11-28T12:59:39.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.289+0000 7f911e81d700 1 -- 172.21.15.111:0/2898681592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91180cc250 con 0x7f911815a9e0 2023-11-28T12:59:39.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.290+0000 7f9114ff9700 1 -- 172.21.15.111:0/2898681592 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f911001f070 con 0x7f911815a9e0 2023-11-28T12:59:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.290+0000 7f9114ff9700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90f80682f0 0x7f90f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.290+0000 7f9114ff9700 1 -- 172.21.15.111:0/2898681592 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9110086090 con 0x7f911815a9e0 2023-11-28T12:59:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.290+0000 7f9117fff700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90f80682f0 0x7f90f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.290+0000 7f9117fff700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90f80682f0 0x7f90f806a7a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9108003e90 tx=0x7f9108009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.294+0000 7f9114ff9700 1 -- 172.21.15.111:0/2898681592 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9110052600 con 0x7f911815a9e0 2023-11-28T12:59:39.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.478+0000 7f911e81d700 1 -- 172.21.15.111:0/2898681592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9118067170 con 0x7f911815a9e0 2023-11-28T12:59:39.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.478+0000 7f9114ff9700 1 -- 172.21.15.111:0/2898681592 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9110055c20 con 0x7f911815a9e0 2023-11-28T12:59:39.480 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:39.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.480+0000 7f911e81d700 1 -- 172.21.15.111:0/2898681592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90f80682f0 msgr2=0x7f90f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.481+0000 7f911e81d700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90f80682f0 0x7f90f806a7a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9108003e90 tx=0x7f9108009040 comp rx=0 tx=0).stop 2023-11-28T12:59:39.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.481+0000 7f911e81d700 1 -- 172.21.15.111:0/2898681592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f911815a9e0 msgr2=0x7f91180c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.481+0000 7f911e81d700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f911815a9e0 0x7f91180c90d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f911000cc30 tx=0x7f911000a1f0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.481+0000 7f911e81d700 1 -- 172.21.15.111:0/2898681592 shutdown_connections 2023-11-28T12:59:39.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.481+0000 7f911e81d700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f911815a9e0 0x7f91180c90d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.481+0000 7f911e81d700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91180b0250 0x7f911814d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.481+0000 7f911e81d700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90f80682f0 0x7f90f806a7a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.481+0000 7f911e81d700 1 --2- 172.21.15.111:0/2898681592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91180b15c0 0x7f91180c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.481+0000 7f911e81d700 1 -- 172.21.15.111:0/2898681592 >> 172.21.15.111:0/2898681592 conn(0x7f9118098ee0 msgr2=0x7f911806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.481+0000 7f911e81d700 1 -- 172.21.15.111:0/2898681592 shutdown_connections 2023-11-28T12:59:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.481+0000 7f911e81d700 1 -- 172.21.15.111:0/2898681592 wait complete. 2023-11-28T12:59:39.495 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:39.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.614+0000 7fe5594c8700 1 Processor -- start 2023-11-28T12:59:39.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.614+0000 7fe5594c8700 1 -- start start 2023-11-28T12:59:39.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.614+0000 7fe5594c8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe554142e80 0x7fe55406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.614+0000 7fe5594c8700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe5540ca600 0x7fe5540cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.614+0000 7fe5594c8700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55406ac00 con 0x7fe55406a7b0 2023-11-28T12:59:39.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.614+0000 7fe5594c8700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5540c9380 con 0x7fe5540ca600 2023-11-28T12:59:39.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.614+0000 7fe5594c8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe554159b10 con 0x7fe554142e80 2023-11-28T12:59:39.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe5527fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe55406a7b0 0x7fe5540c9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34790/0 (socket says 172.21.15.111:34790) 2023-11-28T12:59:39.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe5527fc700 1 -- 172.21.15.111:0/2444521774 learned_addr learned my addr 172.21.15.111:0/2444521774 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:39.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe552ffd700 1 --2- 172.21.15.111:0/2444521774 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe554142e80 0x7fe55406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe5537fe700 1 --2- 172.21.15.111:0/2444521774 >> v2:172.21.15.149:3300/0 conn(0x7fe5540ca600 0x7fe5540cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654230634 0 0) 0x7fe548001110 con 0x7fe55406a7b0 2023-11-28T12:59:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c0035b0 con 0x7fe55406a7b0 2023-11-28T12:59:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 66066498 0 0) 0x7fe53c0035b0 con 0x7fe55406a7b0 2023-11-28T12:59:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe554142e80 msgr2=0x7fe55406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 --2- 172.21.15.111:0/2444521774 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe554142e80 0x7fe55406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 >> v2:172.21.15.149:3300/0 conn(0x7fe5540ca600 msgr2=0x7fe5540cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 --2- 172.21.15.111:0/2444521774 >> v2:172.21.15.149:3300/0 conn(0x7fe5540ca600 0x7fe5540cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe554143250 con 0x7fe55406a7b0 2023-11-28T12:59:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fe548001800 con 0x7fe55406a7b0 2023-11-28T12:59:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 >> v1:172.21.15.111:6789/0 conn(0x7fe55406a7b0 legacy=0x7fe5540c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 --2- 172.21.15.111:0/2444521774 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe53c003a00 0x7fe53c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 --2- 172.21.15.111:0/2444521774 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe554142e80 0x7fe53c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 --2- 172.21.15.111:0/2444521774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe53c004d90 0x7fe53c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540c9380 con 0x7fe53c004d90 2023-11-28T12:59:39.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe554159b10 con 0x7fe53c003a00 2023-11-28T12:59:39.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c00ba10 con 0x7fe554142e80 2023-11-28T12:59:39.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe5537fe700 1 --2- 172.21.15.111:0/2444521774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe53c004d90 0x7fe53c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe552ffd700 1 --2- 172.21.15.111:0/2444521774 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe53c003a00 0x7fe53c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe5527fc700 1 --2- 172.21.15.111:0/2444521774 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe554142e80 0x7fe53c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe5537fe700 1 -- 172.21.15.111:0/2444521774 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe554142e80 msgr2=0x7fe53c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:39.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.615+0000 7fe5537fe700 1 --2- 172.21.15.111:0/2444521774 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe554142e80 0x7fe53c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe5537fe700 1 -- 172.21.15.111:0/2444521774 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe53c003a00 msgr2=0x7fe53c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:39.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe5537fe700 1 --2- 172.21.15.111:0/2444521774 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe53c003a00 0x7fe53c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe5537fe700 1 -- 172.21.15.111:0/2444521774 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c0035b0 con 0x7fe53c004d90 2023-11-28T12:59:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe5537fe700 1 --2- 172.21.15.111:0/2444521774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe53c004d90 0x7fe53c005200 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe54c00e170 tx=0x7fe53c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a462f44c27a037c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c00ec30 con 0x7fe53c004d90 2023-11-28T12:59:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54c00f040 con 0x7fe53c004d90 2023-11-28T12:59:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe551ffb700 1 -- 172.21.15.111:0/2444521774 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c014480 con 0x7fe53c004d90 2023-11-28T12:59:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe5594c8700 1 -- 172.21.15.111:0/2444521774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe53c004d90 msgr2=0x7fe53c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe5594c8700 1 --2- 172.21.15.111:0/2444521774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe53c004d90 0x7fe53c005200 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe54c00e170 tx=0x7fe53c005880 comp rx=0 tx=0).stop 2023-11-28T12:59:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe5537fe700 1 -- 172.21.15.111:0/2444521774 reap_dead start 2023-11-28T12:59:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe5594c8700 1 -- 172.21.15.111:0/2444521774 shutdown_connections 2023-11-28T12:59:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe5594c8700 1 -- 172.21.15.111:0/2444521774 >> 172.21.15.111:0/2444521774 conn(0x7fe5540cb490 msgr2=0x7fe5540cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe5594c8700 1 -- 172.21.15.111:0/2444521774 shutdown_connections 2023-11-28T12:59:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.616+0000 7fe5594c8700 1 -- 172.21.15.111:0/2444521774 wait complete. 2023-11-28T12:59:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.618+0000 7fe5594c8700 1 Processor -- start 2023-11-28T12:59:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.618+0000 7fe5594c8700 1 -- start start 2023-11-28T12:59:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.618+0000 7fe5594c8700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe554153a20 0x7fe5540c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.618+0000 7fe5594c8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5540b39e0 0x7fe5540b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.618+0000 7fe5594c8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe554155980 0x7fe554155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.618+0000 7fe5594c8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540b4390 con 0x7fe554155980 2023-11-28T12:59:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.618+0000 7fe5594c8700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540c79f0 con 0x7fe554153a20 2023-11-28T12:59:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.618+0000 7fe5594c8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5541554d0 con 0x7fe5540b39e0 2023-11-28T12:59:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe5537fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe554155980 0x7fe554155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe5537fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe554155980 0x7fe554155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34028/0 (socket says 172.21.15.111:34028) 2023-11-28T12:59:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe5537fe700 1 -- 172.21.15.111:0/2935195565 learned_addr learned my addr 172.21.15.111:0/2935195565 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe5527fc700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5540b39e0 0x7fe5540b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe552ffd700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe554153a20 0x7fe5540c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe5537fe700 1 -- 172.21.15.111:0/2935195565 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5540b39e0 msgr2=0x7fe5540b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe5537fe700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5540b39e0 0x7fe5540b3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe5537fe700 1 -- 172.21.15.111:0/2935195565 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe554153a20 msgr2=0x7fe5540c74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe5537fe700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe554153a20 0x7fe5540c74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe5537fe700 1 -- 172.21.15.111:0/2935195565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5540cc5f0 con 0x7fe554155980 2023-11-28T12:59:39.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe5537fe700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe554155980 0x7fe554155df0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe54c00e760 tx=0x7fe54c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:39.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe53b7fe700 1 -- 172.21.15.111:0/2935195565 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c007000 con 0x7fe554155980 2023-11-28T12:59:39.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe53b7fe700 1 -- 172.21.15.111:0/2935195565 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54c003e50 con 0x7fe554155980 2023-11-28T12:59:39.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe53b7fe700 1 -- 172.21.15.111:0/2935195565 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c015b90 con 0x7fe554155980 2023-11-28T12:59:39.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe5594c8700 1 -- 172.21.15.111:0/2935195565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5540cab00 con 0x7fe554155980 2023-11-28T12:59:39.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.619+0000 7fe5594c8700 1 -- 172.21.15.111:0/2935195565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5540cc8d0 con 0x7fe554155980 2023-11-28T12:59:39.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.620+0000 7fe5594c8700 1 -- 172.21.15.111:0/2935195565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe554155660 con 0x7fe554155980 2023-11-28T12:59:39.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.621+0000 7fe53b7fe700 1 -- 172.21.15.111:0/2935195565 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe54c02e070 con 0x7fe554155980 2023-11-28T12:59:39.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.621+0000 7fe53b7fe700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5400682f0 0x7fe54006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.621+0000 7fe53b7fe700 1 -- 172.21.15.111:0/2935195565 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe54c087330 con 0x7fe554155980 2023-11-28T12:59:39.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.621+0000 7fe552ffd700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5400682f0 0x7fe54006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.621+0000 7fe552ffd700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5400682f0 0x7fe54006a7a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe5540aff40 tx=0x7fe544008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:39.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.625+0000 7fe53b7fe700 1 -- 172.21.15.111:0/2935195565 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe54c0538a0 con 0x7fe554155980 2023-11-28T12:59:39.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.812+0000 7fe5594c8700 1 -- 172.21.15.111:0/2935195565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe554067880 con 0x7fe554155980 2023-11-28T12:59:39.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.813+0000 7fe53b7fe700 1 -- 172.21.15.111:0/2935195565 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+19796 (secure 0 0 0) 0x7fe54c056ec0 con 0x7fe554155980 2023-11-28T12:59:39.815 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:39.815 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:00.053943+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:39.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.815+0000 7fe5594c8700 1 -- 172.21.15.111:0/2935195565 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5400682f0 msgr2=0x7fe54006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.815+0000 7fe5594c8700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5400682f0 0x7fe54006a7a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe5540aff40 tx=0x7fe544008040 comp rx=0 tx=0).stop 2023-11-28T12:59:39.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.815+0000 7fe5594c8700 1 -- 172.21.15.111:0/2935195565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe554155980 msgr2=0x7fe554155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.815+0000 7fe5594c8700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe554155980 0x7fe554155df0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe54c00e760 tx=0x7fe54c004b40 comp rx=0 tx=0).stop 2023-11-28T12:59:39.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.815+0000 7fe5594c8700 1 -- 172.21.15.111:0/2935195565 shutdown_connections 2023-11-28T12:59:39.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.815+0000 7fe5594c8700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe554155980 0x7fe554155df0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.815+0000 7fe5594c8700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5400682f0 0x7fe54006a7a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.815+0000 7fe5594c8700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe554153a20 0x7fe5540c74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.815+0000 7fe5594c8700 1 --2- 172.21.15.111:0/2935195565 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5540b39e0 0x7fe5540b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.815+0000 7fe5594c8700 1 -- 172.21.15.111:0/2935195565 >> 172.21.15.111:0/2935195565 conn(0x7fe5540cb490 msgr2=0x7fe5540c2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:39.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.815+0000 7fe5594c8700 1 -- 172.21.15.111:0/2935195565 shutdown_connections 2023-11-28T12:59:39.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.815+0000 7fe5594c8700 1 -- 172.21.15.111:0/2935195565 wait complete. 2023-11-28T12:59:39.829 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T12:58:53.275206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T12:58:54.588499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T12:58:57.666033+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T12:59:39.830 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:59:39.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.952+0000 7f81f2aef700 1 Processor -- start 2023-11-28T12:59:39.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.952+0000 7f81f2aef700 1 -- start start 2023-11-28T12:59:39.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81f2aef700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81ec0b0d70 0x7f81ec1604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81f2aef700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f81ec0b15b0 0x7f81ec0cc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81f2aef700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ec15cd90 con 0x7f81ec0b1140 2023-11-28T12:59:39.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81f2aef700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81ec0cedd0 con 0x7f81ec0b15b0 2023-11-28T12:59:39.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81f2aef700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec1613e0 con 0x7f81ec0b0d70 2023-11-28T12:59:39.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81ebfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f81ec0b1140 0x7f81ec0ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34810/0 (socket says 172.21.15.111:34810) 2023-11-28T12:59:39.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81ebfff700 1 -- 172.21.15.111:0/2529738766 learned_addr learned my addr 172.21.15.111:0/2529738766 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:39.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81f088b700 1 --2- 172.21.15.111:0/2529738766 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81ec0b0d70 0x7f81ec1604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81f108c700 1 --2- 172.21.15.111:0/2529738766 >> v2:172.21.15.149:3300/0 conn(0x7f81ec0b15b0 0x7f81ec0cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273752423 0 0) 0x7f81d8001110 con 0x7f81ec0b1140 2023-11-28T12:59:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81dc0035b0 con 0x7f81ec0b1140 2023-11-28T12:59:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4145921784 0 0) 0x7f81dc0035b0 con 0x7f81ec0b1140 2023-11-28T12:59:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81ec0b0d70 msgr2=0x7f81ec1604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81eb7fe700 1 --2- 172.21.15.111:0/2529738766 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81ec0b0d70 0x7f81ec1604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 >> v2:172.21.15.149:3300/0 conn(0x7f81ec0b15b0 msgr2=0x7f81ec0cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81eb7fe700 1 --2- 172.21.15.111:0/2529738766 >> v2:172.21.15.149:3300/0 conn(0x7f81ec0b15b0 0x7f81ec0cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec160b40 con 0x7f81ec0b1140 2023-11-28T12:59:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f81d8001800 con 0x7f81ec0b1140 2023-11-28T12:59:39.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.953+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 >> v1:172.21.15.111:6789/0 conn(0x7f81ec0b1140 legacy=0x7f81ec0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81eb7fe700 1 --2- 172.21.15.111:0/2529738766 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81ec0b0d70 0x7f81dc003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81eb7fe700 1 --2- 172.21.15.111:0/2529738766 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81dc003f80 0x7f81dc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81eb7fe700 1 --2- 172.21.15.111:0/2529738766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81dc004d90 0x7f81dc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec0cedd0 con 0x7f81dc004d90 2023-11-28T12:59:39.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec1613e0 con 0x7f81dc003f80 2023-11-28T12:59:39.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc00ba10 con 0x7f81ec0b0d70 2023-11-28T12:59:39.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81f108c700 1 --2- 172.21.15.111:0/2529738766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81dc004d90 0x7f81dc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81f088b700 1 --2- 172.21.15.111:0/2529738766 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81ec0b0d70 0x7f81dc003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81ebfff700 1 --2- 172.21.15.111:0/2529738766 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81dc003f80 0x7f81dc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81f108c700 1 -- 172.21.15.111:0/2529738766 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81ec0b0d70 msgr2=0x7f81dc003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:39.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81f108c700 1 --2- 172.21.15.111:0/2529738766 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81ec0b0d70 0x7f81dc003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81f108c700 1 -- 172.21.15.111:0/2529738766 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81dc003f80 msgr2=0x7f81dc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:39.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81f108c700 1 --2- 172.21.15.111:0/2529738766 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81dc003f80 0x7f81dc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81f108c700 1 -- 172.21.15.111:0/2529738766 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc0035b0 con 0x7f81dc004d90 2023-11-28T12:59:39.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81f108c700 1 --2- 172.21.15.111:0/2529738766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81dc004d90 0x7f81dc005200 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f81ec071fd0 tx=0x7f81dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198dbfd16c2202ec server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:39.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e400ebd0 con 0x7f81dc004d90 2023-11-28T12:59:39.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e400f040 con 0x7f81dc004d90 2023-11-28T12:59:39.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.954+0000 7f81eb7fe700 1 -- 172.21.15.111:0/2529738766 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e4014480 con 0x7f81dc004d90 2023-11-28T12:59:39.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.955+0000 7f81f2aef700 1 -- 172.21.15.111:0/2529738766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81dc004d90 msgr2=0x7f81dc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.955+0000 7f81f2aef700 1 --2- 172.21.15.111:0/2529738766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81dc004d90 0x7f81dc005200 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f81ec071fd0 tx=0x7f81dc005880 comp rx=0 tx=0).stop 2023-11-28T12:59:39.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.955+0000 7f81f108c700 1 -- 172.21.15.111:0/2529738766 reap_dead start 2023-11-28T12:59:39.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.955+0000 7f81f2aef700 1 -- 172.21.15.111:0/2529738766 shutdown_connections 2023-11-28T12:59:39.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.955+0000 7f81f2aef700 1 -- 172.21.15.111:0/2529738766 >> 172.21.15.111:0/2529738766 conn(0x7f81ec098ee0 msgr2=0x7f81ec06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:39.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.955+0000 7f81f2aef700 1 -- 172.21.15.111:0/2529738766 shutdown_connections 2023-11-28T12:59:39.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.955+0000 7f81f2aef700 1 -- 172.21.15.111:0/2529738766 wait complete. 2023-11-28T12:59:39.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.956+0000 7f81f2aef700 1 Processor -- start 2023-11-28T12:59:39.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.956+0000 7f81f2aef700 1 -- start start 2023-11-28T12:59:39.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f2aef700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81dc004d90 0x7f81ec0b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f2aef700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81ec0ade50 0x7f81ec156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f2aef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81ec0ae2c0 0x7f81ec156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f2aef700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec156bd0 con 0x7f81ec0ae2c0 2023-11-28T12:59:39.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f2aef700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec0b42d0 con 0x7f81ec0ade50 2023-11-28T12:59:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f2aef700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec0b45b0 con 0x7f81dc004d90 2023-11-28T12:59:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f108c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81ec0ae2c0 0x7f81ec156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f108c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81ec0ae2c0 0x7f81ec156690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34048/0 (socket says 172.21.15.111:34048) 2023-11-28T12:59:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f108c700 1 -- 172.21.15.111:0/1662627490 learned_addr learned my addr 172.21.15.111:0/1662627490 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f088b700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81dc004d90 0x7f81ec0b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81ebfff700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81ec0ade50 0x7f81ec156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f108c700 1 -- 172.21.15.111:0/1662627490 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81dc004d90 msgr2=0x7f81ec0b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f108c700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81dc004d90 0x7f81ec0b5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f108c700 1 -- 172.21.15.111:0/1662627490 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81ec0ade50 msgr2=0x7f81ec156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f108c700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81ec0ade50 0x7f81ec156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f108c700 1 -- 172.21.15.111:0/1662627490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec161100 con 0x7f81ec0ae2c0 2023-11-28T12:59:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81f108c700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81ec0ae2c0 0x7f81ec156690 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f81e4002230 tx=0x7f81e4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:39.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.957+0000 7f81e97fa700 1 -- 172.21.15.111:0/1662627490 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e4007000 con 0x7f81ec0ae2c0 2023-11-28T12:59:39.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.958+0000 7f81e97fa700 1 -- 172.21.15.111:0/1662627490 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e4022070 con 0x7f81ec0ae2c0 2023-11-28T12:59:39.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.958+0000 7f81f2aef700 1 -- 172.21.15.111:0/1662627490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ec160e20 con 0x7f81ec0ae2c0 2023-11-28T12:59:39.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.958+0000 7f81e97fa700 1 -- 172.21.15.111:0/1662627490 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e4004690 con 0x7f81ec0ae2c0 2023-11-28T12:59:39.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.958+0000 7f81f2aef700 1 -- 172.21.15.111:0/1662627490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ec0af340 con 0x7f81ec0ae2c0 2023-11-28T12:59:39.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.958+0000 7f81f2aef700 1 -- 172.21.15.111:0/1662627490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81ec1604b0 con 0x7f81ec0ae2c0 2023-11-28T12:59:39.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.959+0000 7f81e97fa700 1 -- 172.21.15.111:0/1662627490 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f81e40047f0 con 0x7f81ec0ae2c0 2023-11-28T12:59:39.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.959+0000 7f81e97fa700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f81d00682f0 0x7f81d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:39.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.959+0000 7f81f088b700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f81d00682f0 0x7f81d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:39.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.959+0000 7f81e97fa700 1 -- 172.21.15.111:0/1662627490 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f81e4024070 con 0x7f81ec0ae2c0 2023-11-28T12:59:39.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.960+0000 7f81f088b700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f81d00682f0 0x7f81d006a7a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009c40 tx=0x7f81e00091c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:39.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:39.964+0000 7f81e97fa700 1 -- 172.21.15.111:0/1662627490 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f81e40544f0 con 0x7f81ec0ae2c0 2023-11-28T12:59:40.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.144+0000 7f81f2aef700 1 -- 172.21.15.111:0/1662627490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f81ec067170 con 0x7f81ec0ae2c0 2023-11-28T12:59:40.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.145+0000 7f81e97fa700 1 -- 172.21.15.111:0/1662627490 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f81e4057b10 con 0x7f81ec0ae2c0 2023-11-28T12:59:40.147 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:59:40.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.147+0000 7f81f2aef700 1 -- 172.21.15.111:0/1662627490 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f81d00682f0 msgr2=0x7f81d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.147+0000 7f81f2aef700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f81d00682f0 0x7f81d006a7a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009c40 tx=0x7f81e00091c0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.147+0000 7f81f2aef700 1 -- 172.21.15.111:0/1662627490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81ec0ae2c0 msgr2=0x7f81ec156690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.147+0000 7f81f2aef700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81ec0ae2c0 0x7f81ec156690 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f81e4002230 tx=0x7f81e4009750 comp rx=0 tx=0).stop 2023-11-28T12:59:40.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.147+0000 7f81f2aef700 1 -- 172.21.15.111:0/1662627490 shutdown_connections 2023-11-28T12:59:40.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.147+0000 7f81f2aef700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81ec0ae2c0 0x7f81ec156690 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.147+0000 7f81f2aef700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81dc004d90 0x7f81ec0b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.147+0000 7f81f2aef700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f81d00682f0 0x7f81d006a7a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.147+0000 7f81f2aef700 1 --2- 172.21.15.111:0/1662627490 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81ec0ade50 0x7f81ec156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.147+0000 7f81f2aef700 1 -- 172.21.15.111:0/1662627490 >> 172.21.15.111:0/1662627490 conn(0x7f81ec098ee0 msgr2=0x7f81ec06dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:40.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.147+0000 7f81f2aef700 1 -- 172.21.15.111:0/1662627490 shutdown_connections 2023-11-28T12:59:40.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.147+0000 7f81f2aef700 1 -- 172.21.15.111:0/1662627490 wait complete. 2023-11-28T12:59:40.160 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:59:40.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.283+0000 7f2774982700 1 Processor -- start 2023-11-28T12:59:40.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.283+0000 7f2774982700 1 -- start start 2023-11-28T12:59:40.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.284+0000 7f2774982700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f2770159520 0x7f27700cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.284+0000 7f2774982700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f277006c0a0 0x7f27700cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f2774982700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27700b1510 con 0x7f27700aeb10 2023-11-28T12:59:40.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f2774982700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27701598f0 con 0x7f2770159520 2023-11-28T12:59:40.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f2774982700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27700b0f50 con 0x7f277006c0a0 2023-11-28T12:59:40.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f276e59c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f2770159520 0x7f27700cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f276dd9b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f277006c0a0 0x7f27700cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f276e59c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f2770159520 0x7f27700cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:51952/0 (socket says 172.21.15.111:51952) 2023-11-28T12:59:40.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f276e59c700 1 -- 172.21.15.111:0/630372492 learned_addr learned my addr 172.21.15.111:0/630372492 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:40.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f276ed9d700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f27700aeb10 0x7f27700af530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:43376/0 (socket says 172.21.15.111:43376) 2023-11-28T12:59:40.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f276e59c700 1 -- 172.21.15.111:0/630372492 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f277006c0a0 msgr2=0x7f27700cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:40.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f276e59c700 1 --2- 172.21.15.111:0/630372492 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f277006c0a0 0x7f27700cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f276e59c700 1 -- 172.21.15.111:0/630372492 >> v1:172.21.15.149:6789/0 conn(0x7f27700aeb10 legacy=0x7f27700af530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f276e59c700 1 -- 172.21.15.111:0/630372492 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27700b1230 con 0x7f2770159520 2023-11-28T12:59:40.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f276e59c700 1 --2- 172.21.15.111:0/630372492 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2770159520 0x7f27700cd7a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f27640098f0 tx=0x7f2764009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=153522a0d214bcf2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:40.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.285+0000 7f276d59a700 1 -- 172.21.15.111:0/630372492 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2764011830 con 0x7f2770159520 2023-11-28T12:59:40.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.286+0000 7f276d59a700 1 -- 172.21.15.111:0/630372492 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2764011990 con 0x7f2770159520 2023-11-28T12:59:40.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.286+0000 7f276d59a700 1 -- 172.21.15.111:0/630372492 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2764011d20 con 0x7f2770159520 2023-11-28T12:59:40.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.286+0000 7f2774982700 1 -- 172.21.15.111:0/630372492 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2770159520 msgr2=0x7f27700cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.286+0000 7f2774982700 1 --2- 172.21.15.111:0/630372492 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2770159520 0x7f27700cd7a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f27640098f0 tx=0x7f2764009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:40.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.286+0000 7f2774982700 1 -- 172.21.15.111:0/630372492 shutdown_connections 2023-11-28T12:59:40.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.286+0000 7f2774982700 1 --2- 172.21.15.111:0/630372492 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2770159520 0x7f27700cd7a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.286+0000 7f2774982700 1 --2- 172.21.15.111:0/630372492 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f277006c0a0 0x7f27700cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.286+0000 7f2774982700 1 -- 172.21.15.111:0/630372492 >> 172.21.15.111:0/630372492 conn(0x7f277006bca0 msgr2=0x7f27700cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:40.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.286+0000 7f2774982700 1 -- 172.21.15.111:0/630372492 shutdown_connections 2023-11-28T12:59:40.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.286+0000 7f2774982700 1 -- 172.21.15.111:0/630372492 wait complete. 2023-11-28T12:59:40.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.288+0000 7f2774982700 1 Processor -- start 2023-11-28T12:59:40.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.288+0000 7f2774982700 1 -- start start 2023-11-28T12:59:40.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f2774982700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f277006c0a0 0x7f27700c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f2774982700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f27700aeb10 0x7f2770154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f2774982700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2770159520 0x7f277009ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f2774982700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27700c6270 con 0x7f277006c0a0 2023-11-28T12:59:40.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f2774982700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27700c40c0 con 0x7f27700aeb10 2023-11-28T12:59:40.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f2774982700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27700c5030 con 0x7f2770159520 2023-11-28T12:59:40.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f276e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f277006c0a0 0x7f27700c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f276e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f277006c0a0 0x7f27700c4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34058/0 (socket says 172.21.15.111:34058) 2023-11-28T12:59:40.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f276e59c700 1 -- 172.21.15.111:0/803360935 learned_addr learned my addr 172.21.15.111:0/803360935 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:40.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f276dd9b700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f27700aeb10 0x7f2770154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f276ed9d700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2770159520 0x7f277009ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f276e59c700 1 -- 172.21.15.111:0/803360935 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2770159520 msgr2=0x7f277009ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f276e59c700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2770159520 0x7f277009ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f276e59c700 1 -- 172.21.15.111:0/803360935 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f27700aeb10 msgr2=0x7f2770154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f276e59c700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f27700aeb10 0x7f2770154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f276e59c700 1 -- 172.21.15.111:0/803360935 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2764009550 con 0x7f277006c0a0 2023-11-28T12:59:40.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f276ed9d700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2770159520 0x7f277009ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:40.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f276dd9b700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f27700aeb10 0x7f2770154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:40.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f276e59c700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f277006c0a0 0x7f27700c4af0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2764000c00 tx=0x7f2764004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:40.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.289+0000 7f275effd700 1 -- 172.21.15.111:0/803360935 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2764004570 con 0x7f277006c0a0 2023-11-28T12:59:40.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.290+0000 7f275effd700 1 -- 172.21.15.111:0/803360935 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276401f070 con 0x7f277006c0a0 2023-11-28T12:59:40.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.290+0000 7f2774982700 1 -- 172.21.15.111:0/803360935 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27700acc90 con 0x7f277006c0a0 2023-11-28T12:59:40.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.290+0000 7f275effd700 1 -- 172.21.15.111:0/803360935 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27640038c0 con 0x7f277006c0a0 2023-11-28T12:59:40.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.290+0000 7f2774982700 1 -- 172.21.15.111:0/803360935 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27700ac9b0 con 0x7f277006c0a0 2023-11-28T12:59:40.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.290+0000 7f2774982700 1 -- 172.21.15.111:0/803360935 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27700acf70 con 0x7f277006c0a0 2023-11-28T12:59:40.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.291+0000 7f275effd700 1 -- 172.21.15.111:0/803360935 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f276402a020 con 0x7f277006c0a0 2023-11-28T12:59:40.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.291+0000 7f275effd700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f27500682f0 0x7f275006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.292+0000 7f276dd9b700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f27500682f0 0x7f275006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.292+0000 7f275effd700 1 -- 172.21.15.111:0/803360935 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f2764087590 con 0x7f277006c0a0 2023-11-28T12:59:40.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.292+0000 7f276dd9b700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f27500682f0 0x7f275006a7a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2770158020 tx=0x7f2760008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:40.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.296+0000 7f275effd700 1 -- 172.21.15.111:0/803360935 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2764053b00 con 0x7f277006c0a0 2023-11-28T12:59:40.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.483+0000 7f2774982700 1 -- 172.21.15.111:0/803360935 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2770067170 con 0x7f277006c0a0 2023-11-28T12:59:40.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.483+0000 7f275effd700 1 -- 172.21.15.111:0/803360935 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2764057120 con 0x7f277006c0a0 2023-11-28T12:59:40.485 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:59:40.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.485+0000 7f2774982700 1 -- 172.21.15.111:0/803360935 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f27500682f0 msgr2=0x7f275006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.485+0000 7f2774982700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f27500682f0 0x7f275006a7a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2770158020 tx=0x7f2760008040 comp rx=0 tx=0).stop 2023-11-28T12:59:40.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.485+0000 7f2774982700 1 -- 172.21.15.111:0/803360935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f277006c0a0 msgr2=0x7f27700c4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.485+0000 7f2774982700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f277006c0a0 0x7f27700c4af0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2764000c00 tx=0x7f2764004750 comp rx=0 tx=0).stop 2023-11-28T12:59:40.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.485+0000 7f2774982700 1 -- 172.21.15.111:0/803360935 shutdown_connections 2023-11-28T12:59:40.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.485+0000 7f2774982700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2770159520 0x7f277009ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.485+0000 7f2774982700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f277006c0a0 0x7f27700c4af0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.485+0000 7f2774982700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f27500682f0 0x7f275006a7a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.485+0000 7f2774982700 1 --2- 172.21.15.111:0/803360935 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f27700aeb10 0x7f2770154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.485+0000 7f2774982700 1 -- 172.21.15.111:0/803360935 >> 172.21.15.111:0/803360935 conn(0x7f277006bca0 msgr2=0x7f27700cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:40.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.485+0000 7f2774982700 1 -- 172.21.15.111:0/803360935 shutdown_connections 2023-11-28T12:59:40.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.485+0000 7f2774982700 1 -- 172.21.15.111:0/803360935 wait complete. 2023-11-28T12:59:40.499 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T12:59:40.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.621+0000 7fd43e609700 1 Processor -- start 2023-11-28T12:59:40.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.621+0000 7fd43e609700 1 -- start start 2023-11-28T12:59:40.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.622+0000 7fd43e609700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd4380cef60 0x7fd4380ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.622+0000 7fd43e609700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4380b0160 0x7fd4380adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.622+0000 7fd43e609700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4380cc390 con 0x7fd43815a5d0 2023-11-28T12:59:40.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.622+0000 7fd43e609700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4380cc1b0 con 0x7fd4380cef60 2023-11-28T12:59:40.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.622+0000 7fd43e609700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4380cc670 con 0x7fd4380b0160 2023-11-28T12:59:40.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.622+0000 7fd43cba6700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4380b0160 0x7fd4380adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.622+0000 7fd4377fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd4380cef60 0x7fd4380ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.622+0000 7fd437fff700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7fd43815a5d0 0x7fd4380ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:43390/0 (socket says 172.21.15.111:43390) 2023-11-28T12:59:40.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.622+0000 7fd437fff700 1 -- 172.21.15.111:0/449712932 learned_addr learned my addr 172.21.15.111:0/449712932 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:40.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.623+0000 7fd43cba6700 1 -- 172.21.15.111:0/449712932 >> v2:172.21.15.149:3300/0 conn(0x7fd4380cef60 msgr2=0x7fd4380ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:40.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.623+0000 7fd43cba6700 1 --2- 172.21.15.111:0/449712932 >> v2:172.21.15.149:3300/0 conn(0x7fd4380cef60 0x7fd4380ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.623+0000 7fd43cba6700 1 -- 172.21.15.111:0/449712932 >> v1:172.21.15.149:6789/0 conn(0x7fd43815a5d0 legacy=0x7fd4380ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.623+0000 7fd43cba6700 1 -- 172.21.15.111:0/449712932 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4380cc950 con 0x7fd4380b0160 2023-11-28T12:59:40.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.623+0000 7fd4377fe700 1 --2- 172.21.15.111:0/449712932 >> v2:172.21.15.149:3300/0 conn(0x7fd4380cef60 0x7fd4380ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:40.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.623+0000 7fd43cba6700 1 --2- 172.21.15.111:0/449712932 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4380b0160 0x7fd4380adde0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd430009d90 tx=0x7fd43000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b188bbd349a648ef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:40.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.623+0000 7fd436ffd700 1 -- 172.21.15.111:0/449712932 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43000e070 con 0x7fd4380b0160 2023-11-28T12:59:40.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.623+0000 7fd436ffd700 1 -- 172.21.15.111:0/449712932 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43000bb20 con 0x7fd4380b0160 2023-11-28T12:59:40.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.623+0000 7fd436ffd700 1 -- 172.21.15.111:0/449712932 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430012430 con 0x7fd4380b0160 2023-11-28T12:59:40.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.623+0000 7fd43e609700 1 -- 172.21.15.111:0/449712932 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4380b0160 msgr2=0x7fd4380adde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.624+0000 7fd43e609700 1 --2- 172.21.15.111:0/449712932 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4380b0160 0x7fd4380adde0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd430009d90 tx=0x7fd43000b330 comp rx=0 tx=0).stop 2023-11-28T12:59:40.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.624+0000 7fd43e609700 1 -- 172.21.15.111:0/449712932 shutdown_connections 2023-11-28T12:59:40.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.624+0000 7fd43e609700 1 --2- 172.21.15.111:0/449712932 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4380b0160 0x7fd4380adde0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.624+0000 7fd43e609700 1 --2- 172.21.15.111:0/449712932 >> v2:172.21.15.149:3300/0 conn(0x7fd4380cef60 0x7fd4380ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.624+0000 7fd43e609700 1 -- 172.21.15.111:0/449712932 >> 172.21.15.111:0/449712932 conn(0x7fd43809a040 msgr2=0x7fd4380d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:40.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.624+0000 7fd43e609700 1 -- 172.21.15.111:0/449712932 shutdown_connections 2023-11-28T12:59:40.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.624+0000 7fd43e609700 1 -- 172.21.15.111:0/449712932 wait complete. 2023-11-28T12:59:40.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.625+0000 7fd43e609700 1 Processor -- start 2023-11-28T12:59:40.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.625+0000 7fd43e609700 1 -- start start 2023-11-28T12:59:40.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.626+0000 7fd43e609700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4380b0160 0x7fd4380c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.626+0000 7fd43e609700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4380cef60 0x7fd438153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.626+0000 7fd43e609700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd43815a5d0 0x7fd4380c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.626+0000 7fd43e609700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd438155d60 con 0x7fd43815a5d0 2023-11-28T12:59:40.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.626+0000 7fd43e609700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd438155150 con 0x7fd4380cef60 2023-11-28T12:59:40.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.626+0000 7fd43e609700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd438155430 con 0x7fd4380b0160 2023-11-28T12:59:40.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.626+0000 7fd43cba6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd43815a5d0 0x7fd4380c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.626+0000 7fd43cba6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd43815a5d0 0x7fd4380c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34080/0 (socket says 172.21.15.111:34080) 2023-11-28T12:59:40.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.626+0000 7fd43cba6700 1 -- 172.21.15.111:0/2793142991 learned_addr learned my addr 172.21.15.111:0/2793142991 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:40.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.626+0000 7fd437fff700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4380b0160 0x7fd4380c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.626+0000 7fd4377fe700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4380cef60 0x7fd438153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.627+0000 7fd43cba6700 1 -- 172.21.15.111:0/2793142991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4380b0160 msgr2=0x7fd4380c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.627+0000 7fd43cba6700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4380b0160 0x7fd4380c6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.627+0000 7fd43cba6700 1 -- 172.21.15.111:0/2793142991 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4380cef60 msgr2=0x7fd438153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.627+0000 7fd43cba6700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4380cef60 0x7fd438153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.627+0000 7fd43cba6700 1 -- 172.21.15.111:0/2793142991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42c0094a0 con 0x7fd43815a5d0 2023-11-28T12:59:40.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.627+0000 7fd43cba6700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd43815a5d0 0x7fd4380c9a60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd430014040 tx=0x7fd430004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:40.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.627+0000 7fd434ff9700 1 -- 172.21.15.111:0/2793142991 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430012430 con 0x7fd43815a5d0 2023-11-28T12:59:40.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.627+0000 7fd434ff9700 1 -- 172.21.15.111:0/2793142991 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd430020070 con 0x7fd43815a5d0 2023-11-28T12:59:40.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.627+0000 7fd43e609700 1 -- 172.21.15.111:0/2793142991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd430009a70 con 0x7fd43815a5d0 2023-11-28T12:59:40.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.627+0000 7fd434ff9700 1 -- 172.21.15.111:0/2793142991 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd430003ea0 con 0x7fd43815a5d0 2023-11-28T12:59:40.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.627+0000 7fd43e609700 1 -- 172.21.15.111:0/2793142991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4380ccf10 con 0x7fd43815a5d0 2023-11-28T12:59:40.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.628+0000 7fd43e609700 1 -- 172.21.15.111:0/2793142991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4380ccc30 con 0x7fd43815a5d0 2023-11-28T12:59:40.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.628+0000 7fd434ff9700 1 -- 172.21.15.111:0/2793142991 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd43002e070 con 0x7fd43815a5d0 2023-11-28T12:59:40.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.629+0000 7fd434ff9700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4240682f0 0x7fd42406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.629+0000 7fd434ff9700 1 -- 172.21.15.111:0/2793142991 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd430022070 con 0x7fd43815a5d0 2023-11-28T12:59:40.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.629+0000 7fd437fff700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4240682f0 0x7fd42406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.629+0000 7fd437fff700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4240682f0 0x7fd42406a7a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd428005960 tx=0x7fd428009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:40.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.633+0000 7fd434ff9700 1 -- 172.21.15.111:0/2793142991 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd430054140 con 0x7fd43815a5d0 2023-11-28T12:59:40.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.814+0000 7fd43e609700 1 -- 172.21.15.111:0/2793142991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd43815a9a0 con 0x7fd43815a5d0 2023-11-28T12:59:40.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.815+0000 7fd434ff9700 1 -- 172.21.15.111:0/2793142991 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd43002c020 con 0x7fd43815a5d0 2023-11-28T12:59:40.817 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:40.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.817+0000 7fd43e609700 1 -- 172.21.15.111:0/2793142991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4240682f0 msgr2=0x7fd42406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.817+0000 7fd43e609700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4240682f0 0x7fd42406a7a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd428005960 tx=0x7fd428009040 comp rx=0 tx=0).stop 2023-11-28T12:59:40.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.817+0000 7fd43e609700 1 -- 172.21.15.111:0/2793142991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd43815a5d0 msgr2=0x7fd4380c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.817+0000 7fd43e609700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd43815a5d0 0x7fd4380c9a60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd430014040 tx=0x7fd430004ad0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.817+0000 7fd43e609700 1 -- 172.21.15.111:0/2793142991 shutdown_connections 2023-11-28T12:59:40.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.817+0000 7fd43e609700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd43815a5d0 0x7fd4380c9a60 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.817+0000 7fd43e609700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4380b0160 0x7fd4380c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.817+0000 7fd43e609700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4240682f0 0x7fd42406a7a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.817+0000 7fd43e609700 1 --2- 172.21.15.111:0/2793142991 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd4380cef60 0x7fd438153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.817+0000 7fd43e609700 1 -- 172.21.15.111:0/2793142991 >> 172.21.15.111:0/2793142991 conn(0x7fd43809a040 msgr2=0x7fd4380d1440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:40.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.817+0000 7fd43e609700 1 -- 172.21.15.111:0/2793142991 shutdown_connections 2023-11-28T12:59:40.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.817+0000 7fd43e609700 1 -- 172.21.15.111:0/2793142991 wait complete. 2023-11-28T12:59:40.831 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T12:59:40.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.955+0000 7f9c8e6bc700 1 Processor -- start 2023-11-28T12:59:40.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.955+0000 7f9c8e6bc700 1 -- start start 2023-11-28T12:59:40.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.956+0000 7f9c8e6bc700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9c8815d6e0 0x7f9c88164360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.956+0000 7f9c8e6bc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c8815de70 con 0x7f9c88159310 2023-11-28T12:59:40.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.956+0000 7f9c8e6bc700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c880ced10 con 0x7f9c881596e0 2023-11-28T12:59:40.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.956+0000 7f9c8e6bc700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c880ceff0 con 0x7f9c8815d6e0 2023-11-28T12:59:40.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.956+0000 7f9c87fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9c88159310 0x7f9c880ce1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34862/0 (socket says 172.21.15.111:34862) 2023-11-28T12:59:40.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.956+0000 7f9c87fff700 1 -- 172.21.15.111:0/2717310311 learned_addr learned my addr 172.21.15.111:0/2717310311 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:40.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118828696 0 0) 0x7f9c8815de70 con 0x7f9c88159310 2023-11-28T12:59:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c8cc59700 1 --2- 172.21.15.111:0/2717310311 >> v2:172.21.15.149:3300/0 conn(0x7f9c8815d6e0 0x7f9c88164360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c700035b0 con 0x7f9c88159310 2023-11-28T12:59:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3823049147 0 0) 0x7f9c700035b0 con 0x7f9c88159310 2023-11-28T12:59:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 >> v2:172.21.15.149:3300/0 conn(0x7f9c8815d6e0 msgr2=0x7f9c88164360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 --2- 172.21.15.111:0/2717310311 >> v2:172.21.15.149:3300/0 conn(0x7f9c8815d6e0 0x7f9c88164360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 >> v1:172.21.15.149:6789/0 conn(0x7f9c881596e0 legacy=0x7f9c8815cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c880cf2d0 con 0x7f9c88159310 2023-11-28T12:59:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f9c78001670 con 0x7f9c88159310 2023-11-28T12:59:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 >> v1:172.21.15.111:6789/0 conn(0x7f9c88159310 legacy=0x7f9c880ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 --2- 172.21.15.111:0/2717310311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9c70003a00 0x7f9c70003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 --2- 172.21.15.111:0/2717310311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c70004850 0x7f9c70004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 --2- 172.21.15.111:0/2717310311 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c70005200 0x7f9c7000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c880ceff0 con 0x7f9c70004850 2023-11-28T12:59:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7000c190 con 0x7f9c70005200 2023-11-28T12:59:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7000c350 con 0x7f9c70003a00 2023-11-28T12:59:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c877fe700 1 --2- 172.21.15.111:0/2717310311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c70004850 0x7f9c70004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c87fff700 1 --2- 172.21.15.111:0/2717310311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9c70003a00 0x7f9c70003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c8cc59700 1 --2- 172.21.15.111:0/2717310311 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c70005200 0x7f9c7000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c877fe700 1 -- 172.21.15.111:0/2717310311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9c70003a00 msgr2=0x7f9c70003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c877fe700 1 --2- 172.21.15.111:0/2717310311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9c70003a00 0x7f9c70003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c877fe700 1 -- 172.21.15.111:0/2717310311 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c70005200 msgr2=0x7f9c7000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c877fe700 1 --2- 172.21.15.111:0/2717310311 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c70005200 0x7f9c7000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c877fe700 1 -- 172.21.15.111:0/2717310311 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c700035b0 con 0x7f9c70004850 2023-11-28T12:59:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c8cc59700 1 --2- 172.21.15.111:0/2717310311 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c70005200 0x7f9c7000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c87fff700 1 --2- 172.21.15.111:0/2717310311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9c70003a00 0x7f9c70003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c8cc59700 1 -- 172.21.15.111:0/2717310311 reap_dead start 2023-11-28T12:59:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c877fe700 1 --2- 172.21.15.111:0/2717310311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c70004850 0x7f9c70004cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c007fd0 tx=0x7f9c70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12b60fa692cfad5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.957+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c0012f0 con 0x7f9c70004850 2023-11-28T12:59:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.958+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7c0052d0 con 0x7f9c70004850 2023-11-28T12:59:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.958+0000 7f9c86ffd700 1 -- 172.21.15.111:0/2717310311 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c006860 con 0x7f9c70004850 2023-11-28T12:59:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.958+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/2717310311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c70004850 msgr2=0x7f9c70004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.958+0000 7f9c8e6bc700 1 --2- 172.21.15.111:0/2717310311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c70004850 0x7f9c70004cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c007fd0 tx=0x7f9c70005880 comp rx=0 tx=0).stop 2023-11-28T12:59:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.958+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/2717310311 shutdown_connections 2023-11-28T12:59:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.958+0000 7f9c8e6bc700 1 --2- 172.21.15.111:0/2717310311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c70004850 0x7f9c70004cc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.958+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/2717310311 >> 172.21.15.111:0/2717310311 conn(0x7f9c8806db10 msgr2=0x7f9c8806df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.958+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/2717310311 shutdown_connections 2023-11-28T12:59:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.958+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/2717310311 wait complete. 2023-11-28T12:59:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.959+0000 7f9c8e6bc700 1 Processor -- start 2023-11-28T12:59:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.959+0000 7f9c8e6bc700 1 -- start start 2023-11-28T12:59:40.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8e6bc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9c70004850 0x7f9c880c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8e6bc700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c881f1380 0x7f9c881f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8e6bc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c880c3e20 0x7f9c880c4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8e6bc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c881f1d30 con 0x7f9c880c3e20 2023-11-28T12:59:40.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8e6bc700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c880c36b0 con 0x7f9c881f1380 2023-11-28T12:59:40.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8e6bc700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c880c3990 con 0x7f9c70004850 2023-11-28T12:59:40.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8cc59700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c880c3e20 0x7f9c880c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8cc59700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c880c3e20 0x7f9c880c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34096/0 (socket says 172.21.15.111:34096) 2023-11-28T12:59:40.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8cc59700 1 -- 172.21.15.111:0/3153722397 learned_addr learned my addr 172.21.15.111:0/3153722397 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:40.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c877fe700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c881f1380 0x7f9c881f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c87fff700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9c70004850 0x7f9c880c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8cc59700 1 -- 172.21.15.111:0/3153722397 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9c70004850 msgr2=0x7f9c880c6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8cc59700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9c70004850 0x7f9c880c6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8cc59700 1 -- 172.21.15.111:0/3153722397 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c881f1380 msgr2=0x7f9c881f17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8cc59700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c881f1380 0x7f9c881f17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.960+0000 7f9c8cc59700 1 -- 172.21.15.111:0/3153722397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c880cb3d0 con 0x7f9c880c3e20 2023-11-28T12:59:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.961+0000 7f9c8cc59700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c880c3e20 0x7f9c880c4290 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9c8815e0b0 tx=0x7f9c80012ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.961+0000 7f9c84ff9700 1 -- 172.21.15.111:0/3153722397 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80003c30 con 0x7f9c880c3e20 2023-11-28T12:59:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.961+0000 7f9c84ff9700 1 -- 172.21.15.111:0/3153722397 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8000e040 con 0x7f9c880c3e20 2023-11-28T12:59:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.961+0000 7f9c84ff9700 1 -- 172.21.15.111:0/3153722397 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80010690 con 0x7f9c880c3e20 2023-11-28T12:59:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.961+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/3153722397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c880cb0f0 con 0x7f9c880c3e20 2023-11-28T12:59:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.961+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/3153722397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c8815cfd0 con 0x7f9c880c3e20 2023-11-28T12:59:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.962+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/3153722397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c8806c4d0 con 0x7f9c880c3e20 2023-11-28T12:59:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.962+0000 7f9c84ff9700 1 -- 172.21.15.111:0/3153722397 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9c800107f0 con 0x7f9c880c3e20 2023-11-28T12:59:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.963+0000 7f9c84ff9700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9c740682f0 0x7f9c7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.963+0000 7f9c87fff700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9c740682f0 0x7f9c7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:40.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.963+0000 7f9c84ff9700 1 -- 172.21.15.111:0/3153722397 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9c800888e0 con 0x7f9c880c3e20 2023-11-28T12:59:40.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.963+0000 7f9c87fff700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9c740682f0 0x7f9c7406a7a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9c78000ff0 tx=0x7f9c78001de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:40.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:40.967+0000 7f9c84ff9700 1 -- 172.21.15.111:0/3153722397 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9c80054e10 con 0x7f9c880c3e20 2023-11-28T12:59:41.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.148+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/3153722397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9c880c3ad0 con 0x7f9c880c3e20 2023-11-28T12:59:41.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.148+0000 7f9c84ff9700 1 -- 172.21.15.111:0/3153722397 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9c80058430 con 0x7f9c880c3e20 2023-11-28T12:59:41.150 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T12:59:41.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.150+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/3153722397 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9c740682f0 msgr2=0x7f9c7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.150+0000 7f9c8e6bc700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9c740682f0 0x7f9c7406a7a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9c78000ff0 tx=0x7f9c78001de0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.150+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/3153722397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c880c3e20 msgr2=0x7f9c880c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.150+0000 7f9c8e6bc700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c880c3e20 0x7f9c880c4290 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9c8815e0b0 tx=0x7f9c80012ef0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.150+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/3153722397 shutdown_connections 2023-11-28T12:59:41.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.150+0000 7f9c8e6bc700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c880c3e20 0x7f9c880c4290 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.150+0000 7f9c8e6bc700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9c70004850 0x7f9c880c6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.150+0000 7f9c8e6bc700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9c740682f0 0x7f9c7406a7a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.150+0000 7f9c8e6bc700 1 --2- 172.21.15.111:0/3153722397 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9c881f1380 0x7f9c881f17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.150+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/3153722397 >> 172.21.15.111:0/3153722397 conn(0x7f9c8806db10 msgr2=0x7f9c880cdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:41.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.150+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/3153722397 shutdown_connections 2023-11-28T12:59:41.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.150+0000 7f9c8e6bc700 1 -- 172.21.15.111:0/3153722397 wait complete. 2023-11-28T12:59:41.164 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T12:59:41.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.286+0000 7ff360b36700 1 Processor -- start 2023-11-28T12:59:41.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.286+0000 7ff360b36700 1 -- start start 2023-11-28T12:59:41.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.287+0000 7ff360b36700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff35c0cdd90 0x7ff35c0b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.287+0000 7ff360b36700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff35c0aecc0 0x7ff35c0ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.287+0000 7ff360b36700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c0ad420 con 0x7ff35c159960 2023-11-28T12:59:41.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.287+0000 7ff360b36700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35c159d30 con 0x7ff35c0aecc0 2023-11-28T12:59:41.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.287+0000 7ff360b36700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0ac810 con 0x7ff35c0cdd90 2023-11-28T12:59:41.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff35ad9d700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff35c0aecc0 0x7ff35c0ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff359d9b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff35c0cdd90 0x7ff35c0b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff35ad9d700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff35c0aecc0 0x7ff35c0ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:52004/0 (socket says 172.21.15.111:52004) 2023-11-28T12:59:41.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff359d9b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff35c0cdd90 0x7ff35c0b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55850/0 (socket says 172.21.15.111:55850) 2023-11-28T12:59:41.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff35ad9d700 1 -- 172.21.15.111:0/2932314718 learned_addr learned my addr 172.21.15.111:0/2932314718 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff35a59c700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7ff35c159960 0x7ff35c0cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:43420/0 (socket says 172.21.15.111:43420) 2023-11-28T12:59:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff359d9b700 1 -- 172.21.15.111:0/2932314718 >> v2:172.21.15.149:3300/0 conn(0x7ff35c0aecc0 msgr2=0x7ff35c0ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff359d9b700 1 --2- 172.21.15.111:0/2932314718 >> v2:172.21.15.149:3300/0 conn(0x7ff35c0aecc0 0x7ff35c0ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff359d9b700 1 -- 172.21.15.111:0/2932314718 >> v1:172.21.15.149:6789/0 conn(0x7ff35c159960 legacy=0x7ff35c0cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff359d9b700 1 -- 172.21.15.111:0/2932314718 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0acaf0 con 0x7ff35c0cdd90 2023-11-28T12:59:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff35959a700 1 -- 172.21.15.111:0/2932314718 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277444940 0 0) 0x7ff35c0ad420 con 0x7ff35c159960 2023-11-28T12:59:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff35ad9d700 1 --2- 172.21.15.111:0/2932314718 >> v2:172.21.15.149:3300/0 conn(0x7ff35c0aecc0 0x7ff35c0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff359d9b700 1 --2- 172.21.15.111:0/2932314718 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff35c0cdd90 0x7ff35c0b1fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff34c009840 tx=0x7ff34c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=297f9d834c7e3ebb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff35959a700 1 -- 172.21.15.111:0/2932314718 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c011830 con 0x7ff35c0cdd90 2023-11-28T12:59:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff35959a700 1 -- 172.21.15.111:0/2932314718 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c011990 con 0x7ff35c0cdd90 2023-11-28T12:59:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.288+0000 7ff35959a700 1 -- 172.21.15.111:0/2932314718 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c011d20 con 0x7ff35c0cdd90 2023-11-28T12:59:41.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.289+0000 7ff360b36700 1 -- 172.21.15.111:0/2932314718 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff35c0cdd90 msgr2=0x7ff35c0b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.289+0000 7ff360b36700 1 --2- 172.21.15.111:0/2932314718 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff35c0cdd90 0x7ff35c0b1fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff34c009840 tx=0x7ff34c009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:41.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.289+0000 7ff360b36700 1 -- 172.21.15.111:0/2932314718 shutdown_connections 2023-11-28T12:59:41.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.289+0000 7ff360b36700 1 --2- 172.21.15.111:0/2932314718 >> v2:172.21.15.149:3300/0 conn(0x7ff35c0aecc0 0x7ff35c0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.289+0000 7ff360b36700 1 --2- 172.21.15.111:0/2932314718 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff35c0cdd90 0x7ff35c0b1fd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.289+0000 7ff360b36700 1 -- 172.21.15.111:0/2932314718 >> 172.21.15.111:0/2932314718 conn(0x7ff35c098ee0 msgr2=0x7ff35c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:41.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.289+0000 7ff360b36700 1 -- 172.21.15.111:0/2932314718 shutdown_connections 2023-11-28T12:59:41.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.289+0000 7ff360b36700 1 -- 172.21.15.111:0/2932314718 wait complete. 2023-11-28T12:59:41.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.291+0000 7ff360b36700 1 Processor -- start 2023-11-28T12:59:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.291+0000 7ff360b36700 1 -- start start 2023-11-28T12:59:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.291+0000 7ff360b36700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff35c0aecc0 0x7ff35c1545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.291+0000 7ff360b36700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff35c0cdd90 0x7ff35c0c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.291+0000 7ff360b36700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff35c159960 0x7ff35c15c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.291+0000 7ff360b36700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c154b00 con 0x7ff35c159960 2023-11-28T12:59:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.291+0000 7ff360b36700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0c3680 con 0x7ff35c0aecc0 2023-11-28T12:59:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.291+0000 7ff360b36700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0c3960 con 0x7ff35c0cdd90 2023-11-28T12:59:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.291+0000 7ff35ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff35c159960 0x7ff35c15c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.291+0000 7ff35ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff35c159960 0x7ff35c15c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34110/0 (socket says 172.21.15.111:34110) 2023-11-28T12:59:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.291+0000 7ff35ad9d700 1 -- 172.21.15.111:0/2013369361 learned_addr learned my addr 172.21.15.111:0/2013369361 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.292+0000 7ff35a59c700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff35c0aecc0 0x7ff35c1545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.292+0000 7ff359d9b700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff35c0cdd90 0x7ff35c0c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.292+0000 7ff35ad9d700 1 -- 172.21.15.111:0/2013369361 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff35c0cdd90 msgr2=0x7ff35c0c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.292+0000 7ff35ad9d700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff35c0cdd90 0x7ff35c0c5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.292+0000 7ff35ad9d700 1 -- 172.21.15.111:0/2013369361 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff35c0aecc0 msgr2=0x7ff35c1545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.292+0000 7ff35ad9d700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff35c0aecc0 0x7ff35c1545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.292+0000 7ff35ad9d700 1 -- 172.21.15.111:0/2013369361 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff354009b30 con 0x7ff35c159960 2023-11-28T12:59:41.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.292+0000 7ff35ad9d700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff35c159960 0x7ff35c15c080 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff35400efc0 tx=0x7ff354009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:41.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.292+0000 7ff34affd700 1 -- 172.21.15.111:0/2013369361 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354018070 con 0x7ff35c159960 2023-11-28T12:59:41.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.292+0000 7ff34affd700 1 -- 172.21.15.111:0/2013369361 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35400f040 con 0x7ff35c159960 2023-11-28T12:59:41.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.292+0000 7ff34affd700 1 -- 172.21.15.111:0/2013369361 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff354003c40 con 0x7ff35c159960 2023-11-28T12:59:41.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.292+0000 7ff360b36700 1 -- 172.21.15.111:0/2013369361 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34c0094a0 con 0x7ff35c159960 2023-11-28T12:59:41.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.292+0000 7ff360b36700 1 -- 172.21.15.111:0/2013369361 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35c0ad0b0 con 0x7ff35c159960 2023-11-28T12:59:41.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.293+0000 7ff360b36700 1 -- 172.21.15.111:0/2013369361 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35c0acdd0 con 0x7ff35c159960 2023-11-28T12:59:41.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.293+0000 7ff34affd700 1 -- 172.21.15.111:0/2013369361 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff35400c9e0 con 0x7ff35c159960 2023-11-28T12:59:41.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.294+0000 7ff34affd700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff33c0682f0 0x7ff33c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.294+0000 7ff35a59c700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff33c0682f0 0x7ff33c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.294+0000 7ff34affd700 1 -- 172.21.15.111:0/2013369361 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff3540863d0 con 0x7ff35c159960 2023-11-28T12:59:41.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.294+0000 7ff35a59c700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff33c0682f0 0x7ff33c06a7a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff350007fd0 tx=0x7ff350007d60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:41.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.298+0000 7ff34affd700 1 -- 172.21.15.111:0/2013369361 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff354052900 con 0x7ff35c159960 2023-11-28T12:59:41.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.482+0000 7ff360b36700 1 -- 172.21.15.111:0/2013369361 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff35c0addf0 con 0x7ff35c159960 2023-11-28T12:59:41.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.482+0000 7ff34affd700 1 -- 172.21.15.111:0/2013369361 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff354055f20 con 0x7ff35c159960 2023-11-28T12:59:41.484 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:41.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.484+0000 7ff360b36700 1 -- 172.21.15.111:0/2013369361 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff33c0682f0 msgr2=0x7ff33c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.484+0000 7ff360b36700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff33c0682f0 0x7ff33c06a7a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff350007fd0 tx=0x7ff350007d60 comp rx=0 tx=0).stop 2023-11-28T12:59:41.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.484+0000 7ff360b36700 1 -- 172.21.15.111:0/2013369361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff35c159960 msgr2=0x7ff35c15c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.484+0000 7ff360b36700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff35c159960 0x7ff35c15c080 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff35400efc0 tx=0x7ff354009f90 comp rx=0 tx=0).stop 2023-11-28T12:59:41.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.484+0000 7ff360b36700 1 -- 172.21.15.111:0/2013369361 shutdown_connections 2023-11-28T12:59:41.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.484+0000 7ff360b36700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff35c159960 0x7ff35c15c080 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.484+0000 7ff360b36700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff33c0682f0 0x7ff33c06a7a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.484+0000 7ff360b36700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff35c0aecc0 0x7ff35c1545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.484+0000 7ff360b36700 1 --2- 172.21.15.111:0/2013369361 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff35c0cdd90 0x7ff35c0c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.484+0000 7ff360b36700 1 -- 172.21.15.111:0/2013369361 >> 172.21.15.111:0/2013369361 conn(0x7ff35c098ee0 msgr2=0x7ff35c06edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:41.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.485+0000 7ff360b36700 1 -- 172.21.15.111:0/2013369361 shutdown_connections 2023-11-28T12:59:41.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.485+0000 7ff360b36700 1 -- 172.21.15.111:0/2013369361 wait complete. 2023-11-28T12:59:41.498 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:41.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.616+0000 7f5950669700 1 Processor -- start 2023-11-28T12:59:41.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.616+0000 7f5950669700 1 -- start start 2023-11-28T12:59:41.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.617+0000 7f5950669700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f59481594f0 0x7f594815a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.617+0000 7f5950669700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f59480712b0 0x7f5948071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.617+0000 7f5950669700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59481598c0 con 0x7f594815cf50 2023-11-28T12:59:41.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.617+0000 7f5950669700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594815a1c0 con 0x7f59481594f0 2023-11-28T12:59:41.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.617+0000 7f5950669700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59480725c0 con 0x7f59480712b0 2023-11-28T12:59:41.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594e405700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f59481594f0 0x7f594815a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594e405700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f59481594f0 0x7f594815a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:52014/0 (socket says 172.21.15.111:52014) 2023-11-28T12:59:41.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594ec06700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f594815cf50 0x7f594806a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:43438/0 (socket says 172.21.15.111:43438) 2023-11-28T12:59:41.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594e405700 1 -- 172.21.15.111:0/3650849812 learned_addr learned my addr 172.21.15.111:0/3650849812 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:41.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594dc04700 1 --2- 172.21.15.111:0/3650849812 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f59480712b0 0x7f5948071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594d403700 1 -- 172.21.15.111:0/3650849812 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327632131 0 0) 0x7f5940001490 con 0x7f594815cf50 2023-11-28T12:59:41.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594d403700 1 -- 172.21.15.111:0/3650849812 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59340035b0 con 0x7f594815cf50 2023-11-28T12:59:41.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594e405700 1 -- 172.21.15.111:0/3650849812 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f59480712b0 msgr2=0x7f5948071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:41.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594e405700 1 --2- 172.21.15.111:0/3650849812 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f59480712b0 0x7f5948071700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594e405700 1 -- 172.21.15.111:0/3650849812 >> v1:172.21.15.149:6789/0 conn(0x7f594815cf50 legacy=0x7f594806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594e405700 1 -- 172.21.15.111:0/3650849812 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5948071c40 con 0x7f59481594f0 2023-11-28T12:59:41.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594e405700 1 --2- 172.21.15.111:0/3650849812 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f59481594f0 0x7f594815a5e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f59380098f0 tx=0x7f5938009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52eccb7424908730 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:41.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594d403700 1 -- 172.21.15.111:0/3650849812 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5938011830 con 0x7f59481594f0 2023-11-28T12:59:41.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594d403700 1 -- 172.21.15.111:0/3650849812 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5938011990 con 0x7f59481594f0 2023-11-28T12:59:41.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.618+0000 7f594d403700 1 -- 172.21.15.111:0/3650849812 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5938011d00 con 0x7f59481594f0 2023-11-28T12:59:41.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.619+0000 7f5950669700 1 -- 172.21.15.111:0/3650849812 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f59481594f0 msgr2=0x7f594815a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.619+0000 7f5950669700 1 --2- 172.21.15.111:0/3650849812 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f59481594f0 0x7f594815a5e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f59380098f0 tx=0x7f5938009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:41.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.619+0000 7f5950669700 1 -- 172.21.15.111:0/3650849812 shutdown_connections 2023-11-28T12:59:41.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.619+0000 7f5950669700 1 --2- 172.21.15.111:0/3650849812 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f59481594f0 0x7f594815a5e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.619+0000 7f5950669700 1 --2- 172.21.15.111:0/3650849812 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f59480712b0 0x7f5948071700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.619+0000 7f5950669700 1 -- 172.21.15.111:0/3650849812 >> 172.21.15.111:0/3650849812 conn(0x7f59480cf2e0 msgr2=0x7f59480cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:41.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.619+0000 7f5950669700 1 -- 172.21.15.111:0/3650849812 shutdown_connections 2023-11-28T12:59:41.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.619+0000 7f5950669700 1 -- 172.21.15.111:0/3650849812 wait complete. 2023-11-28T12:59:41.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.621+0000 7f5950669700 1 Processor -- start 2023-11-28T12:59:41.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.621+0000 7f5950669700 1 -- start start 2023-11-28T12:59:41.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f5950669700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f59480712b0 0x7f59481eed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f5950669700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59481594f0 0x7f59480aa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f5950669700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f594815cf50 0x7f59480b4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f5950669700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59480a8020 con 0x7f59481594f0 2023-11-28T12:59:41.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f5950669700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59480a8300 con 0x7f594815cf50 2023-11-28T12:59:41.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f5950669700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59480a85e0 con 0x7f59480712b0 2023-11-28T12:59:41.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f594dc04700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59481594f0 0x7f59480aa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f594dc04700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59481594f0 0x7f59480aa8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34122/0 (socket says 172.21.15.111:34122) 2023-11-28T12:59:41.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f594dc04700 1 -- 172.21.15.111:0/1472181127 learned_addr learned my addr 172.21.15.111:0/1472181127 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:41.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f594ec06700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f594815cf50 0x7f59480b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f594e405700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f59480712b0 0x7f59481eed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f594dc04700 1 -- 172.21.15.111:0/1472181127 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f59480712b0 msgr2=0x7f59481eed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f594dc04700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f59480712b0 0x7f59481eed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f594dc04700 1 -- 172.21.15.111:0/1472181127 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f594815cf50 msgr2=0x7f59480b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f594dc04700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f594815cf50 0x7f59480b4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f594dc04700 1 -- 172.21.15.111:0/1472181127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5938009550 con 0x7f59481594f0 2023-11-28T12:59:41.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f594dc04700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59481594f0 0x7f59480aa8a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f594400e9b0 tx=0x7f594400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:41.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f593effd700 1 -- 172.21.15.111:0/1472181127 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5944009710 con 0x7f59481594f0 2023-11-28T12:59:41.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f593effd700 1 -- 172.21.15.111:0/1472181127 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594400c850 con 0x7f59481594f0 2023-11-28T12:59:41.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f5950669700 1 -- 172.21.15.111:0/1472181127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594815db60 con 0x7f59481594f0 2023-11-28T12:59:41.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f593effd700 1 -- 172.21.15.111:0/1472181127 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594400cbc0 con 0x7f59481594f0 2023-11-28T12:59:41.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.622+0000 7f5950669700 1 -- 172.21.15.111:0/1472181127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594815de40 con 0x7f59481594f0 2023-11-28T12:59:41.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.623+0000 7f5950669700 1 -- 172.21.15.111:0/1472181127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5948159b00 con 0x7f59481594f0 2023-11-28T12:59:41.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.624+0000 7f593effd700 1 -- 172.21.15.111:0/1472181127 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5944019030 con 0x7f59481594f0 2023-11-28T12:59:41.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.624+0000 7f593effd700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f592c0682f0 0x7f592c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.624+0000 7f593effd700 1 -- 172.21.15.111:0/1472181127 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f59440860c0 con 0x7f59481594f0 2023-11-28T12:59:41.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.624+0000 7f594e405700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f592c0682f0 0x7f592c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.625+0000 7f594e405700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f592c0682f0 0x7f592c06a7a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5938009c40 tx=0x7f5938008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:41.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.628+0000 7f593effd700 1 -- 172.21.15.111:0/1472181127 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5944052500 con 0x7f59481594f0 2023-11-28T12:59:41.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.811+0000 7f5950669700 1 -- 172.21.15.111:0/1472181127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5948067170 con 0x7f59481594f0 2023-11-28T12:59:41.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.812+0000 7f593effd700 1 -- 172.21.15.111:0/1472181127 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+19890 (secure 0 0 0) 0x7f5944021090 con 0x7f59481594f0 2023-11-28T12:59:41.814 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:41.814 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":27,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:40.874365+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:41.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.814+0000 7f5950669700 1 -- 172.21.15.111:0/1472181127 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f592c0682f0 msgr2=0x7f592c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.814+0000 7f5950669700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f592c0682f0 0x7f592c06a7a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5938009c40 tx=0x7f5938008030 comp rx=0 tx=0).stop 2023-11-28T12:59:41.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.814+0000 7f5950669700 1 -- 172.21.15.111:0/1472181127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59481594f0 msgr2=0x7f59480aa8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.814+0000 7f5950669700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59481594f0 0x7f59480aa8a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f594400e9b0 tx=0x7f594400ed70 comp rx=0 tx=0).stop 2023-11-28T12:59:41.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.815+0000 7f5950669700 1 -- 172.21.15.111:0/1472181127 shutdown_connections 2023-11-28T12:59:41.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.815+0000 7f5950669700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f592c0682f0 0x7f592c06a7a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.815+0000 7f5950669700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f594815cf50 0x7f59480b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.815+0000 7f5950669700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f59480712b0 0x7f59481eed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.815+0000 7f5950669700 1 --2- 172.21.15.111:0/1472181127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59481594f0 0x7f59480aa8a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.815+0000 7f5950669700 1 -- 172.21.15.111:0/1472181127 >> 172.21.15.111:0/1472181127 conn(0x7f59480cf2e0 msgr2=0x7f59480b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:41.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.815+0000 7f5950669700 1 -- 172.21.15.111:0/1472181127 shutdown_connections 2023-11-28T12:59:41.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.815+0000 7f5950669700 1 -- 172.21.15.111:0/1472181127 wait complete. 2023-11-28T12:59:41.828 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T12:58:53.275206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T12:58:54.588499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T12:58:57.666033+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T12:59:41.829 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:59:41.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.956+0000 7f4b90ed3700 1 Processor -- start 2023-11-28T12:59:41.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.956+0000 7f4b90ed3700 1 -- start start 2023-11-28T12:59:41.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b90ed3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4b8c0b20f0 0x7f4b8c0b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b90ed3700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b8c06ba90 0x7f4b8c0b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b90ed3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8c0b5300 con 0x7f4b8c0b40c0 2023-11-28T12:59:41.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b90ed3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8c0cfbb0 con 0x7f4b8c0b20f0 2023-11-28T12:59:41.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b90ed3700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c0b46f0 con 0x7f4b8c06ba90 2023-11-28T12:59:41.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b8a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4b8c0b20f0 0x7f4b8c0b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b89d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4b8c0b40c0 0x7f4b8c06b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34908/0 (socket says 172.21.15.111:34908) 2023-11-28T12:59:41.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b89d9b700 1 -- 172.21.15.111:0/623570604 learned_addr learned my addr 172.21.15.111:0/623570604 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:41.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b8a59c700 1 -- 172.21.15.111:0/623570604 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b8c06ba90 msgr2=0x7f4b8c0b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:41.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b8a59c700 1 --2- 172.21.15.111:0/623570604 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b8c06ba90 0x7f4b8c0b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b8a59c700 1 -- 172.21.15.111:0/623570604 >> v1:172.21.15.111:6789/0 conn(0x7f4b8c0b40c0 legacy=0x7f4b8c06b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b8959a700 1 -- 172.21.15.111:0/623570604 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648183367 0 0) 0x7f4b8c0b5300 con 0x7f4b8c0b40c0 2023-11-28T12:59:41.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b8a59c700 1 -- 172.21.15.111:0/623570604 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c0b49d0 con 0x7f4b8c0b20f0 2023-11-28T12:59:41.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.957+0000 7f4b8a59c700 1 --2- 172.21.15.111:0/623570604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c0b20f0 0x7f4b8c0b3b80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009840 tx=0x7f4b80009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb19b8a49eef5057 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:41.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.958+0000 7f4b8959a700 1 -- 172.21.15.111:0/623570604 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80011830 con 0x7f4b8c0b20f0 2023-11-28T12:59:41.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.958+0000 7f4b8959a700 1 -- 172.21.15.111:0/623570604 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b80011990 con 0x7f4b8c0b20f0 2023-11-28T12:59:41.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.958+0000 7f4b8959a700 1 -- 172.21.15.111:0/623570604 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80011d20 con 0x7f4b8c0b20f0 2023-11-28T12:59:41.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.958+0000 7f4b90ed3700 1 -- 172.21.15.111:0/623570604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c0b20f0 msgr2=0x7f4b8c0b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.958+0000 7f4b90ed3700 1 --2- 172.21.15.111:0/623570604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c0b20f0 0x7f4b8c0b3b80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009840 tx=0x7f4b80009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:41.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.958+0000 7f4b90ed3700 1 -- 172.21.15.111:0/623570604 shutdown_connections 2023-11-28T12:59:41.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.958+0000 7f4b90ed3700 1 --2- 172.21.15.111:0/623570604 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b8c06ba90 0x7f4b8c0b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.958+0000 7f4b90ed3700 1 --2- 172.21.15.111:0/623570604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c0b20f0 0x7f4b8c0b3b80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.958+0000 7f4b90ed3700 1 -- 172.21.15.111:0/623570604 >> 172.21.15.111:0/623570604 conn(0x7f4b8c09a040 msgr2=0x7f4b8c0d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:41.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.958+0000 7f4b90ed3700 1 -- 172.21.15.111:0/623570604 shutdown_connections 2023-11-28T12:59:41.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.958+0000 7f4b90ed3700 1 -- 172.21.15.111:0/623570604 wait complete. 2023-11-28T12:59:41.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.960+0000 7f4b90ed3700 1 Processor -- start 2023-11-28T12:59:41.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.960+0000 7f4b90ed3700 1 -- start start 2023-11-28T12:59:41.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.960+0000 7f4b90ed3700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b8c06ba90 0x7f4b8c0c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.960+0000 7f4b90ed3700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4b8c0b20f0 0x7f4b8c0cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.960+0000 7f4b90ed3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c0b40c0 0x7f4b8c14cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.960+0000 7f4b90ed3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c0cd850 con 0x7f4b8c0b40c0 2023-11-28T12:59:41.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.960+0000 7f4b90ed3700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c0cccd0 con 0x7f4b8c0b20f0 2023-11-28T12:59:41.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.960+0000 7f4b90ed3700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c0ccfb0 con 0x7f4b8c06ba90 2023-11-28T12:59:41.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.960+0000 7f4b8ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c0b40c0 0x7f4b8c14cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.960+0000 7f4b8ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c0b40c0 0x7f4b8c14cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34138/0 (socket says 172.21.15.111:34138) 2023-11-28T12:59:41.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.960+0000 7f4b8ad9d700 1 -- 172.21.15.111:0/4037173719 learned_addr learned my addr 172.21.15.111:0/4037173719 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:41.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b89d9b700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4b8c0b20f0 0x7f4b8c0cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b8a59c700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b8c06ba90 0x7f4b8c0c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b8ad9d700 1 -- 172.21.15.111:0/4037173719 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b8c06ba90 msgr2=0x7f4b8c0c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b8ad9d700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b8c06ba90 0x7f4b8c0c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b8ad9d700 1 -- 172.21.15.111:0/4037173719 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4b8c0b20f0 msgr2=0x7f4b8c0cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:41.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b8ad9d700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4b8c0b20f0 0x7f4b8c0cc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:41.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b8ad9d700 1 -- 172.21.15.111:0/4037173719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b800094a0 con 0x7f4b8c0b40c0 2023-11-28T12:59:41.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b8a59c700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b8c06ba90 0x7f4b8c0c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:41.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b8ad9d700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c0b40c0 0x7f4b8c14cd60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400b970 tx=0x7f4b8400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:41.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b7affd700 1 -- 172.21.15.111:0/4037173719 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8400c9b0 con 0x7f4b8c0b40c0 2023-11-28T12:59:41.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b7affd700 1 -- 172.21.15.111:0/4037173719 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8400cb30 con 0x7f4b8c0b40c0 2023-11-28T12:59:41.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b90ed3700 1 -- 172.21.15.111:0/4037173719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c0b4f90 con 0x7f4b8c0b40c0 2023-11-28T12:59:41.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b7affd700 1 -- 172.21.15.111:0/4037173719 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b84014440 con 0x7f4b8c0b40c0 2023-11-28T12:59:41.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.961+0000 7f4b90ed3700 1 -- 172.21.15.111:0/4037173719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c0b4cb0 con 0x7f4b8c0b40c0 2023-11-28T12:59:41.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.962+0000 7f4b90ed3700 1 -- 172.21.15.111:0/4037173719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c0b24c0 con 0x7f4b8c0b40c0 2023-11-28T12:59:41.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.962+0000 7f4b7affd700 1 -- 172.21.15.111:0/4037173719 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4b8401b030 con 0x7f4b8c0b40c0 2023-11-28T12:59:41.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.963+0000 7f4b7affd700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4b6c0682f0 0x7f4b6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:41.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.963+0000 7f4b7affd700 1 -- 172.21.15.111:0/4037173719 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4b84085c40 con 0x7f4b8c0b40c0 2023-11-28T12:59:41.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.963+0000 7f4b8a59c700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4b6c0682f0 0x7f4b6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:41.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.963+0000 7f4b8a59c700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4b6c0682f0 0x7f4b6c06a7a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009b90 tx=0x7f4b80008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:41.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:41.967+0000 7f4b7affd700 1 -- 172.21.15.111:0/4037173719 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4b84052170 con 0x7f4b8c0b40c0 2023-11-28T12:59:42.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.148+0000 7f4b90ed3700 1 -- 172.21.15.111:0/4037173719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4b8c067170 con 0x7f4b8c0b40c0 2023-11-28T12:59:42.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.149+0000 7f4b7affd700 1 -- 172.21.15.111:0/4037173719 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4b840190a0 con 0x7f4b8c0b40c0 2023-11-28T12:59:42.150 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:59:42.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.151+0000 7f4b90ed3700 1 -- 172.21.15.111:0/4037173719 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4b6c0682f0 msgr2=0x7f4b6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.151+0000 7f4b90ed3700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4b6c0682f0 0x7f4b6c06a7a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009b90 tx=0x7f4b80008030 comp rx=0 tx=0).stop 2023-11-28T12:59:42.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.151+0000 7f4b90ed3700 1 -- 172.21.15.111:0/4037173719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c0b40c0 msgr2=0x7f4b8c14cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.151+0000 7f4b90ed3700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c0b40c0 0x7f4b8c14cd60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400b970 tx=0x7f4b8400bd30 comp rx=0 tx=0).stop 2023-11-28T12:59:42.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.151+0000 7f4b90ed3700 1 -- 172.21.15.111:0/4037173719 shutdown_connections 2023-11-28T12:59:42.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.151+0000 7f4b90ed3700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b8c0b40c0 0x7f4b8c14cd60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.151+0000 7f4b90ed3700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b8c06ba90 0x7f4b8c0c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.151+0000 7f4b90ed3700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4b6c0682f0 0x7f4b6c06a7a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.151+0000 7f4b90ed3700 1 --2- 172.21.15.111:0/4037173719 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4b8c0b20f0 0x7f4b8c0cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.151+0000 7f4b90ed3700 1 -- 172.21.15.111:0/4037173719 >> 172.21.15.111:0/4037173719 conn(0x7f4b8c09a040 msgr2=0x7f4b8c0d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:42.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.151+0000 7f4b90ed3700 1 -- 172.21.15.111:0/4037173719 shutdown_connections 2023-11-28T12:59:42.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.151+0000 7f4b90ed3700 1 -- 172.21.15.111:0/4037173719 wait complete. 2023-11-28T12:59:42.165 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:59:42.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.286+0000 7f67a836a700 1 Processor -- start 2023-11-28T12:59:42.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.286+0000 7f67a836a700 1 -- start start 2023-11-28T12:59:42.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.287+0000 7f67a836a700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f67a00cdef0 0x7f67a00d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.287+0000 7f67a836a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67a00cc5a0 0x7f67a00ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.287+0000 7f67a836a700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a00d05a0 con 0x7f67a00cfc90 2023-11-28T12:59:42.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.287+0000 7f67a836a700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a00ca260 con 0x7f67a00cdef0 2023-11-28T12:59:42.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.287+0000 7f67a836a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00ccaa0 con 0x7f67a00cc5a0 2023-11-28T12:59:42.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.288+0000 7f67a6907700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67a00cc5a0 0x7f67a00ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.288+0000 7f67a5905700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f67a00cdef0 0x7f67a00d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.288+0000 7f67a6907700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67a00cc5a0 0x7f67a00ca6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:55894/0 (socket says 172.21.15.111:55894) 2023-11-28T12:59:42.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.288+0000 7f67a6106700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f67a00cfc90 0x7f67a00cd7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:43462/0 (socket says 172.21.15.111:43462) 2023-11-28T12:59:42.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.288+0000 7f67a6907700 1 -- 172.21.15.111:0/2930715824 learned_addr learned my addr 172.21.15.111:0/2930715824 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:42.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.288+0000 7f67a5104700 1 -- 172.21.15.111:0/2930715824 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769479706 0 0) 0x7f6790001110 con 0x7f67a00cfc90 2023-11-28T12:59:42.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.288+0000 7f67a5104700 1 -- 172.21.15.111:0/2930715824 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f678c0035b0 con 0x7f67a00cfc90 2023-11-28T12:59:42.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.288+0000 7f67a6907700 1 -- 172.21.15.111:0/2930715824 >> v2:172.21.15.149:3300/0 conn(0x7f67a00cdef0 msgr2=0x7f67a00d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:42.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.288+0000 7f67a6907700 1 --2- 172.21.15.111:0/2930715824 >> v2:172.21.15.149:3300/0 conn(0x7f67a00cdef0 0x7f67a00d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.288+0000 7f67a6907700 1 -- 172.21.15.111:0/2930715824 >> v1:172.21.15.149:6789/0 conn(0x7f67a00cfc90 legacy=0x7f67a00cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.288+0000 7f67a6907700 1 -- 172.21.15.111:0/2930715824 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a00ccd80 con 0x7f67a00cc5a0 2023-11-28T12:59:42.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.288+0000 7f67a6907700 1 --2- 172.21.15.111:0/2930715824 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67a00cc5a0 0x7f67a00ca6f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f679800d150 tx=0x7f679800d460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16f3fd3bfd1d408a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:42.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.289+0000 7f67a5104700 1 -- 172.21.15.111:0/2930715824 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6798010070 con 0x7f67a00cc5a0 2023-11-28T12:59:42.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.289+0000 7f67a5104700 1 -- 172.21.15.111:0/2930715824 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679800dd10 con 0x7f67a00cc5a0 2023-11-28T12:59:42.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.289+0000 7f67a5104700 1 -- 172.21.15.111:0/2930715824 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679800b2d0 con 0x7f67a00cc5a0 2023-11-28T12:59:42.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.289+0000 7f67a836a700 1 -- 172.21.15.111:0/2930715824 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67a00cc5a0 msgr2=0x7f67a00ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.289+0000 7f67a836a700 1 --2- 172.21.15.111:0/2930715824 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67a00cc5a0 0x7f67a00ca6f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f679800d150 tx=0x7f679800d460 comp rx=0 tx=0).stop 2023-11-28T12:59:42.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.289+0000 7f67a836a700 1 -- 172.21.15.111:0/2930715824 shutdown_connections 2023-11-28T12:59:42.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.289+0000 7f67a836a700 1 --2- 172.21.15.111:0/2930715824 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67a00cc5a0 0x7f67a00ca6f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.289+0000 7f67a836a700 1 --2- 172.21.15.111:0/2930715824 >> v2:172.21.15.149:3300/0 conn(0x7f67a00cdef0 0x7f67a00d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.289+0000 7f67a836a700 1 -- 172.21.15.111:0/2930715824 >> 172.21.15.111:0/2930715824 conn(0x7f67a0098ee0 msgr2=0x7f67a006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:42.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.289+0000 7f67a836a700 1 -- 172.21.15.111:0/2930715824 shutdown_connections 2023-11-28T12:59:42.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.289+0000 7f67a836a700 1 -- 172.21.15.111:0/2930715824 wait complete. 2023-11-28T12:59:42.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.290+0000 7f67a836a700 1 Processor -- start 2023-11-28T12:59:42.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.290+0000 7f67a836a700 1 -- start start 2023-11-28T12:59:42.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.291+0000 7f67a836a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f67a00cc5a0 0x7f67a00b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.291+0000 7f67a836a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67a00cdef0 0x7f67a015bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.291+0000 7f67a836a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67a00cfc90 0x7f67a01ed240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.291+0000 7f67a836a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00b4670 con 0x7f67a00cdef0 2023-11-28T12:59:42.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.291+0000 7f67a836a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00b4950 con 0x7f67a00cc5a0 2023-11-28T12:59:42.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.291+0000 7f67a836a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00b4c30 con 0x7f67a00cfc90 2023-11-28T12:59:42.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.291+0000 7f67a5905700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67a00cdef0 0x7f67a015bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.291+0000 7f67a5905700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67a00cdef0 0x7f67a015bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34150/0 (socket says 172.21.15.111:34150) 2023-11-28T12:59:42.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.291+0000 7f67a5905700 1 -- 172.21.15.111:0/431860220 learned_addr learned my addr 172.21.15.111:0/431860220 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:42.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.291+0000 7f67a6106700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f67a00cc5a0 0x7f67a00b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.291+0000 7f67a6907700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67a00cfc90 0x7f67a01ed240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.291+0000 7f67a5905700 1 -- 172.21.15.111:0/431860220 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67a00cfc90 msgr2=0x7f67a01ed240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.291+0000 7f67a5905700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67a00cfc90 0x7f67a01ed240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.292+0000 7f67a5905700 1 -- 172.21.15.111:0/431860220 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f67a00cc5a0 msgr2=0x7f67a00b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.292+0000 7f67a5905700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f67a00cc5a0 0x7f67a00b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.292+0000 7f67a5905700 1 -- 172.21.15.111:0/431860220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679800bd40 con 0x7f67a00cdef0 2023-11-28T12:59:42.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.292+0000 7f67a6106700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f67a00cc5a0 0x7f67a00b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:42.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.292+0000 7f67a5905700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67a00cdef0 0x7f67a015bfa0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f679c00e980 tx=0x7f679c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:42.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.292+0000 7f6796ffd700 1 -- 172.21.15.111:0/431860220 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c009710 con 0x7f67a00cdef0 2023-11-28T12:59:42.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.292+0000 7f6796ffd700 1 -- 172.21.15.111:0/431860220 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679c00c850 con 0x7f67a00cdef0 2023-11-28T12:59:42.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.292+0000 7f67a836a700 1 -- 172.21.15.111:0/431860220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a00cd340 con 0x7f67a00cdef0 2023-11-28T12:59:42.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.292+0000 7f6796ffd700 1 -- 172.21.15.111:0/431860220 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c00cc00 con 0x7f67a00cdef0 2023-11-28T12:59:42.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.292+0000 7f67a836a700 1 -- 172.21.15.111:0/431860220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a00cd060 con 0x7f67a00cdef0 2023-11-28T12:59:42.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.293+0000 7f67a836a700 1 -- 172.21.15.111:0/431860220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a0143280 con 0x7f67a00cdef0 2023-11-28T12:59:42.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.293+0000 7f6796ffd700 1 -- 172.21.15.111:0/431860220 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f679c019030 con 0x7f67a00cdef0 2023-11-28T12:59:42.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.294+0000 7f6796ffd700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f67840682f0 0x7f678406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.294+0000 7f67a6106700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f67840682f0 0x7f678406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.294+0000 7f6796ffd700 1 -- 172.21.15.111:0/431860220 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f679c086020 con 0x7f67a00cdef0 2023-11-28T12:59:42.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.294+0000 7f67a6106700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f67840682f0 0x7f678406a7a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6790000f60 tx=0x7f6790000bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:42.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.298+0000 7f6796ffd700 1 -- 172.21.15.111:0/431860220 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f679c052580 con 0x7f67a00cdef0 2023-11-28T12:59:42.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.480+0000 7f67a836a700 1 -- 172.21.15.111:0/431860220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f67a0067170 con 0x7f67a00cdef0 2023-11-28T12:59:42.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.480+0000 7f6796ffd700 1 -- 172.21.15.111:0/431860220 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7f679c0210a0 con 0x7f67a00cdef0 2023-11-28T12:59:42.482 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:59:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.482+0000 7f67a836a700 1 -- 172.21.15.111:0/431860220 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f67840682f0 msgr2=0x7f678406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.482+0000 7f67a836a700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f67840682f0 0x7f678406a7a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6790000f60 tx=0x7f6790000bc0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.482+0000 7f67a836a700 1 -- 172.21.15.111:0/431860220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67a00cdef0 msgr2=0x7f67a015bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.482+0000 7f67a836a700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67a00cdef0 0x7f67a015bfa0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f679c00e980 tx=0x7f679c00ed40 comp rx=0 tx=0).stop 2023-11-28T12:59:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.482+0000 7f67a836a700 1 -- 172.21.15.111:0/431860220 shutdown_connections 2023-11-28T12:59:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.482+0000 7f67a836a700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67a00cfc90 0x7f67a01ed240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.482+0000 7f67a836a700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f67840682f0 0x7f678406a7a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.482+0000 7f67a836a700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f67a00cc5a0 0x7f67a00b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.482+0000 7f67a836a700 1 --2- 172.21.15.111:0/431860220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67a00cdef0 0x7f67a015bfa0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.482+0000 7f67a836a700 1 -- 172.21.15.111:0/431860220 >> 172.21.15.111:0/431860220 conn(0x7f67a0098ee0 msgr2=0x7f67a00adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:42.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.482+0000 7f67a836a700 1 -- 172.21.15.111:0/431860220 shutdown_connections 2023-11-28T12:59:42.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.482+0000 7f67a836a700 1 -- 172.21.15.111:0/431860220 wait complete. 2023-11-28T12:59:42.495 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T12:59:42.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.617+0000 7f75e1e7e700 1 Processor -- start 2023-11-28T12:59:42.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.617+0000 7f75e1e7e700 1 -- start start 2023-11-28T12:59:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75e1e7e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75dc0b0d70 0x7f75dc1604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75e1e7e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f75dc0b1140 0x7f75dc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75e1e7e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc15cd90 con 0x7f75dc0b15b0 2023-11-28T12:59:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75e1e7e700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75dc1613e0 con 0x7f75dc0b1140 2023-11-28T12:59:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75e1e7e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc160b40 con 0x7f75dc0b0d70 2023-11-28T12:59:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75dbfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f75dc0b15b0 0x7f75dc0cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34938/0 (socket says 172.21.15.111:34938) 2023-11-28T12:59:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75dbfff700 1 -- 172.21.15.111:0/2346458216 learned_addr learned my addr 172.21.15.111:0/2346458216 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75db7fe700 1 --2- 172.21.15.111:0/2346458216 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75dc0b0d70 0x7f75dc1604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75daffd700 1 --2- 172.21.15.111:0/2346458216 >> v2:172.21.15.149:3300/0 conn(0x7f75dc0b1140 0x7f75dc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75da7fc700 1 -- 172.21.15.111:0/2346458216 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917970942 0 0) 0x7f75d4001490 con 0x7f75dc0b15b0 2023-11-28T12:59:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75da7fc700 1 -- 172.21.15.111:0/2346458216 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c40035b0 con 0x7f75dc0b15b0 2023-11-28T12:59:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75da7fc700 1 -- 172.21.15.111:0/2346458216 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3220665942 0 0) 0x7f75c40035b0 con 0x7f75dc0b15b0 2023-11-28T12:59:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75db7fe700 1 -- 172.21.15.111:0/2346458216 >> v2:172.21.15.149:3300/0 conn(0x7f75dc0b1140 msgr2=0x7f75dc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75db7fe700 1 --2- 172.21.15.111:0/2346458216 >> v2:172.21.15.149:3300/0 conn(0x7f75dc0b1140 0x7f75dc0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75db7fe700 1 -- 172.21.15.111:0/2346458216 >> v1:172.21.15.111:6789/0 conn(0x7f75dc0b15b0 legacy=0x7f75dc0cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.618+0000 7f75db7fe700 1 -- 172.21.15.111:0/2346458216 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc160e20 con 0x7f75dc0b0d70 2023-11-28T12:59:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.619+0000 7f75db7fe700 1 --2- 172.21.15.111:0/2346458216 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75dc0b0d70 0x7f75dc1604b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009820 tx=0x7f75d0009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8f258cdf981e072 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.619+0000 7f75da7fc700 1 -- 172.21.15.111:0/2346458216 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d4001b80 con 0x7f75dc0b0d70 2023-11-28T12:59:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.619+0000 7f75da7fc700 1 -- 172.21.15.111:0/2346458216 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d0005c30 con 0x7f75dc0b0d70 2023-11-28T12:59:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.619+0000 7f75da7fc700 1 -- 172.21.15.111:0/2346458216 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d0004030 con 0x7f75dc0b0d70 2023-11-28T12:59:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.619+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2346458216 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75dc0b0d70 msgr2=0x7f75dc1604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.619+0000 7f75e1e7e700 1 --2- 172.21.15.111:0/2346458216 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75dc0b0d70 0x7f75dc1604b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009820 tx=0x7f75d0009b30 comp rx=0 tx=0).stop 2023-11-28T12:59:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.619+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2346458216 shutdown_connections 2023-11-28T12:59:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.619+0000 7f75e1e7e700 1 --2- 172.21.15.111:0/2346458216 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75dc0b0d70 0x7f75dc1604b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.619+0000 7f75e1e7e700 1 --2- 172.21.15.111:0/2346458216 >> v2:172.21.15.149:3300/0 conn(0x7f75dc0b1140 0x7f75dc0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.619+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2346458216 >> 172.21.15.111:0/2346458216 conn(0x7f75dc098ee0 msgr2=0x7f75dc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.619+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2346458216 shutdown_connections 2023-11-28T12:59:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.619+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2346458216 wait complete. 2023-11-28T12:59:42.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.621+0000 7f75e1e7e700 1 Processor -- start 2023-11-28T12:59:42.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.621+0000 7f75e1e7e700 1 -- start start 2023-11-28T12:59:42.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.621+0000 7f75e1e7e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75dc0b0d70 0x7f75dc15f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75e1e7e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75dc0b1140 0x7f75dc0b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75e1e7e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc0b15b0 0x7f75dc155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75e1e7e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc15fe40 con 0x7f75dc0b15b0 2023-11-28T12:59:42.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75e1e7e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc0c45a0 con 0x7f75dc0b1140 2023-11-28T12:59:42.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75e1e7e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc0c4880 con 0x7f75dc0b0d70 2023-11-28T12:59:42.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75dbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc0b15b0 0x7f75dc155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75dbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc0b15b0 0x7f75dc155770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34166/0 (socket says 172.21.15.111:34166) 2023-11-28T12:59:42.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75dbfff700 1 -- 172.21.15.111:0/2160807343 learned_addr learned my addr 172.21.15.111:0/2160807343 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:42.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75db7fe700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75dc0b0d70 0x7f75dc15f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75daffd700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75dc0b1140 0x7f75dc0b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75dbfff700 1 -- 172.21.15.111:0/2160807343 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75dc0b0d70 msgr2=0x7f75dc15f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75dbfff700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75dc0b0d70 0x7f75dc15f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75dbfff700 1 -- 172.21.15.111:0/2160807343 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75dc0b1140 msgr2=0x7f75dc0b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75dbfff700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75dc0b1140 0x7f75dc0b2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75dbfff700 1 -- 172.21.15.111:0/2160807343 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d0009430 con 0x7f75dc0b15b0 2023-11-28T12:59:42.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75dbfff700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc0b15b0 0x7f75dc155770 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f75d400cc00 tx=0x7f75d400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75c3fff700 1 -- 172.21.15.111:0/2160807343 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d400ab20 con 0x7f75dc0b15b0 2023-11-28T12:59:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75c3fff700 1 -- 172.21.15.111:0/2160807343 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d400aca0 con 0x7f75dc0b15b0 2023-11-28T12:59:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2160807343 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75dc0af340 con 0x7f75dc0b15b0 2023-11-28T12:59:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75c3fff700 1 -- 172.21.15.111:0/2160807343 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d4007310 con 0x7f75dc0b15b0 2023-11-28T12:59:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.622+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2160807343 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75dc161100 con 0x7f75dc0b15b0 2023-11-28T12:59:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.623+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2160807343 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75dc0cec00 con 0x7f75dc0b15b0 2023-11-28T12:59:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.624+0000 7f75c3fff700 1 -- 172.21.15.111:0/2160807343 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f75d401a030 con 0x7f75dc0b15b0 2023-11-28T12:59:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.624+0000 7f75c3fff700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75c80682f0 0x7f75c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.624+0000 7f75c3fff700 1 -- 172.21.15.111:0/2160807343 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f75d4085b40 con 0x7f75dc0b15b0 2023-11-28T12:59:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.624+0000 7f75db7fe700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75c80682f0 0x7f75c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.624+0000 7f75db7fe700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75c80682f0 0x7f75c806a7a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f75d0000c00 tx=0x7f75d0008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:42.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.628+0000 7f75c3fff700 1 -- 172.21.15.111:0/2160807343 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f75d4052070 con 0x7f75dc0b15b0 2023-11-28T12:59:42.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.811+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2160807343 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f75dc15cd90 con 0x7f75dc0b15b0 2023-11-28T12:59:42.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.811+0000 7f75c3fff700 1 -- 172.21.15.111:0/2160807343 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f75dc15cd90 con 0x7f75dc0b15b0 2023-11-28T12:59:42.813 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:42.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.813+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2160807343 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75c80682f0 msgr2=0x7f75c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.813+0000 7f75e1e7e700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75c80682f0 0x7f75c806a7a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f75d0000c00 tx=0x7f75d0008070 comp rx=0 tx=0).stop 2023-11-28T12:59:42.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.813+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2160807343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc0b15b0 msgr2=0x7f75dc155770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.813+0000 7f75e1e7e700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc0b15b0 0x7f75dc155770 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f75d400cc00 tx=0x7f75d400a060 comp rx=0 tx=0).stop 2023-11-28T12:59:42.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.813+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2160807343 shutdown_connections 2023-11-28T12:59:42.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.813+0000 7f75e1e7e700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc0b15b0 0x7f75dc155770 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.813+0000 7f75e1e7e700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75dc0b0d70 0x7f75dc15f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.813+0000 7f75e1e7e700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75c80682f0 0x7f75c806a7a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.813+0000 7f75e1e7e700 1 --2- 172.21.15.111:0/2160807343 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75dc0b1140 0x7f75dc0b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.813+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2160807343 >> 172.21.15.111:0/2160807343 conn(0x7f75dc098ee0 msgr2=0x7f75dc06dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:42.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.813+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2160807343 shutdown_connections 2023-11-28T12:59:42.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.813+0000 7f75e1e7e700 1 -- 172.21.15.111:0/2160807343 wait complete. 2023-11-28T12:59:42.827 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T12:59:42.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.949+0000 7fe5d2351700 1 Processor -- start 2023-11-28T12:59:42.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.949+0000 7fe5d2351700 1 -- start start 2023-11-28T12:59:42.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5d2351700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe5cc0cc550 0x7fe5cc0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5d2351700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5cc15cd90 con 0x7fe5cc1604b0 2023-11-28T12:59:42.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5d2351700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5cc0cced0 con 0x7fe5cc0b0d70 2023-11-28T12:59:42.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5d2351700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5cc160950 con 0x7fe5cc0cc550 2023-11-28T12:59:42.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5d08ee700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe5cc0cc550 0x7fe5cc0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5cb7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe5cc1604b0 0x7fe5cc0ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34948/0 (socket says 172.21.15.111:34948) 2023-11-28T12:59:42.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5cb7fe700 1 -- 172.21.15.111:0/3642196429 learned_addr learned my addr 172.21.15.111:0/3642196429 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:42.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5d08ee700 1 -- 172.21.15.111:0/3642196429 >> v1:172.21.15.149:6789/0 conn(0x7fe5cc0b0d70 legacy=0x7fe5cc0b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:42.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5d08ee700 1 -- 172.21.15.111:0/3642196429 >> v1:172.21.15.111:6789/0 conn(0x7fe5cc1604b0 legacy=0x7fe5cc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5caffd700 1 -- 172.21.15.111:0/3642196429 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131031324 0 0) 0x7fe5bc001110 con 0x7fe5cc1604b0 2023-11-28T12:59:42.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5d08ee700 1 -- 172.21.15.111:0/3642196429 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5cc1613e0 con 0x7fe5cc0cc550 2023-11-28T12:59:42.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5d08ee700 1 --2- 172.21.15.111:0/3642196429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5cc0cc550 0x7fe5cc0cc990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4009d90 tx=0x7fe5c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f9d2b1cf0ac862 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:42.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5caffd700 1 -- 172.21.15.111:0/3642196429 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c400e070 con 0x7fe5cc0cc550 2023-11-28T12:59:42.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.950+0000 7fe5caffd700 1 -- 172.21.15.111:0/3642196429 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c400bb90 con 0x7fe5cc0cc550 2023-11-28T12:59:42.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.951+0000 7fe5caffd700 1 -- 172.21.15.111:0/3642196429 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4003a20 con 0x7fe5cc0cc550 2023-11-28T12:59:42.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.951+0000 7fe5d2351700 1 -- 172.21.15.111:0/3642196429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5cc0cc550 msgr2=0x7fe5cc0cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.951+0000 7fe5d2351700 1 --2- 172.21.15.111:0/3642196429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5cc0cc550 0x7fe5cc0cc990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4009d90 tx=0x7fe5c400b330 comp rx=0 tx=0).stop 2023-11-28T12:59:42.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.951+0000 7fe5d2351700 1 -- 172.21.15.111:0/3642196429 shutdown_connections 2023-11-28T12:59:42.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.951+0000 7fe5d2351700 1 --2- 172.21.15.111:0/3642196429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5cc0cc550 0x7fe5cc0cc990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.951+0000 7fe5d2351700 1 -- 172.21.15.111:0/3642196429 >> 172.21.15.111:0/3642196429 conn(0x7fe5cc098ee0 msgr2=0x7fe5cc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:42.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.951+0000 7fe5d2351700 1 -- 172.21.15.111:0/3642196429 shutdown_connections 2023-11-28T12:59:42.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.951+0000 7fe5d2351700 1 -- 172.21.15.111:0/3642196429 wait complete. 2023-11-28T12:59:42.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.952+0000 7fe5d2351700 1 Processor -- start 2023-11-28T12:59:42.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.952+0000 7fe5d2351700 1 -- start start 2023-11-28T12:59:42.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.953+0000 7fe5d2351700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5cc0b0d70 0x7fe5cc0c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.953+0000 7fe5d2351700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5cc0cc550 0x7fe5cc150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.953+0000 7fe5d2351700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5cc1604b0 0x7fe5cc0ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.953+0000 7fe5d2351700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc151340 con 0x7fe5cc0cc550 2023-11-28T12:59:42.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.953+0000 7fe5d2351700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0c99d0 con 0x7fe5cc1604b0 2023-11-28T12:59:42.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.953+0000 7fe5d2351700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc0c8e30 con 0x7fe5cc0b0d70 2023-11-28T12:59:42.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.953+0000 7fe5cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5cc0cc550 0x7fe5cc150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.953+0000 7fe5cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5cc0cc550 0x7fe5cc150280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34176/0 (socket says 172.21.15.111:34176) 2023-11-28T12:59:42.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.953+0000 7fe5cb7fe700 1 -- 172.21.15.111:0/2549711395 learned_addr learned my addr 172.21.15.111:0/2549711395 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:42.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.953+0000 7fe5cbfff700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5cc0b0d70 0x7fe5cc0c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.953+0000 7fe5d08ee700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5cc1604b0 0x7fe5cc0ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.954+0000 7fe5cb7fe700 1 -- 172.21.15.111:0/2549711395 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5cc0b0d70 msgr2=0x7fe5cc0c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.954+0000 7fe5cb7fe700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5cc0b0d70 0x7fe5cc0c9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.954+0000 7fe5cb7fe700 1 -- 172.21.15.111:0/2549711395 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5cc1604b0 msgr2=0x7fe5cc0ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.954+0000 7fe5cb7fe700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5cc1604b0 0x7fe5cc0ad6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.954+0000 7fe5cb7fe700 1 -- 172.21.15.111:0/2549711395 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c4009a70 con 0x7fe5cc0cc550 2023-11-28T12:59:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.954+0000 7fe5cbfff700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5cc0b0d70 0x7fe5cc0c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.954+0000 7fe5d08ee700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5cc1604b0 0x7fe5cc0ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.954+0000 7fe5cb7fe700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5cc0cc550 0x7fe5cc150280 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00c550 tx=0x7fe5bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.954+0000 7fe5c8ff9700 1 -- 172.21.15.111:0/2549711395 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc004c80 con 0x7fe5cc0cc550 2023-11-28T12:59:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.954+0000 7fe5c8ff9700 1 -- 172.21.15.111:0/2549711395 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5bc004e00 con 0x7fe5cc0cc550 2023-11-28T12:59:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.954+0000 7fe5c8ff9700 1 -- 172.21.15.111:0/2549711395 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc007310 con 0x7fe5cc0cc550 2023-11-28T12:59:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.954+0000 7fe5d2351700 1 -- 172.21.15.111:0/2549711395 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5cc160e20 con 0x7fe5cc0cc550 2023-11-28T12:59:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.954+0000 7fe5d2351700 1 -- 172.21.15.111:0/2549711395 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5cc160b40 con 0x7fe5cc0cc550 2023-11-28T12:59:42.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.955+0000 7fe5d2351700 1 -- 172.21.15.111:0/2549711395 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5cc161100 con 0x7fe5cc0cc550 2023-11-28T12:59:42.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.955+0000 7fe5c8ff9700 1 -- 172.21.15.111:0/2549711395 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe5bc01a020 con 0x7fe5cc0cc550 2023-11-28T12:59:42.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.956+0000 7fe5c8ff9700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5b80682f0 0x7fe5b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:42.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.956+0000 7fe5c8ff9700 1 -- 172.21.15.111:0/2549711395 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fe5bc085ab0 con 0x7fe5cc0cc550 2023-11-28T12:59:42.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.956+0000 7fe5cbfff700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5b80682f0 0x7fe5b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:42.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.956+0000 7fe5cbfff700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5b80682f0 0x7fe5b806a7a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe5c00017f0 tx=0x7fe5c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:42.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:42.960+0000 7fe5c8ff9700 1 -- 172.21.15.111:0/2549711395 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe5bc051ef0 con 0x7fe5cc0cc550 2023-11-28T12:59:43.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.140+0000 7fe5d2351700 1 -- 172.21.15.111:0/2549711395 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe5cc15cfc0 con 0x7fe5cc0cc550 2023-11-28T12:59:43.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.140+0000 7fe5c8ff9700 1 -- 172.21.15.111:0/2549711395 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe5cc15cfc0 con 0x7fe5cc0cc550 2023-11-28T12:59:43.142 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T12:59:43.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.142+0000 7fe5d2351700 1 -- 172.21.15.111:0/2549711395 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5b80682f0 msgr2=0x7fe5b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.142+0000 7fe5d2351700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5b80682f0 0x7fe5b806a7a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe5c00017f0 tx=0x7fe5c0009040 comp rx=0 tx=0).stop 2023-11-28T12:59:43.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.142+0000 7fe5d2351700 1 -- 172.21.15.111:0/2549711395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5cc0cc550 msgr2=0x7fe5cc150280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.142+0000 7fe5d2351700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5cc0cc550 0x7fe5cc150280 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00c550 tx=0x7fe5bc00c910 comp rx=0 tx=0).stop 2023-11-28T12:59:43.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.142+0000 7fe5d2351700 1 -- 172.21.15.111:0/2549711395 shutdown_connections 2023-11-28T12:59:43.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.142+0000 7fe5d2351700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5b80682f0 0x7fe5b806a7a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.142+0000 7fe5d2351700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5cc1604b0 0x7fe5cc0ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.142+0000 7fe5d2351700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5cc0b0d70 0x7fe5cc0c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.142+0000 7fe5d2351700 1 --2- 172.21.15.111:0/2549711395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5cc0cc550 0x7fe5cc150280 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.142+0000 7fe5d2351700 1 -- 172.21.15.111:0/2549711395 >> 172.21.15.111:0/2549711395 conn(0x7fe5cc098ee0 msgr2=0x7fe5cc0b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:43.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.142+0000 7fe5d2351700 1 -- 172.21.15.111:0/2549711395 shutdown_connections 2023-11-28T12:59:43.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.142+0000 7fe5d2351700 1 -- 172.21.15.111:0/2549711395 wait complete. 2023-11-28T12:59:43.156 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T12:59:43.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.278+0000 7fbfd97ce700 1 Processor -- start 2023-11-28T12:59:43.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.278+0000 7fbfd97ce700 1 -- start start 2023-11-28T12:59:43.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.279+0000 7fbfd97ce700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbfd4159960 0x7fbfd40b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:43.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.279+0000 7fbfd97ce700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfd40cdd90 0x7fbfd40ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:43.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.279+0000 7fbfd97ce700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd40ad420 con 0x7fbfd40aecc0 2023-11-28T12:59:43.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.279+0000 7fbfd97ce700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd4159d30 con 0x7fbfd4159960 2023-11-28T12:59:43.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.279+0000 7fbfd97ce700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40ac810 con 0x7fbfd40cdd90 2023-11-28T12:59:43.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.279+0000 7fbfd2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbfd4159960 0x7fbfd40b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:43.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.279+0000 7fbfd27fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbfd40aecc0 0x7fbfd40cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34960/0 (socket says 172.21.15.111:34960) 2023-11-28T12:59:43.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.279+0000 7fbfd27fc700 1 -- 172.21.15.111:0/2809059286 learned_addr learned my addr 172.21.15.111:0/2809059286 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:43.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd37fe700 1 --2- 172.21.15.111:0/2809059286 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfd40cdd90 0x7fbfd40ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:43.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd2ffd700 1 -- 172.21.15.111:0/2809059286 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfd40cdd90 msgr2=0x7fbfd40ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:43.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd2ffd700 1 --2- 172.21.15.111:0/2809059286 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfd40cdd90 0x7fbfd40ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd2ffd700 1 -- 172.21.15.111:0/2809059286 >> v1:172.21.15.111:6789/0 conn(0x7fbfd40aecc0 legacy=0x7fbfd40cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd2ffd700 1 -- 172.21.15.111:0/2809059286 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd40acaf0 con 0x7fbfd4159960 2023-11-28T12:59:43.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd2ffd700 1 --2- 172.21.15.111:0/2809059286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfd4159960 0x7fbfd40b1fd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009820 tx=0x7fbfc8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e2cbd779de38d4b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:43.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd1ffb700 1 -- 172.21.15.111:0/2809059286 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8005e30 con 0x7fbfd4159960 2023-11-28T12:59:43.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd1ffb700 1 -- 172.21.15.111:0/2809059286 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc8006030 con 0x7fbfd4159960 2023-11-28T12:59:43.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd1ffb700 1 -- 172.21.15.111:0/2809059286 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc80063e0 con 0x7fbfd4159960 2023-11-28T12:59:43.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd97ce700 1 -- 172.21.15.111:0/2809059286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfd4159960 msgr2=0x7fbfd40b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd97ce700 1 --2- 172.21.15.111:0/2809059286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfd4159960 0x7fbfd40b1fd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009820 tx=0x7fbfc8009b30 comp rx=0 tx=0).stop 2023-11-28T12:59:43.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd97ce700 1 -- 172.21.15.111:0/2809059286 shutdown_connections 2023-11-28T12:59:43.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd97ce700 1 --2- 172.21.15.111:0/2809059286 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfd40cdd90 0x7fbfd40ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd97ce700 1 --2- 172.21.15.111:0/2809059286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfd4159960 0x7fbfd40b1fd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.280+0000 7fbfd97ce700 1 -- 172.21.15.111:0/2809059286 >> 172.21.15.111:0/2809059286 conn(0x7fbfd4098ee0 msgr2=0x7fbfd406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:43.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.281+0000 7fbfd97ce700 1 -- 172.21.15.111:0/2809059286 shutdown_connections 2023-11-28T12:59:43.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.281+0000 7fbfd97ce700 1 -- 172.21.15.111:0/2809059286 wait complete. 2023-11-28T12:59:43.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.282+0000 7fbfd97ce700 1 Processor -- start 2023-11-28T12:59:43.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.282+0000 7fbfd97ce700 1 -- start start 2023-11-28T12:59:43.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd97ce700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbfd40aecc0 0x7fbfd41543c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:43.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd97ce700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfd40cdd90 0x7fbfd41f35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:43.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd97ce700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfd4159960 0x7fbfd409b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:43.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd97ce700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd4154900 con 0x7fbfd40cdd90 2023-11-28T12:59:43.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd97ce700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd41f3e00 con 0x7fbfd40aecc0 2023-11-28T12:59:43.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd97ce700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd41f40e0 con 0x7fbfd4159960 2023-11-28T12:59:43.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfd40cdd90 0x7fbfd41f35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfd40cdd90 0x7fbfd41f35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34190/0 (socket says 172.21.15.111:34190) 2023-11-28T12:59:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd27fc700 1 -- 172.21.15.111:0/1660638019 learned_addr learned my addr 172.21.15.111:0/1660638019 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd2ffd700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbfd40aecc0 0x7fbfd41543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd37fe700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfd4159960 0x7fbfd409b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd27fc700 1 -- 172.21.15.111:0/1660638019 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfd4159960 msgr2=0x7fbfd409b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd27fc700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfd4159960 0x7fbfd409b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd27fc700 1 -- 172.21.15.111:0/1660638019 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbfd40aecc0 msgr2=0x7fbfd41543c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd27fc700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbfd40aecc0 0x7fbfd41543c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd27fc700 1 -- 172.21.15.111:0/1660638019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc8009430 con 0x7fbfd40cdd90 2023-11-28T12:59:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd2ffd700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbfd40aecc0 0x7fbfd41543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd27fc700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfd40cdd90 0x7fbfd41f35d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400c520 tx=0x7fbfc400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfbb7fe700 1 -- 172.21.15.111:0/1660638019 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4005d70 con 0x7fbfd40cdd90 2023-11-28T12:59:43.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfbb7fe700 1 -- 172.21.15.111:0/1660638019 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc40070d0 con 0x7fbfd40cdd90 2023-11-28T12:59:43.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd97ce700 1 -- 172.21.15.111:0/1660638019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd40ad0b0 con 0x7fbfd40cdd90 2023-11-28T12:59:43.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfbb7fe700 1 -- 172.21.15.111:0/1660638019 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4007440 con 0x7fbfd40cdd90 2023-11-28T12:59:43.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.283+0000 7fbfd97ce700 1 -- 172.21.15.111:0/1660638019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd40acdd0 con 0x7fbfd40cdd90 2023-11-28T12:59:43.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.284+0000 7fbfd97ce700 1 -- 172.21.15.111:0/1660638019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd40addf0 con 0x7fbfd40cdd90 2023-11-28T12:59:43.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.285+0000 7fbfbb7fe700 1 -- 172.21.15.111:0/1660638019 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbfc4019030 con 0x7fbfd40cdd90 2023-11-28T12:59:43.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.285+0000 7fbfbb7fe700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbfc00682f0 0x7fbfc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:43.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.285+0000 7fbfbb7fe700 1 -- 172.21.15.111:0/1660638019 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fbfc4085260 con 0x7fbfd40cdd90 2023-11-28T12:59:43.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.285+0000 7fbfd2ffd700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbfc00682f0 0x7fbfc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:43.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.286+0000 7fbfd2ffd700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbfc00682f0 0x7fbfc006a7a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8013040 tx=0x7fbfc8008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:43.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.289+0000 7fbfbb7fe700 1 -- 172.21.15.111:0/1660638019 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbfc404b5e0 con 0x7fbfd40cdd90 2023-11-28T12:59:43.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.470+0000 7fbfd97ce700 1 -- 172.21.15.111:0/1660638019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbfd4067170 con 0x7fbfd40cdd90 2023-11-28T12:59:43.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.471+0000 7fbfbb7fe700 1 -- 172.21.15.111:0/1660638019 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbfc4051840 con 0x7fbfd40cdd90 2023-11-28T12:59:43.472 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:43.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.472+0000 7fbfd97ce700 1 -- 172.21.15.111:0/1660638019 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbfc00682f0 msgr2=0x7fbfc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.472+0000 7fbfd97ce700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbfc00682f0 0x7fbfc006a7a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8013040 tx=0x7fbfc8008070 comp rx=0 tx=0).stop 2023-11-28T12:59:43.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.473+0000 7fbfd97ce700 1 -- 172.21.15.111:0/1660638019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfd40cdd90 msgr2=0x7fbfd41f35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.473+0000 7fbfd97ce700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfd40cdd90 0x7fbfd41f35d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400c520 tx=0x7fbfc400c8e0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.473+0000 7fbfd97ce700 1 -- 172.21.15.111:0/1660638019 shutdown_connections 2023-11-28T12:59:43.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.473+0000 7fbfd97ce700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfd4159960 0x7fbfd409b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.473+0000 7fbfd97ce700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbfc00682f0 0x7fbfc006a7a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.473+0000 7fbfd97ce700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbfd40aecc0 0x7fbfd41543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.473+0000 7fbfd97ce700 1 --2- 172.21.15.111:0/1660638019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfd40cdd90 0x7fbfd41f35d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.473+0000 7fbfd97ce700 1 -- 172.21.15.111:0/1660638019 >> 172.21.15.111:0/1660638019 conn(0x7fbfd4098ee0 msgr2=0x7fbfd406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:43.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.473+0000 7fbfd97ce700 1 -- 172.21.15.111:0/1660638019 shutdown_connections 2023-11-28T12:59:43.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.473+0000 7fbfd97ce700 1 -- 172.21.15.111:0/1660638019 wait complete. 2023-11-28T12:59:43.494 INFO:tasks.cephfs_test_runner:Starting test: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-28T12:59:43.494 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2023-11-28T12:59:43.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.613+0000 7f7ef02c0700 1 Processor -- start 2023-11-28T12:59:43.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.614+0000 7f7ef02c0700 1 -- start start 2023-11-28T12:59:43.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.614+0000 7f7ef02c0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7ee80cc5a0 0x7f7ee80d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:43.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.614+0000 7f7ef02c0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee80d05a0 con 0x7f7ee80cdef0 2023-11-28T12:59:43.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.614+0000 7f7ef02c0700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee80ca260 con 0x7f7ee80cfc90 2023-11-28T12:59:43.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.614+0000 7f7ef02c0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee80ccaa0 con 0x7f7ee80cc5a0 2023-11-28T12:59:43.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.615+0000 7f7eee85d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7ee80cc5a0 0x7f7ee80d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:43.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.615+0000 7f7eed85b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7ee80cdef0 0x7f7ee80ca6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34976/0 (socket says 172.21.15.111:34976) 2023-11-28T12:59:43.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.615+0000 7f7eed85b700 1 -- 172.21.15.111:0/3452729112 learned_addr learned my addr 172.21.15.111:0/3452729112 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:43.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.615+0000 7f7eee85d700 1 -- 172.21.15.111:0/3452729112 >> v1:172.21.15.149:6789/0 conn(0x7f7ee80cfc90 legacy=0x7f7ee80cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:43.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.615+0000 7f7eee85d700 1 -- 172.21.15.111:0/3452729112 >> v1:172.21.15.111:6789/0 conn(0x7f7ee80cdef0 legacy=0x7f7ee80ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.615+0000 7f7eee85d700 1 -- 172.21.15.111:0/3452729112 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee80ccd80 con 0x7f7ee80cc5a0 2023-11-28T12:59:43.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.615+0000 7f7eed05a700 1 -- 172.21.15.111:0/3452729112 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990574878 0 0) 0x7f7ee4001110 con 0x7f7ee80cdef0 2023-11-28T12:59:43.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.615+0000 7f7eee85d700 1 --2- 172.21.15.111:0/3452729112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ee80cc5a0 0x7f7ee80d0060 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009d90 tx=0x7f7ee000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c95646db595bdf5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:43.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.615+0000 7f7eed05a700 1 -- 172.21.15.111:0/3452729112 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee000e070 con 0x7f7ee80cc5a0 2023-11-28T12:59:43.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.615+0000 7f7eed05a700 1 -- 172.21.15.111:0/3452729112 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee000bb90 con 0x7f7ee80cc5a0 2023-11-28T12:59:43.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.615+0000 7f7eed05a700 1 -- 172.21.15.111:0/3452729112 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0003a20 con 0x7f7ee80cc5a0 2023-11-28T12:59:43.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.616+0000 7f7ef02c0700 1 -- 172.21.15.111:0/3452729112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ee80cc5a0 msgr2=0x7f7ee80d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.616+0000 7f7ef02c0700 1 --2- 172.21.15.111:0/3452729112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ee80cc5a0 0x7f7ee80d0060 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009d90 tx=0x7f7ee000b330 comp rx=0 tx=0).stop 2023-11-28T12:59:43.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.616+0000 7f7ef02c0700 1 -- 172.21.15.111:0/3452729112 shutdown_connections 2023-11-28T12:59:43.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.616+0000 7f7ef02c0700 1 --2- 172.21.15.111:0/3452729112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ee80cc5a0 0x7f7ee80d0060 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.616+0000 7f7ef02c0700 1 -- 172.21.15.111:0/3452729112 >> 172.21.15.111:0/3452729112 conn(0x7f7ee8098ee0 msgr2=0x7f7ee806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:43.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.616+0000 7f7ef02c0700 1 -- 172.21.15.111:0/3452729112 shutdown_connections 2023-11-28T12:59:43.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.616+0000 7f7ef02c0700 1 -- 172.21.15.111:0/3452729112 wait complete. 2023-11-28T12:59:43.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.617+0000 7f7ef02c0700 1 Processor -- start 2023-11-28T12:59:43.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.617+0000 7f7ef02c0700 1 -- start start 2023-11-28T12:59:43.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7ef02c0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7ee80cc5a0 0x7f7ee81ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:43.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7ef02c0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ee80cdef0 0x7f7ee80b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:43.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7ef02c0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7ee80cfc90 0x7f7ee80c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:43.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7ef02c0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee8159db0 con 0x7f7ee80cdef0 2023-11-28T12:59:43.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7ef02c0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee815c0d0 con 0x7f7ee80cfc90 2023-11-28T12:59:43.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7ef02c0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee815c300 con 0x7f7ee80cc5a0 2023-11-28T12:59:43.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7eed85b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ee80cdef0 0x7f7ee80b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:43.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7eed85b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ee80cdef0 0x7f7ee80b45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34204/0 (socket says 172.21.15.111:34204) 2023-11-28T12:59:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7eed85b700 1 -- 172.21.15.111:0/1469642872 learned_addr learned my addr 172.21.15.111:0/1469642872 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7eee85d700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7ee80cfc90 0x7f7ee80c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7eee05c700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7ee80cc5a0 0x7f7ee81ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7eed85b700 1 -- 172.21.15.111:0/1469642872 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7ee80cc5a0 msgr2=0x7f7ee81ee3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7eed85b700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7ee80cc5a0 0x7f7ee81ee3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7eed85b700 1 -- 172.21.15.111:0/1469642872 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7ee80cfc90 msgr2=0x7f7ee80c8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7eed85b700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7ee80cfc90 0x7f7ee80c8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7eed85b700 1 -- 172.21.15.111:0/1469642872 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee0009a70 con 0x7f7ee80cdef0 2023-11-28T12:59:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7eee85d700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7ee80cfc90 0x7f7ee80c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7eed85b700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ee80cdef0 0x7f7ee80b45c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c550 tx=0x7f7ee400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7edeffd700 1 -- 172.21.15.111:0/1469642872 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4004c80 con 0x7f7ee80cdef0 2023-11-28T12:59:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7edeffd700 1 -- 172.21.15.111:0/1469642872 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee4004e00 con 0x7f7ee80cdef0 2023-11-28T12:59:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7ef02c0700 1 -- 172.21.15.111:0/1469642872 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee80cd340 con 0x7f7ee80cdef0 2023-11-28T12:59:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7edeffd700 1 -- 172.21.15.111:0/1469642872 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4007310 con 0x7f7ee80cdef0 2023-11-28T12:59:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.618+0000 7f7ef02c0700 1 -- 172.21.15.111:0/1469642872 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee80cd060 con 0x7f7ee80cdef0 2023-11-28T12:59:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.619+0000 7f7ef02c0700 1 -- 172.21.15.111:0/1469642872 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee8143280 con 0x7f7ee80cdef0 2023-11-28T12:59:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.620+0000 7f7edeffd700 1 -- 172.21.15.111:0/1469642872 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7ee401a020 con 0x7f7ee80cdef0 2023-11-28T12:59:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.620+0000 7f7edeffd700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7ecc0682f0 0x7f7ecc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.620+0000 7f7edeffd700 1 -- 172.21.15.111:0/1469642872 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f7ee4085a70 con 0x7f7ee80cdef0 2023-11-28T12:59:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.620+0000 7f7eee05c700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7ecc0682f0 0x7f7ecc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.620+0000 7f7eee05c700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7ecc0682f0 0x7f7ecc06a7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8001670 tx=0x7f7ed8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:43.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.624+0000 7f7edeffd700 1 -- 172.21.15.111:0/1469642872 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7ee4051fa0 con 0x7f7ee80cdef0 2023-11-28T12:59:43.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.884+0000 7f7ef02c0700 1 -- 172.21.15.111:0/1469642872 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) v1 -- 0x7f7ee80d05a0 con 0x7f7ee80cdef0 2023-11-28T12:59:43.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.897+0000 7f7edeffd700 1 -- 172.21.15.111:0/1469642872 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v61) v1 ==== 145+0+0 (secure 0 0 0) 0x7f7ee80d05a0 con 0x7f7ee80cdef0 2023-11-28T12:59:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.899+0000 7f7ef02c0700 1 -- 172.21.15.111:0/1469642872 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7ecc0682f0 msgr2=0x7f7ecc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.899+0000 7f7ef02c0700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7ecc0682f0 0x7f7ecc06a7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8001670 tx=0x7f7ed8009040 comp rx=0 tx=0).stop 2023-11-28T12:59:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.899+0000 7f7ef02c0700 1 -- 172.21.15.111:0/1469642872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ee80cdef0 msgr2=0x7f7ee80b45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.899+0000 7f7ef02c0700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ee80cdef0 0x7f7ee80b45c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c550 tx=0x7f7ee400c910 comp rx=0 tx=0).stop 2023-11-28T12:59:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.899+0000 7f7ef02c0700 1 -- 172.21.15.111:0/1469642872 shutdown_connections 2023-11-28T12:59:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.899+0000 7f7ef02c0700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7ecc0682f0 0x7f7ecc06a7a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.899+0000 7f7ef02c0700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7ee80cfc90 0x7f7ee80c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.899+0000 7f7ef02c0700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7ee80cc5a0 0x7f7ee81ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.899+0000 7f7ef02c0700 1 --2- 172.21.15.111:0/1469642872 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ee80cdef0 0x7f7ee80b45c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.899+0000 7f7ef02c0700 1 -- 172.21.15.111:0/1469642872 >> 172.21.15.111:0/1469642872 conn(0x7f7ee8098ee0 msgr2=0x7f7ee806f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.899+0000 7f7ef02c0700 1 -- 172.21.15.111:0/1469642872 shutdown_connections 2023-11-28T12:59:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:43.899+0000 7f7ef02c0700 1 -- 172.21.15.111:0/1469642872 wait complete. 2023-11-28T12:59:43.912 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T12:59:44.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.034+0000 7f250def0700 1 Processor -- start 2023-11-28T12:59:44.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.034+0000 7f250def0700 1 -- start start 2023-11-28T12:59:44.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.034+0000 7f250def0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f25080cdef0 0x7f25080d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.034+0000 7f250def0700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f25080cc5a0 0x7f25080ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.034+0000 7f250def0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25080d05a0 con 0x7f25080cfc90 2023-11-28T12:59:44.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.034+0000 7f250def0700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25080ca260 con 0x7f25080cc5a0 2023-11-28T12:59:44.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.034+0000 7f250def0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25080ccaa0 con 0x7f25080cdef0 2023-11-28T12:59:44.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25077fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f25080cfc90 0x7f25080cd7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:34988/0 (socket says 172.21.15.111:34988) 2023-11-28T12:59:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25077fe700 1 -- 172.21.15.111:0/4250415657 learned_addr learned my addr 172.21.15.111:0/4250415657 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f2506ffd700 1 --2- 172.21.15.111:0/4250415657 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f25080cdef0 0x7f25080d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148226201 0 0) 0x7f24f8001110 con 0x7f25080cfc90 2023-11-28T12:59:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f2507fff700 1 --2- 172.21.15.111:0/4250415657 >> v2:172.21.15.149:3300/0 conn(0x7f25080cc5a0 0x7f25080ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f00035b0 con 0x7f25080cfc90 2023-11-28T12:59:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2180398647 0 0) 0x7f24f00035b0 con 0x7f25080cfc90 2023-11-28T12:59:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f25080cdef0 msgr2=0x7f25080d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 --2- 172.21.15.111:0/4250415657 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f25080cdef0 0x7f25080d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 >> v2:172.21.15.149:3300/0 conn(0x7f25080cc5a0 msgr2=0x7f25080ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 --2- 172.21.15.111:0/4250415657 >> v2:172.21.15.149:3300/0 conn(0x7f25080cc5a0 0x7f25080ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25080ccd80 con 0x7f25080cfc90 2023-11-28T12:59:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f24f8001800 con 0x7f25080cfc90 2023-11-28T12:59:44.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 >> v1:172.21.15.111:6789/0 conn(0x7f25080cfc90 legacy=0x7f25080cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 --2- 172.21.15.111:0/4250415657 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f25080cdef0 0x7f24f0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 --2- 172.21.15.111:0/4250415657 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24f0003f80 0x7f24f0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 --2- 172.21.15.111:0/4250415657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24f0004d90 0x7f24f0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25080ca260 con 0x7f24f0004d90 2023-11-28T12:59:44.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25080ccaa0 con 0x7f24f0003f80 2023-11-28T12:59:44.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f000ba10 con 0x7f25080cdef0 2023-11-28T12:59:44.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f2507fff700 1 --2- 172.21.15.111:0/4250415657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24f0004d90 0x7f24f0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f2506ffd700 1 --2- 172.21.15.111:0/4250415657 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24f0003f80 0x7f24f0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.035+0000 7f25077fe700 1 --2- 172.21.15.111:0/4250415657 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f25080cdef0 0x7f24f0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f2507fff700 1 -- 172.21.15.111:0/4250415657 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f25080cdef0 msgr2=0x7f24f0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f2507fff700 1 --2- 172.21.15.111:0/4250415657 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f25080cdef0 0x7f24f0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f2507fff700 1 -- 172.21.15.111:0/4250415657 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24f0003f80 msgr2=0x7f24f0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f2507fff700 1 --2- 172.21.15.111:0/4250415657 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24f0003f80 0x7f24f0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f2507fff700 1 -- 172.21.15.111:0/4250415657 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f00035b0 con 0x7f24f0004d90 2023-11-28T12:59:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f25077fe700 1 --2- 172.21.15.111:0/4250415657 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f25080cdef0 0x7f24f0003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f2507fff700 1 --2- 172.21.15.111:0/4250415657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24f0004d90 0x7f24f0005200 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2500009fc0 tx=0x7f24f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ffffd0b5e7cbc5c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250000ec30 con 0x7f24f0004d90 2023-11-28T12:59:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250000f040 con 0x7f24f0004d90 2023-11-28T12:59:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f25067fc700 1 -- 172.21.15.111:0/4250415657 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2500014480 con 0x7f24f0004d90 2023-11-28T12:59:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f250def0700 1 -- 172.21.15.111:0/4250415657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24f0004d90 msgr2=0x7f24f0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f250def0700 1 --2- 172.21.15.111:0/4250415657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24f0004d90 0x7f24f0005200 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2500009fc0 tx=0x7f24f0005880 comp rx=0 tx=0).stop 2023-11-28T12:59:44.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f2507fff700 1 -- 172.21.15.111:0/4250415657 reap_dead start 2023-11-28T12:59:44.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f250def0700 1 -- 172.21.15.111:0/4250415657 shutdown_connections 2023-11-28T12:59:44.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f250def0700 1 -- 172.21.15.111:0/4250415657 >> 172.21.15.111:0/4250415657 conn(0x7f2508098ee0 msgr2=0x7f250806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:44.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f250def0700 1 -- 172.21.15.111:0/4250415657 shutdown_connections 2023-11-28T12:59:44.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.036+0000 7f250def0700 1 -- 172.21.15.111:0/4250415657 wait complete. 2023-11-28T12:59:44.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.038+0000 7f250def0700 1 Processor -- start 2023-11-28T12:59:44.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.038+0000 7f250def0700 1 -- start start 2023-11-28T12:59:44.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f250def0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24f0004d90 0x7f2508157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f250def0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f250815b940 0x7f250815bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f250def0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f25080c4c00 0x7f25080c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f250def0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25080b3910 con 0x7f24f0004d90 2023-11-28T12:59:44.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f250def0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250815c2f0 con 0x7f25080c4c00 2023-11-28T12:59:44.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f250def0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25080b2a40 con 0x7f250815b940 2023-11-28T12:59:44.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f25077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24f0004d90 0x7f2508157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f25077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24f0004d90 0x7f2508157be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34222/0 (socket says 172.21.15.111:34222) 2023-11-28T12:59:44.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f25077fe700 1 -- 172.21.15.111:0/3377735391 learned_addr learned my addr 172.21.15.111:0/3377735391 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:44.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f2507fff700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f25080c4c00 0x7f25080c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f2506ffd700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f250815b940 0x7f250815bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f25077fe700 1 -- 172.21.15.111:0/3377735391 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f250815b940 msgr2=0x7f250815bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f25077fe700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f250815b940 0x7f250815bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f25077fe700 1 -- 172.21.15.111:0/3377735391 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f25080c4c00 msgr2=0x7f25080c5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f25077fe700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f25080c4c00 0x7f25080c5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f25077fe700 1 -- 172.21.15.111:0/3377735391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25080cd340 con 0x7f24f0004d90 2023-11-28T12:59:44.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f2507fff700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f25080c4c00 0x7f25080c5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:44.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f25077fe700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24f0004d90 0x7f2508157be0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f24f8007360 tx=0x7f24f8007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:44.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f24effff700 1 -- 172.21.15.111:0/3377735391 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f800e860 con 0x7f24f0004d90 2023-11-28T12:59:44.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f24effff700 1 -- 172.21.15.111:0/3377735391 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f800e9c0 con 0x7f24f0004d90 2023-11-28T12:59:44.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f250def0700 1 -- 172.21.15.111:0/3377735391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25080cd060 con 0x7f24f0004d90 2023-11-28T12:59:44.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f24effff700 1 -- 172.21.15.111:0/3377735391 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f800ed50 con 0x7f24f0004d90 2023-11-28T12:59:44.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.039+0000 7f250def0700 1 -- 172.21.15.111:0/3377735391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2508143280 con 0x7f24f0004d90 2023-11-28T12:59:44.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.040+0000 7f250def0700 1 -- 172.21.15.111:0/3377735391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250806c4a0 con 0x7f24f0004d90 2023-11-28T12:59:44.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.041+0000 7f24effff700 1 -- 172.21.15.111:0/3377735391 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f24f801d460 con 0x7f24f0004d90 2023-11-28T12:59:44.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.041+0000 7f24effff700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24f4068320 0x7f24f406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.041+0000 7f24effff700 1 -- 172.21.15.111:0/3377735391 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f24f8086100 con 0x7f24f0004d90 2023-11-28T12:59:44.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.041+0000 7f2506ffd700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24f4068320 0x7f24f406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.041+0000 7f2506ffd700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24f4068320 0x7f24f406a7d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f25080b1f60 tx=0x7f24fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:44.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.045+0000 7f24effff700 1 -- 172.21.15.111:0/3377735391 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f24f80525d0 con 0x7f24f0004d90 2023-11-28T12:59:44.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.228+0000 7f250def0700 1 -- 172.21.15.111:0/3377735391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f25080d05a0 con 0x7f24f0004d90 2023-11-28T12:59:44.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.232+0000 7f24effff700 1 -- 172.21.15.111:0/3377735391 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f25080d05a0 con 0x7f24f0004d90 2023-11-28T12:59:44.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.234+0000 7f250def0700 1 -- 172.21.15.111:0/3377735391 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24f4068320 msgr2=0x7f24f406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.234+0000 7f250def0700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24f4068320 0x7f24f406a7d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f25080b1f60 tx=0x7f24fc008040 comp rx=0 tx=0).stop 2023-11-28T12:59:44.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.234+0000 7f250def0700 1 -- 172.21.15.111:0/3377735391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24f0004d90 msgr2=0x7f2508157be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.234+0000 7f250def0700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24f0004d90 0x7f2508157be0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f24f8007360 tx=0x7f24f8007720 comp rx=0 tx=0).stop 2023-11-28T12:59:44.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.235+0000 7f250def0700 1 -- 172.21.15.111:0/3377735391 shutdown_connections 2023-11-28T12:59:44.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.235+0000 7f250def0700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24f4068320 0x7f24f406a7d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.235+0000 7f250def0700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f25080c4c00 0x7f25080c5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.235+0000 7f250def0700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24f0004d90 0x7f2508157be0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.235+0000 7f250def0700 1 --2- 172.21.15.111:0/3377735391 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f250815b940 0x7f250815bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.235+0000 7f250def0700 1 -- 172.21.15.111:0/3377735391 >> 172.21.15.111:0/3377735391 conn(0x7f2508098ee0 msgr2=0x7f250806ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:44.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.235+0000 7f250def0700 1 -- 172.21.15.111:0/3377735391 shutdown_connections 2023-11-28T12:59:44.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.235+0000 7f250def0700 1 -- 172.21.15.111:0/3377735391 wait complete. 2023-11-28T12:59:44.248 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T12:59:44.248 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T12:59:44.289 DEBUG:teuthology.orchestra.run.smithi149:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T12:59:44.340 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T12:59:44.341 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:59:44.341 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:44.359 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:59:44.359 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:44.416 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:59:44.417 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:44.474 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:44.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.634+0000 7f1be4bcb700 1 Processor -- start 2023-11-28T12:59:44.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.634+0000 7f1be4bcb700 1 -- start start 2023-11-28T12:59:44.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.634+0000 7f1be4bcb700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f1be00cf010 0x7f1be00cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.634+0000 7f1be4bcb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1be00cb380 0x7f1be00b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.634+0000 7f1be4bcb700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be00cc7b0 con 0x7f1be00b2220 2023-11-28T12:59:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.634+0000 7f1be4bcb700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be008ae90 con 0x7f1be00cb380 2023-11-28T12:59:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.634+0000 7f1be4bcb700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be00b25f0 con 0x7f1be00cf010 2023-11-28T12:59:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.634+0000 7f1bded9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1be00cb380 0x7f1be00b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.635+0000 7f1bded9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1be00cb380 0x7f1be00b07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34234/0 (socket says 172.21.15.111:34234) 2023-11-28T12:59:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.635+0000 7f1bded9d700 1 -- 172.21.15.111:0/1273326659 learned_addr learned my addr 172.21.15.111:0/1273326659 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.635+0000 7f1bd7fff700 1 --2- 172.21.15.111:0/1273326659 >> v2:172.21.15.149:3300/0 conn(0x7f1be00cf010 0x7f1be00cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.635+0000 7f1bded9d700 1 -- 172.21.15.111:0/1273326659 >> v2:172.21.15.149:3300/0 conn(0x7f1be00cf010 msgr2=0x7f1be00cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.635+0000 7f1bded9d700 1 --2- 172.21.15.111:0/1273326659 >> v2:172.21.15.149:3300/0 conn(0x7f1be00cf010 0x7f1be00cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.635+0000 7f1bded9d700 1 -- 172.21.15.111:0/1273326659 >> v1:172.21.15.149:6789/0 conn(0x7f1be00b2220 legacy=0x7f1be00ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.635+0000 7f1bded9d700 1 -- 172.21.15.111:0/1273326659 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be00cca10 con 0x7f1be00cb380 2023-11-28T12:59:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.635+0000 7f1bded9d700 1 --2- 172.21.15.111:0/1273326659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be00cb380 0x7f1be00b07b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1be008a070 tx=0x7f1bd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b7f13d9fa18b79 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.635+0000 7f1bddd9b700 1 -- 172.21.15.111:0/1273326659 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd800e070 con 0x7f1be00cb380 2023-11-28T12:59:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.635+0000 7f1bddd9b700 1 -- 172.21.15.111:0/1273326659 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd8003a20 con 0x7f1be00cb380 2023-11-28T12:59:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.635+0000 7f1bddd9b700 1 -- 172.21.15.111:0/1273326659 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8003dd0 con 0x7f1be00cb380 2023-11-28T12:59:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.636+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1273326659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be00cb380 msgr2=0x7f1be00b07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.636+0000 7f1be4bcb700 1 --2- 172.21.15.111:0/1273326659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be00cb380 0x7f1be00b07b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1be008a070 tx=0x7f1bd800b330 comp rx=0 tx=0).stop 2023-11-28T12:59:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.636+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1273326659 shutdown_connections 2023-11-28T12:59:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.636+0000 7f1be4bcb700 1 --2- 172.21.15.111:0/1273326659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be00cb380 0x7f1be00b07b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.636+0000 7f1be4bcb700 1 --2- 172.21.15.111:0/1273326659 >> v2:172.21.15.149:3300/0 conn(0x7f1be00cf010 0x7f1be00cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.636+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1273326659 >> 172.21.15.111:0/1273326659 conn(0x7f1be009a040 msgr2=0x7f1be00d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.636+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1273326659 shutdown_connections 2023-11-28T12:59:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.636+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1273326659 wait complete. 2023-11-28T12:59:44.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.637+0000 7f1be4bcb700 1 Processor -- start 2023-11-28T12:59:44.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.637+0000 7f1be4bcb700 1 -- start start 2023-11-28T12:59:44.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1be4bcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be00b2220 0x7f1be01f16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1be4bcb700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1be00cb380 0x7f1be00ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1be4bcb700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1be00cf010 0x7f1be00ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1be4bcb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be006b6d0 con 0x7f1be00b2220 2023-11-28T12:59:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1be4bcb700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be006b380 con 0x7f1be00cf010 2023-11-28T12:59:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1be4bcb700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be00caaa0 con 0x7f1be00cb380 2023-11-28T12:59:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bde59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be00b2220 0x7f1be01f16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bde59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be00b2220 0x7f1be01f16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34236/0 (socket says 172.21.15.111:34236) 2023-11-28T12:59:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bde59c700 1 -- 172.21.15.111:0/1310718833 learned_addr learned my addr 172.21.15.111:0/1310718833 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bd7fff700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1be00cb380 0x7f1be00ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bded9d700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1be00cf010 0x7f1be00ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bde59c700 1 -- 172.21.15.111:0/1310718833 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1be00cb380 msgr2=0x7f1be00ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bde59c700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1be00cb380 0x7f1be00ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bde59c700 1 -- 172.21.15.111:0/1310718833 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1be00cf010 msgr2=0x7f1be00ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bde59c700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1be00cf010 0x7f1be00ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bde59c700 1 -- 172.21.15.111:0/1310718833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd8009b30 con 0x7f1be00b2220 2023-11-28T12:59:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bded9d700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1be00cf010 0x7f1be00ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bde59c700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be00b2220 0x7f1be01f16b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1bd000c730 tx=0x7f1bd000ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bd6ffd700 1 -- 172.21.15.111:0/1310718833 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0011790 con 0x7f1be00b2220 2023-11-28T12:59:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bd6ffd700 1 -- 172.21.15.111:0/1310718833 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd0011910 con 0x7f1be00b2220 2023-11-28T12:59:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1bd6ffd700 1 -- 172.21.15.111:0/1310718833 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0011cc0 con 0x7f1be00b2220 2023-11-28T12:59:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.638+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1310718833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be00ccfd0 con 0x7f1be00b2220 2023-11-28T12:59:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.639+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1310718833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be00cccf0 con 0x7f1be00b2220 2023-11-28T12:59:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.639+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1310718833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be00cd2b0 con 0x7f1be00b2220 2023-11-28T12:59:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.640+0000 7f1bd6ffd700 1 -- 172.21.15.111:0/1310718833 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1bd0010050 con 0x7f1be00b2220 2023-11-28T12:59:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.640+0000 7f1bd6ffd700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1bc00682f0 0x7f1bc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.640+0000 7f1bd6ffd700 1 -- 172.21.15.111:0/1310718833 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1bd0085c30 con 0x7f1be00b2220 2023-11-28T12:59:44.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.640+0000 7f1bd7fff700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1bc00682f0 0x7f1bc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.641+0000 7f1bd7fff700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1bc00682f0 0x7f1bc006a7a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1be00b17b0 tx=0x7f1bc8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:44.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.645+0000 7f1bd6ffd700 1 -- 172.21.15.111:0/1310718833 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1bd0052160 con 0x7f1be00b2220 2023-11-28T12:59:44.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.830+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1310718833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1be00c3b70 con 0x7f1be00b2220 2023-11-28T12:59:44.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.831+0000 7f1bd6ffd700 1 -- 172.21.15.111:0/1310718833 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+19890 (secure 0 0 0) 0x7f1bd0055780 con 0x7f1be00b2220 2023-11-28T12:59:44.832 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:44.833 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":28,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:41.880678+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.833+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1310718833 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1bc00682f0 msgr2=0x7f1bc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.833+0000 7f1be4bcb700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1bc00682f0 0x7f1bc006a7a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1be00b17b0 tx=0x7f1bc8008040 comp rx=0 tx=0).stop 2023-11-28T12:59:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.833+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1310718833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be00b2220 msgr2=0x7f1be01f16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.833+0000 7f1be4bcb700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be00b2220 0x7f1be01f16b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1bd000c730 tx=0x7f1bd000ca40 comp rx=0 tx=0).stop 2023-11-28T12:59:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.833+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1310718833 shutdown_connections 2023-11-28T12:59:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.833+0000 7f1be4bcb700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1bc00682f0 0x7f1bc006a7a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.833+0000 7f1be4bcb700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1be00cf010 0x7f1be00ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.833+0000 7f1be4bcb700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be00b2220 0x7f1be01f16b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.833+0000 7f1be4bcb700 1 --2- 172.21.15.111:0/1310718833 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1be00cb380 0x7f1be00ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.833+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1310718833 >> 172.21.15.111:0/1310718833 conn(0x7f1be009a040 msgr2=0x7f1be00cde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.833+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1310718833 shutdown_connections 2023-11-28T12:59:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.833+0000 7f1be4bcb700 1 -- 172.21.15.111:0/1310718833 wait complete. 2023-11-28T12:59:44.847 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T12:58:53.275206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T12:58:54.588499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T12:58:57.666033+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T12:59:44.847 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:59:44.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.970+0000 7f0f1dc27700 1 Processor -- start 2023-11-28T12:59:44.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.970+0000 7f0f1dc27700 1 -- start start 2023-11-28T12:59:44.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.970+0000 7f0f1dc27700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f180cdd90 0x7f0f180b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.970+0000 7f0f1dc27700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f180ad420 con 0x7f0f18159960 2023-11-28T12:59:44.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.970+0000 7f0f1dc27700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f180ac810 con 0x7f0f180aecc0 2023-11-28T12:59:44.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.970+0000 7f0f1dc27700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f18159d30 con 0x7f0f180cdd90 2023-11-28T12:59:44.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f177fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0f18159960 0x7f0f180cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35022/0 (socket says 172.21.15.111:35022) 2023-11-28T12:59:44.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f177fe700 1 -- 172.21.15.111:0/1388196314 learned_addr learned my addr 172.21.15.111:0/1388196314 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:44.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f16ffd700 1 --2- 172.21.15.111:0/1388196314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f180cdd90 0x7f0f180b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332746853 0 0) 0x7f0f180ad420 con 0x7f0f18159960 2023-11-28T12:59:44.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f000035b0 con 0x7f0f18159960 2023-11-28T12:59:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021647993 0 0) 0x7f0f180ac810 con 0x7f0f180aecc0 2023-11-28T12:59:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f180ad420 con 0x7f0f180aecc0 2023-11-28T12:59:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3168119994 0 0) 0x7f0f000035b0 con 0x7f0f18159960 2023-11-28T12:59:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f180cdd90 msgr2=0x7f0f180b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 --2- 172.21.15.111:0/1388196314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f180cdd90 0x7f0f180b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 >> v1:172.21.15.149:6789/0 conn(0x7f0f180aecc0 legacy=0x7f0f180ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f180acaf0 con 0x7f0f18159960 2023-11-28T12:59:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f16ffd700 1 --2- 172.21.15.111:0/1388196314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f180cdd90 0x7f0f180b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f0f0c0017f0 con 0x7f0f18159960 2023-11-28T12:59:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 >> v1:172.21.15.111:6789/0 conn(0x7f0f18159960 legacy=0x7f0f180cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 --2- 172.21.15.111:0/1388196314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f180cdd90 0x7f0f00003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 --2- 172.21.15.111:0/1388196314 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f00004150 0x7f0f00004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 --2- 172.21.15.111:0/1388196314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f00004d90 0x7f0f00005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f18159d30 con 0x7f0f00004d90 2023-11-28T12:59:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0000bba0 con 0x7f0f00004150 2023-11-28T12:59:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0000bd60 con 0x7f0f180cdd90 2023-11-28T12:59:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f177fe700 1 --2- 172.21.15.111:0/1388196314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f180cdd90 0x7f0f00003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f17fff700 1 --2- 172.21.15.111:0/1388196314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f00004d90 0x7f0f00005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.971+0000 7f0f16ffd700 1 --2- 172.21.15.111:0/1388196314 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f00004150 0x7f0f00004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f17fff700 1 -- 172.21.15.111:0/1388196314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f180cdd90 msgr2=0x7f0f00003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f17fff700 1 --2- 172.21.15.111:0/1388196314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f180cdd90 0x7f0f00003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f17fff700 1 -- 172.21.15.111:0/1388196314 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f00004150 msgr2=0x7f0f00004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f17fff700 1 --2- 172.21.15.111:0/1388196314 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f00004150 0x7f0f00004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f17fff700 1 -- 172.21.15.111:0/1388196314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f080094a0 con 0x7f0f00004d90 2023-11-28T12:59:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f17fff700 1 --2- 172.21.15.111:0/1388196314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f00004d90 0x7f0f00005200 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0f10002050 tx=0x7f0f00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04e6ef9f97978d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:44.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1000ce60 con 0x7f0f00004d90 2023-11-28T12:59:44.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f10003a00 con 0x7f0f00004d90 2023-11-28T12:59:44.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f167fc700 1 -- 172.21.15.111:0/1388196314 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10003db0 con 0x7f0f00004d90 2023-11-28T12:59:44.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1388196314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f00004d90 msgr2=0x7f0f00005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f1dc27700 1 --2- 172.21.15.111:0/1388196314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f00004d90 0x7f0f00005200 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0f10002050 tx=0x7f0f00005880 comp rx=0 tx=0).stop 2023-11-28T12:59:44.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f17fff700 1 -- 172.21.15.111:0/1388196314 reap_dead start 2023-11-28T12:59:44.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1388196314 shutdown_connections 2023-11-28T12:59:44.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1388196314 >> 172.21.15.111:0/1388196314 conn(0x7f0f18098ee0 msgr2=0x7f0f1806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:44.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1388196314 shutdown_connections 2023-11-28T12:59:44.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.972+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1388196314 wait complete. 2023-11-28T12:59:44.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.974+0000 7f0f1dc27700 1 Processor -- start 2023-11-28T12:59:44.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.974+0000 7f0f1dc27700 1 -- start start 2023-11-28T12:59:44.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f1dc27700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f181ed5a0 0x7f0f1815c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f1dc27700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f181ed970 0x7f0f18142cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f1dc27700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f18143230 0x7f0f181436a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f1dc27700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1815c9d0 con 0x7f0f181ed5a0 2023-11-28T12:59:44.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f1dc27700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1815b3f0 con 0x7f0f181ed970 2023-11-28T12:59:44.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f1dc27700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1815b6d0 con 0x7f0f18143230 2023-11-28T12:59:44.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f181ed5a0 0x7f0f1815c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f181ed5a0 0x7f0f1815c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34256/0 (socket says 172.21.15.111:34256) 2023-11-28T12:59:44.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f177fe700 1 -- 172.21.15.111:0/1282551333 learned_addr learned my addr 172.21.15.111:0/1282551333 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:44.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f16ffd700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f181ed970 0x7f0f18142cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f17fff700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f18143230 0x7f0f181436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f177fe700 1 -- 172.21.15.111:0/1282551333 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f18143230 msgr2=0x7f0f181436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f177fe700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f18143230 0x7f0f181436a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f177fe700 1 -- 172.21.15.111:0/1282551333 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f181ed970 msgr2=0x7f0f18142cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:44.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f177fe700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f181ed970 0x7f0f18142cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:44.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f177fe700 1 -- 172.21.15.111:0/1282551333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f180ad0b0 con 0x7f0f181ed5a0 2023-11-28T12:59:44.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.975+0000 7f0f16ffd700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f181ed970 0x7f0f18142cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:44.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.976+0000 7f0f177fe700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f181ed5a0 0x7f0f1815c490 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00f800 tx=0x7f0f0c00fbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:44.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.976+0000 7f0efffff700 1 -- 172.21.15.111:0/1282551333 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c0072e0 con 0x7f0f181ed5a0 2023-11-28T12:59:44.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.976+0000 7f0efffff700 1 -- 172.21.15.111:0/1282551333 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0c007460 con 0x7f0f181ed5a0 2023-11-28T12:59:44.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.976+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1282551333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f180acdd0 con 0x7f0f181ed5a0 2023-11-28T12:59:44.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.976+0000 7f0efffff700 1 -- 172.21.15.111:0/1282551333 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c005830 con 0x7f0f181ed5a0 2023-11-28T12:59:44.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.976+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1282551333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f180b1fd0 con 0x7f0f181ed5a0 2023-11-28T12:59:44.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.977+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1282551333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1806c4f0 con 0x7f0f181ed5a0 2023-11-28T12:59:44.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.977+0000 7f0efffff700 1 -- 172.21.15.111:0/1282551333 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0f0c005990 con 0x7f0f181ed5a0 2023-11-28T12:59:44.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.978+0000 7f0efffff700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0f040682f0 0x7f0f0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:44.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.978+0000 7f0f16ffd700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0f040682f0 0x7f0f0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:44.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.978+0000 7f0efffff700 1 -- 172.21.15.111:0/1282551333 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f0f0c086100 con 0x7f0f181ed5a0 2023-11-28T12:59:44.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.978+0000 7f0f16ffd700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0f040682f0 0x7f0f0406a7a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0f181546e0 tx=0x7f0f08008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:44.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:44.982+0000 7f0efffff700 1 -- 172.21.15.111:0/1282551333 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0f0c0525d0 con 0x7f0f181ed5a0 2023-11-28T12:59:45.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.164+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1282551333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0f18067170 con 0x7f0f181ed5a0 2023-11-28T12:59:45.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.165+0000 7f0efffff700 1 -- 172.21.15.111:0/1282551333 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0f18067170 con 0x7f0f181ed5a0 2023-11-28T12:59:45.166 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:59:45.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.167+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1282551333 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0f040682f0 msgr2=0x7f0f0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.167+0000 7f0f1dc27700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0f040682f0 0x7f0f0406a7a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0f181546e0 tx=0x7f0f08008040 comp rx=0 tx=0).stop 2023-11-28T12:59:45.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.167+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1282551333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f181ed5a0 msgr2=0x7f0f1815c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.167+0000 7f0f1dc27700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f181ed5a0 0x7f0f1815c490 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00f800 tx=0x7f0f0c00fbc0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.167+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1282551333 shutdown_connections 2023-11-28T12:59:45.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.167+0000 7f0f1dc27700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0f18143230 0x7f0f181436a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.167+0000 7f0f1dc27700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f181ed5a0 0x7f0f1815c490 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.167+0000 7f0f1dc27700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0f040682f0 0x7f0f0406a7a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.167+0000 7f0f1dc27700 1 --2- 172.21.15.111:0/1282551333 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0f181ed970 0x7f0f18142cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.167+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1282551333 >> 172.21.15.111:0/1282551333 conn(0x7f0f18098ee0 msgr2=0x7f0f180cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:45.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.167+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1282551333 shutdown_connections 2023-11-28T12:59:45.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.167+0000 7f0f1dc27700 1 -- 172.21.15.111:0/1282551333 wait complete. 2023-11-28T12:59:45.180 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:59:45.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.301+0000 7f487b9bd700 1 Processor -- start 2023-11-28T12:59:45.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.302+0000 7f487b9bd700 1 -- start start 2023-11-28T12:59:45.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.302+0000 7f487b9bd700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f487415a5d0 0x7f48740ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f487b9bd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48740ae8b0 con 0x7f48740b0160 2023-11-28T12:59:45.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f487b9bd700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f487415ef40 con 0x7f48740cb2d0 2023-11-28T12:59:45.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f487b9bd700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48740cc1b0 con 0x7f487415a5d0 2023-11-28T12:59:45.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f4878f58700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f48740b0160 0x7f487415e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35044/0 (socket says 172.21.15.111:35044) 2023-11-28T12:59:45.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f4878f58700 1 -- 172.21.15.111:0/2424952355 learned_addr learned my addr 172.21.15.111:0/2424952355 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:45.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f4879759700 1 --2- 172.21.15.111:0/2424952355 >> v2:172.21.15.149:3300/0 conn(0x7f487415a5d0 0x7f48740ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f486bfff700 1 -- 172.21.15.111:0/2424952355 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027260265 0 0) 0x7f48740ae8b0 con 0x7f48740b0160 2023-11-28T12:59:45.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f486bfff700 1 -- 172.21.15.111:0/2424952355 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48600035b0 con 0x7f48740b0160 2023-11-28T12:59:45.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f486bfff700 1 -- 172.21.15.111:0/2424952355 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 443510800 0 0) 0x7f48600035b0 con 0x7f48740b0160 2023-11-28T12:59:45.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f4879759700 1 -- 172.21.15.111:0/2424952355 >> v1:172.21.15.149:6789/0 conn(0x7f48740cb2d0 legacy=0x7f487415e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f4879759700 1 -- 172.21.15.111:0/2424952355 >> v1:172.21.15.111:6789/0 conn(0x7f48740b0160 legacy=0x7f487415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f4879759700 1 -- 172.21.15.111:0/2424952355 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48740cc390 con 0x7f487415a5d0 2023-11-28T12:59:45.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f4879759700 1 --2- 172.21.15.111:0/2424952355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f487415a5d0 0x7f48740ce850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4864009820 tx=0x7f4864009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f3a05aaf7adef2d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:45.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.303+0000 7f486bfff700 1 -- 172.21.15.111:0/2424952355 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4870001670 con 0x7f487415a5d0 2023-11-28T12:59:45.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.304+0000 7f486bfff700 1 -- 172.21.15.111:0/2424952355 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4864005c30 con 0x7f487415a5d0 2023-11-28T12:59:45.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.304+0000 7f486bfff700 1 -- 172.21.15.111:0/2424952355 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4864004030 con 0x7f487415a5d0 2023-11-28T12:59:45.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.304+0000 7f487b9bd700 1 -- 172.21.15.111:0/2424952355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f487415a5d0 msgr2=0x7f48740ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.304+0000 7f487b9bd700 1 --2- 172.21.15.111:0/2424952355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f487415a5d0 0x7f48740ce850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4864009820 tx=0x7f4864009b30 comp rx=0 tx=0).stop 2023-11-28T12:59:45.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.304+0000 7f487b9bd700 1 -- 172.21.15.111:0/2424952355 shutdown_connections 2023-11-28T12:59:45.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.304+0000 7f487b9bd700 1 --2- 172.21.15.111:0/2424952355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f487415a5d0 0x7f48740ce850 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.304+0000 7f487b9bd700 1 -- 172.21.15.111:0/2424952355 >> 172.21.15.111:0/2424952355 conn(0x7f487409a040 msgr2=0x7f48740d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:45.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.304+0000 7f487b9bd700 1 -- 172.21.15.111:0/2424952355 shutdown_connections 2023-11-28T12:59:45.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.304+0000 7f487b9bd700 1 -- 172.21.15.111:0/2424952355 wait complete. 2023-11-28T12:59:45.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.305+0000 7f487b9bd700 1 Processor -- start 2023-11-28T12:59:45.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.305+0000 7f487b9bd700 1 -- start start 2023-11-28T12:59:45.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.306+0000 7f487b9bd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48740b0160 0x7f48741ef870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.306+0000 7f487b9bd700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f48740cb2d0 0x7f48741f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.306+0000 7f487b9bd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f487415a5d0 0x7f48740c48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.306+0000 7f487b9bd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487406b890 con 0x7f48740b0160 2023-11-28T12:59:45.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.306+0000 7f487b9bd700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48741f31c0 con 0x7f48740cb2d0 2023-11-28T12:59:45.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.306+0000 7f487b9bd700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48741f2c00 con 0x7f487415a5d0 2023-11-28T12:59:45.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.306+0000 7f4879759700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48740b0160 0x7f48741ef870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.306+0000 7f4879759700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48740b0160 0x7f48741ef870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34272/0 (socket says 172.21.15.111:34272) 2023-11-28T12:59:45.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.306+0000 7f4879759700 1 -- 172.21.15.111:0/372158403 learned_addr learned my addr 172.21.15.111:0/372158403 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:45.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.306+0000 7f4878f58700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f48740cb2d0 0x7f48741f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.306+0000 7f4879f5a700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f487415a5d0 0x7f48740c48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.306+0000 7f4879759700 1 -- 172.21.15.111:0/372158403 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f487415a5d0 msgr2=0x7f48740c48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.307+0000 7f4879759700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f487415a5d0 0x7f48740c48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.307+0000 7f4879759700 1 -- 172.21.15.111:0/372158403 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f48740cb2d0 msgr2=0x7f48741f21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.307+0000 7f4879759700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f48740cb2d0 0x7f48741f21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.307+0000 7f4879759700 1 -- 172.21.15.111:0/372158403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4864009430 con 0x7f48740b0160 2023-11-28T12:59:45.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.307+0000 7f4878f58700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f48740cb2d0 0x7f48741f21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:45.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.307+0000 7f4879759700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48740b0160 0x7f48741ef870 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f487406d8f0 tx=0x7f4864004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:45.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.307+0000 7f4869ffb700 1 -- 172.21.15.111:0/372158403 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4864003c30 con 0x7f48740b0160 2023-11-28T12:59:45.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.307+0000 7f4869ffb700 1 -- 172.21.15.111:0/372158403 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4864003d90 con 0x7f48740b0160 2023-11-28T12:59:45.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.307+0000 7f487b9bd700 1 -- 172.21.15.111:0/372158403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48740cc950 con 0x7f48740b0160 2023-11-28T12:59:45.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.307+0000 7f4869ffb700 1 -- 172.21.15.111:0/372158403 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486401d650 con 0x7f48740b0160 2023-11-28T12:59:45.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.307+0000 7f487b9bd700 1 -- 172.21.15.111:0/372158403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48740cc670 con 0x7f48740b0160 2023-11-28T12:59:45.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.308+0000 7f487b9bd700 1 -- 172.21.15.111:0/372158403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48740ccc30 con 0x7f48740b0160 2023-11-28T12:59:45.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.308+0000 7f4869ffb700 1 -- 172.21.15.111:0/372158403 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f486402d070 con 0x7f48740b0160 2023-11-28T12:59:45.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.309+0000 7f4869ffb700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f48580682f0 0x7f485806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.309+0000 7f4869ffb700 1 -- 172.21.15.111:0/372158403 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4864087240 con 0x7f48740b0160 2023-11-28T12:59:45.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.309+0000 7f4878f58700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f48580682f0 0x7f485806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.309+0000 7f4878f58700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f48580682f0 0x7f485806a7a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f48740c4160 tx=0x7f4870001ca0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:45.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.313+0000 7f4869ffb700 1 -- 172.21.15.111:0/372158403 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f48640536f0 con 0x7f48740b0160 2023-11-28T12:59:45.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.494+0000 7f487b9bd700 1 -- 172.21.15.111:0/372158403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4874067170 con 0x7f48740b0160 2023-11-28T12:59:45.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.494+0000 7f4869ffb700 1 -- 172.21.15.111:0/372158403 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4864056d10 con 0x7f48740b0160 2023-11-28T12:59:45.496 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:59:45.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.496+0000 7f487b9bd700 1 -- 172.21.15.111:0/372158403 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f48580682f0 msgr2=0x7f485806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.496+0000 7f487b9bd700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f48580682f0 0x7f485806a7a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f48740c4160 tx=0x7f4870001ca0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.496+0000 7f487b9bd700 1 -- 172.21.15.111:0/372158403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48740b0160 msgr2=0x7f48741ef870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.496+0000 7f487b9bd700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48740b0160 0x7f48741ef870 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f487406d8f0 tx=0x7f4864004750 comp rx=0 tx=0).stop 2023-11-28T12:59:45.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.496+0000 7f487b9bd700 1 -- 172.21.15.111:0/372158403 shutdown_connections 2023-11-28T12:59:45.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.496+0000 7f487b9bd700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f487415a5d0 0x7f48740c48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.496+0000 7f487b9bd700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48740b0160 0x7f48741ef870 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.496+0000 7f487b9bd700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f48580682f0 0x7f485806a7a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.496+0000 7f487b9bd700 1 --2- 172.21.15.111:0/372158403 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f48740cb2d0 0x7f48741f21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.497+0000 7f487b9bd700 1 -- 172.21.15.111:0/372158403 >> 172.21.15.111:0/372158403 conn(0x7f487409a040 msgr2=0x7f48740ada60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:45.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.497+0000 7f487b9bd700 1 -- 172.21.15.111:0/372158403 shutdown_connections 2023-11-28T12:59:45.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.497+0000 7f487b9bd700 1 -- 172.21.15.111:0/372158403 wait complete. 2023-11-28T12:59:45.510 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T12:59:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.632+0000 7fb98f935700 1 Processor -- start 2023-11-28T12:59:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.632+0000 7fb98f935700 1 -- start start 2023-11-28T12:59:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.633+0000 7fb98f935700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb9880cfc90 0x7fb9880d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.633+0000 7fb98f935700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9880cc5a0 0x7fb9880cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.633+0000 7fb98f935700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9880d05a0 con 0x7fb9880cdd20 2023-11-28T12:59:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.633+0000 7fb98f935700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9880ce1c0 con 0x7fb9880cfc90 2023-11-28T12:59:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.633+0000 7fb98f935700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9880ca260 con 0x7fb9880cc5a0 2023-11-28T12:59:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb98d6d1700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb9880cfc90 0x7fb9880d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb98ced0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9880cc5a0 0x7fb9880cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb98d6d1700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb9880cfc90 0x7fb9880d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:52178/0 (socket says 172.21.15.111:52178) 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb98d6d1700 1 -- 172.21.15.111:0/3321845065 learned_addr learned my addr 172.21.15.111:0/3321845065 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb98ded2700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7fb9880cdd20 0x7fb98815d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:43602/0 (socket says 172.21.15.111:43602) 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb98ced0700 1 -- 172.21.15.111:0/3321845065 >> v2:172.21.15.149:3300/0 conn(0x7fb9880cfc90 msgr2=0x7fb9880d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb98ced0700 1 --2- 172.21.15.111:0/3321845065 >> v2:172.21.15.149:3300/0 conn(0x7fb9880cfc90 0x7fb9880d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb98ced0700 1 -- 172.21.15.111:0/3321845065 >> v1:172.21.15.149:6789/0 conn(0x7fb9880cdd20 legacy=0x7fb98815d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb98ced0700 1 -- 172.21.15.111:0/3321845065 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9880ccaa0 con 0x7fb9880cc5a0 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb98d6d1700 1 --2- 172.21.15.111:0/3321845065 >> v2:172.21.15.149:3300/0 conn(0x7fb9880cfc90 0x7fb9880d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb98ced0700 1 --2- 172.21.15.111:0/3321845065 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9880cc5a0 0x7fb9880cd7e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb984009840 tx=0x7fb984009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c5b85d727f36c8a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb97ffff700 1 -- 172.21.15.111:0/3321845065 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984011830 con 0x7fb9880cc5a0 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb97ffff700 1 -- 172.21.15.111:0/3321845065 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb984011990 con 0x7fb9880cc5a0 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.634+0000 7fb97ffff700 1 -- 172.21.15.111:0/3321845065 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984011d20 con 0x7fb9880cc5a0 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.635+0000 7fb98f935700 1 -- 172.21.15.111:0/3321845065 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9880cc5a0 msgr2=0x7fb9880cd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.635+0000 7fb98f935700 1 --2- 172.21.15.111:0/3321845065 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9880cc5a0 0x7fb9880cd7e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb984009840 tx=0x7fb984009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:45.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.635+0000 7fb98f935700 1 -- 172.21.15.111:0/3321845065 shutdown_connections 2023-11-28T12:59:45.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.635+0000 7fb98f935700 1 --2- 172.21.15.111:0/3321845065 >> v2:172.21.15.149:3300/0 conn(0x7fb9880cfc90 0x7fb9880d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.635+0000 7fb98f935700 1 --2- 172.21.15.111:0/3321845065 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9880cc5a0 0x7fb9880cd7e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.635+0000 7fb98f935700 1 -- 172.21.15.111:0/3321845065 >> 172.21.15.111:0/3321845065 conn(0x7fb988098ee0 msgr2=0x7fb98806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:45.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.635+0000 7fb98f935700 1 -- 172.21.15.111:0/3321845065 shutdown_connections 2023-11-28T12:59:45.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.635+0000 7fb98f935700 1 -- 172.21.15.111:0/3321845065 wait complete. 2023-11-28T12:59:45.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.636+0000 7fb98f935700 1 Processor -- start 2023-11-28T12:59:45.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.636+0000 7fb98f935700 1 -- start start 2023-11-28T12:59:45.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98f935700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9880cc5a0 0x7fb988158390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98f935700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9880cdd20 0x7fb9880c3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98f935700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9880cfc90 0x7fb98815b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98f935700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9881588d0 con 0x7fb9880cdd20 2023-11-28T12:59:45.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98f935700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9880c3bb0 con 0x7fb9880cfc90 2023-11-28T12:59:45.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98f935700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9880c3e90 con 0x7fb9880cc5a0 2023-11-28T12:59:45.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98ced0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9880cdd20 0x7fb9880c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98ced0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9880cdd20 0x7fb9880c3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34286/0 (socket says 172.21.15.111:34286) 2023-11-28T12:59:45.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98ced0700 1 -- 172.21.15.111:0/2908745412 learned_addr learned my addr 172.21.15.111:0/2908745412 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:45.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98d6d1700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9880cc5a0 0x7fb988158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98ded2700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9880cfc90 0x7fb98815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98ced0700 1 -- 172.21.15.111:0/2908745412 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9880cc5a0 msgr2=0x7fb988158390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98ced0700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9880cc5a0 0x7fb988158390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98ced0700 1 -- 172.21.15.111:0/2908745412 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9880cfc90 msgr2=0x7fb98815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98ced0700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9880cfc90 0x7fb98815b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.637+0000 7fb98ced0700 1 -- 172.21.15.111:0/2908745412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97800b7f0 con 0x7fb9880cdd20 2023-11-28T12:59:45.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.638+0000 7fb98ced0700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9880cdd20 0x7fb9880c3240 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb9840090b0 tx=0x7fb984004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:45.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.638+0000 7fb97dffb700 1 -- 172.21.15.111:0/2908745412 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984004550 con 0x7fb9880cdd20 2023-11-28T12:59:45.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.638+0000 7fb97dffb700 1 -- 172.21.15.111:0/2908745412 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98401f070 con 0x7fb9880cdd20 2023-11-28T12:59:45.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.638+0000 7fb98f935700 1 -- 172.21.15.111:0/2908745412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9840094a0 con 0x7fb9880cdd20 2023-11-28T12:59:45.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.638+0000 7fb97dffb700 1 -- 172.21.15.111:0/2908745412 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9840038e0 con 0x7fb9880cdd20 2023-11-28T12:59:45.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.638+0000 7fb98f935700 1 -- 172.21.15.111:0/2908745412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9880cd060 con 0x7fb9880cdd20 2023-11-28T12:59:45.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.639+0000 7fb98f935700 1 -- 172.21.15.111:0/2908745412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9880ccd80 con 0x7fb9880cdd20 2023-11-28T12:59:45.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.639+0000 7fb97dffb700 1 -- 172.21.15.111:0/2908745412 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb98400f050 con 0x7fb9880cdd20 2023-11-28T12:59:45.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.640+0000 7fb97dffb700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb96c0682f0 0x7fb96c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.640+0000 7fb97dffb700 1 -- 172.21.15.111:0/2908745412 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fb984021070 con 0x7fb9880cdd20 2023-11-28T12:59:45.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.640+0000 7fb98d6d1700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb96c0682f0 0x7fb96c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.640+0000 7fb98d6d1700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb96c0682f0 0x7fb96c06a7a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb978006d30 tx=0x7fb97800b4d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:45.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.644+0000 7fb97dffb700 1 -- 172.21.15.111:0/2908745412 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb984053ce0 con 0x7fb9880cdd20 2023-11-28T12:59:45.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.826+0000 7fb98f935700 1 -- 172.21.15.111:0/2908745412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb9880cd340 con 0x7fb9880cdd20 2023-11-28T12:59:45.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.826+0000 7fb97dffb700 1 -- 172.21.15.111:0/2908745412 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb984015020 con 0x7fb9880cdd20 2023-11-28T12:59:45.828 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:45.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.828+0000 7fb98f935700 1 -- 172.21.15.111:0/2908745412 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb96c0682f0 msgr2=0x7fb96c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.828+0000 7fb98f935700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb96c0682f0 0x7fb96c06a7a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb978006d30 tx=0x7fb97800b4d0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.828+0000 7fb98f935700 1 -- 172.21.15.111:0/2908745412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9880cdd20 msgr2=0x7fb9880c3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.828+0000 7fb98f935700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9880cdd20 0x7fb9880c3240 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb9840090b0 tx=0x7fb984004750 comp rx=0 tx=0).stop 2023-11-28T12:59:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.828+0000 7fb98f935700 1 -- 172.21.15.111:0/2908745412 shutdown_connections 2023-11-28T12:59:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.828+0000 7fb98f935700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb96c0682f0 0x7fb96c06a7a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.828+0000 7fb98f935700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9880cfc90 0x7fb98815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.828+0000 7fb98f935700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9880cc5a0 0x7fb988158390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.828+0000 7fb98f935700 1 --2- 172.21.15.111:0/2908745412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9880cdd20 0x7fb9880c3240 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.828+0000 7fb98f935700 1 -- 172.21.15.111:0/2908745412 >> 172.21.15.111:0/2908745412 conn(0x7fb988098ee0 msgr2=0x7fb98806f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:45.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.828+0000 7fb98f935700 1 -- 172.21.15.111:0/2908745412 shutdown_connections 2023-11-28T12:59:45.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.828+0000 7fb98f935700 1 -- 172.21.15.111:0/2908745412 wait complete. 2023-11-28T12:59:45.842 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T12:59:45.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.964+0000 7f4184699700 1 Processor -- start 2023-11-28T12:59:45.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.964+0000 7f4184699700 1 -- start start 2023-11-28T12:59:45.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4184699700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f417c15a5d0 0x7f417c0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4184699700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f417c0cb2d0 0x7f417c0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4184699700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f417c0ae8b0 con 0x7f417c0b0160 2023-11-28T12:59:45.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4184699700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f417c0cc1b0 con 0x7f417c15a5d0 2023-11-28T12:59:45.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4184699700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417c0cc390 con 0x7f417c0cb2d0 2023-11-28T12:59:45.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181c34700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f417c0b0160 0x7f417c15e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35070/0 (socket says 172.21.15.111:35070) 2023-11-28T12:59:45.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181c34700 1 -- 172.21.15.111:0/1669448270 learned_addr learned my addr 172.21.15.111:0/1669448270 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:45.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4182435700 1 --2- 172.21.15.111:0/1669448270 >> v2:172.21.15.149:3300/0 conn(0x7f417c15a5d0 0x7f417c0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4182c36700 1 --2- 172.21.15.111:0/1669448270 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f417c0cb2d0 0x7f417c0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335250961 0 0) 0x7f417c0ae8b0 con 0x7f417c0b0160 2023-11-28T12:59:45.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41680035b0 con 0x7f417c0b0160 2023-11-28T12:59:45.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 381111044 0 0) 0x7f41680035b0 con 0x7f417c0b0160 2023-11-28T12:59:45.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f417c0cb2d0 msgr2=0x7f417c0ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:45.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181433700 1 --2- 172.21.15.111:0/1669448270 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f417c0cb2d0 0x7f417c0ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 >> v2:172.21.15.149:3300/0 conn(0x7f417c15a5d0 msgr2=0x7f417c0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:45.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181433700 1 --2- 172.21.15.111:0/1669448270 >> v2:172.21.15.149:3300/0 conn(0x7f417c15a5d0 0x7f417c0ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c0cc670 con 0x7f417c0b0160 2023-11-28T12:59:45.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f416c001670 con 0x7f417c0b0160 2023-11-28T12:59:45.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 >> v1:172.21.15.111:6789/0 conn(0x7f417c0b0160 legacy=0x7f417c15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181433700 1 --2- 172.21.15.111:0/1669448270 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4168003a00 0x7f4168003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.965+0000 7f4181433700 1 --2- 172.21.15.111:0/1669448270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4168004430 0x7f41680048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181433700 1 --2- 172.21.15.111:0/1669448270 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f417c0cb2d0 0x7f4168004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c0cc1b0 con 0x7f4168004430 2023-11-28T12:59:45.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c0cc390 con 0x7f4168003a00 2023-11-28T12:59:45.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416800ba10 con 0x7f417c0cb2d0 2023-11-28T12:59:45.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181c34700 1 --2- 172.21.15.111:0/1669448270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4168004430 0x7f41680048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4182435700 1 --2- 172.21.15.111:0/1669448270 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4168003a00 0x7f4168003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4182c36700 1 --2- 172.21.15.111:0/1669448270 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f417c0cb2d0 0x7f4168004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181c34700 1 -- 172.21.15.111:0/1669448270 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f417c0cb2d0 msgr2=0x7f4168004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:45.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181c34700 1 --2- 172.21.15.111:0/1669448270 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f417c0cb2d0 0x7f4168004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181c34700 1 -- 172.21.15.111:0/1669448270 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4168003a00 msgr2=0x7f4168003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:45.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181c34700 1 --2- 172.21.15.111:0/1669448270 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4168003a00 0x7f4168003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181c34700 1 -- 172.21.15.111:0/1669448270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41680035b0 con 0x7f4168004430 2023-11-28T12:59:45.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181c34700 1 --2- 172.21.15.111:0/1669448270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4168004430 0x7f41680048f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f416c00c320 tx=0x7f4168005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9779f94b33d6193f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:45.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416c00ce60 con 0x7f4168004430 2023-11-28T12:59:45.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f416c005960 con 0x7f4168004430 2023-11-28T12:59:45.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.966+0000 7f4181433700 1 -- 172.21.15.111:0/1669448270 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416c005cf0 con 0x7f4168004430 2023-11-28T12:59:45.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.967+0000 7f4184699700 1 -- 172.21.15.111:0/1669448270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4168004430 msgr2=0x7f41680048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.967+0000 7f4184699700 1 --2- 172.21.15.111:0/1669448270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4168004430 0x7f41680048f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f416c00c320 tx=0x7f4168005880 comp rx=0 tx=0).stop 2023-11-28T12:59:45.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.967+0000 7f4182c36700 1 -- 172.21.15.111:0/1669448270 reap_dead start 2023-11-28T12:59:45.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.967+0000 7f4184699700 1 -- 172.21.15.111:0/1669448270 shutdown_connections 2023-11-28T12:59:45.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.967+0000 7f4184699700 1 -- 172.21.15.111:0/1669448270 >> 172.21.15.111:0/1669448270 conn(0x7f417c09a040 msgr2=0x7f417c0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:45.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.967+0000 7f4184699700 1 -- 172.21.15.111:0/1669448270 shutdown_connections 2023-11-28T12:59:45.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.967+0000 7f4184699700 1 -- 172.21.15.111:0/1669448270 wait complete. 2023-11-28T12:59:45.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.968+0000 7f4184699700 1 Processor -- start 2023-11-28T12:59:45.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.968+0000 7f4184699700 1 -- start start 2023-11-28T12:59:45.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4184699700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4168004430 0x7f417c0ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4184699700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f417c154370 0x7f417c1f0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4184699700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f417c1f10c0 0x7f417c0c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4184699700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c1547e0 con 0x7f4168004430 2023-11-28T12:59:45.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4184699700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c0c47d0 con 0x7f417c1f10c0 2023-11-28T12:59:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4184699700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417c0c4ab0 con 0x7f417c154370 2023-11-28T12:59:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4182435700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4168004430 0x7f417c0ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4182435700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4168004430 0x7f417c0ca3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34302/0 (socket says 172.21.15.111:34302) 2023-11-28T12:59:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4182435700 1 -- 172.21.15.111:0/3071073803 learned_addr learned my addr 172.21.15.111:0/3071073803 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4181c34700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f417c154370 0x7f417c1f0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4182c36700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f417c1f10c0 0x7f417c0c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4182435700 1 -- 172.21.15.111:0/3071073803 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f417c154370 msgr2=0x7f417c1f0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4182435700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f417c154370 0x7f417c1f0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4182435700 1 -- 172.21.15.111:0/3071073803 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f417c1f10c0 msgr2=0x7f417c0c6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4182435700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f417c1f10c0 0x7f417c0c6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4182435700 1 -- 172.21.15.111:0/3071073803 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c0ccc30 con 0x7f4168004430 2023-11-28T12:59:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4182c36700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f417c1f10c0 0x7f417c0c6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4182435700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4168004430 0x7f417c0ca3a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4178011a70 tx=0x7f4178011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4172ffd700 1 -- 172.21.15.111:0/3071073803 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417800f940 con 0x7f4168004430 2023-11-28T12:59:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4172ffd700 1 -- 172.21.15.111:0/3071073803 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417800fac0 con 0x7f4168004430 2023-11-28T12:59:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4184699700 1 -- 172.21.15.111:0/3071073803 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417c0cc950 con 0x7f4168004430 2023-11-28T12:59:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4172ffd700 1 -- 172.21.15.111:0/3071073803 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417800fe70 con 0x7f4168004430 2023-11-28T12:59:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.969+0000 7f4184699700 1 -- 172.21.15.111:0/3071073803 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f417c0ccf10 con 0x7f4168004430 2023-11-28T12:59:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.970+0000 7f4184699700 1 -- 172.21.15.111:0/3071073803 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f417c0c53e0 con 0x7f4168004430 2023-11-28T12:59:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.971+0000 7f4172ffd700 1 -- 172.21.15.111:0/3071073803 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4178014440 con 0x7f4168004430 2023-11-28T12:59:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.971+0000 7f4172ffd700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41600682f0 0x7f416006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.971+0000 7f4172ffd700 1 -- 172.21.15.111:0/3071073803 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4178086930 con 0x7f4168004430 2023-11-28T12:59:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.971+0000 7f4181c34700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41600682f0 0x7f416006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.972+0000 7f4181c34700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41600682f0 0x7f416006a7a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f416c008fd0 tx=0x7f416c00a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:45.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:45.975+0000 7f4172ffd700 1 -- 172.21.15.111:0/3071073803 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4178052d10 con 0x7f4168004430 2023-11-28T12:59:46.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.155+0000 7f4184699700 1 -- 172.21.15.111:0/3071073803 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f417c0c4910 con 0x7f4168004430 2023-11-28T12:59:46.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.155+0000 7f4172ffd700 1 -- 172.21.15.111:0/3071073803 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f417c0c4910 con 0x7f4168004430 2023-11-28T12:59:46.157 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T12:59:46.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.157+0000 7f4184699700 1 -- 172.21.15.111:0/3071073803 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41600682f0 msgr2=0x7f416006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.157+0000 7f4184699700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41600682f0 0x7f416006a7a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f416c008fd0 tx=0x7f416c00a000 comp rx=0 tx=0).stop 2023-11-28T12:59:46.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.157+0000 7f4184699700 1 -- 172.21.15.111:0/3071073803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4168004430 msgr2=0x7f417c0ca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.157+0000 7f4184699700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4168004430 0x7f417c0ca3a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4178011a70 tx=0x7f4178011d80 comp rx=0 tx=0).stop 2023-11-28T12:59:46.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.157+0000 7f4184699700 1 -- 172.21.15.111:0/3071073803 shutdown_connections 2023-11-28T12:59:46.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.157+0000 7f4184699700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41600682f0 0x7f416006a7a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.157+0000 7f4184699700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f417c1f10c0 0x7f417c0c6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.157+0000 7f4184699700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4168004430 0x7f417c0ca3a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.157+0000 7f4184699700 1 --2- 172.21.15.111:0/3071073803 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f417c154370 0x7f417c1f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.157+0000 7f4184699700 1 -- 172.21.15.111:0/3071073803 >> 172.21.15.111:0/3071073803 conn(0x7f417c09a040 msgr2=0x7f417c0cdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:46.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.157+0000 7f4184699700 1 -- 172.21.15.111:0/3071073803 shutdown_connections 2023-11-28T12:59:46.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.157+0000 7f4184699700 1 -- 172.21.15.111:0/3071073803 wait complete. 2023-11-28T12:59:46.171 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T12:59:46.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.293+0000 7fb5a2ad6700 1 Processor -- start 2023-11-28T12:59:46.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.293+0000 7fb5a2ad6700 1 -- start start 2023-11-28T12:59:46.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.293+0000 7fb5a2ad6700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c06b720 0x7fb59c06a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.293+0000 7fb5a2ad6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c15c9b0 con 0x7fb59c06e970 2023-11-28T12:59:46.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.293+0000 7fb5a2ad6700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c160e60 con 0x7fb59c15c5e0 2023-11-28T12:59:46.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.293+0000 7fb5a2ad6700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0cce90 con 0x7fb59c06b720 2023-11-28T12:59:46.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb5a1073700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb59c06e970 0x7fb59c06ee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35092/0 (socket says 172.21.15.111:35092) 2023-11-28T12:59:46.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb5a1073700 1 -- 172.21.15.111:0/473052653 learned_addr learned my addr 172.21.15.111:0/473052653 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:46.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59bfff700 1 --2- 172.21.15.111:0/473052653 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c06b720 0x7fb59c06a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378672733 0 0) 0x7fb59c15c9b0 con 0x7fb59c06e970 2023-11-28T12:59:46.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb58c0035b0 con 0x7fb59c06e970 2023-11-28T12:59:46.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3086627815 0 0) 0x7fb58c0035b0 con 0x7fb59c06e970 2023-11-28T12:59:46.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c06b720 msgr2=0x7fb59c06a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:46.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 --2- 172.21.15.111:0/473052653 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c06b720 0x7fb59c06a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 >> v1:172.21.15.149:6789/0 conn(0x7fb59c15c5e0 legacy=0x7fb59c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c0cd0a0 con 0x7fb59c06e970 2023-11-28T12:59:46.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fb5940019f0 con 0x7fb59c06e970 2023-11-28T12:59:46.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59bfff700 1 --2- 172.21.15.111:0/473052653 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c06b720 0x7fb59c06a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:46.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 >> v1:172.21.15.111:6789/0 conn(0x7fb59c06e970 legacy=0x7fb59c06ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 --2- 172.21.15.111:0/473052653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb58c003a30 0x7fb58c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 --2- 172.21.15.111:0/473052653 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c06b720 0x7fb58c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 --2- 172.21.15.111:0/473052653 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb58c004d90 0x7fb58c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0cce90 con 0x7fb58c003a30 2023-11-28T12:59:46.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58c00bac0 con 0x7fb58c004d90 2023-11-28T12:59:46.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb58c00bc00 con 0x7fb59c06b720 2023-11-28T12:59:46.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb5a0872700 1 --2- 172.21.15.111:0/473052653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb58c003a30 0x7fb58c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb5a1073700 1 --2- 172.21.15.111:0/473052653 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb58c004d90 0x7fb58c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb59bfff700 1 --2- 172.21.15.111:0/473052653 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c06b720 0x7fb58c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.294+0000 7fb5a0872700 1 -- 172.21.15.111:0/473052653 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c06b720 msgr2=0x7fb58c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:46.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb5a0872700 1 --2- 172.21.15.111:0/473052653 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c06b720 0x7fb58c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb5a0872700 1 -- 172.21.15.111:0/473052653 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb58c004d90 msgr2=0x7fb58c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:46.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb5a0872700 1 --2- 172.21.15.111:0/473052653 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb58c004d90 0x7fb58c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb5a0872700 1 -- 172.21.15.111:0/473052653 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5880094a0 con 0x7fb58c003a30 2023-11-28T12:59:46.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb5a0872700 1 --2- 172.21.15.111:0/473052653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb58c003a30 0x7fb58c003e40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb5900017c0 tx=0x7fb58c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2018f8760e8e983 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:46.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5900052d0 con 0x7fb58c003a30 2023-11-28T12:59:46.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb590005430 con 0x7fb58c003a30 2023-11-28T12:59:46.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb59b7fe700 1 -- 172.21.15.111:0/473052653 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5900038c0 con 0x7fb58c003a30 2023-11-28T12:59:46.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/473052653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb58c003a30 msgr2=0x7fb58c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb5a2ad6700 1 --2- 172.21.15.111:0/473052653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb58c003a30 0x7fb58c003e40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb5900017c0 tx=0x7fb58c005880 comp rx=0 tx=0).stop 2023-11-28T12:59:46.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb5a1073700 1 -- 172.21.15.111:0/473052653 reap_dead start 2023-11-28T12:59:46.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/473052653 shutdown_connections 2023-11-28T12:59:46.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/473052653 >> 172.21.15.111:0/473052653 conn(0x7fb59c0b0db0 msgr2=0x7fb59c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:46.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/473052653 shutdown_connections 2023-11-28T12:59:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.295+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/473052653 wait complete. 2023-11-28T12:59:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.297+0000 7fb5a2ad6700 1 Processor -- start 2023-11-28T12:59:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.297+0000 7fb5a2ad6700 1 -- start start 2023-11-28T12:59:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.298+0000 7fb5a2ad6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb58c003a30 0x7fb59c14f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.298+0000 7fb5a2ad6700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb59c150030 0x7fb59c15cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.298+0000 7fb5a2ad6700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c15d0e0 0x7fb59c0aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.298+0000 7fb5a2ad6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c14fe30 con 0x7fb58c003a30 2023-11-28T12:59:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.298+0000 7fb5a2ad6700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0c4220 con 0x7fb59c150030 2023-11-28T12:59:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.298+0000 7fb5a2ad6700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0c4500 con 0x7fb59c15d0e0 2023-11-28T12:59:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.298+0000 7fb5a0872700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb58c003a30 0x7fb59c14f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.298+0000 7fb5a0872700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb58c003a30 0x7fb59c14f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34322/0 (socket says 172.21.15.111:34322) 2023-11-28T12:59:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.298+0000 7fb5a0872700 1 -- 172.21.15.111:0/3761636134 learned_addr learned my addr 172.21.15.111:0/3761636134 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.298+0000 7fb5a1073700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c15d0e0 0x7fb59c0aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.298+0000 7fb59bfff700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb59c150030 0x7fb59c15cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.299+0000 7fb5a0872700 1 -- 172.21.15.111:0/3761636134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c15d0e0 msgr2=0x7fb59c0aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.299+0000 7fb5a0872700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c15d0e0 0x7fb59c0aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.299+0000 7fb5a0872700 1 -- 172.21.15.111:0/3761636134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb59c150030 msgr2=0x7fb59c15cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.299+0000 7fb5a0872700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb59c150030 0x7fb59c15cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.299+0000 7fb5a0872700 1 -- 172.21.15.111:0/3761636134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c0cd660 con 0x7fb58c003a30 2023-11-28T12:59:46.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.299+0000 7fb5a1073700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c15d0e0 0x7fb59c0aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:46.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.299+0000 7fb5a0872700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb58c003a30 0x7fb59c14f8f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb590000c00 tx=0x7fb590001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:46.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.299+0000 7fb5997fa700 1 -- 172.21.15.111:0/3761636134 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb590007240 con 0x7fb58c003a30 2023-11-28T12:59:46.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.299+0000 7fb5997fa700 1 -- 172.21.15.111:0/3761636134 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5900073a0 con 0x7fb58c003a30 2023-11-28T12:59:46.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.299+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/3761636134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59c0cd380 con 0x7fb58c003a30 2023-11-28T12:59:46.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.299+0000 7fb5997fa700 1 -- 172.21.15.111:0/3761636134 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb590001290 con 0x7fb58c003a30 2023-11-28T12:59:46.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.299+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/3761636134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59c160af0 con 0x7fb58c003a30 2023-11-28T12:59:46.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.300+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/3761636134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59c067170 con 0x7fb58c003a30 2023-11-28T12:59:46.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.300+0000 7fb5997fa700 1 -- 172.21.15.111:0/3761636134 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb59002e070 con 0x7fb58c003a30 2023-11-28T12:59:46.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.301+0000 7fb5997fa700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb580068320 0x7fb58006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.301+0000 7fb5997fa700 1 -- 172.21.15.111:0/3761636134 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fb590086d90 con 0x7fb58c003a30 2023-11-28T12:59:46.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.301+0000 7fb59bfff700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb580068320 0x7fb58006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.301+0000 7fb59bfff700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb580068320 0x7fb58006a7d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0b25d0 tx=0x7fb588008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:46.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.305+0000 7fb5997fa700 1 -- 172.21.15.111:0/3761636134 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb5900532c0 con 0x7fb58c003a30 2023-11-28T12:59:46.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.490+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/3761636134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb59c14fe30 con 0x7fb58c003a30 2023-11-28T12:59:46.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.490+0000 7fb5997fa700 1 -- 172.21.15.111:0/3761636134 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb5900568e0 con 0x7fb58c003a30 2023-11-28T12:59:46.492 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:46.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.492+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/3761636134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb580068320 msgr2=0x7fb58006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.492+0000 7fb5a2ad6700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb580068320 0x7fb58006a7d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0b25d0 tx=0x7fb588008040 comp rx=0 tx=0).stop 2023-11-28T12:59:46.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.492+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/3761636134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb58c003a30 msgr2=0x7fb59c14f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.492+0000 7fb5a2ad6700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb58c003a30 0x7fb59c14f8f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb590000c00 tx=0x7fb590001120 comp rx=0 tx=0).stop 2023-11-28T12:59:46.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.492+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/3761636134 shutdown_connections 2023-11-28T12:59:46.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.492+0000 7fb5a2ad6700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb59c15d0e0 0x7fb59c0aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.492+0000 7fb5a2ad6700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb58c003a30 0x7fb59c14f8f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.492+0000 7fb5a2ad6700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb580068320 0x7fb58006a7d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.492+0000 7fb5a2ad6700 1 --2- 172.21.15.111:0/3761636134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb59c150030 0x7fb59c15cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.492+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/3761636134 >> 172.21.15.111:0/3761636134 conn(0x7fb59c0b0db0 msgr2=0x7fb59c1573e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:46.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.492+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/3761636134 shutdown_connections 2023-11-28T12:59:46.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.492+0000 7fb5a2ad6700 1 -- 172.21.15.111:0/3761636134 wait complete. 2023-11-28T12:59:46.506 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T12:59:46.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.626+0000 7f4cfd398700 1 Processor -- start 2023-11-28T12:59:46.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.627+0000 7f4cfd398700 1 -- start start 2023-11-28T12:59:46.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.627+0000 7f4cfd398700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4cf80ae0b0 0x7f4cf815d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.627+0000 7f4cfd398700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cf80cea90 0x7f4cf806d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.627+0000 7f4cfd398700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf806d370 con 0x7f4cf806dd40 2023-11-28T12:59:46.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.627+0000 7f4cfd398700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf80ae480 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.627+0000 7f4cfd398700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf815dba0 con 0x7f4cf80cea90 2023-11-28T12:59:46.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf6ffd700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4cf80ae0b0 0x7f4cf815d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf67fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cf80cea90 0x7f4cf806d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf77fe700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f4cf806dd40 0x7f4cf80b0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:43654/0 (socket says 172.21.15.111:43654) 2023-11-28T12:59:46.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf6ffd700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4cf80ae0b0 0x7f4cf815d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:52230/0 (socket says 172.21.15.111:52230) 2023-11-28T12:59:46.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf67fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cf80cea90 0x7f4cf806d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:56080/0 (socket says 172.21.15.111:56080) 2023-11-28T12:59:46.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf77fe700 1 -- 172.21.15.111:0/3984774000 learned_addr learned my addr 172.21.15.111:0/3984774000 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:46.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf67fc700 1 -- 172.21.15.111:0/3984774000 >> v2:172.21.15.149:3300/0 conn(0x7f4cf80ae0b0 msgr2=0x7f4cf815d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:46.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf67fc700 1 --2- 172.21.15.111:0/3984774000 >> v2:172.21.15.149:3300/0 conn(0x7f4cf80ae0b0 0x7f4cf815d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf67fc700 1 -- 172.21.15.111:0/3984774000 >> v1:172.21.15.149:6789/0 conn(0x7f4cf806dd40 legacy=0x7f4cf80b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf67fc700 1 -- 172.21.15.111:0/3984774000 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf80cca50 con 0x7f4cf80cea90 2023-11-28T12:59:46.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf67fc700 1 --2- 172.21.15.111:0/3984774000 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cf80cea90 0x7f4cf806d800 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009840 tx=0x7f4ce8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f294ba62ba625495 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:46.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf5ffb700 1 -- 172.21.15.111:0/3984774000 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8011830 con 0x7f4cf80cea90 2023-11-28T12:59:46.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf5ffb700 1 -- 172.21.15.111:0/3984774000 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce8011990 con 0x7f4cf80cea90 2023-11-28T12:59:46.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.628+0000 7f4cf5ffb700 1 -- 172.21.15.111:0/3984774000 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8011d20 con 0x7f4cf80cea90 2023-11-28T12:59:46.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.629+0000 7f4cfd398700 1 -- 172.21.15.111:0/3984774000 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cf80cea90 msgr2=0x7f4cf806d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.629+0000 7f4cfd398700 1 --2- 172.21.15.111:0/3984774000 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cf80cea90 0x7f4cf806d800 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009840 tx=0x7f4ce8009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:46.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.629+0000 7f4cfd398700 1 -- 172.21.15.111:0/3984774000 shutdown_connections 2023-11-28T12:59:46.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.629+0000 7f4cfd398700 1 --2- 172.21.15.111:0/3984774000 >> v2:172.21.15.149:3300/0 conn(0x7f4cf80ae0b0 0x7f4cf815d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.629+0000 7f4cfd398700 1 --2- 172.21.15.111:0/3984774000 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cf80cea90 0x7f4cf806d800 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.629+0000 7f4cfd398700 1 -- 172.21.15.111:0/3984774000 >> 172.21.15.111:0/3984774000 conn(0x7f4cf806aeb0 msgr2=0x7f4cf80cb900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:46.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.629+0000 7f4cfd398700 1 -- 172.21.15.111:0/3984774000 shutdown_connections 2023-11-28T12:59:46.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.629+0000 7f4cfd398700 1 -- 172.21.15.111:0/3984774000 wait complete. 2023-11-28T12:59:46.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.630+0000 7f4cfd398700 1 Processor -- start 2023-11-28T12:59:46.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.630+0000 7f4cfd398700 1 -- start start 2023-11-28T12:59:46.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cfd398700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cf806dd40 0x7f4cf814d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cfd398700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cf80ae0b0 0x7f4cf81f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cfd398700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cf80cea90 0x7f4cf80ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cfd398700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf81f2840 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cfd398700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf80c33a0 con 0x7f4cf80cea90 2023-11-28T12:59:46.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cfd398700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf81f62e0 con 0x7f4cf806dd40 2023-11-28T12:59:46.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cf67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cf80ae0b0 0x7f4cf81f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cf67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cf80ae0b0 0x7f4cf81f5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34340/0 (socket says 172.21.15.111:34340) 2023-11-28T12:59:46.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cf67fc700 1 -- 172.21.15.111:0/2066885436 learned_addr learned my addr 172.21.15.111:0/2066885436 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:46.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cf6ffd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cf806dd40 0x7f4cf814d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cf77fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cf80cea90 0x7f4cf80ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cf67fc700 1 -- 172.21.15.111:0/2066885436 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cf806dd40 msgr2=0x7f4cf814d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cf67fc700 1 --2- 172.21.15.111:0/2066885436 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cf806dd40 0x7f4cf814d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cf67fc700 1 -- 172.21.15.111:0/2066885436 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cf80cea90 msgr2=0x7f4cf80ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cf67fc700 1 --2- 172.21.15.111:0/2066885436 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cf80cea90 0x7f4cf80ad1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.631+0000 7f4cf67fc700 1 -- 172.21.15.111:0/2066885436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce80094a0 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.632+0000 7f4cf67fc700 1 --2- 172.21.15.111:0/2066885436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cf80ae0b0 0x7f4cf81f5a90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009fd0 tx=0x7f4ce8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:46.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.632+0000 7f4cdf7fe700 1 -- 172.21.15.111:0/2066885436 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8004550 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.632+0000 7f4cdf7fe700 1 -- 172.21.15.111:0/2066885436 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce801e070 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.632+0000 7f4cfd398700 1 -- 172.21.15.111:0/2066885436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf80ccff0 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.632+0000 7f4cdf7fe700 1 -- 172.21.15.111:0/2066885436 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce80038e0 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.632+0000 7f4cfd398700 1 -- 172.21.15.111:0/2066885436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf80ccd10 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.633+0000 7f4cfd398700 1 -- 172.21.15.111:0/2066885436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf80cd2d0 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.633+0000 7f4cdf7fe700 1 -- 172.21.15.111:0/2066885436 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ce800f050 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.634+0000 7f4cdf7fe700 1 --2- 172.21.15.111:0/2066885436 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4ce40682f0 0x7f4ce406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:46.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.634+0000 7f4cf6ffd700 1 --2- 172.21.15.111:0/2066885436 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4ce40682f0 0x7f4ce406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:46.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.634+0000 7f4cdf7fe700 1 -- 172.21.15.111:0/2066885436 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4ce8086e60 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.634+0000 7f4cf6ffd700 1 --2- 172.21.15.111:0/2066885436 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4ce40682f0 0x7f4ce406a7a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009be0 tx=0x7f4cec009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:46.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.638+0000 7f4cdf7fe700 1 -- 172.21.15.111:0/2066885436 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ce8053390 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.836+0000 7f4cfd398700 1 -- 172.21.15.111:0/2066885436 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4cf8067880 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.837+0000 7f4cdf7fe700 1 -- 172.21.15.111:0/2066885436 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4ce8020080 con 0x7f4cf80ae0b0 2023-11-28T12:59:46.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.839+0000 7f4cfd398700 1 -- 172.21.15.111:0/2066885436 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4ce40682f0 msgr2=0x7f4ce406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.839+0000 7f4cfd398700 1 --2- 172.21.15.111:0/2066885436 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4ce40682f0 0x7f4ce406a7a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009be0 tx=0x7f4cec009210 comp rx=0 tx=0).stop 2023-11-28T12:59:46.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.839+0000 7f4cfd398700 1 -- 172.21.15.111:0/2066885436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cf80ae0b0 msgr2=0x7f4cf81f5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:46.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.839+0000 7f4cfd398700 1 --2- 172.21.15.111:0/2066885436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cf80ae0b0 0x7f4cf81f5a90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009fd0 tx=0x7f4ce8004750 comp rx=0 tx=0).stop 2023-11-28T12:59:46.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.839+0000 7f4cfd398700 1 -- 172.21.15.111:0/2066885436 shutdown_connections 2023-11-28T12:59:46.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.839+0000 7f4cfd398700 1 --2- 172.21.15.111:0/2066885436 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4ce40682f0 0x7f4ce406a7a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.839+0000 7f4cfd398700 1 --2- 172.21.15.111:0/2066885436 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cf80cea90 0x7f4cf80ad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.839+0000 7f4cfd398700 1 --2- 172.21.15.111:0/2066885436 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cf806dd40 0x7f4cf814d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.839+0000 7f4cfd398700 1 --2- 172.21.15.111:0/2066885436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cf80ae0b0 0x7f4cf81f5a90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:46.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.839+0000 7f4cfd398700 1 -- 172.21.15.111:0/2066885436 >> 172.21.15.111:0/2066885436 conn(0x7f4cf806aeb0 msgr2=0x7f4cf8070e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:46.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.839+0000 7f4cfd398700 1 -- 172.21.15.111:0/2066885436 shutdown_connections 2023-11-28T12:59:46.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:46.839+0000 7f4cfd398700 1 -- 172.21.15.111:0/2066885436 wait complete. 2023-11-28T12:59:46.842 INFO:teuthology.orchestra.run.smithi111.stderr:listed 0 entries 2023-11-28T12:59:46.852 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T12:59:46.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:46.852 DEBUG:teuthology.orchestra.run.smithi111:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-11-28T12:59:46.906 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T12:59:46.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:46.906 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T12:59:46.924 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:59:46.924 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:46.944 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:59:46.945 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:47.005 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:59:47.005 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:47.064 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:47.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.182+0000 7f7422468700 1 Processor -- start 2023-11-28T12:59:47.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.182+0000 7f7422468700 1 -- start start 2023-11-28T12:59:47.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f7422468700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f741c15cad0 0x7f741c0af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f7422468700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f741c0b0b10 0x7f741c0ca230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f7422468700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741c0cc1a0 con 0x7f741c0ccf50 2023-11-28T12:59:47.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f7422468700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f741c0b0270 con 0x7f741c0b0b10 2023-11-28T12:59:47.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f7422468700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f741c0cf400 con 0x7f741c15cad0 2023-11-28T12:59:47.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f7420a05700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f741c0b0b10 0x7f741c0ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f7420a05700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f741c0b0b10 0x7f741c0ca230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34348/0 (socket says 172.21.15.111:34348) 2023-11-28T12:59:47.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f7420a05700 1 -- 172.21.15.111:0/464546856 learned_addr learned my addr 172.21.15.111:0/464546856 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:47.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f741bfff700 1 --2- 172.21.15.111:0/464546856 >> v2:172.21.15.149:3300/0 conn(0x7f741c15cad0 0x7f741c0af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f7420a05700 1 -- 172.21.15.111:0/464546856 >> v2:172.21.15.149:3300/0 conn(0x7f741c15cad0 msgr2=0x7f741c0af8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:47.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f7420a05700 1 --2- 172.21.15.111:0/464546856 >> v2:172.21.15.149:3300/0 conn(0x7f741c15cad0 0x7f741c0af8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f7420a05700 1 -- 172.21.15.111:0/464546856 >> v1:172.21.15.149:6789/0 conn(0x7f741c0ccf50 legacy=0x7f741c0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:47.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f7420a05700 1 -- 172.21.15.111:0/464546856 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741c0ce860 con 0x7f741c0b0b10 2023-11-28T12:59:47.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f741bfff700 1 --2- 172.21.15.111:0/464546856 >> v2:172.21.15.149:3300/0 conn(0x7f741c15cad0 0x7f741c0af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:47.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f7420a05700 1 --2- 172.21.15.111:0/464546856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f741c0b0b10 0x7f741c0ca230 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f741c072a50 tx=0x7f741400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fde502f395747f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:47.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f741affd700 1 -- 172.21.15.111:0/464546856 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741400e070 con 0x7f741c0b0b10 2023-11-28T12:59:47.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.183+0000 7f741affd700 1 -- 172.21.15.111:0/464546856 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7414003a20 con 0x7f741c0b0b10 2023-11-28T12:59:47.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.184+0000 7f741affd700 1 -- 172.21.15.111:0/464546856 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414003dd0 con 0x7f741c0b0b10 2023-11-28T12:59:47.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.184+0000 7f7422468700 1 -- 172.21.15.111:0/464546856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f741c0b0b10 msgr2=0x7f741c0ca230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.184+0000 7f7422468700 1 --2- 172.21.15.111:0/464546856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f741c0b0b10 0x7f741c0ca230 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f741c072a50 tx=0x7f741400b330 comp rx=0 tx=0).stop 2023-11-28T12:59:47.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.184+0000 7f7422468700 1 -- 172.21.15.111:0/464546856 shutdown_connections 2023-11-28T12:59:47.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.184+0000 7f7422468700 1 --2- 172.21.15.111:0/464546856 >> v2:172.21.15.149:3300/0 conn(0x7f741c15cad0 0x7f741c0af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.184+0000 7f7422468700 1 --2- 172.21.15.111:0/464546856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f741c0b0b10 0x7f741c0ca230 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.184+0000 7f7422468700 1 -- 172.21.15.111:0/464546856 >> 172.21.15.111:0/464546856 conn(0x7f741c06bca0 msgr2=0x7f741c0cfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:47.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.184+0000 7f7422468700 1 -- 172.21.15.111:0/464546856 shutdown_connections 2023-11-28T12:59:47.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.184+0000 7f7422468700 1 -- 172.21.15.111:0/464546856 wait complete. 2023-11-28T12:59:47.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.185+0000 7f7422468700 1 Processor -- start 2023-11-28T12:59:47.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.185+0000 7f7422468700 1 -- start start 2023-11-28T12:59:47.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f7422468700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f741c0b0b10 0x7f741c155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f7422468700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f741c0ccf50 0x7f741c0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f7422468700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f741c15cad0 0x7f741c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f7422468700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741c15f2c0 con 0x7f741c0ccf50 2023-11-28T12:59:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f7422468700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741c15f560 con 0x7f741c15cad0 2023-11-28T12:59:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f7422468700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741c15f840 con 0x7f741c0b0b10 2023-11-28T12:59:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f741b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f741c0ccf50 0x7f741c0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f741b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f741c0ccf50 0x7f741c0cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34352/0 (socket says 172.21.15.111:34352) 2023-11-28T12:59:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f741b7fe700 1 -- 172.21.15.111:0/2725023123 learned_addr learned my addr 172.21.15.111:0/2725023123 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f741bfff700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f741c0b0b10 0x7f741c155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f7420a05700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f741c15cad0 0x7f741c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f741b7fe700 1 -- 172.21.15.111:0/2725023123 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f741c0b0b10 msgr2=0x7f741c155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f741b7fe700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f741c0b0b10 0x7f741c155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f741b7fe700 1 -- 172.21.15.111:0/2725023123 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f741c15cad0 msgr2=0x7f741c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.186+0000 7f741b7fe700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f741c15cad0 0x7f741c0cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.187+0000 7f741b7fe700 1 -- 172.21.15.111:0/2725023123 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7414009b30 con 0x7f741c0ccf50 2023-11-28T12:59:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.187+0000 7f741bfff700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f741c0b0b10 0x7f741c155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.187+0000 7f741b7fe700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f741c0ccf50 0x7f741c0cc330 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f740c00c4d0 tx=0x7f740c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.187+0000 7f7418ff9700 1 -- 172.21.15.111:0/2725023123 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c011790 con 0x7f741c0ccf50 2023-11-28T12:59:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.187+0000 7f7418ff9700 1 -- 172.21.15.111:0/2725023123 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740c0118f0 con 0x7f741c0ccf50 2023-11-28T12:59:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.187+0000 7f7422468700 1 -- 172.21.15.111:0/2725023123 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f741c0cead0 con 0x7f741c0ccf50 2023-11-28T12:59:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.187+0000 7f7418ff9700 1 -- 172.21.15.111:0/2725023123 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c011ca0 con 0x7f741c0ccf50 2023-11-28T12:59:47.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.187+0000 7f7422468700 1 -- 172.21.15.111:0/2725023123 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741c15cea0 con 0x7f741c0ccf50 2023-11-28T12:59:47.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.188+0000 7f7422468700 1 -- 172.21.15.111:0/2725023123 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741c0cedb0 con 0x7f741c0ccf50 2023-11-28T12:59:47.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.188+0000 7f7418ff9700 1 -- 172.21.15.111:0/2725023123 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f740c010050 con 0x7f741c0ccf50 2023-11-28T12:59:47.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.188+0000 7f7418ff9700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74080682f0 0x7f740806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.189+0000 7f7418ff9700 1 -- 172.21.15.111:0/2725023123 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f740c085d40 con 0x7f741c0ccf50 2023-11-28T12:59:47.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.189+0000 7f741bfff700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74080682f0 0x7f740806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.189+0000 7f741bfff700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74080682f0 0x7f740806a7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f741000aa20 tx=0x7f7410008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:47.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.193+0000 7f7418ff9700 1 -- 172.21.15.111:0/2725023123 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f740c052270 con 0x7f741c0ccf50 2023-11-28T12:59:47.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.376+0000 7f7422468700 1 -- 172.21.15.111:0/2725023123 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f741c067170 con 0x7f741c0ccf50 2023-11-28T12:59:47.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.377+0000 7f7418ff9700 1 -- 172.21.15.111:0/2725023123 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+19890 (secure 0 0 0) 0x7f740c055890 con 0x7f741c0ccf50 2023-11-28T12:59:47.379 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:47.379 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":28,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:41.880678+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:47.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.379+0000 7f7422468700 1 -- 172.21.15.111:0/2725023123 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74080682f0 msgr2=0x7f740806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.379+0000 7f7422468700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74080682f0 0x7f740806a7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f741000aa20 tx=0x7f7410008040 comp rx=0 tx=0).stop 2023-11-28T12:59:47.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.379+0000 7f7422468700 1 -- 172.21.15.111:0/2725023123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f741c0ccf50 msgr2=0x7f741c0cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.379+0000 7f7422468700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f741c0ccf50 0x7f741c0cc330 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f740c00c4d0 tx=0x7f740c00c890 comp rx=0 tx=0).stop 2023-11-28T12:59:47.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.379+0000 7f7422468700 1 -- 172.21.15.111:0/2725023123 shutdown_connections 2023-11-28T12:59:47.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.379+0000 7f7422468700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74080682f0 0x7f740806a7a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.379+0000 7f7422468700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f741c15cad0 0x7f741c0cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.379+0000 7f7422468700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f741c0b0b10 0x7f741c155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.379+0000 7f7422468700 1 --2- 172.21.15.111:0/2725023123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f741c0ccf50 0x7f741c0cc330 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.379+0000 7f7422468700 1 -- 172.21.15.111:0/2725023123 >> 172.21.15.111:0/2725023123 conn(0x7f741c06bca0 msgr2=0x7f741c15c030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:47.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.380+0000 7f7422468700 1 -- 172.21.15.111:0/2725023123 shutdown_connections 2023-11-28T12:59:47.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.380+0000 7f7422468700 1 -- 172.21.15.111:0/2725023123 wait complete. 2023-11-28T12:59:47.394 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T12:58:53.275206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T12:58:54.588499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T12:58:57.666033+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T12:59:47.394 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:59:47.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.515+0000 7f251dd3c700 1 Processor -- start 2023-11-28T12:59:47.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.515+0000 7f251dd3c700 1 -- start start 2023-11-28T12:59:47.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f251dd3c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f25180b0d70 0x7f25181604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f251dd3c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251815cd90 con 0x7f25180cc550 2023-11-28T12:59:47.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f251dd3c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25181613e0 con 0x7f25180b1140 2023-11-28T12:59:47.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f251dd3c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25180cedd0 con 0x7f25180b0d70 2023-11-28T12:59:47.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f25177fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f25180b0d70 0x7f25181604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f25177fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f25180b0d70 0x7f25181604b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34358/0 (socket says 172.21.15.111:34358) 2023-11-28T12:59:47.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f25177fe700 1 -- 172.21.15.111:0/1232446827 learned_addr learned my addr 172.21.15.111:0/1232446827 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:47.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f25177fe700 1 -- 172.21.15.111:0/1232446827 >> v1:172.21.15.149:6789/0 conn(0x7f25180b1140 legacy=0x7f25180ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:47.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f25177fe700 1 -- 172.21.15.111:0/1232446827 >> v1:172.21.15.111:6789/0 conn(0x7f25180cc550 legacy=0x7f25180af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f25177fe700 1 -- 172.21.15.111:0/1232446827 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2518160b40 con 0x7f25180b0d70 2023-11-28T12:59:47.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f25177fe700 1 --2- 172.21.15.111:0/1232446827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25180b0d70 0x7f25181604b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2508006fd0 tx=0x7f2508009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abca6997bc8dc9b2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:47.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f25167fc700 1 -- 172.21.15.111:0/1232446827 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2508005e30 con 0x7f25180b0d70 2023-11-28T12:59:47.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f25167fc700 1 -- 172.21.15.111:0/1232446827 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2508006030 con 0x7f25180b0d70 2023-11-28T12:59:47.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.516+0000 7f25167fc700 1 -- 172.21.15.111:0/1232446827 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25080063e0 con 0x7f25180b0d70 2023-11-28T12:59:47.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.517+0000 7f251dd3c700 1 -- 172.21.15.111:0/1232446827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25180b0d70 msgr2=0x7f25181604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.517+0000 7f251dd3c700 1 --2- 172.21.15.111:0/1232446827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25180b0d70 0x7f25181604b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2508006fd0 tx=0x7f2508009bd0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.517+0000 7f251dd3c700 1 -- 172.21.15.111:0/1232446827 shutdown_connections 2023-11-28T12:59:47.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.517+0000 7f251dd3c700 1 --2- 172.21.15.111:0/1232446827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25180b0d70 0x7f25181604b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.517+0000 7f251dd3c700 1 -- 172.21.15.111:0/1232446827 >> 172.21.15.111:0/1232446827 conn(0x7f2518098ee0 msgr2=0x7f251806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:47.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.517+0000 7f251dd3c700 1 -- 172.21.15.111:0/1232446827 shutdown_connections 2023-11-28T12:59:47.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.517+0000 7f251dd3c700 1 -- 172.21.15.111:0/1232446827 wait complete. 2023-11-28T12:59:47.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.518+0000 7f251dd3c700 1 Processor -- start 2023-11-28T12:59:47.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.518+0000 7f251dd3c700 1 -- start start 2023-11-28T12:59:47.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f251dd3c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f25180b0d70 0x7f251815b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f251dd3c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f25180b1140 0x7f2518149730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f251dd3c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25180cc550 0x7f2518149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f251dd3c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25180b3060 con 0x7f25180cc550 2023-11-28T12:59:47.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f251dd3c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25180b5a20 con 0x7f25180b1140 2023-11-28T12:59:47.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f251dd3c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25180b5740 con 0x7f25180b0d70 2023-11-28T12:59:47.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f2517fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25180cc550 0x7f2518149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f2517fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25180cc550 0x7f2518149c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34368/0 (socket says 172.21.15.111:34368) 2023-11-28T12:59:47.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f2517fff700 1 -- 172.21.15.111:0/3155730549 learned_addr learned my addr 172.21.15.111:0/3155730549 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:47.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f2516ffd700 1 --2- 172.21.15.111:0/3155730549 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f25180b1140 0x7f2518149730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f2517fff700 1 -- 172.21.15.111:0/3155730549 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f25180b0d70 msgr2=0x7f251815b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f2517fff700 1 --2- 172.21.15.111:0/3155730549 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f25180b0d70 0x7f251815b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f2517fff700 1 -- 172.21.15.111:0/3155730549 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f25180b1140 msgr2=0x7f2518149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f2517fff700 1 --2- 172.21.15.111:0/3155730549 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f25180b1140 0x7f2518149730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f2517fff700 1 -- 172.21.15.111:0/3155730549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2508009500 con 0x7f25180cc550 2023-11-28T12:59:47.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.519+0000 7f2517fff700 1 --2- 172.21.15.111:0/3155730549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25180cc550 0x7f2518149c70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f251000cb50 tx=0x7f251000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:47.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.520+0000 7f24fffff700 1 -- 172.21.15.111:0/3155730549 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2510003c00 con 0x7f25180cc550 2023-11-28T12:59:47.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.520+0000 7f24fffff700 1 -- 172.21.15.111:0/3155730549 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2510003d60 con 0x7f25180cc550 2023-11-28T12:59:47.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.520+0000 7f251dd3c700 1 -- 172.21.15.111:0/3155730549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2518161100 con 0x7f25180cc550 2023-11-28T12:59:47.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.520+0000 7f24fffff700 1 -- 172.21.15.111:0/3155730549 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251000ab70 con 0x7f25180cc550 2023-11-28T12:59:47.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.520+0000 7f251dd3c700 1 -- 172.21.15.111:0/3155730549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2518160e20 con 0x7f25180cc550 2023-11-28T12:59:47.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.520+0000 7f251dd3c700 1 -- 172.21.15.111:0/3155730549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25180af340 con 0x7f25180cc550 2023-11-28T12:59:47.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.521+0000 7f24fffff700 1 -- 172.21.15.111:0/3155730549 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2510010050 con 0x7f25180cc550 2023-11-28T12:59:47.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.521+0000 7f24fffff700 1 --2- 172.21.15.111:0/3155730549 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f25040682f0 0x7f250406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.521+0000 7f24fffff700 1 -- 172.21.15.111:0/3155730549 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f2510085d80 con 0x7f25180cc550 2023-11-28T12:59:47.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.521+0000 7f25177fe700 1 --2- 172.21.15.111:0/3155730549 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f25040682f0 0x7f250406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.522+0000 7f25177fe700 1 --2- 172.21.15.111:0/3155730549 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f25040682f0 0x7f250406a7a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f251815fc40 tx=0x7f2508010000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:47.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.525+0000 7f24fffff700 1 -- 172.21.15.111:0/3155730549 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f25100522b0 con 0x7f25180cc550 2023-11-28T12:59:47.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.710+0000 7f251dd3c700 1 -- 172.21.15.111:0/3155730549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f251815cd90 con 0x7f25180cc550 2023-11-28T12:59:47.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.710+0000 7f24fffff700 1 -- 172.21.15.111:0/3155730549 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2510087020 con 0x7f25180cc550 2023-11-28T12:59:47.712 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:59:47.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.712+0000 7f251dd3c700 1 -- 172.21.15.111:0/3155730549 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f25040682f0 msgr2=0x7f250406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.712+0000 7f251dd3c700 1 --2- 172.21.15.111:0/3155730549 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f25040682f0 0x7f250406a7a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f251815fc40 tx=0x7f2508010000 comp rx=0 tx=0).stop 2023-11-28T12:59:47.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.712+0000 7f251dd3c700 1 -- 172.21.15.111:0/3155730549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25180cc550 msgr2=0x7f2518149c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.712+0000 7f251dd3c700 1 --2- 172.21.15.111:0/3155730549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25180cc550 0x7f2518149c70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f251000cb50 tx=0x7f251000cf10 comp rx=0 tx=0).stop 2023-11-28T12:59:47.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.712+0000 7f251dd3c700 1 -- 172.21.15.111:0/3155730549 shutdown_connections 2023-11-28T12:59:47.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.712+0000 7f251dd3c700 1 --2- 172.21.15.111:0/3155730549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25180cc550 0x7f2518149c70 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.712+0000 7f251dd3c700 1 --2- 172.21.15.111:0/3155730549 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f25180b0d70 0x7f251815b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.712+0000 7f251dd3c700 1 --2- 172.21.15.111:0/3155730549 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f25040682f0 0x7f250406a7a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.712+0000 7f251dd3c700 1 --2- 172.21.15.111:0/3155730549 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f25180b1140 0x7f2518149730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.712+0000 7f251dd3c700 1 -- 172.21.15.111:0/3155730549 >> 172.21.15.111:0/3155730549 conn(0x7f2518098ee0 msgr2=0x7f25180af760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:47.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.712+0000 7f251dd3c700 1 -- 172.21.15.111:0/3155730549 shutdown_connections 2023-11-28T12:59:47.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.712+0000 7f251dd3c700 1 -- 172.21.15.111:0/3155730549 wait complete. 2023-11-28T12:59:47.726 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:59:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.847+0000 7f73e3533700 1 Processor -- start 2023-11-28T12:59:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.847+0000 7f73e3533700 1 -- start start 2023-11-28T12:59:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73e3533700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73dc0b0440 0x7f73dc0b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73e3533700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f73dc0ca2f0 0x7f73dc0ca770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73e3533700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73dc0cc750 con 0x7f73dc15c6c0 2023-11-28T12:59:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73e3533700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73dc0b0260 con 0x7f73dc0ca2f0 2023-11-28T12:59:47.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73e3533700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc0b0df0 con 0x7f73dc0b0440 2023-11-28T12:59:47.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73e12cf700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f73dc15c6c0 0x7f73dc0afb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35150/0 (socket says 172.21.15.111:35150) 2023-11-28T12:59:47.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73e12cf700 1 -- 172.21.15.111:0/3136428981 learned_addr learned my addr 172.21.15.111:0/3136428981 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:47.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73e0ace700 1 --2- 172.21.15.111:0/3136428981 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73dc0b0440 0x7f73dc0b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73e1ad0700 1 --2- 172.21.15.111:0/3136428981 >> v2:172.21.15.149:3300/0 conn(0x7f73dc0ca2f0 0x7f73dc0ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73d3fff700 1 -- 172.21.15.111:0/3136428981 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342457198 0 0) 0x7f73dc0cc750 con 0x7f73dc15c6c0 2023-11-28T12:59:47.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73d3fff700 1 -- 172.21.15.111:0/3136428981 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73c80035b0 con 0x7f73dc15c6c0 2023-11-28T12:59:47.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73e0ace700 1 -- 172.21.15.111:0/3136428981 >> v2:172.21.15.149:3300/0 conn(0x7f73dc0ca2f0 msgr2=0x7f73dc0ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:47.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73e0ace700 1 --2- 172.21.15.111:0/3136428981 >> v2:172.21.15.149:3300/0 conn(0x7f73dc0ca2f0 0x7f73dc0ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73e0ace700 1 -- 172.21.15.111:0/3136428981 >> v1:172.21.15.111:6789/0 conn(0x7f73dc15c6c0 legacy=0x7f73dc0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.848+0000 7f73e0ace700 1 -- 172.21.15.111:0/3136428981 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73dc0cca30 con 0x7f73dc0b0440 2023-11-28T12:59:47.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.849+0000 7f73e0ace700 1 --2- 172.21.15.111:0/3136428981 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73dc0b0440 0x7f73dc0b08b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009840 tx=0x7f73cc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc9ee05392bb4713 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:47.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.849+0000 7f73d3fff700 1 -- 172.21.15.111:0/3136428981 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc011830 con 0x7f73dc0b0440 2023-11-28T12:59:47.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.849+0000 7f73d3fff700 1 -- 172.21.15.111:0/3136428981 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73cc011990 con 0x7f73dc0b0440 2023-11-28T12:59:47.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.849+0000 7f73d3fff700 1 -- 172.21.15.111:0/3136428981 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc011d00 con 0x7f73dc0b0440 2023-11-28T12:59:47.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.849+0000 7f73e3533700 1 -- 172.21.15.111:0/3136428981 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73dc0b0440 msgr2=0x7f73dc0b08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.849+0000 7f73e3533700 1 --2- 172.21.15.111:0/3136428981 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73dc0b0440 0x7f73dc0b08b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009840 tx=0x7f73cc009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:47.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.849+0000 7f73e3533700 1 -- 172.21.15.111:0/3136428981 shutdown_connections 2023-11-28T12:59:47.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.849+0000 7f73e3533700 1 --2- 172.21.15.111:0/3136428981 >> v2:172.21.15.149:3300/0 conn(0x7f73dc0ca2f0 0x7f73dc0ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.849+0000 7f73e3533700 1 --2- 172.21.15.111:0/3136428981 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73dc0b0440 0x7f73dc0b08b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.849+0000 7f73e3533700 1 -- 172.21.15.111:0/3136428981 >> 172.21.15.111:0/3136428981 conn(0x7f73dc098ee0 msgr2=0x7f73dc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:47.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.849+0000 7f73e3533700 1 -- 172.21.15.111:0/3136428981 shutdown_connections 2023-11-28T12:59:47.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.849+0000 7f73e3533700 1 -- 172.21.15.111:0/3136428981 wait complete. 2023-11-28T12:59:47.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.851+0000 7f73e3533700 1 Processor -- start 2023-11-28T12:59:47.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.851+0000 7f73e3533700 1 -- start start 2023-11-28T12:59:47.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e3533700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f73dc0b0440 0x7f73dc154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e3533700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0ca2f0 0x7f73dc15d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e3533700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73dc15c6c0 0x7f73dc15dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e3533700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc0c8040 con 0x7f73dc0ca2f0 2023-11-28T12:59:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e3533700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc0c7d60 con 0x7f73dc0b0440 2023-11-28T12:59:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e3533700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc0c8860 con 0x7f73dc15c6c0 2023-11-28T12:59:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e0ace700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0ca2f0 0x7f73dc15d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e0ace700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0ca2f0 0x7f73dc15d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34380/0 (socket says 172.21.15.111:34380) 2023-11-28T12:59:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e0ace700 1 -- 172.21.15.111:0/995420493 learned_addr learned my addr 172.21.15.111:0/995420493 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e12cf700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f73dc0b0440 0x7f73dc154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e1ad0700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73dc15c6c0 0x7f73dc15dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e0ace700 1 -- 172.21.15.111:0/995420493 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73dc15c6c0 msgr2=0x7f73dc15dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e0ace700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73dc15c6c0 0x7f73dc15dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e0ace700 1 -- 172.21.15.111:0/995420493 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f73dc0b0440 msgr2=0x7f73dc154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:47.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e0ace700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f73dc0b0440 0x7f73dc154810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e0ace700 1 -- 172.21.15.111:0/995420493 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc0094a0 con 0x7f73dc0ca2f0 2023-11-28T12:59:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e12cf700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f73dc0b0440 0x7f73dc154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e1ad0700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73dc15c6c0 0x7f73dc15dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.852+0000 7f73e0ace700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0ca2f0 0x7f73dc15d9a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f73cc0090b0 tx=0x7f73cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.853+0000 7f73d1ffb700 1 -- 172.21.15.111:0/995420493 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc0036a0 con 0x7f73dc0ca2f0 2023-11-28T12:59:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.853+0000 7f73d1ffb700 1 -- 172.21.15.111:0/995420493 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73cc01e070 con 0x7f73dc0ca2f0 2023-11-28T12:59:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.853+0000 7f73e3533700 1 -- 172.21.15.111:0/995420493 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73dc0ccff0 con 0x7f73dc0ca2f0 2023-11-28T12:59:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.853+0000 7f73d1ffb700 1 -- 172.21.15.111:0/995420493 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc003a20 con 0x7f73dc0ca2f0 2023-11-28T12:59:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.853+0000 7f73e3533700 1 -- 172.21.15.111:0/995420493 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73dc0ccd10 con 0x7f73dc0ca2f0 2023-11-28T12:59:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.854+0000 7f73e3533700 1 -- 172.21.15.111:0/995420493 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73dc0cd2d0 con 0x7f73dc0ca2f0 2023-11-28T12:59:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.854+0000 7f73d1ffb700 1 -- 172.21.15.111:0/995420493 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f73cc00f050 con 0x7f73dc0ca2f0 2023-11-28T12:59:47.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.855+0000 7f73d1ffb700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f73c00682f0 0x7f73c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:47.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.855+0000 7f73e12cf700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f73c00682f0 0x7f73c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:47.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.855+0000 7f73d1ffb700 1 -- 172.21.15.111:0/995420493 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f73cc086ff0 con 0x7f73dc0ca2f0 2023-11-28T12:59:47.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.855+0000 7f73e12cf700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f73c00682f0 0x7f73c006a7a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f73d8005910 tx=0x7f73d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:47.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:47.859+0000 7f73d1ffb700 1 -- 172.21.15.111:0/995420493 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f73cc0534a0 con 0x7f73dc0ca2f0 2023-11-28T12:59:48.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.041+0000 7f73e3533700 1 -- 172.21.15.111:0/995420493 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f73dc0cdf80 con 0x7f73dc0ca2f0 2023-11-28T12:59:48.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.041+0000 7f73d1ffb700 1 -- 172.21.15.111:0/995420493 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7f73cc020020 con 0x7f73dc0ca2f0 2023-11-28T12:59:48.043 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:59:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.043+0000 7f73e3533700 1 -- 172.21.15.111:0/995420493 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f73c00682f0 msgr2=0x7f73c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.043+0000 7f73e3533700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f73c00682f0 0x7f73c006a7a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f73d8005910 tx=0x7f73d8009040 comp rx=0 tx=0).stop 2023-11-28T12:59:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.043+0000 7f73e3533700 1 -- 172.21.15.111:0/995420493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0ca2f0 msgr2=0x7f73dc15d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.043+0000 7f73e3533700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0ca2f0 0x7f73dc15d9a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f73cc0090b0 tx=0x7f73cc004750 comp rx=0 tx=0).stop 2023-11-28T12:59:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.043+0000 7f73e3533700 1 -- 172.21.15.111:0/995420493 shutdown_connections 2023-11-28T12:59:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.043+0000 7f73e3533700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f73dc15c6c0 0x7f73dc15dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.043+0000 7f73e3533700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f73c00682f0 0x7f73c006a7a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.043+0000 7f73e3533700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f73dc0b0440 0x7f73dc154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.043+0000 7f73e3533700 1 --2- 172.21.15.111:0/995420493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0ca2f0 0x7f73dc15d9a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.043+0000 7f73e3533700 1 -- 172.21.15.111:0/995420493 >> 172.21.15.111:0/995420493 conn(0x7f73dc098ee0 msgr2=0x7f73dc06ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:48.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.043+0000 7f73e3533700 1 -- 172.21.15.111:0/995420493 shutdown_connections 2023-11-28T12:59:48.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.043+0000 7f73e3533700 1 -- 172.21.15.111:0/995420493 wait complete. 2023-11-28T12:59:48.056 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T12:59:48.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.177+0000 7fb89bd44700 1 Processor -- start 2023-11-28T12:59:48.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.177+0000 7fb89bd44700 1 -- start start 2023-11-28T12:59:48.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.178+0000 7fb89bd44700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8940cc550 0x7fb8940cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.178+0000 7fb89bd44700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89415cd90 con 0x7fb8940b0d70 2023-11-28T12:59:48.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.178+0000 7fb89bd44700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8941613e0 con 0x7fb8941604b0 2023-11-28T12:59:48.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.178+0000 7fb89bd44700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb894160950 con 0x7fb8940cc550 2023-11-28T12:59:48.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.178+0000 7fb899ae0700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb8940b0d70 0x7fb8940b1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35164/0 (socket says 172.21.15.111:35164) 2023-11-28T12:59:48.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.178+0000 7fb899ae0700 1 -- 172.21.15.111:0/3592817399 learned_addr learned my addr 172.21.15.111:0/3592817399 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:48.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.178+0000 7fb89a2e1700 1 --2- 172.21.15.111:0/3592817399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8940cc550 0x7fb8940cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.178+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226819762 0 0) 0x7fb884001290 con 0x7fb8940b0d70 2023-11-28T12:59:48.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.178+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8800035b0 con 0x7fb8940b0d70 2023-11-28T12:59:48.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1570823016 0 0) 0x7fb8800035b0 con 0x7fb8940b0d70 2023-11-28T12:59:48.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8940cc550 msgr2=0x7fb8940cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:48.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 --2- 172.21.15.111:0/3592817399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8940cc550 0x7fb8940cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 >> v1:172.21.15.149:6789/0 conn(0x7fb8941604b0 legacy=0x7fb8940ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb894160b40 con 0x7fb8940b0d70 2023-11-28T12:59:48.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fb884001980 con 0x7fb8940b0d70 2023-11-28T12:59:48.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 >> v1:172.21.15.111:6789/0 conn(0x7fb8940b0d70 legacy=0x7fb8940b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 --2- 172.21.15.111:0/3592817399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8940cc550 0x7fb880003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 --2- 172.21.15.111:0/3592817399 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb880003fb0 0x7fb880004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 --2- 172.21.15.111:0/3592817399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb880004d90 0x7fb880005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88000bac0 con 0x7fb880004d90 2023-11-28T12:59:48.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88000bc30 con 0x7fb880003fb0 2023-11-28T12:59:48.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88000bdf0 con 0x7fb8940cc550 2023-11-28T12:59:48.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb89a2e1700 1 --2- 172.21.15.111:0/3592817399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb880004d90 0x7fb880005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb8992df700 1 --2- 172.21.15.111:0/3592817399 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb880003fb0 0x7fb880004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb899ae0700 1 --2- 172.21.15.111:0/3592817399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8940cc550 0x7fb880003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb89a2e1700 1 -- 172.21.15.111:0/3592817399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8940cc550 msgr2=0x7fb880003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:48.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb89a2e1700 1 --2- 172.21.15.111:0/3592817399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8940cc550 0x7fb880003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb89a2e1700 1 -- 172.21.15.111:0/3592817399 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb880003fb0 msgr2=0x7fb880004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:48.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb89a2e1700 1 --2- 172.21.15.111:0/3592817399 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb880003fb0 0x7fb880004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb89a2e1700 1 -- 172.21.15.111:0/3592817399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8800035b0 con 0x7fb880004d90 2023-11-28T12:59:48.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb8992df700 1 --2- 172.21.15.111:0/3592817399 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb880003fb0 0x7fb880004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:48.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb89a2e1700 1 --2- 172.21.15.111:0/3592817399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb880004d90 0x7fb880005200 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009f90 tx=0x7fb880005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae7cbd8645219ed server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:48.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c00ec30 con 0x7fb880004d90 2023-11-28T12:59:48.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c00f040 con 0x7fb880004d90 2023-11-28T12:59:48.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.179+0000 7fb898ade700 1 -- 172.21.15.111:0/3592817399 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c014480 con 0x7fb880004d90 2023-11-28T12:59:48.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.180+0000 7fb89bd44700 1 -- 172.21.15.111:0/3592817399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb880004d90 msgr2=0x7fb880005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.180+0000 7fb89bd44700 1 --2- 172.21.15.111:0/3592817399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb880004d90 0x7fb880005200 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009f90 tx=0x7fb880005880 comp rx=0 tx=0).stop 2023-11-28T12:59:48.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.180+0000 7fb89a2e1700 1 -- 172.21.15.111:0/3592817399 reap_dead start 2023-11-28T12:59:48.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.180+0000 7fb89bd44700 1 -- 172.21.15.111:0/3592817399 shutdown_connections 2023-11-28T12:59:48.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.180+0000 7fb89bd44700 1 -- 172.21.15.111:0/3592817399 >> 172.21.15.111:0/3592817399 conn(0x7fb894098ee0 msgr2=0x7fb89406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:48.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.180+0000 7fb89bd44700 1 -- 172.21.15.111:0/3592817399 shutdown_connections 2023-11-28T12:59:48.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.180+0000 7fb89bd44700 1 -- 172.21.15.111:0/3592817399 wait complete. 2023-11-28T12:59:48.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.181+0000 7fb89bd44700 1 Processor -- start 2023-11-28T12:59:48.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.181+0000 7fb89bd44700 1 -- start start 2023-11-28T12:59:48.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb89bd44700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb89414af70 0x7fb89415fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb89bd44700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8940c4610 0x7fb89415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb89bd44700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8940c5c20 0x7fb89415d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb89bd44700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940c4b80 con 0x7fb8940c4610 2023-11-28T12:59:48.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb89bd44700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940c4e60 con 0x7fb89414af70 2023-11-28T12:59:48.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb89bd44700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8940c5140 con 0x7fb8940c5c20 2023-11-28T12:59:48.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb8992df700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8940c4610 0x7fb89415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb8992df700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8940c4610 0x7fb89415cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34400/0 (socket says 172.21.15.111:34400) 2023-11-28T12:59:48.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb8992df700 1 -- 172.21.15.111:0/1274109785 learned_addr learned my addr 172.21.15.111:0/1274109785 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:48.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb899ae0700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb89414af70 0x7fb89415fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb89a2e1700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8940c5c20 0x7fb89415d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb8992df700 1 -- 172.21.15.111:0/1274109785 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8940c5c20 msgr2=0x7fb89415d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb8992df700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8940c5c20 0x7fb89415d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb8992df700 1 -- 172.21.15.111:0/1274109785 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb89414af70 msgr2=0x7fb89415fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb8992df700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb89414af70 0x7fb89415fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb8992df700 1 -- 172.21.15.111:0/1274109785 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb894161100 con 0x7fb8940c4610 2023-11-28T12:59:48.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.182+0000 7fb8992df700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8940c4610 0x7fb89415cf30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb89000e7b0 tx=0x7fb89000eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:48.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.183+0000 7fb88a7fc700 1 -- 172.21.15.111:0/1274109785 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8900052e0 con 0x7fb8940c4610 2023-11-28T12:59:48.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.183+0000 7fb88a7fc700 1 -- 172.21.15.111:0/1274109785 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb890005440 con 0x7fb8940c4610 2023-11-28T12:59:48.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.183+0000 7fb89bd44700 1 -- 172.21.15.111:0/1274109785 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb894160e20 con 0x7fb8940c4610 2023-11-28T12:59:48.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.183+0000 7fb88a7fc700 1 -- 172.21.15.111:0/1274109785 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb890003880 con 0x7fb8940c4610 2023-11-28T12:59:48.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.183+0000 7fb89bd44700 1 -- 172.21.15.111:0/1274109785 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8940af340 con 0x7fb8940c4610 2023-11-28T12:59:48.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.184+0000 7fb89bd44700 1 -- 172.21.15.111:0/1274109785 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb894067170 con 0x7fb8940c4610 2023-11-28T12:59:48.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.184+0000 7fb88a7fc700 1 -- 172.21.15.111:0/1274109785 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb8900039e0 con 0x7fb8940c4610 2023-11-28T12:59:48.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.184+0000 7fb88a7fc700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8780682f0 0x7fb87806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.184+0000 7fb899ae0700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8780682f0 0x7fb87806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.184+0000 7fb88a7fc700 1 -- 172.21.15.111:0/1274109785 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fb890085520 con 0x7fb8940c4610 2023-11-28T12:59:48.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.185+0000 7fb899ae0700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8780682f0 0x7fb87806a7a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb8840011d0 tx=0x7fb884009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:48.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.188+0000 7fb88a7fc700 1 -- 172.21.15.111:0/1274109785 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb890051a50 con 0x7fb8940c4610 2023-11-28T12:59:48.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.370+0000 7fb89bd44700 1 -- 172.21.15.111:0/1274109785 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb89406d9a0 con 0x7fb8940c4610 2023-11-28T12:59:48.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.371+0000 7fb88a7fc700 1 -- 172.21.15.111:0/1274109785 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb890055070 con 0x7fb8940c4610 2023-11-28T12:59:48.372 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:48.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.373+0000 7fb89bd44700 1 -- 172.21.15.111:0/1274109785 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8780682f0 msgr2=0x7fb87806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.373+0000 7fb89bd44700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8780682f0 0x7fb87806a7a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb8840011d0 tx=0x7fb884009040 comp rx=0 tx=0).stop 2023-11-28T12:59:48.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.373+0000 7fb89bd44700 1 -- 172.21.15.111:0/1274109785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8940c4610 msgr2=0x7fb89415cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.373+0000 7fb89bd44700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8940c4610 0x7fb89415cf30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb89000e7b0 tx=0x7fb89000eac0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.373+0000 7fb89bd44700 1 -- 172.21.15.111:0/1274109785 shutdown_connections 2023-11-28T12:59:48.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.373+0000 7fb89bd44700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8940c5c20 0x7fb89415d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.373+0000 7fb89bd44700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8780682f0 0x7fb87806a7a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.373+0000 7fb89bd44700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb89414af70 0x7fb89415fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.373+0000 7fb89bd44700 1 --2- 172.21.15.111:0/1274109785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8940c4610 0x7fb89415cf30 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.373+0000 7fb89bd44700 1 -- 172.21.15.111:0/1274109785 >> 172.21.15.111:0/1274109785 conn(0x7fb894098ee0 msgr2=0x7fb8940af760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:48.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.373+0000 7fb89bd44700 1 -- 172.21.15.111:0/1274109785 shutdown_connections 2023-11-28T12:59:48.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.373+0000 7fb89bd44700 1 -- 172.21.15.111:0/1274109785 wait complete. 2023-11-28T12:59:48.386 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T12:59:48.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.509+0000 7fd11164b700 1 Processor -- start 2023-11-28T12:59:48.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.509+0000 7fd11164b700 1 -- start start 2023-11-28T12:59:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.510+0000 7fd11164b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd10c0ce850 0x7fd10c0cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.510+0000 7fd11164b700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd10c15a920 0x7fd10c0cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.510+0000 7fd11164b700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd10c0ad8a0 con 0x7fd10c0cb810 2023-11-28T12:59:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.510+0000 7fd11164b700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd10c0cf160 con 0x7fd10c15a920 2023-11-28T12:59:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.510+0000 7fd11164b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0adaf0 con 0x7fd10c0ce850 2023-11-28T12:59:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.510+0000 7fd10b7fe700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7fd10c0cb810 0x7fd10c0cbc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:43730/0 (socket says 172.21.15.111:43730) 2023-11-28T12:59:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.510+0000 7fd10b7fe700 1 -- 172.21.15.111:0/3613931064 learned_addr learned my addr 172.21.15.111:0/3613931064 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.510+0000 7fd10affd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd10c0ce850 0x7fd10c0cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.510+0000 7fd10a7fc700 1 --2- 172.21.15.111:0/3613931064 >> v2:172.21.15.149:3300/0 conn(0x7fd10c15a920 0x7fd10c0cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.511+0000 7fd10a7fc700 1 -- 172.21.15.111:0/3613931064 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd10c0ce850 msgr2=0x7fd10c0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:48.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.511+0000 7fd10a7fc700 1 --2- 172.21.15.111:0/3613931064 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd10c0ce850 0x7fd10c0cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.511+0000 7fd10a7fc700 1 -- 172.21.15.111:0/3613931064 >> v1:172.21.15.149:6789/0 conn(0x7fd10c0cb810 legacy=0x7fd10c0cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.511+0000 7fd10a7fc700 1 -- 172.21.15.111:0/3613931064 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10c0addd0 con 0x7fd10c15a920 2023-11-28T12:59:48.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.511+0000 7fd10affd700 1 --2- 172.21.15.111:0/3613931064 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd10c0ce850 0x7fd10c0cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T12:59:48.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.511+0000 7fd10a7fc700 1 --2- 172.21.15.111:0/3613931064 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd10c15a920 0x7fd10c0cb2d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd100009840 tx=0x7fd100009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3810fad2e54336cd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:48.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.511+0000 7fd109ffb700 1 -- 172.21.15.111:0/3613931064 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100011830 con 0x7fd10c15a920 2023-11-28T12:59:48.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.511+0000 7fd109ffb700 1 -- 172.21.15.111:0/3613931064 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd100011990 con 0x7fd10c15a920 2023-11-28T12:59:48.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.511+0000 7fd109ffb700 1 -- 172.21.15.111:0/3613931064 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100011d20 con 0x7fd10c15a920 2023-11-28T12:59:48.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.512+0000 7fd11164b700 1 -- 172.21.15.111:0/3613931064 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd10c15a920 msgr2=0x7fd10c0cb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.512+0000 7fd11164b700 1 --2- 172.21.15.111:0/3613931064 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd10c15a920 0x7fd10c0cb2d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd100009840 tx=0x7fd100009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:48.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.512+0000 7fd11164b700 1 -- 172.21.15.111:0/3613931064 shutdown_connections 2023-11-28T12:59:48.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.512+0000 7fd11164b700 1 --2- 172.21.15.111:0/3613931064 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd10c0ce850 0x7fd10c0cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.512+0000 7fd11164b700 1 --2- 172.21.15.111:0/3613931064 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd10c15a920 0x7fd10c0cb2d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.512+0000 7fd11164b700 1 -- 172.21.15.111:0/3613931064 >> 172.21.15.111:0/3613931064 conn(0x7fd10c09a040 msgr2=0x7fd10c0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:48.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.512+0000 7fd11164b700 1 -- 172.21.15.111:0/3613931064 shutdown_connections 2023-11-28T12:59:48.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.512+0000 7fd11164b700 1 -- 172.21.15.111:0/3613931064 wait complete. 2023-11-28T12:59:48.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.513+0000 7fd11164b700 1 Processor -- start 2023-11-28T12:59:48.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.513+0000 7fd11164b700 1 -- start start 2023-11-28T12:59:48.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd11164b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd10c0cb810 0x7fd10c0b5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd11164b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd10c0ce850 0x7fd10c0c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd11164b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd10c0c5790 0x7fd10c0b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd11164b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0c9cd0 con 0x7fd10c0ce850 2023-11-28T12:59:48.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd11164b700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0b6120 con 0x7fd10c0cb810 2023-11-28T12:59:48.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd11164b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0c9330 con 0x7fd10c0c5790 2023-11-28T12:59:48.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd10a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd10c0ce850 0x7fd10c0c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd10a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd10c0ce850 0x7fd10c0c5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34414/0 (socket says 172.21.15.111:34414) 2023-11-28T12:59:48.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd10a7fc700 1 -- 172.21.15.111:0/3844759485 learned_addr learned my addr 172.21.15.111:0/3844759485 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:48.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd10affd700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd10c0cb810 0x7fd10c0b5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd10b7fe700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd10c0c5790 0x7fd10c0b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd10a7fc700 1 -- 172.21.15.111:0/3844759485 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd10c0c5790 msgr2=0x7fd10c0b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd10a7fc700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd10c0c5790 0x7fd10c0b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd10a7fc700 1 -- 172.21.15.111:0/3844759485 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd10c0cb810 msgr2=0x7fd10c0b5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd10a7fc700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd10c0cb810 0x7fd10c0b5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd10a7fc700 1 -- 172.21.15.111:0/3844759485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc0094a0 con 0x7fd10c0ce850 2023-11-28T12:59:48.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd10a7fc700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd10c0ce850 0x7fd10c0c5250 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd100009fd0 tx=0x7fd100004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:48.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd0f37fe700 1 -- 172.21.15.111:0/3844759485 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1000036a0 con 0x7fd10c0ce850 2023-11-28T12:59:48.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.514+0000 7fd0f37fe700 1 -- 172.21.15.111:0/3844759485 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10001e070 con 0x7fd10c0ce850 2023-11-28T12:59:48.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.515+0000 7fd11164b700 1 -- 172.21.15.111:0/3844759485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1000094a0 con 0x7fd10c0ce850 2023-11-28T12:59:48.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.515+0000 7fd0f37fe700 1 -- 172.21.15.111:0/3844759485 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100003a60 con 0x7fd10c0ce850 2023-11-28T12:59:48.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.515+0000 7fd11164b700 1 -- 172.21.15.111:0/3844759485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10c0ae390 con 0x7fd10c0ce850 2023-11-28T12:59:48.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.515+0000 7fd11164b700 1 -- 172.21.15.111:0/3844759485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd10c0ae0b0 con 0x7fd10c0ce850 2023-11-28T12:59:48.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.516+0000 7fd0f37fe700 1 -- 172.21.15.111:0/3844759485 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd10000f050 con 0x7fd10c0ce850 2023-11-28T12:59:48.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.516+0000 7fd0f37fe700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd0f80682f0 0x7fd0f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.516+0000 7fd0f37fe700 1 -- 172.21.15.111:0/3844759485 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fd100086f40 con 0x7fd10c0ce850 2023-11-28T12:59:48.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.516+0000 7fd10affd700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd0f80682f0 0x7fd0f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.517+0000 7fd10affd700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd0f80682f0 0x7fd0f806a7a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd10c06b4f0 tx=0x7fd0fc009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:48.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.520+0000 7fd0f37fe700 1 -- 172.21.15.111:0/3844759485 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd1000b85e0 con 0x7fd10c0ce850 2023-11-28T12:59:48.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.701+0000 7fd11164b700 1 -- 172.21.15.111:0/3844759485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd10c0ae670 con 0x7fd10c0ce850 2023-11-28T12:59:48.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.701+0000 7fd0f37fe700 1 -- 172.21.15.111:0/3844759485 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd100020020 con 0x7fd10c0ce850 2023-11-28T12:59:48.703 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T12:59:48.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.703+0000 7fd11164b700 1 -- 172.21.15.111:0/3844759485 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd0f80682f0 msgr2=0x7fd0f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.703+0000 7fd11164b700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd0f80682f0 0x7fd0f806a7a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd10c06b4f0 tx=0x7fd0fc009210 comp rx=0 tx=0).stop 2023-11-28T12:59:48.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.703+0000 7fd11164b700 1 -- 172.21.15.111:0/3844759485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd10c0ce850 msgr2=0x7fd10c0c5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.703+0000 7fd11164b700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd10c0ce850 0x7fd10c0c5250 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd100009fd0 tx=0x7fd100004750 comp rx=0 tx=0).stop 2023-11-28T12:59:48.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.703+0000 7fd11164b700 1 -- 172.21.15.111:0/3844759485 shutdown_connections 2023-11-28T12:59:48.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.703+0000 7fd11164b700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd10c0c5790 0x7fd10c0b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.703+0000 7fd11164b700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd0f80682f0 0x7fd0f806a7a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.703+0000 7fd11164b700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd10c0cb810 0x7fd10c0b5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.703+0000 7fd11164b700 1 --2- 172.21.15.111:0/3844759485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd10c0ce850 0x7fd10c0c5250 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.703+0000 7fd11164b700 1 -- 172.21.15.111:0/3844759485 >> 172.21.15.111:0/3844759485 conn(0x7fd10c09a040 msgr2=0x7fd10c159250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:48.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.703+0000 7fd11164b700 1 -- 172.21.15.111:0/3844759485 shutdown_connections 2023-11-28T12:59:48.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.703+0000 7fd11164b700 1 -- 172.21.15.111:0/3844759485 wait complete. 2023-11-28T12:59:48.717 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T12:59:48.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.838+0000 7f4698a8b700 1 Processor -- start 2023-11-28T12:59:48.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.838+0000 7f4698a8b700 1 -- start start 2023-11-28T12:59:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.839+0000 7f4698a8b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f46940aecc0 0x7f46940b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.839+0000 7f4698a8b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46940ad420 con 0x7f46940cdd90 2023-11-28T12:59:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.839+0000 7f4698a8b700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46940ac810 con 0x7f4694159960 2023-11-28T12:59:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.839+0000 7f4698a8b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4694159d30 con 0x7f46940aecc0 2023-11-28T12:59:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.840+0000 7f4692d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f46940aecc0 0x7f46940b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.840+0000 7f4692d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f46940aecc0 0x7f46940b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34424/0 (socket says 172.21.15.111:34424) 2023-11-28T12:59:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.840+0000 7f4692d9d700 1 -- 172.21.15.111:0/118460700 learned_addr learned my addr 172.21.15.111:0/118460700 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.840+0000 7f4691d9b700 1 --1- 172.21.15.111:0/118460700 >> v1:172.21.15.111:6789/0 conn(0x7f46940cdd90 0x7f46940ad8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35198/0 (socket says 172.21.15.111:35198) 2023-11-28T12:59:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.840+0000 7f4692d9d700 1 -- 172.21.15.111:0/118460700 >> v1:172.21.15.149:6789/0 conn(0x7f4694159960 legacy=0x7f46940cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.840+0000 7f4692d9d700 1 -- 172.21.15.111:0/118460700 >> v1:172.21.15.111:6789/0 conn(0x7f46940cdd90 legacy=0x7f46940ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.840+0000 7f4692d9d700 1 -- 172.21.15.111:0/118460700 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46940acaf0 con 0x7f46940aecc0 2023-11-28T12:59:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.840+0000 7f469159a700 1 -- 172.21.15.111:0/118460700 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337034384 0 0) 0x7f46940ad420 con 0x7f46940cdd90 2023-11-28T12:59:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.840+0000 7f4692d9d700 1 --2- 172.21.15.111:0/118460700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46940aecc0 0x7f46940b1fd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f468c009e50 tx=0x7f468c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121f6594ce0ef050 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.840+0000 7f469159a700 1 -- 172.21.15.111:0/118460700 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c00e070 con 0x7f46940aecc0 2023-11-28T12:59:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.840+0000 7f469159a700 1 -- 172.21.15.111:0/118460700 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c003a20 con 0x7f46940aecc0 2023-11-28T12:59:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.840+0000 7f469159a700 1 -- 172.21.15.111:0/118460700 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c003dd0 con 0x7f46940aecc0 2023-11-28T12:59:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.841+0000 7f4698a8b700 1 -- 172.21.15.111:0/118460700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46940aecc0 msgr2=0x7f46940b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.841+0000 7f4698a8b700 1 --2- 172.21.15.111:0/118460700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46940aecc0 0x7f46940b1fd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f468c009e50 tx=0x7f468c00b330 comp rx=0 tx=0).stop 2023-11-28T12:59:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.841+0000 7f4698a8b700 1 -- 172.21.15.111:0/118460700 shutdown_connections 2023-11-28T12:59:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.841+0000 7f4698a8b700 1 --2- 172.21.15.111:0/118460700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46940aecc0 0x7f46940b1fd0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.841+0000 7f4698a8b700 1 -- 172.21.15.111:0/118460700 >> 172.21.15.111:0/118460700 conn(0x7f4694098ee0 msgr2=0x7f469406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.841+0000 7f4698a8b700 1 -- 172.21.15.111:0/118460700 shutdown_connections 2023-11-28T12:59:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.841+0000 7f4698a8b700 1 -- 172.21.15.111:0/118460700 wait complete. 2023-11-28T12:59:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.842+0000 7f4698a8b700 1 Processor -- start 2023-11-28T12:59:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.842+0000 7f4698a8b700 1 -- start start 2023-11-28T12:59:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4698a8b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f46940aecc0 0x7f46941f0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4698a8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46940cdd90 0x7f4694157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4698a8b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4694159960 0x7f469415c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4698a8b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46941f0570 con 0x7f46940cdd90 2023-11-28T12:59:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4698a8b700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46940c3980 con 0x7f46940aecc0 2023-11-28T12:59:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4698a8b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46940c3c60 con 0x7f4694159960 2023-11-28T12:59:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4691d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46940cdd90 0x7f4694157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4691d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46940cdd90 0x7f4694157a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34428/0 (socket says 172.21.15.111:34428) 2023-11-28T12:59:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4691d9b700 1 -- 172.21.15.111:0/2407232019 learned_addr learned my addr 172.21.15.111:0/2407232019 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f469259c700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f46940aecc0 0x7f46941f0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4692d9d700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4694159960 0x7f469415c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4691d9b700 1 -- 172.21.15.111:0/2407232019 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4694159960 msgr2=0x7f469415c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4691d9b700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4694159960 0x7f469415c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4691d9b700 1 -- 172.21.15.111:0/2407232019 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f46940aecc0 msgr2=0x7f46941f0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4691d9b700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f46940aecc0 0x7f46941f0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4691d9b700 1 -- 172.21.15.111:0/2407232019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468c009b30 con 0x7f46940cdd90 2023-11-28T12:59:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4692d9d700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4694159960 0x7f469415c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.843+0000 7f4691d9b700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46940cdd90 0x7f4694157a30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f468400c540 tx=0x7f468400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.844+0000 7f4682ffd700 1 -- 172.21.15.111:0/2407232019 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684004c80 con 0x7f46940cdd90 2023-11-28T12:59:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.844+0000 7f4682ffd700 1 -- 172.21.15.111:0/2407232019 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4684004de0 con 0x7f46940cdd90 2023-11-28T12:59:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.844+0000 7f4698a8b700 1 -- 172.21.15.111:0/2407232019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46940ad0b0 con 0x7f46940cdd90 2023-11-28T12:59:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.844+0000 7f4682ffd700 1 -- 172.21.15.111:0/2407232019 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684007310 con 0x7f46940cdd90 2023-11-28T12:59:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.844+0000 7f4698a8b700 1 -- 172.21.15.111:0/2407232019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46940acdd0 con 0x7f46940cdd90 2023-11-28T12:59:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.845+0000 7f4698a8b700 1 -- 172.21.15.111:0/2407232019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46940adfc0 con 0x7f46940cdd90 2023-11-28T12:59:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.845+0000 7f4682ffd700 1 -- 172.21.15.111:0/2407232019 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f468401a020 con 0x7f46940cdd90 2023-11-28T12:59:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.845+0000 7f4682ffd700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f46740682f0 0x7f467406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.846+0000 7f4682ffd700 1 -- 172.21.15.111:0/2407232019 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f4684085b20 con 0x7f46940cdd90 2023-11-28T12:59:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.846+0000 7f469259c700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f46740682f0 0x7f467406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.846+0000 7f469259c700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f46740682f0 0x7f467406a7a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4688006fd0 tx=0x7f46880090e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:48.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:48.850+0000 7f4682ffd700 1 -- 172.21.15.111:0/2407232019 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4684052050 con 0x7f46940cdd90 2023-11-28T12:59:49.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.036+0000 7f4698a8b700 1 -- 172.21.15.111:0/2407232019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4694067170 con 0x7f46940cdd90 2023-11-28T12:59:49.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.037+0000 7f4682ffd700 1 -- 172.21.15.111:0/2407232019 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4684055670 con 0x7f46940cdd90 2023-11-28T12:59:49.038 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:49.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.039+0000 7f4698a8b700 1 -- 172.21.15.111:0/2407232019 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f46740682f0 msgr2=0x7f467406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:49.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.039+0000 7f4698a8b700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f46740682f0 0x7f467406a7a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4688006fd0 tx=0x7f46880090e0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.039+0000 7f4698a8b700 1 -- 172.21.15.111:0/2407232019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46940cdd90 msgr2=0x7f4694157a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:49.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.039+0000 7f4698a8b700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46940cdd90 0x7f4694157a30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f468400c540 tx=0x7f468400c850 comp rx=0 tx=0).stop 2023-11-28T12:59:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.039+0000 7f4698a8b700 1 -- 172.21.15.111:0/2407232019 shutdown_connections 2023-11-28T12:59:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.039+0000 7f4698a8b700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4694159960 0x7f469415c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.039+0000 7f4698a8b700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f46740682f0 0x7f467406a7a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.039+0000 7f4698a8b700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f46940aecc0 0x7f46941f0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.039+0000 7f4698a8b700 1 --2- 172.21.15.111:0/2407232019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46940cdd90 0x7f4694157a30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.039+0000 7f4698a8b700 1 -- 172.21.15.111:0/2407232019 >> 172.21.15.111:0/2407232019 conn(0x7f4694098ee0 msgr2=0x7f46940b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.039+0000 7f4698a8b700 1 -- 172.21.15.111:0/2407232019 shutdown_connections 2023-11-28T12:59:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.039+0000 7f4698a8b700 1 -- 172.21.15.111:0/2407232019 wait complete. 2023-11-28T12:59:49.053 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T12:59:49.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.172+0000 7effa582d700 1 Processor -- start 2023-11-28T12:59:49.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.172+0000 7effa582d700 1 -- start start 2023-11-28T12:59:49.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.173+0000 7effa582d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7effa015a5d0 0x7effa00ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.173+0000 7effa582d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7effa00b0160 0x7effa00ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.173+0000 7effa582d700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa00ae8b0 con 0x7effa00cb2d0 2023-11-28T12:59:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.173+0000 7effa582d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa00cc1b0 con 0x7effa00b0160 2023-11-28T12:59:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.173+0000 7effa582d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa00cc390 con 0x7effa015a5d0 2023-11-28T12:59:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.173+0000 7eff9e7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7effa00b0160 0x7effa00ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.173+0000 7eff9e7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7effa00b0160 0x7effa00ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34436/0 (socket says 172.21.15.111:34436) 2023-11-28T12:59:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.173+0000 7eff9e7fc700 1 -- 172.21.15.111:0/1709905645 learned_addr learned my addr 172.21.15.111:0/1709905645 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.174+0000 7eff9effd700 1 --2- 172.21.15.111:0/1709905645 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7effa015a5d0 0x7effa00ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.174+0000 7eff9e7fc700 1 -- 172.21.15.111:0/1709905645 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7effa015a5d0 msgr2=0x7effa00ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.174+0000 7eff9e7fc700 1 --2- 172.21.15.111:0/1709905645 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7effa015a5d0 0x7effa00ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.174+0000 7eff9e7fc700 1 -- 172.21.15.111:0/1709905645 >> v1:172.21.15.149:6789/0 conn(0x7effa00cb2d0 legacy=0x7effa015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.174+0000 7eff9e7fc700 1 -- 172.21.15.111:0/1709905645 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa00cc670 con 0x7effa00b0160 2023-11-28T12:59:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.174+0000 7eff9e7fc700 1 --2- 172.21.15.111:0/1709905645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa00b0160 0x7effa00ced90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7eff900098f0 tx=0x7eff90009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b29696c2aec7362 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.174+0000 7eff9dffb700 1 -- 172.21.15.111:0/1709905645 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90004230 con 0x7effa00b0160 2023-11-28T12:59:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.174+0000 7eff9dffb700 1 -- 172.21.15.111:0/1709905645 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff90004390 con 0x7effa00b0160 2023-11-28T12:59:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.174+0000 7eff9dffb700 1 -- 172.21.15.111:0/1709905645 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff900036a0 con 0x7effa00b0160 2023-11-28T12:59:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.174+0000 7effa582d700 1 -- 172.21.15.111:0/1709905645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa00b0160 msgr2=0x7effa00ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.174+0000 7effa582d700 1 --2- 172.21.15.111:0/1709905645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa00b0160 0x7effa00ced90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7eff900098f0 tx=0x7eff90009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.175+0000 7effa582d700 1 -- 172.21.15.111:0/1709905645 shutdown_connections 2023-11-28T12:59:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.175+0000 7effa582d700 1 --2- 172.21.15.111:0/1709905645 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7effa015a5d0 0x7effa00ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.175+0000 7effa582d700 1 --2- 172.21.15.111:0/1709905645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa00b0160 0x7effa00ced90 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.175+0000 7effa582d700 1 -- 172.21.15.111:0/1709905645 >> 172.21.15.111:0/1709905645 conn(0x7effa009a040 msgr2=0x7effa00d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:49.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.175+0000 7effa582d700 1 -- 172.21.15.111:0/1709905645 shutdown_connections 2023-11-28T12:59:49.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.175+0000 7effa582d700 1 -- 172.21.15.111:0/1709905645 wait complete. 2023-11-28T12:59:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.176+0000 7effa582d700 1 Processor -- start 2023-11-28T12:59:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.176+0000 7effa582d700 1 -- start start 2023-11-28T12:59:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7effa582d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7effa00b0160 0x7effa0155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7effa582d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa00cb2d0 0x7effa00b6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7effa582d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7effa015a5d0 0x7effa014cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7effa582d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa00c9460 con 0x7effa00cb2d0 2023-11-28T12:59:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7effa582d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa00c96d0 con 0x7effa00b0160 2023-11-28T12:59:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7effa582d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa00c99b0 con 0x7effa015a5d0 2023-11-28T12:59:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff9e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa00cb2d0 0x7effa00b6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff9e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa00cb2d0 0x7effa00b6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34442/0 (socket says 172.21.15.111:34442) 2023-11-28T12:59:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff9e7fc700 1 -- 172.21.15.111:0/1029184274 learned_addr learned my addr 172.21.15.111:0/1029184274 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff9f7fe700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7effa015a5d0 0x7effa014cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff9effd700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7effa00b0160 0x7effa0155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff9e7fc700 1 -- 172.21.15.111:0/1029184274 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7effa015a5d0 msgr2=0x7effa014cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff9e7fc700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7effa015a5d0 0x7effa014cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff9e7fc700 1 -- 172.21.15.111:0/1029184274 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7effa00b0160 msgr2=0x7effa0155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:49.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff9e7fc700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7effa00b0160 0x7effa0155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff9e7fc700 1 -- 172.21.15.111:0/1029184274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff90009550 con 0x7effa00cb2d0 2023-11-28T12:59:49.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff9f7fe700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7effa015a5d0 0x7effa014cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:49.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff9e7fc700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa00cb2d0 0x7effa00b6330 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7eff90009130 tx=0x7eff90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:49.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff87fff700 1 -- 172.21.15.111:0/1029184274 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90003e30 con 0x7effa00cb2d0 2023-11-28T12:59:49.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff87fff700 1 -- 172.21.15.111:0/1029184274 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9001f070 con 0x7effa00cb2d0 2023-11-28T12:59:49.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7effa582d700 1 -- 172.21.15.111:0/1029184274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa00ccc30 con 0x7effa00cb2d0 2023-11-28T12:59:49.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7eff87fff700 1 -- 172.21.15.111:0/1029184274 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9000b8a0 con 0x7effa00cb2d0 2023-11-28T12:59:49.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.177+0000 7effa582d700 1 -- 172.21.15.111:0/1029184274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa00cc950 con 0x7effa00cb2d0 2023-11-28T12:59:49.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.178+0000 7effa582d700 1 -- 172.21.15.111:0/1029184274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa00ccf10 con 0x7effa00cb2d0 2023-11-28T12:59:49.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.179+0000 7eff87fff700 1 -- 172.21.15.111:0/1029184274 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7eff9002a020 con 0x7effa00cb2d0 2023-11-28T12:59:49.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.179+0000 7eff87fff700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7eff8c0682f0 0x7eff8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:49.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.179+0000 7eff9effd700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7eff8c0682f0 0x7eff8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:49.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.179+0000 7eff87fff700 1 -- 172.21.15.111:0/1029184274 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7eff90087830 con 0x7effa00cb2d0 2023-11-28T12:59:49.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.180+0000 7eff9effd700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7eff8c0682f0 0x7eff8c06a7a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7eff94006fd0 tx=0x7eff94008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:49.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.183+0000 7eff87fff700 1 -- 172.21.15.111:0/1029184274 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7eff90053d60 con 0x7effa00cb2d0 2023-11-28T12:59:49.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.381+0000 7effa582d700 1 -- 172.21.15.111:0/1029184274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7effa00ccf10 con 0x7effa00cb2d0 2023-11-28T12:59:49.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.381+0000 7eff87fff700 1 -- 172.21.15.111:0/1029184274 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) v1 ==== 80+0+0 (secure 0 0 0) 0x7eff90057380 con 0x7effa00cb2d0 2023-11-28T12:59:49.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.383+0000 7effa582d700 1 -- 172.21.15.111:0/1029184274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7eff8c0682f0 msgr2=0x7eff8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:49.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.383+0000 7effa582d700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7eff8c0682f0 0x7eff8c06a7a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7eff94006fd0 tx=0x7eff94008040 comp rx=0 tx=0).stop 2023-11-28T12:59:49.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.383+0000 7effa582d700 1 -- 172.21.15.111:0/1029184274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa00cb2d0 msgr2=0x7effa00b6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:49.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.383+0000 7effa582d700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa00cb2d0 0x7effa00b6330 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7eff90009130 tx=0x7eff90004750 comp rx=0 tx=0).stop 2023-11-28T12:59:49.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.383+0000 7effa582d700 1 -- 172.21.15.111:0/1029184274 shutdown_connections 2023-11-28T12:59:49.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.383+0000 7effa582d700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7effa015a5d0 0x7effa014cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.383+0000 7effa582d700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7eff8c0682f0 0x7eff8c06a7a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.383+0000 7effa582d700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7effa00b0160 0x7effa0155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.383+0000 7effa582d700 1 --2- 172.21.15.111:0/1029184274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7effa00cb2d0 0x7effa00b6330 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.383+0000 7effa582d700 1 -- 172.21.15.111:0/1029184274 >> 172.21.15.111:0/1029184274 conn(0x7effa009a040 msgr2=0x7effa006f090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.383+0000 7effa582d700 1 -- 172.21.15.111:0/1029184274 shutdown_connections 2023-11-28T12:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.383+0000 7effa582d700 1 -- 172.21.15.111:0/1029184274 wait complete. 2023-11-28T12:59:49.386 INFO:teuthology.orchestra.run.smithi111.stderr:listed 0 entries 2023-11-28T12:59:49.397 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-28T12:59:49.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:49.397 DEBUG:teuthology.orchestra.run.smithi149:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-11-28T12:59:49.441 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi149.front.sepia.ceph.com 2023-11-28T12:59:49.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:49.441 DEBUG:teuthology.orchestra.run.smithi149:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-28T12:59:49.509 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:59:49.509 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:49.527 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:59:49.527 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:49.584 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T12:59:49.584 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:49.642 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:49.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.799+0000 7fb827a54700 1 Processor -- start 2023-11-28T12:59:49.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.800+0000 7fb827a54700 1 -- start start 2023-11-28T12:59:49.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.800+0000 7fb827a54700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb82015a3c0 0x7fb82006f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:49.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.800+0000 7fb827a54700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb8200cb790 0x7fb82006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:49.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.800+0000 7fb827a54700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82006b490 con 0x7fb82006a7b0 2023-11-28T12:59:49.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.800+0000 7fb827a54700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82006abf0 con 0x7fb82015a3c0 2023-11-28T12:59:49.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.800+0000 7fb827a54700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82006f680 con 0x7fb8200cb790 2023-11-28T12:59:49.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.801+0000 7fb8257f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb82015a3c0 0x7fb82006f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:49.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.801+0000 7fb8257f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb82015a3c0 0x7fb82006f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34448/0 (socket says 172.21.15.111:34448) 2023-11-28T12:59:49.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.801+0000 7fb8257f0700 1 -- 172.21.15.111:0/1214276146 learned_addr learned my addr 172.21.15.111:0/1214276146 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:49.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.801+0000 7fb824fef700 1 --2- 172.21.15.111:0/1214276146 >> v2:172.21.15.149:3300/0 conn(0x7fb8200cb790 0x7fb82006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:49.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.801+0000 7fb8257f0700 1 -- 172.21.15.111:0/1214276146 >> v2:172.21.15.149:3300/0 conn(0x7fb8200cb790 msgr2=0x7fb82006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:49.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.801+0000 7fb8257f0700 1 --2- 172.21.15.111:0/1214276146 >> v2:172.21.15.149:3300/0 conn(0x7fb8200cb790 0x7fb82006a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.801+0000 7fb8257f0700 1 -- 172.21.15.111:0/1214276146 >> v1:172.21.15.149:6789/0 conn(0x7fb82006a7b0 legacy=0x7fb82015df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:49.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.801+0000 7fb8257f0700 1 -- 172.21.15.111:0/1214276146 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82015a790 con 0x7fb82015a3c0 2023-11-28T12:59:49.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.801+0000 7fb8257f0700 1 --2- 172.21.15.111:0/1214276146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb82015a3c0 0x7fb82006f140 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb820066420 tx=0x7fb810009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75d7eefb8fdf898 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:49.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.801+0000 7fb817fff700 1 -- 172.21.15.111:0/1214276146 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb810011830 con 0x7fb82015a3c0 2023-11-28T12:59:49.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.801+0000 7fb817fff700 1 -- 172.21.15.111:0/1214276146 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb810011990 con 0x7fb82015a3c0 2023-11-28T12:59:49.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.801+0000 7fb817fff700 1 -- 172.21.15.111:0/1214276146 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb810011d20 con 0x7fb82015a3c0 2023-11-28T12:59:49.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.802+0000 7fb827a54700 1 -- 172.21.15.111:0/1214276146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb82015a3c0 msgr2=0x7fb82006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:49.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.802+0000 7fb827a54700 1 --2- 172.21.15.111:0/1214276146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb82015a3c0 0x7fb82006f140 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb820066420 tx=0x7fb810009bd0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.802+0000 7fb827a54700 1 -- 172.21.15.111:0/1214276146 shutdown_connections 2023-11-28T12:59:49.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.802+0000 7fb827a54700 1 --2- 172.21.15.111:0/1214276146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb82015a3c0 0x7fb82006f140 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.802+0000 7fb827a54700 1 --2- 172.21.15.111:0/1214276146 >> v2:172.21.15.149:3300/0 conn(0x7fb8200cb790 0x7fb82006a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.802+0000 7fb827a54700 1 -- 172.21.15.111:0/1214276146 >> 172.21.15.111:0/1214276146 conn(0x7fb8200b0db0 msgr2=0x7fb8200b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:49.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.802+0000 7fb827a54700 1 -- 172.21.15.111:0/1214276146 shutdown_connections 2023-11-28T12:59:49.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.802+0000 7fb827a54700 1 -- 172.21.15.111:0/1214276146 wait complete. 2023-11-28T12:59:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.803+0000 7fb827a54700 1 Processor -- start 2023-11-28T12:59:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.803+0000 7fb827a54700 1 -- start start 2023-11-28T12:59:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb827a54700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb82006a7b0 0x7fb82015d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb827a54700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8200cb790 0x7fb8200c4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb827a54700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb82015a3c0 0x7fb8200c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb827a54700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8200c5730 con 0x7fb8200cb790 2023-11-28T12:59:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb827a54700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8200c5170 con 0x7fb82006a7b0 2023-11-28T12:59:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb827a54700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8200c5450 con 0x7fb82015a3c0 2023-11-28T12:59:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb824fef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8200cb790 0x7fb8200c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb824fef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8200cb790 0x7fb8200c4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34456/0 (socket says 172.21.15.111:34456) 2023-11-28T12:59:49.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb824fef700 1 -- 172.21.15.111:0/4195321258 learned_addr learned my addr 172.21.15.111:0/4195321258 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb8257f0700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb82006a7b0 0x7fb82015d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb825ff1700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb82015a3c0 0x7fb8200c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb824fef700 1 -- 172.21.15.111:0/4195321258 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb82015a3c0 msgr2=0x7fb8200c6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb824fef700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb82015a3c0 0x7fb8200c6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb824fef700 1 -- 172.21.15.111:0/4195321258 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb82006a7b0 msgr2=0x7fb82015d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb824fef700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb82006a7b0 0x7fb82015d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb824fef700 1 -- 172.21.15.111:0/4195321258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb810009550 con 0x7fb8200cb790 2023-11-28T12:59:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb824fef700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8200cb790 0x7fb8200c4820 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00e870 tx=0x7fb81c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb815ffb700 1 -- 172.21.15.111:0/4195321258 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c009590 con 0x7fb8200cb790 2023-11-28T12:59:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.804+0000 7fb815ffb700 1 -- 172.21.15.111:0/4195321258 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c013680 con 0x7fb8200cb790 2023-11-28T12:59:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.805+0000 7fb827a54700 1 -- 172.21.15.111:0/4195321258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82006b9c0 con 0x7fb8200cb790 2023-11-28T12:59:49.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.805+0000 7fb815ffb700 1 -- 172.21.15.111:0/4195321258 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c0139f0 con 0x7fb8200cb790 2023-11-28T12:59:49.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.805+0000 7fb827a54700 1 -- 172.21.15.111:0/4195321258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82006b6e0 con 0x7fb8200cb790 2023-11-28T12:59:49.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.806+0000 7fb827a54700 1 -- 172.21.15.111:0/4195321258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8200c9ab0 con 0x7fb8200cb790 2023-11-28T12:59:49.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.806+0000 7fb815ffb700 1 -- 172.21.15.111:0/4195321258 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb81c019030 con 0x7fb8200cb790 2023-11-28T12:59:49.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.806+0000 7fb815ffb700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8040682f0 0x7fb80406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:49.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.806+0000 7fb815ffb700 1 -- 172.21.15.111:0/4195321258 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fb81c085c80 con 0x7fb8200cb790 2023-11-28T12:59:49.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.806+0000 7fb8257f0700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8040682f0 0x7fb80406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:49.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.807+0000 7fb8257f0700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8040682f0 0x7fb80406a7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb810009c10 tx=0x7fb810008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:49.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.811+0000 7fb815ffb700 1 -- 172.21.15.111:0/4195321258 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb81c0521b0 con 0x7fb8200cb790 2023-11-28T12:59:49.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.997+0000 7fb827a54700 1 -- 172.21.15.111:0/4195321258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb820067170 con 0x7fb8200cb790 2023-11-28T12:59:49.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:49.998+0000 7fb815ffb700 1 -- 172.21.15.111:0/4195321258 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+19890 (secure 0 0 0) 0x7fb81c0557d0 con 0x7fb8200cb790 2023-11-28T12:59:50.000 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:50.000 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":28,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:41.880678+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:50.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.000+0000 7fb827a54700 1 -- 172.21.15.111:0/4195321258 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8040682f0 msgr2=0x7fb80406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.000+0000 7fb827a54700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8040682f0 0x7fb80406a7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb810009c10 tx=0x7fb810008030 comp rx=0 tx=0).stop 2023-11-28T12:59:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.000+0000 7fb827a54700 1 -- 172.21.15.111:0/4195321258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8200cb790 msgr2=0x7fb8200c4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.000+0000 7fb827a54700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8200cb790 0x7fb8200c4820 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00e870 tx=0x7fb81c00eb80 comp rx=0 tx=0).stop 2023-11-28T12:59:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.000+0000 7fb827a54700 1 -- 172.21.15.111:0/4195321258 shutdown_connections 2023-11-28T12:59:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.000+0000 7fb827a54700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb82015a3c0 0x7fb8200c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.000+0000 7fb827a54700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8040682f0 0x7fb80406a7a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.000+0000 7fb827a54700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb82006a7b0 0x7fb82015d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.000+0000 7fb827a54700 1 --2- 172.21.15.111:0/4195321258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8200cb790 0x7fb8200c4820 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.000+0000 7fb827a54700 1 -- 172.21.15.111:0/4195321258 >> 172.21.15.111:0/4195321258 conn(0x7fb8200b0db0 msgr2=0x7fb8200b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.000+0000 7fb827a54700 1 -- 172.21.15.111:0/4195321258 shutdown_connections 2023-11-28T12:59:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.000+0000 7fb827a54700 1 -- 172.21.15.111:0/4195321258 wait complete. 2023-11-28T12:59:50.014 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T12:58:53.275206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T12:58:54.588499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T12:58:57.666033+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T12:59:50.015 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:59:50.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.136+0000 7f5f768cf700 1 Processor -- start 2023-11-28T12:59:50.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.136+0000 7f5f768cf700 1 -- start start 2023-11-28T12:59:50.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.136+0000 7f5f768cf700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5f700cc550 0x7f5f700cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.136+0000 7f5f768cf700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f7015cd90 con 0x7f5f701604b0 2023-11-28T12:59:50.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.136+0000 7f5f768cf700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f700cced0 con 0x7f5f700b0d70 2023-11-28T12:59:50.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.136+0000 7f5f768cf700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f70160950 con 0x7f5f700cc550 2023-11-28T12:59:50.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.137+0000 7f5f74e6c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5f700cc550 0x7f5f700cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.137+0000 7f5f74e6c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5f700cc550 0x7f5f700cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34466/0 (socket says 172.21.15.111:34466) 2023-11-28T12:59:50.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.137+0000 7f5f74e6c700 1 -- 172.21.15.111:0/3321601457 learned_addr learned my addr 172.21.15.111:0/3321601457 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:50.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.137+0000 7f5f74e6c700 1 -- 172.21.15.111:0/3321601457 >> v1:172.21.15.149:6789/0 conn(0x7f5f700b0d70 legacy=0x7f5f700b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:50.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.137+0000 7f5f74e6c700 1 -- 172.21.15.111:0/3321601457 >> v1:172.21.15.111:6789/0 conn(0x7f5f701604b0 legacy=0x7f5f700ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.137+0000 7f5f74e6c700 1 -- 172.21.15.111:0/3321601457 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f701613e0 con 0x7f5f700cc550 2023-11-28T12:59:50.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.137+0000 7f5f74e6c700 1 --2- 172.21.15.111:0/3321601457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f700cc550 0x7f5f700cc990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5f68009e50 tx=0x7f5f6800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8bccfc744ea94fa server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:50.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.137+0000 7f5f6effd700 1 -- 172.21.15.111:0/3321601457 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6800e070 con 0x7f5f700cc550 2023-11-28T12:59:50.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.137+0000 7f5f6effd700 1 -- 172.21.15.111:0/3321601457 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f68003a20 con 0x7f5f700cc550 2023-11-28T12:59:50.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.137+0000 7f5f6effd700 1 -- 172.21.15.111:0/3321601457 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f68003dd0 con 0x7f5f700cc550 2023-11-28T12:59:50.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.138+0000 7f5f768cf700 1 -- 172.21.15.111:0/3321601457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f700cc550 msgr2=0x7f5f700cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.138+0000 7f5f768cf700 1 --2- 172.21.15.111:0/3321601457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f700cc550 0x7f5f700cc990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5f68009e50 tx=0x7f5f6800b330 comp rx=0 tx=0).stop 2023-11-28T12:59:50.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.138+0000 7f5f768cf700 1 -- 172.21.15.111:0/3321601457 shutdown_connections 2023-11-28T12:59:50.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.138+0000 7f5f768cf700 1 --2- 172.21.15.111:0/3321601457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f700cc550 0x7f5f700cc990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.138+0000 7f5f768cf700 1 -- 172.21.15.111:0/3321601457 >> 172.21.15.111:0/3321601457 conn(0x7f5f70098ee0 msgr2=0x7f5f7006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:50.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.138+0000 7f5f768cf700 1 -- 172.21.15.111:0/3321601457 shutdown_connections 2023-11-28T12:59:50.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.138+0000 7f5f768cf700 1 -- 172.21.15.111:0/3321601457 wait complete. 2023-11-28T12:59:50.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.139+0000 7f5f768cf700 1 Processor -- start 2023-11-28T12:59:50.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.139+0000 7f5f768cf700 1 -- start start 2023-11-28T12:59:50.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f768cf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f700b0d70 0x7f5f701507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f768cf700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5f700cc550 0x7f5f700c5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f768cf700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5f701604b0 0x7f5f7006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f768cf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7015fd90 con 0x7f5f700b0d70 2023-11-28T12:59:50.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f768cf700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7015f550 con 0x7f5f700cc550 2023-11-28T12:59:50.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f768cf700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7015f7d0 con 0x7f5f701604b0 2023-11-28T12:59:50.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f6ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f700b0d70 0x7f5f701507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f6ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f700b0d70 0x7f5f701507b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34468/0 (socket says 172.21.15.111:34468) 2023-11-28T12:59:50.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f6ffff700 1 -- 172.21.15.111:0/2212534477 learned_addr learned my addr 172.21.15.111:0/2212534477 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:50.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f6f7fe700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5f700cc550 0x7f5f700c5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f74e6c700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5f701604b0 0x7f5f7006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f6ffff700 1 -- 172.21.15.111:0/2212534477 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5f701604b0 msgr2=0x7f5f7006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f6ffff700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5f701604b0 0x7f5f7006e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f6ffff700 1 -- 172.21.15.111:0/2212534477 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5f700cc550 msgr2=0x7f5f700c5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f6ffff700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5f700cc550 0x7f5f700c5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f6ffff700 1 -- 172.21.15.111:0/2212534477 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f68009b30 con 0x7f5f700b0d70 2023-11-28T12:59:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f6f7fe700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5f700cc550 0x7f5f700c5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f74e6c700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5f701604b0 0x7f5f7006e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.140+0000 7f5f6ffff700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f700b0d70 0x7f5f701507b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000c4e0 tx=0x7f5f6000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.141+0000 7f5f6cff9700 1 -- 172.21.15.111:0/2212534477 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f60011790 con 0x7f5f700b0d70 2023-11-28T12:59:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.141+0000 7f5f6cff9700 1 -- 172.21.15.111:0/2212534477 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f600118f0 con 0x7f5f700b0d70 2023-11-28T12:59:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.141+0000 7f5f6cff9700 1 -- 172.21.15.111:0/2212534477 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f60011ca0 con 0x7f5f700b0d70 2023-11-28T12:59:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.141+0000 7f5f768cf700 1 -- 172.21.15.111:0/2212534477 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f70160e20 con 0x7f5f700b0d70 2023-11-28T12:59:50.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.141+0000 7f5f768cf700 1 -- 172.21.15.111:0/2212534477 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f70160b40 con 0x7f5f700b0d70 2023-11-28T12:59:50.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.142+0000 7f5f768cf700 1 -- 172.21.15.111:0/2212534477 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f70161100 con 0x7f5f700b0d70 2023-11-28T12:59:50.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.142+0000 7f5f6cff9700 1 -- 172.21.15.111:0/2212534477 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5f60010050 con 0x7f5f700b0d70 2023-11-28T12:59:50.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.142+0000 7f5f6cff9700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5f5c0682f0 0x7f5f5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.142+0000 7f5f6cff9700 1 -- 172.21.15.111:0/2212534477 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f5f60085a30 con 0x7f5f700b0d70 2023-11-28T12:59:50.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.142+0000 7f5f6f7fe700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5f5c0682f0 0x7f5f5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.143+0000 7f5f6f7fe700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5f5c0682f0 0x7f5f5c06a7a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5f700c49c0 tx=0x7f5f64009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:50.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.147+0000 7f5f6cff9700 1 -- 172.21.15.111:0/2212534477 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5f60051f00 con 0x7f5f700b0d70 2023-11-28T12:59:50.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.329+0000 7f5f768cf700 1 -- 172.21.15.111:0/2212534477 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5f70160190 con 0x7f5f700b0d70 2023-11-28T12:59:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.329+0000 7f5f6cff9700 1 -- 172.21.15.111:0/2212534477 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5f60055520 con 0x7f5f700b0d70 2023-11-28T12:59:50.331 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:59:50.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.331+0000 7f5f768cf700 1 -- 172.21.15.111:0/2212534477 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5f5c0682f0 msgr2=0x7f5f5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.331+0000 7f5f768cf700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5f5c0682f0 0x7f5f5c06a7a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5f700c49c0 tx=0x7f5f64009040 comp rx=0 tx=0).stop 2023-11-28T12:59:50.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.331+0000 7f5f768cf700 1 -- 172.21.15.111:0/2212534477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f700b0d70 msgr2=0x7f5f701507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.331+0000 7f5f768cf700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f700b0d70 0x7f5f701507b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000c4e0 tx=0x7f5f6000c8a0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.331+0000 7f5f768cf700 1 -- 172.21.15.111:0/2212534477 shutdown_connections 2023-11-28T12:59:50.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.331+0000 7f5f768cf700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5f701604b0 0x7f5f7006e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.331+0000 7f5f768cf700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f700b0d70 0x7f5f701507b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.331+0000 7f5f768cf700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5f5c0682f0 0x7f5f5c06a7a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.331+0000 7f5f768cf700 1 --2- 172.21.15.111:0/2212534477 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5f700cc550 0x7f5f700c5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.331+0000 7f5f768cf700 1 -- 172.21.15.111:0/2212534477 >> 172.21.15.111:0/2212534477 conn(0x7f5f70098ee0 msgr2=0x7f5f700b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:50.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.331+0000 7f5f768cf700 1 -- 172.21.15.111:0/2212534477 shutdown_connections 2023-11-28T12:59:50.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.331+0000 7f5f768cf700 1 -- 172.21.15.111:0/2212534477 wait complete. 2023-11-28T12:59:50.344 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:59:50.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.466+0000 7f7576af8700 1 Processor -- start 2023-11-28T12:59:50.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.466+0000 7f7576af8700 1 -- start start 2023-11-28T12:59:50.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.467+0000 7f7576af8700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f75700b8510 0x7f757006b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.467+0000 7f7576af8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75700cd460 0x7f757006beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.467+0000 7f7576af8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f757006c3f0 0x7f757015cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.467+0000 7f7576af8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75700cdbd0 con 0x7f757006c3f0 2023-11-28T12:59:50.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.467+0000 7f7576af8700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75700d02d0 con 0x7f75700b8510 2023-11-28T12:59:50.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.467+0000 7f7576af8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75700b88e0 con 0x7f75700cd460 2023-11-28T12:59:50.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.467+0000 7f7575095700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f757006c3f0 0x7f757015cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.467+0000 7f7575095700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f757006c3f0 0x7f757015cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34480/0 (socket says 172.21.15.111:34480) 2023-11-28T12:59:50.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.467+0000 7f7575095700 1 -- 172.21.15.111:0/3872850593 learned_addr learned my addr 172.21.15.111:0/3872850593 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:50.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.467+0000 7f756ffff700 1 --2- 172.21.15.111:0/3872850593 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75700cd460 0x7f757006beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.467+0000 7f7574894700 1 --2- 172.21.15.111:0/3872850593 >> v2:172.21.15.149:3300/0 conn(0x7f75700b8510 0x7f757006b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7575095700 1 -- 172.21.15.111:0/3872850593 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75700cd460 msgr2=0x7f757006beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:50.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7575095700 1 --2- 172.21.15.111:0/3872850593 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75700cd460 0x7f757006beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7575095700 1 -- 172.21.15.111:0/3872850593 >> v2:172.21.15.149:3300/0 conn(0x7f75700b8510 msgr2=0x7f757006b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:50.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7575095700 1 --2- 172.21.15.111:0/3872850593 >> v2:172.21.15.149:3300/0 conn(0x7f75700b8510 0x7f757006b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7575095700 1 -- 172.21.15.111:0/3872850593 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75700cd8f0 con 0x7f757006c3f0 2023-11-28T12:59:50.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7575095700 1 --2- 172.21.15.111:0/3872850593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f757006c3f0 0x7f757015cc60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f75700b5a50 tx=0x7f756800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99369bf63b9aa175 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:50.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f756f7fe700 1 -- 172.21.15.111:0/3872850593 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756800e070 con 0x7f757006c3f0 2023-11-28T12:59:50.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f756f7fe700 1 -- 172.21.15.111:0/3872850593 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756800b980 con 0x7f757006c3f0 2023-11-28T12:59:50.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f756f7fe700 1 -- 172.21.15.111:0/3872850593 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756800bd30 con 0x7f757006c3f0 2023-11-28T12:59:50.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7576af8700 1 -- 172.21.15.111:0/3872850593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f757006c3f0 msgr2=0x7f757015cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7576af8700 1 --2- 172.21.15.111:0/3872850593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f757006c3f0 0x7f757015cc60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f75700b5a50 tx=0x7f756800b330 comp rx=0 tx=0).stop 2023-11-28T12:59:50.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7576af8700 1 -- 172.21.15.111:0/3872850593 shutdown_connections 2023-11-28T12:59:50.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7576af8700 1 --2- 172.21.15.111:0/3872850593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f757006c3f0 0x7f757015cc60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7576af8700 1 --2- 172.21.15.111:0/3872850593 >> v2:172.21.15.149:3300/0 conn(0x7f75700b8510 0x7f757006b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7576af8700 1 --2- 172.21.15.111:0/3872850593 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75700cd460 0x7f757006beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7576af8700 1 -- 172.21.15.111:0/3872850593 >> 172.21.15.111:0/3872850593 conn(0x7f75700719a0 msgr2=0x7f7570071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:50.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7576af8700 1 -- 172.21.15.111:0/3872850593 shutdown_connections 2023-11-28T12:59:50.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.468+0000 7f7576af8700 1 -- 172.21.15.111:0/3872850593 wait complete. 2023-11-28T12:59:50.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.470+0000 7f7576af8700 1 Processor -- start 2023-11-28T12:59:50.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.470+0000 7f7576af8700 1 -- start start 2023-11-28T12:59:50.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.470+0000 7f7576af8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f757006c3f0 0x7f75700c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.470+0000 7f7576af8700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75700b8510 0x7f75700cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7576af8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75700cd460 0x7f75700d0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7576af8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75700c9300 con 0x7f75700cd460 2023-11-28T12:59:50.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7576af8700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75701e5240 con 0x7f75700b8510 2023-11-28T12:59:50.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7576af8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75701e4630 con 0x7f757006c3f0 2023-11-28T12:59:50.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7575095700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75700cd460 0x7f75700d0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7575095700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75700cd460 0x7f75700d0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34486/0 (socket says 172.21.15.111:34486) 2023-11-28T12:59:50.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7575095700 1 -- 172.21.15.111:0/2345369808 learned_addr learned my addr 172.21.15.111:0/2345369808 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:50.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f756ffff700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75700b8510 0x7f75700cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7574894700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f757006c3f0 0x7f75700c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7575095700 1 -- 172.21.15.111:0/2345369808 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f757006c3f0 msgr2=0x7f75700c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7575095700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f757006c3f0 0x7f75700c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7575095700 1 -- 172.21.15.111:0/2345369808 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75700b8510 msgr2=0x7f75700cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7575095700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75700b8510 0x7f75700cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7575095700 1 -- 172.21.15.111:0/2345369808 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7568009b30 con 0x7f75700cd460 2023-11-28T12:59:50.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7575095700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75700cd460 0x7f75700d0c10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7570072610 tx=0x7f7568004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:50.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f756d7fa700 1 -- 172.21.15.111:0/2345369808 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7568004250 con 0x7f75700cd460 2023-11-28T12:59:50.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f756d7fa700 1 -- 172.21.15.111:0/2345369808 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75680043b0 con 0x7f75700cd460 2023-11-28T12:59:50.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f756d7fa700 1 -- 172.21.15.111:0/2345369808 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7568004760 con 0x7f75700cd460 2023-11-28T12:59:50.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7576af8700 1 -- 172.21.15.111:0/2345369808 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75700cfff0 con 0x7f75700cd460 2023-11-28T12:59:50.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.471+0000 7f7576af8700 1 -- 172.21.15.111:0/2345369808 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75700cfd10 con 0x7f75700cd460 2023-11-28T12:59:50.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.472+0000 7f7576af8700 1 -- 172.21.15.111:0/2345369808 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75701e53f0 con 0x7f75700cd460 2023-11-28T12:59:50.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.473+0000 7f756d7fa700 1 -- 172.21.15.111:0/2345369808 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f756802c070 con 0x7f75700cd460 2023-11-28T12:59:50.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.473+0000 7f756d7fa700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75540682f0 0x7f755406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.473+0000 7f756d7fa700 1 -- 172.21.15.111:0/2345369808 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f7568087720 con 0x7f75700cd460 2023-11-28T12:59:50.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.473+0000 7f7574894700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75540682f0 0x7f755406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.474+0000 7f7574894700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75540682f0 0x7f755406a7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f756400c900 tx=0x7f7564008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:50.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.477+0000 7f756d7fa700 1 -- 172.21.15.111:0/2345369808 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f75701e53f0 con 0x7f75700cd460 2023-11-28T12:59:50.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.660+0000 7f7576af8700 1 -- 172.21.15.111:0/2345369808 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f75701e53b0 con 0x7f75700cd460 2023-11-28T12:59:50.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.660+0000 7f756d7fa700 1 -- 172.21.15.111:0/2345369808 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7568020020 con 0x7f75700cd460 2023-11-28T12:59:50.666 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:59:50.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.662+0000 7f7576af8700 1 -- 172.21.15.111:0/2345369808 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75540682f0 msgr2=0x7f755406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.662+0000 7f7576af8700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75540682f0 0x7f755406a7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f756400c900 tx=0x7f7564008040 comp rx=0 tx=0).stop 2023-11-28T12:59:50.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.662+0000 7f7576af8700 1 -- 172.21.15.111:0/2345369808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75700cd460 msgr2=0x7f75700d0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.662+0000 7f7576af8700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75700cd460 0x7f75700d0c10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7570072610 tx=0x7f7568004ad0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.662+0000 7f7576af8700 1 -- 172.21.15.111:0/2345369808 shutdown_connections 2023-11-28T12:59:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.662+0000 7f7576af8700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75700cd460 0x7f75700d0c10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.662+0000 7f7576af8700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f757006c3f0 0x7f75700c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.662+0000 7f7576af8700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75540682f0 0x7f755406a7a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.662+0000 7f7576af8700 1 --2- 172.21.15.111:0/2345369808 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75700b8510 0x7f75700cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.662+0000 7f7576af8700 1 -- 172.21.15.111:0/2345369808 >> 172.21.15.111:0/2345369808 conn(0x7f75700719a0 msgr2=0x7f75700b3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.662+0000 7f7576af8700 1 -- 172.21.15.111:0/2345369808 shutdown_connections 2023-11-28T12:59:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.662+0000 7f7576af8700 1 -- 172.21.15.111:0/2345369808 wait complete. 2023-11-28T12:59:50.676 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T12:59:50.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.797+0000 7f45312ad700 1 Processor -- start 2023-11-28T12:59:50.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.797+0000 7f45312ad700 1 -- start start 2023-11-28T12:59:50.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f45312ad700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f452c0aecc0 0x7f452c0b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f45312ad700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c0ad420 con 0x7f452c159960 2023-11-28T12:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f45312ad700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452c0ac810 con 0x7f452c0cdd90 2023-11-28T12:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f45312ad700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452c159d30 con 0x7f452c0aecc0 2023-11-28T12:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f452affd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f452c159960 0x7f452c0cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35268/0 (socket says 172.21.15.111:35268) 2023-11-28T12:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f452affd700 1 -- 172.21.15.111:0/1201598590 learned_addr learned my addr 172.21.15.111:0/1201598590 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936591667 0 0) 0x7f452c0ad420 con 0x7f452c159960 2023-11-28T12:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45140035b0 con 0x7f452c159960 2023-11-28T12:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f452b7fe700 1 --2- 172.21.15.111:0/1201598590 >> v2:172.21.15.149:3300/0 conn(0x7f452c0aecc0 0x7f452c0b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2663968944 0 0) 0x7f45140035b0 con 0x7f452c159960 2023-11-28T12:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 >> v2:172.21.15.149:3300/0 conn(0x7f452c0aecc0 msgr2=0x7f452c0b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:50.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f4529ffb700 1 --2- 172.21.15.111:0/1201598590 >> v2:172.21.15.149:3300/0 conn(0x7f452c0aecc0 0x7f452c0b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 >> v1:172.21.15.149:6789/0 conn(0x7f452c0cdd90 legacy=0x7f452c0ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c0acaf0 con 0x7f452c159960 2023-11-28T12:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f451c0017f0 con 0x7f452c159960 2023-11-28T12:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 >> v1:172.21.15.111:6789/0 conn(0x7f452c159960 legacy=0x7f452c0cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f4529ffb700 1 --2- 172.21.15.111:0/1201598590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4514003a00 0x7f4514003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.798+0000 7f4529ffb700 1 --2- 172.21.15.111:0/1201598590 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4514004850 0x7f4514004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f4529ffb700 1 --2- 172.21.15.111:0/1201598590 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4514005200 0x7f451400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c159d30 con 0x7f4514003a00 2023-11-28T12:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451400c080 con 0x7f4514005200 2023-11-28T12:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f451400c240 con 0x7f4514004850 2023-11-28T12:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f452affd700 1 --2- 172.21.15.111:0/1201598590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4514003a00 0x7f4514003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f452b7fe700 1 --2- 172.21.15.111:0/1201598590 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4514005200 0x7f451400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f452a7fc700 1 --2- 172.21.15.111:0/1201598590 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4514004850 0x7f4514004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f452affd700 1 -- 172.21.15.111:0/1201598590 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4514004850 msgr2=0x7f4514004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f452affd700 1 --2- 172.21.15.111:0/1201598590 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4514004850 0x7f4514004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f452affd700 1 -- 172.21.15.111:0/1201598590 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4514005200 msgr2=0x7f451400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f452affd700 1 --2- 172.21.15.111:0/1201598590 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4514005200 0x7f451400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f452affd700 1 -- 172.21.15.111:0/1201598590 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140035b0 con 0x7f4514003a00 2023-11-28T12:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f452b7fe700 1 --2- 172.21.15.111:0/1201598590 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4514005200 0x7f451400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f452b7fe700 1 -- 172.21.15.111:0/1201598590 reap_dead start 2023-11-28T12:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f452a7fc700 1 --2- 172.21.15.111:0/1201598590 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4514004850 0x7f4514004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f452affd700 1 --2- 172.21.15.111:0/1201598590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4514003a00 0x7f4514003e10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f451c00c240 tx=0x7f4514005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f955dfb78451636f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c00cde0 con 0x7f4514003a00 2023-11-28T12:59:50.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451c007b80 con 0x7f4514003a00 2023-11-28T12:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.799+0000 7f4529ffb700 1 -- 172.21.15.111:0/1201598590 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c0072d0 con 0x7f4514003a00 2023-11-28T12:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.800+0000 7f45312ad700 1 -- 172.21.15.111:0/1201598590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4514003a00 msgr2=0x7f4514003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.800+0000 7f45312ad700 1 --2- 172.21.15.111:0/1201598590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4514003a00 0x7f4514003e10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f451c00c240 tx=0x7f4514005880 comp rx=0 tx=0).stop 2023-11-28T12:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.800+0000 7f45312ad700 1 -- 172.21.15.111:0/1201598590 shutdown_connections 2023-11-28T12:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.800+0000 7f45312ad700 1 --2- 172.21.15.111:0/1201598590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4514003a00 0x7f4514003e10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.800+0000 7f45312ad700 1 -- 172.21.15.111:0/1201598590 >> 172.21.15.111:0/1201598590 conn(0x7f452c098ee0 msgr2=0x7f452c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.800+0000 7f45312ad700 1 -- 172.21.15.111:0/1201598590 shutdown_connections 2023-11-28T12:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.800+0000 7f45312ad700 1 -- 172.21.15.111:0/1201598590 wait complete. 2023-11-28T12:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.801+0000 7f45312ad700 1 Processor -- start 2023-11-28T12:59:50.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.801+0000 7f45312ad700 1 -- start start 2023-11-28T12:59:50.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f45312ad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4514003a00 0x7f452c0cba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f45312ad700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f452c0cbf40 0x7f452c06a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f45312ad700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f452c158100 0x7f452c06a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f45312ad700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c151b40 con 0x7f4514003a00 2023-11-28T12:59:50.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f45312ad700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452c0ccc60 con 0x7f452c0cbf40 2023-11-28T12:59:50.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f45312ad700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452c0cc3c0 con 0x7f452c158100 2023-11-28T12:59:50.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f452affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4514003a00 0x7f452c0cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f452affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4514003a00 0x7f452c0cba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34502/0 (socket says 172.21.15.111:34502) 2023-11-28T12:59:50.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f452affd700 1 -- 172.21.15.111:0/2310809151 learned_addr learned my addr 172.21.15.111:0/2310809151 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f452b7fe700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f452c158100 0x7f452c06a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f452affd700 1 -- 172.21.15.111:0/2310809151 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f452c158100 msgr2=0x7f452c06a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f452a7fc700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f452c0cbf40 0x7f452c06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f452affd700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f452c158100 0x7f452c06a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f452affd700 1 -- 172.21.15.111:0/2310809151 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f452c0cbf40 msgr2=0x7f452c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f452affd700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f452c0cbf40 0x7f452c06a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f452affd700 1 -- 172.21.15.111:0/2310809151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c0ad0b0 con 0x7f4514003a00 2023-11-28T12:59:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f452b7fe700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f452c158100 0x7f452c06a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f452affd700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4514003a00 0x7f452c0cba00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f451c007d30 tx=0x7f451c001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f45137fe700 1 -- 172.21.15.111:0/2310809151 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c0074e0 con 0x7f4514003a00 2023-11-28T12:59:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f45137fe700 1 -- 172.21.15.111:0/2310809151 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451c007640 con 0x7f4514003a00 2023-11-28T12:59:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f45137fe700 1 -- 172.21.15.111:0/2310809151 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c005a70 con 0x7f4514003a00 2023-11-28T12:59:50.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.802+0000 7f45312ad700 1 -- 172.21.15.111:0/2310809151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452c0acdd0 con 0x7f4514003a00 2023-11-28T12:59:50.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.803+0000 7f45312ad700 1 -- 172.21.15.111:0/2310809151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452c0b1fd0 con 0x7f4514003a00 2023-11-28T12:59:50.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.803+0000 7f45312ad700 1 -- 172.21.15.111:0/2310809151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452c0ccdf0 con 0x7f4514003a00 2023-11-28T12:59:50.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.804+0000 7f45137fe700 1 -- 172.21.15.111:0/2310809151 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f451c024030 con 0x7f4514003a00 2023-11-28T12:59:50.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.804+0000 7f45137fe700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4518068320 0x7f451806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:50.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.804+0000 7f45137fe700 1 -- 172.21.15.111:0/2310809151 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f451c087060 con 0x7f4514003a00 2023-11-28T12:59:50.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.804+0000 7f452a7fc700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4518068320 0x7f451806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:50.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.805+0000 7f452a7fc700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4518068320 0x7f451806a7d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f452c15a100 tx=0x7f4520009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:50.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.808+0000 7f45137fe700 1 -- 172.21.15.111:0/2310809151 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f451c053650 con 0x7f4514003a00 2023-11-28T12:59:50.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.990+0000 7f45312ad700 1 -- 172.21.15.111:0/2310809151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f452c06c4f0 con 0x7f4514003a00 2023-11-28T12:59:50.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.991+0000 7f45137fe700 1 -- 172.21.15.111:0/2310809151 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f451c056c70 con 0x7f4514003a00 2023-11-28T12:59:50.992 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:50.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.993+0000 7f45312ad700 1 -- 172.21.15.111:0/2310809151 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4518068320 msgr2=0x7f451806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.993+0000 7f45312ad700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4518068320 0x7f451806a7d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f452c15a100 tx=0x7f4520009040 comp rx=0 tx=0).stop 2023-11-28T12:59:50.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.993+0000 7f45312ad700 1 -- 172.21.15.111:0/2310809151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4514003a00 msgr2=0x7f452c0cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:50.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.993+0000 7f45312ad700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4514003a00 0x7f452c0cba00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f451c007d30 tx=0x7f451c001b80 comp rx=0 tx=0).stop 2023-11-28T12:59:50.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.993+0000 7f45312ad700 1 -- 172.21.15.111:0/2310809151 shutdown_connections 2023-11-28T12:59:50.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.993+0000 7f45312ad700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f452c158100 0x7f452c06a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.993+0000 7f45312ad700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4514003a00 0x7f452c0cba00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.993+0000 7f45312ad700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4518068320 0x7f451806a7d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.993+0000 7f45312ad700 1 --2- 172.21.15.111:0/2310809151 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f452c0cbf40 0x7f452c06a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:50.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.993+0000 7f45312ad700 1 -- 172.21.15.111:0/2310809151 >> 172.21.15.111:0/2310809151 conn(0x7f452c098ee0 msgr2=0x7f452c06e490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:50.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.993+0000 7f45312ad700 1 -- 172.21.15.111:0/2310809151 shutdown_connections 2023-11-28T12:59:50.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:50.993+0000 7f45312ad700 1 -- 172.21.15.111:0/2310809151 wait complete. 2023-11-28T12:59:51.006 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T12:59:51.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.127+0000 7f56f48eb700 1 Processor -- start 2023-11-28T12:59:51.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.127+0000 7f56f48eb700 1 -- start start 2023-11-28T12:59:51.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.128+0000 7f56f48eb700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f56f0159520 0x7f56f00cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.128+0000 7f56f48eb700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f00b1510 con 0x7f56f00ae4a0 2023-11-28T12:59:51.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.128+0000 7f56f48eb700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f01598f0 con 0x7f56f006c0a0 2023-11-28T12:59:51.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.128+0000 7f56f48eb700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f00b0350 con 0x7f56f0159520 2023-11-28T12:59:51.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.128+0000 7f56eed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f56f00ae4a0 0x7f56f00afc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35290/0 (socket says 172.21.15.111:35290) 2023-11-28T12:59:51.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.128+0000 7f56eed9d700 1 -- 172.21.15.111:0/3665586839 learned_addr learned my addr 172.21.15.111:0/3665586839 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:51.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ee59c700 1 --2- 172.21.15.111:0/3665586839 >> v2:172.21.15.149:3300/0 conn(0x7f56f0159520 0x7f56f00cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472672788 0 0) 0x7f56f00b1510 con 0x7f56f00ae4a0 2023-11-28T12:59:51.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d80035b0 con 0x7f56f00ae4a0 2023-11-28T12:59:51.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212524187 0 0) 0x7f56f01598f0 con 0x7f56f006c0a0 2023-11-28T12:59:51.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56f00b1510 con 0x7f56f006c0a0 2023-11-28T12:59:51.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2861676362 0 0) 0x7f56d80035b0 con 0x7f56f00ae4a0 2023-11-28T12:59:51.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 >> v2:172.21.15.149:3300/0 conn(0x7f56f0159520 msgr2=0x7f56f00cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:51.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 --2- 172.21.15.111:0/3665586839 >> v2:172.21.15.149:3300/0 conn(0x7f56f0159520 0x7f56f00cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 >> v1:172.21.15.149:6789/0 conn(0x7f56f006c0a0 legacy=0x7f56f00af530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f00b0f50 con 0x7f56f00ae4a0 2023-11-28T12:59:51.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f56e80019f0 con 0x7f56f00ae4a0 2023-11-28T12:59:51.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 >> v1:172.21.15.111:6789/0 conn(0x7f56f00ae4a0 legacy=0x7f56f00afc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 --2- 172.21.15.111:0/3665586839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56d8003be0 0x7f56d8003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 --2- 172.21.15.111:0/3665586839 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f56d8004850 0x7f56d8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 --2- 172.21.15.111:0/3665586839 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f56d8005250 0x7f56d800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00b0350 con 0x7f56d8003be0 2023-11-28T12:59:51.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d800c160 con 0x7f56d8004850 2023-11-28T12:59:51.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d800c320 con 0x7f56d8005250 2023-11-28T12:59:51.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ee59c700 1 --2- 172.21.15.111:0/3665586839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56d8003be0 0x7f56d8003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56edd9b700 1 --2- 172.21.15.111:0/3665586839 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f56d8004850 0x7f56d8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56eed9d700 1 --2- 172.21.15.111:0/3665586839 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f56d8005250 0x7f56d800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ee59c700 1 -- 172.21.15.111:0/3665586839 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f56d8005250 msgr2=0x7f56d800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:51.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ee59c700 1 --2- 172.21.15.111:0/3665586839 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f56d8005250 0x7f56d800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ee59c700 1 -- 172.21.15.111:0/3665586839 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f56d8004850 msgr2=0x7f56d8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:51.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ee59c700 1 --2- 172.21.15.111:0/3665586839 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f56d8004850 0x7f56d8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56eed9d700 1 -- 172.21.15.111:0/3665586839 reap_dead start 2023-11-28T12:59:51.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.129+0000 7f56ee59c700 1 -- 172.21.15.111:0/3665586839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d80035b0 con 0x7f56d8003be0 2023-11-28T12:59:51.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.130+0000 7f56ee59c700 1 --2- 172.21.15.111:0/3665586839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56d8003be0 0x7f56d8003fd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009b30 tx=0x7f56d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0270f07e7fe3231 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:51.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.130+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e400e6c0 con 0x7f56d8003be0 2023-11-28T12:59:51.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.130+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e400e820 con 0x7f56d8003be0 2023-11-28T12:59:51.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.130+0000 7f56ed59a700 1 -- 172.21.15.111:0/3665586839 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e400ebb0 con 0x7f56d8003be0 2023-11-28T12:59:51.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.130+0000 7f56f48eb700 1 -- 172.21.15.111:0/3665586839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56d8003be0 msgr2=0x7f56d8003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.130+0000 7f56f48eb700 1 --2- 172.21.15.111:0/3665586839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56d8003be0 0x7f56d8003fd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009b30 tx=0x7f56d8005880 comp rx=0 tx=0).stop 2023-11-28T12:59:51.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.130+0000 7f56f48eb700 1 -- 172.21.15.111:0/3665586839 shutdown_connections 2023-11-28T12:59:51.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.130+0000 7f56f48eb700 1 --2- 172.21.15.111:0/3665586839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56d8003be0 0x7f56d8003fd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.130+0000 7f56f48eb700 1 -- 172.21.15.111:0/3665586839 >> 172.21.15.111:0/3665586839 conn(0x7f56f006bca0 msgr2=0x7f56f00cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:51.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.130+0000 7f56f48eb700 1 -- 172.21.15.111:0/3665586839 shutdown_connections 2023-11-28T12:59:51.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.130+0000 7f56f48eb700 1 -- 172.21.15.111:0/3665586839 wait complete. 2023-11-28T12:59:51.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.132+0000 7f56f48eb700 1 Processor -- start 2023-11-28T12:59:51.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.132+0000 7f56f48eb700 1 -- start start 2023-11-28T12:59:51.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.132+0000 7f56f48eb700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f56d8003be0 0x7f56f014bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.132+0000 7f56f48eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56f00c3ff0 0x7f56f014b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56f48eb700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f56f006a270 0x7f56f006a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56f48eb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f006ac20 con 0x7f56f00c3ff0 2023-11-28T12:59:51.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56f48eb700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f015cdb0 con 0x7f56d8003be0 2023-11-28T12:59:51.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56f48eb700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00c4460 con 0x7f56f006a270 2023-11-28T12:59:51.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56edd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56f00c3ff0 0x7f56f014b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56edd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56f00c3ff0 0x7f56f014b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34524/0 (socket says 172.21.15.111:34524) 2023-11-28T12:59:51.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56edd9b700 1 -- 172.21.15.111:0/2683608155 learned_addr learned my addr 172.21.15.111:0/2683608155 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:51.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56edd9b700 1 -- 172.21.15.111:0/2683608155 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f56f006a270 msgr2=0x7f56f006a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56ee59c700 1 --2- 172.21.15.111:0/2683608155 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f56d8003be0 0x7f56f014bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56edd9b700 1 --2- 172.21.15.111:0/2683608155 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f56f006a270 0x7f56f006a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56edd9b700 1 -- 172.21.15.111:0/2683608155 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f56d8003be0 msgr2=0x7f56f014bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56edd9b700 1 --2- 172.21.15.111:0/2683608155 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f56d8003be0 0x7f56f014bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56edd9b700 1 -- 172.21.15.111:0/2683608155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f00ac9b0 con 0x7f56f00c3ff0 2023-11-28T12:59:51.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56edd9b700 1 --2- 172.21.15.111:0/2683608155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56f00c3ff0 0x7f56f014b820 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f56e000f760 tx=0x7f56e000fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:51.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56deffd700 1 -- 172.21.15.111:0/2683608155 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e00053e0 con 0x7f56f00c3ff0 2023-11-28T12:59:51.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56deffd700 1 -- 172.21.15.111:0/2683608155 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e0005540 con 0x7f56f00c3ff0 2023-11-28T12:59:51.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56deffd700 1 -- 172.21.15.111:0/2683608155 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e0006aa0 con 0x7f56f00c3ff0 2023-11-28T12:59:51.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56f48eb700 1 -- 172.21.15.111:0/2683608155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f00b1230 con 0x7f56f00c3ff0 2023-11-28T12:59:51.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.133+0000 7f56f48eb700 1 -- 172.21.15.111:0/2683608155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f00acc90 con 0x7f56f00c3ff0 2023-11-28T12:59:51.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.134+0000 7f56f48eb700 1 -- 172.21.15.111:0/2683608155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f0067170 con 0x7f56f00c3ff0 2023-11-28T12:59:51.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.135+0000 7f56deffd700 1 -- 172.21.15.111:0/2683608155 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f56e0006c00 con 0x7f56f00c3ff0 2023-11-28T12:59:51.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.135+0000 7f56deffd700 1 --2- 172.21.15.111:0/2683608155 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56d00682f0 0x7f56d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.135+0000 7f56deffd700 1 -- 172.21.15.111:0/2683608155 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f56e0085590 con 0x7f56f00c3ff0 2023-11-28T12:59:51.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.135+0000 7f56ee59c700 1 --2- 172.21.15.111:0/2683608155 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56d00682f0 0x7f56d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.135+0000 7f56ee59c700 1 --2- 172.21.15.111:0/2683608155 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56d00682f0 0x7f56d006a7a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f56f006f2a0 tx=0x7f56e4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:51.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.139+0000 7f56deffd700 1 -- 172.21.15.111:0/2683608155 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f56e0051ac0 con 0x7f56f00c3ff0 2023-11-28T12:59:51.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.324+0000 7f56f48eb700 1 -- 172.21.15.111:0/2683608155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f56f00acf60 con 0x7f56f00c3ff0 2023-11-28T12:59:51.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.324+0000 7f56deffd700 1 -- 172.21.15.111:0/2683608155 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f56e00550e0 con 0x7f56f00c3ff0 2023-11-28T12:59:51.326 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T12:59:51.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.326+0000 7f56f48eb700 1 -- 172.21.15.111:0/2683608155 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56d00682f0 msgr2=0x7f56d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.326+0000 7f56f48eb700 1 --2- 172.21.15.111:0/2683608155 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56d00682f0 0x7f56d006a7a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f56f006f2a0 tx=0x7f56e4008030 comp rx=0 tx=0).stop 2023-11-28T12:59:51.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.326+0000 7f56f48eb700 1 -- 172.21.15.111:0/2683608155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56f00c3ff0 msgr2=0x7f56f014b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.326+0000 7f56f48eb700 1 --2- 172.21.15.111:0/2683608155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56f00c3ff0 0x7f56f014b820 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f56e000f760 tx=0x7f56e000fb20 comp rx=0 tx=0).stop 2023-11-28T12:59:51.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.326+0000 7f56f48eb700 1 -- 172.21.15.111:0/2683608155 shutdown_connections 2023-11-28T12:59:51.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.326+0000 7f56f48eb700 1 --2- 172.21.15.111:0/2683608155 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f56f006a270 0x7f56f006a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.326+0000 7f56f48eb700 1 --2- 172.21.15.111:0/2683608155 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56d00682f0 0x7f56d006a7a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.326+0000 7f56f48eb700 1 --2- 172.21.15.111:0/2683608155 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f56d8003be0 0x7f56f014bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.326+0000 7f56f48eb700 1 --2- 172.21.15.111:0/2683608155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56f00c3ff0 0x7f56f014b820 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.326+0000 7f56f48eb700 1 -- 172.21.15.111:0/2683608155 >> 172.21.15.111:0/2683608155 conn(0x7f56f006bca0 msgr2=0x7f56f006dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:51.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.326+0000 7f56f48eb700 1 -- 172.21.15.111:0/2683608155 shutdown_connections 2023-11-28T12:59:51.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.326+0000 7f56f48eb700 1 -- 172.21.15.111:0/2683608155 wait complete. 2023-11-28T12:59:51.340 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T12:59:51.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.462+0000 7fdd84821700 1 Processor -- start 2023-11-28T12:59:51.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.462+0000 7fdd84821700 1 -- start start 2023-11-28T12:59:51.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd84821700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdd800b0d70 0x7fdd801604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd84821700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdd800b1140 0x7fdd800ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd84821700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8015cd90 con 0x7fdd800b15b0 2023-11-28T12:59:51.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd84821700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd801613e0 con 0x7fdd800b1140 2023-11-28T12:59:51.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd84821700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd80160b40 con 0x7fdd800b0d70 2023-11-28T12:59:51.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd7dd9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdd800b1140 0x7fdd800ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd7dd9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdd800b1140 0x7fdd800ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34530/0 (socket says 172.21.15.111:34530) 2023-11-28T12:59:51.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd7dd9b700 1 -- 172.21.15.111:0/3492521519 learned_addr learned my addr 172.21.15.111:0/3492521519 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:51.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd7dd9b700 1 -- 172.21.15.111:0/3492521519 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdd800b0d70 msgr2=0x7fdd801604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:51.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd7e59c700 1 --2- 172.21.15.111:0/3492521519 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdd800b0d70 0x7fdd801604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd7dd9b700 1 --2- 172.21.15.111:0/3492521519 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdd800b0d70 0x7fdd801604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd7dd9b700 1 -- 172.21.15.111:0/3492521519 >> v1:172.21.15.149:6789/0 conn(0x7fdd800b15b0 legacy=0x7fdd800cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:51.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd7dd9b700 1 -- 172.21.15.111:0/3492521519 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd80160e20 con 0x7fdd800b1140 2023-11-28T12:59:51.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd7e59c700 1 --2- 172.21.15.111:0/3492521519 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdd800b0d70 0x7fdd801604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:51.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd7dd9b700 1 --2- 172.21.15.111:0/3492521519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd800b1140 0x7fdd800ce6c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdd740098f0 tx=0x7fdd74009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d273742469d1a869 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:51.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd7d59a700 1 -- 172.21.15.111:0/3492521519 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd74004230 con 0x7fdd800b1140 2023-11-28T12:59:51.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd7d59a700 1 -- 172.21.15.111:0/3492521519 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd74004390 con 0x7fdd800b1140 2023-11-28T12:59:51.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.463+0000 7fdd7d59a700 1 -- 172.21.15.111:0/3492521519 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd740036a0 con 0x7fdd800b1140 2023-11-28T12:59:51.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.464+0000 7fdd84821700 1 -- 172.21.15.111:0/3492521519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd800b1140 msgr2=0x7fdd800ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.464+0000 7fdd84821700 1 --2- 172.21.15.111:0/3492521519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd800b1140 0x7fdd800ce6c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdd740098f0 tx=0x7fdd74009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:51.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.464+0000 7fdd84821700 1 -- 172.21.15.111:0/3492521519 shutdown_connections 2023-11-28T12:59:51.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.464+0000 7fdd84821700 1 --2- 172.21.15.111:0/3492521519 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdd800b0d70 0x7fdd801604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.464+0000 7fdd84821700 1 --2- 172.21.15.111:0/3492521519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd800b1140 0x7fdd800ce6c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.464+0000 7fdd84821700 1 -- 172.21.15.111:0/3492521519 >> 172.21.15.111:0/3492521519 conn(0x7fdd80098ee0 msgr2=0x7fdd8006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:51.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.464+0000 7fdd84821700 1 -- 172.21.15.111:0/3492521519 shutdown_connections 2023-11-28T12:59:51.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.464+0000 7fdd84821700 1 -- 172.21.15.111:0/3492521519 wait complete. 2023-11-28T12:59:51.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.465+0000 7fdd84821700 1 Processor -- start 2023-11-28T12:59:51.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.465+0000 7fdd84821700 1 -- start start 2023-11-28T12:59:51.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.466+0000 7fdd84821700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdd800b0d70 0x7fdd800b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.466+0000 7fdd84821700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdd800b1140 0x7fdd800b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.466+0000 7fdd84821700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd800b15b0 0x7fdd800cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.466+0000 7fdd84821700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8015dd40 con 0x7fdd800b15b0 2023-11-28T12:59:51.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.466+0000 7fdd84821700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd800b5490 con 0x7fdd800b0d70 2023-11-28T12:59:51.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.466+0000 7fdd84821700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd800b5770 con 0x7fdd800b1140 2023-11-28T12:59:51.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.466+0000 7fdd7ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd800b15b0 0x7fdd800cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.466+0000 7fdd7ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd800b15b0 0x7fdd800cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34538/0 (socket says 172.21.15.111:34538) 2023-11-28T12:59:51.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.466+0000 7fdd7ed9d700 1 -- 172.21.15.111:0/2942623603 learned_addr learned my addr 172.21.15.111:0/2942623603 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:51.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.466+0000 7fdd7e59c700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdd800b0d70 0x7fdd800b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.466+0000 7fdd7dd9b700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdd800b1140 0x7fdd800b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.467+0000 7fdd7ed9d700 1 -- 172.21.15.111:0/2942623603 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdd800b1140 msgr2=0x7fdd800b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.467+0000 7fdd7ed9d700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdd800b1140 0x7fdd800b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.467+0000 7fdd7ed9d700 1 -- 172.21.15.111:0/2942623603 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdd800b0d70 msgr2=0x7fdd800b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.467+0000 7fdd7ed9d700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdd800b0d70 0x7fdd800b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.467+0000 7fdd7ed9d700 1 -- 172.21.15.111:0/2942623603 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd74009550 con 0x7fdd800b15b0 2023-11-28T12:59:51.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.467+0000 7fdd7dd9b700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdd800b1140 0x7fdd800b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:51.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.467+0000 7fdd7ed9d700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd800b15b0 0x7fdd800cb7b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800be60 tx=0x7fdd78009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:51.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.467+0000 7fdd6effd700 1 -- 172.21.15.111:0/2942623603 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7800cd10 con 0x7fdd800b15b0 2023-11-28T12:59:51.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.467+0000 7fdd6effd700 1 -- 172.21.15.111:0/2942623603 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd78014440 con 0x7fdd800b15b0 2023-11-28T12:59:51.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.467+0000 7fdd84821700 1 -- 172.21.15.111:0/2942623603 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd800af340 con 0x7fdd800b15b0 2023-11-28T12:59:51.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.467+0000 7fdd6effd700 1 -- 172.21.15.111:0/2942623603 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd780147f0 con 0x7fdd800b15b0 2023-11-28T12:59:51.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.467+0000 7fdd84821700 1 -- 172.21.15.111:0/2942623603 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd80161100 con 0x7fdd800b15b0 2023-11-28T12:59:51.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.468+0000 7fdd84821700 1 -- 172.21.15.111:0/2942623603 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd800cec00 con 0x7fdd800b15b0 2023-11-28T12:59:51.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.468+0000 7fdd6effd700 1 -- 172.21.15.111:0/2942623603 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdd7801b030 con 0x7fdd800b15b0 2023-11-28T12:59:51.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.469+0000 7fdd6effd700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdd600682f0 0x7fdd6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.469+0000 7fdd6effd700 1 -- 172.21.15.111:0/2942623603 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fdd78085de0 con 0x7fdd800b15b0 2023-11-28T12:59:51.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.469+0000 7fdd7e59c700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdd600682f0 0x7fdd6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.469+0000 7fdd7e59c700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdd600682f0 0x7fdd6006a7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000abe0 tx=0x7fdd70008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:51.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.473+0000 7fdd6effd700 1 -- 172.21.15.111:0/2942623603 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdd7804d5e0 con 0x7fdd800b15b0 2023-11-28T12:59:51.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.654+0000 7fdd84821700 1 -- 172.21.15.111:0/2942623603 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fdd8015d830 con 0x7fdd800b15b0 2023-11-28T12:59:51.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.655+0000 7fdd6effd700 1 -- 172.21.15.111:0/2942623603 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7fdd8015d830 con 0x7fdd800b15b0 2023-11-28T12:59:51.656 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:51.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.657+0000 7fdd84821700 1 -- 172.21.15.111:0/2942623603 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdd600682f0 msgr2=0x7fdd6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.657+0000 7fdd84821700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdd600682f0 0x7fdd6006a7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000abe0 tx=0x7fdd70008040 comp rx=0 tx=0).stop 2023-11-28T12:59:51.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.657+0000 7fdd84821700 1 -- 172.21.15.111:0/2942623603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd800b15b0 msgr2=0x7fdd800cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.657+0000 7fdd84821700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd800b15b0 0x7fdd800cb7b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800be60 tx=0x7fdd78009f30 comp rx=0 tx=0).stop 2023-11-28T12:59:51.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.657+0000 7fdd84821700 1 -- 172.21.15.111:0/2942623603 shutdown_connections 2023-11-28T12:59:51.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.657+0000 7fdd84821700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd800b15b0 0x7fdd800cb7b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.657+0000 7fdd84821700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdd600682f0 0x7fdd6006a7a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.657+0000 7fdd84821700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdd800b0d70 0x7fdd800b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.657+0000 7fdd84821700 1 --2- 172.21.15.111:0/2942623603 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdd800b1140 0x7fdd800b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.657+0000 7fdd84821700 1 -- 172.21.15.111:0/2942623603 >> 172.21.15.111:0/2942623603 conn(0x7fdd80098ee0 msgr2=0x7fdd8015c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:51.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.657+0000 7fdd84821700 1 -- 172.21.15.111:0/2942623603 shutdown_connections 2023-11-28T12:59:51.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.658+0000 7fdd84821700 1 -- 172.21.15.111:0/2942623603 wait complete. 2023-11-28T12:59:51.672 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T12:59:51.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.791+0000 7f28fbef7700 1 Processor -- start 2023-11-28T12:59:51.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.791+0000 7f28fbef7700 1 -- start start 2023-11-28T12:59:51.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28fbef7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28f40cfc90 0x7f28f40d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28fbef7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28f40cc5a0 0x7f28f40cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28fbef7700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f40d05a0 con 0x7f28f40cdd20 2023-11-28T12:59:51.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28fbef7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f40ce1c0 con 0x7f28f40cfc90 2023-11-28T12:59:51.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28fbef7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40ca260 con 0x7f28f40cc5a0 2023-11-28T12:59:51.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28f9c93700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28f40cfc90 0x7f28f40d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28f9c93700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28f40cfc90 0x7f28f40d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34542/0 (socket says 172.21.15.111:34542) 2023-11-28T12:59:51.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28f9c93700 1 -- 172.21.15.111:0/1200432484 learned_addr learned my addr 172.21.15.111:0/1200432484 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:51.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28f9492700 1 --2- 172.21.15.111:0/1200432484 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28f40cc5a0 0x7f28f40cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28f9c93700 1 -- 172.21.15.111:0/1200432484 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28f40cc5a0 msgr2=0x7f28f40cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:51.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28f9c93700 1 --2- 172.21.15.111:0/1200432484 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28f40cc5a0 0x7f28f40cd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28f9c93700 1 -- 172.21.15.111:0/1200432484 >> v1:172.21.15.111:6789/0 conn(0x7f28f40cdd20 legacy=0x7f28f415d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28f9c93700 1 -- 172.21.15.111:0/1200432484 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f40ccaa0 con 0x7f28f40cfc90 2023-11-28T12:59:51.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.792+0000 7f28f9492700 1 --2- 172.21.15.111:0/1200432484 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28f40cc5a0 0x7f28f40cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.793+0000 7f28f9c93700 1 --2- 172.21.15.111:0/1200432484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f40cfc90 0x7f28f40d0060 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f28e40098f0 tx=0x7f28e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43aaa715e2d598b8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.793+0000 7f28f8c91700 1 -- 172.21.15.111:0/1200432484 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e400b8a0 con 0x7f28f40cfc90 2023-11-28T12:59:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.793+0000 7f28f8c91700 1 -- 172.21.15.111:0/1200432484 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e400ba00 con 0x7f28f40cfc90 2023-11-28T12:59:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.793+0000 7f28f8c91700 1 -- 172.21.15.111:0/1200432484 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e400bd90 con 0x7f28f40cfc90 2023-11-28T12:59:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.793+0000 7f28fbef7700 1 -- 172.21.15.111:0/1200432484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f40cfc90 msgr2=0x7f28f40d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.793+0000 7f28fbef7700 1 --2- 172.21.15.111:0/1200432484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f40cfc90 0x7f28f40d0060 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f28e40098f0 tx=0x7f28e4009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.793+0000 7f28fbef7700 1 -- 172.21.15.111:0/1200432484 shutdown_connections 2023-11-28T12:59:51.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.793+0000 7f28fbef7700 1 --2- 172.21.15.111:0/1200432484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f40cfc90 0x7f28f40d0060 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.793+0000 7f28fbef7700 1 --2- 172.21.15.111:0/1200432484 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28f40cc5a0 0x7f28f40cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.793+0000 7f28fbef7700 1 -- 172.21.15.111:0/1200432484 >> 172.21.15.111:0/1200432484 conn(0x7f28f4098ee0 msgr2=0x7f28f406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:51.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.793+0000 7f28fbef7700 1 -- 172.21.15.111:0/1200432484 shutdown_connections 2023-11-28T12:59:51.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.793+0000 7f28fbef7700 1 -- 172.21.15.111:0/1200432484 wait complete. 2023-11-28T12:59:51.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.795+0000 7f28fbef7700 1 Processor -- start 2023-11-28T12:59:51.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.795+0000 7f28fbef7700 1 -- start start 2023-11-28T12:59:51.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.795+0000 7f28fbef7700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28f40cc5a0 0x7f28f40b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.795+0000 7f28fbef7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28f40cdd20 0x7f28f40c3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.795+0000 7f28fbef7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f40cfc90 0x7f28f40c3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.795+0000 7f28fbef7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40b4050 con 0x7f28f40cfc90 2023-11-28T12:59:51.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.795+0000 7f28fbef7700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40b4330 con 0x7f28f40cc5a0 2023-11-28T12:59:51.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.795+0000 7f28fbef7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40b4610 con 0x7f28f40cdd20 2023-11-28T12:59:51.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.795+0000 7f28fa494700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f40cfc90 0x7f28f40c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.795+0000 7f28fa494700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f40cfc90 0x7f28f40c3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34552/0 (socket says 172.21.15.111:34552) 2023-11-28T12:59:51.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.795+0000 7f28fa494700 1 -- 172.21.15.111:0/3740415528 learned_addr learned my addr 172.21.15.111:0/3740415528 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:51.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28f9c93700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28f40cc5a0 0x7f28f40b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28f9492700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28f40cdd20 0x7f28f40c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28fa494700 1 -- 172.21.15.111:0/3740415528 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28f40cdd20 msgr2=0x7f28f40c3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28fa494700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28f40cdd20 0x7f28f40c3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28fa494700 1 -- 172.21.15.111:0/3740415528 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28f40cc5a0 msgr2=0x7f28f40b50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:51.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28fa494700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28f40cc5a0 0x7f28f40b50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:51.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28fa494700 1 -- 172.21.15.111:0/3740415528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e4009550 con 0x7f28f40cfc90 2023-11-28T12:59:51.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28f9c93700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28f40cc5a0 0x7f28f40b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:51.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28fa494700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f40cfc90 0x7f28f40c3780 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00cc50 tx=0x7f28ec00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:51.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28ea7fc700 1 -- 172.21.15.111:0/3740415528 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec00ad80 con 0x7f28f40cfc90 2023-11-28T12:59:51.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28ea7fc700 1 -- 172.21.15.111:0/3740415528 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ec00ce50 con 0x7f28f40cfc90 2023-11-28T12:59:51.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28fbef7700 1 -- 172.21.15.111:0/3740415528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f40cd060 con 0x7f28f40cfc90 2023-11-28T12:59:51.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28ea7fc700 1 -- 172.21.15.111:0/3740415528 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec003c40 con 0x7f28f40cfc90 2023-11-28T12:59:51.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.796+0000 7f28fbef7700 1 -- 172.21.15.111:0/3740415528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f40ccd80 con 0x7f28f40cfc90 2023-11-28T12:59:51.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.797+0000 7f28fbef7700 1 -- 172.21.15.111:0/3740415528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f40cd340 con 0x7f28f40cfc90 2023-11-28T12:59:51.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.797+0000 7f28ea7fc700 1 -- 172.21.15.111:0/3740415528 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f28ec010050 con 0x7f28f40cfc90 2023-11-28T12:59:51.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.798+0000 7f28ea7fc700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28d80682f0 0x7f28d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:51.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.798+0000 7f28ea7fc700 1 -- 172.21.15.111:0/3740415528 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f28ec085dc0 con 0x7f28f40cfc90 2023-11-28T12:59:51.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.798+0000 7f28f9c93700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28d80682f0 0x7f28d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:51.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.798+0000 7f28f9c93700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28d80682f0 0x7f28d806a7a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f28e4014040 tx=0x7f28e4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:51.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:51.802+0000 7f28ea7fc700 1 -- 172.21.15.111:0/3740415528 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f28ec0522f0 con 0x7f28f40cfc90 2023-11-28T12:59:52.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.001+0000 7f28fbef7700 1 -- 172.21.15.111:0/3740415528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f28f40d05a0 con 0x7f28f40cfc90 2023-11-28T12:59:52.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.001+0000 7f28ea7fc700 1 -- 172.21.15.111:0/3740415528 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) v1 ==== 80+0+0 (secure 0 0 0) 0x7f28f40d05a0 con 0x7f28f40cfc90 2023-11-28T12:59:52.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.004+0000 7f28fbef7700 1 -- 172.21.15.111:0/3740415528 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28d80682f0 msgr2=0x7f28d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.004+0000 7f28fbef7700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28d80682f0 0x7f28d806a7a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f28e4014040 tx=0x7f28e4008030 comp rx=0 tx=0).stop 2023-11-28T12:59:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.004+0000 7f28fbef7700 1 -- 172.21.15.111:0/3740415528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f40cfc90 msgr2=0x7f28f40c3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.004+0000 7f28fbef7700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f40cfc90 0x7f28f40c3780 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00cc50 tx=0x7f28ec00a340 comp rx=0 tx=0).stop 2023-11-28T12:59:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.004+0000 7f28fbef7700 1 -- 172.21.15.111:0/3740415528 shutdown_connections 2023-11-28T12:59:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.004+0000 7f28fbef7700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f40cfc90 0x7f28f40c3780 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.004+0000 7f28fbef7700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28d80682f0 0x7f28d806a7a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.004+0000 7f28fbef7700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28f40cc5a0 0x7f28f40b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.004+0000 7f28fbef7700 1 --2- 172.21.15.111:0/3740415528 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28f40cdd20 0x7f28f40c3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.004+0000 7f28fbef7700 1 -- 172.21.15.111:0/3740415528 >> 172.21.15.111:0/3740415528 conn(0x7f28f4098ee0 msgr2=0x7f28f40acd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:52.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.004+0000 7f28fbef7700 1 -- 172.21.15.111:0/3740415528 shutdown_connections 2023-11-28T12:59:52.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.004+0000 7f28fbef7700 1 -- 172.21.15.111:0/3740415528 wait complete. 2023-11-28T12:59:52.007 INFO:teuthology.orchestra.run.smithi111.stderr:listed 0 entries 2023-11-28T12:59:52.017 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-11-28T12:59:52.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:52.017 DEBUG:teuthology.orchestra.run.smithi111:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-11-28T12:59:52.057 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T12:59:52.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:52.057 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-28T12:59:52.114 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:59:52.114 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:52.136 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:59:52.137 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:52.196 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T12:59:52.197 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T12:59:52.257 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:52.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.375+0000 7f5b756ec700 1 Processor -- start 2023-11-28T12:59:52.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.375+0000 7f5b756ec700 1 -- start start 2023-11-28T12:59:52.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.376+0000 7f5b756ec700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5b7015c800 0x7f5b70160e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.376+0000 7f5b756ec700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f5b700cd790 0x7f5b700cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.376+0000 7f5b756ec700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7015cbd0 con 0x7f5b7006bfd0 2023-11-28T12:59:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.376+0000 7f5b756ec700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b700ce150 con 0x7f5b700cd790 2023-11-28T12:59:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.376+0000 7f5b756ec700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7006af30 con 0x7f5b7015c800 2023-11-28T12:59:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.376+0000 7f5b6effd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5b7015c800 0x7f5b70160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.376+0000 7f5b6f7fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f5b700cd790 0x7f5b700cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.376+0000 7f5b6e7fc700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f5b7006bfd0 0x7f5b7006a6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:43878/0 (socket says 172.21.15.111:43878) 2023-11-28T12:59:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.376+0000 7f5b6f7fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f5b700cd790 0x7f5b700cdc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:52460/0 (socket says 172.21.15.111:52460) 2023-11-28T12:59:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.376+0000 7f5b6e7fc700 1 -- 172.21.15.111:0/1144337710 learned_addr learned my addr 172.21.15.111:0/1144337710 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.376+0000 7f5b6effd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5b7015c800 0x7f5b70160e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:56304/0 (socket says 172.21.15.111:56304) 2023-11-28T12:59:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.377+0000 7f5b6f7fe700 1 -- 172.21.15.111:0/1144337710 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5b7015c800 msgr2=0x7f5b70160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.377+0000 7f5b6f7fe700 1 --2- 172.21.15.111:0/1144337710 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5b7015c800 0x7f5b70160e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.377+0000 7f5b6f7fe700 1 -- 172.21.15.111:0/1144337710 >> v1:172.21.15.149:6789/0 conn(0x7f5b7006bfd0 legacy=0x7f5b7006a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.377+0000 7f5b6f7fe700 1 -- 172.21.15.111:0/1144337710 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7006b210 con 0x7f5b700cd790 2023-11-28T12:59:52.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.377+0000 7f5b6f7fe700 1 --2- 172.21.15.111:0/1144337710 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b700cd790 0x7f5b700cdc10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5b68009d90 tx=0x7f5b6800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3cf29847c249a06 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:52.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.377+0000 7f5b6dffb700 1 -- 172.21.15.111:0/1144337710 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6800e070 con 0x7f5b700cd790 2023-11-28T12:59:52.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.377+0000 7f5b6dffb700 1 -- 172.21.15.111:0/1144337710 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6800bb20 con 0x7f5b700cd790 2023-11-28T12:59:52.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.377+0000 7f5b6dffb700 1 -- 172.21.15.111:0/1144337710 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b68012430 con 0x7f5b700cd790 2023-11-28T12:59:52.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.378+0000 7f5b756ec700 1 -- 172.21.15.111:0/1144337710 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b700cd790 msgr2=0x7f5b700cdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.378+0000 7f5b756ec700 1 --2- 172.21.15.111:0/1144337710 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b700cd790 0x7f5b700cdc10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5b68009d90 tx=0x7f5b6800b330 comp rx=0 tx=0).stop 2023-11-28T12:59:52.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.378+0000 7f5b756ec700 1 -- 172.21.15.111:0/1144337710 shutdown_connections 2023-11-28T12:59:52.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.378+0000 7f5b756ec700 1 --2- 172.21.15.111:0/1144337710 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b700cd790 0x7f5b700cdc10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.378+0000 7f5b756ec700 1 --2- 172.21.15.111:0/1144337710 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5b7015c800 0x7f5b70160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.378+0000 7f5b756ec700 1 -- 172.21.15.111:0/1144337710 >> 172.21.15.111:0/1144337710 conn(0x7f5b7006db10 msgr2=0x7f5b7006df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:52.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.378+0000 7f5b756ec700 1 -- 172.21.15.111:0/1144337710 shutdown_connections 2023-11-28T12:59:52.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.378+0000 7f5b756ec700 1 -- 172.21.15.111:0/1144337710 wait complete. 2023-11-28T12:59:52.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.379+0000 7f5b756ec700 1 Processor -- start 2023-11-28T12:59:52.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.379+0000 7f5b756ec700 1 -- start start 2023-11-28T12:59:52.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b756ec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b7006bfd0 0x7f5b700b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b756ec700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5b700cd790 0x7f5b70153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b756ec700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b7015c800 0x7f5b7015f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b756ec700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7015e560 con 0x7f5b7006bfd0 2023-11-28T12:59:52.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b756ec700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7015f0c0 con 0x7f5b7015c800 2023-11-28T12:59:52.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b756ec700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b700b4080 con 0x7f5b700cd790 2023-11-28T12:59:52.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b6effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b7006bfd0 0x7f5b700b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b6effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b7006bfd0 0x7f5b700b38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34562/0 (socket says 172.21.15.111:34562) 2023-11-28T12:59:52.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b6effd700 1 -- 172.21.15.111:0/1664813603 learned_addr learned my addr 172.21.15.111:0/1664813603 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:52.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b6e7fc700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5b700cd790 0x7f5b70153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b6f7fe700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b7015c800 0x7f5b7015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b6effd700 1 -- 172.21.15.111:0/1664813603 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5b700cd790 msgr2=0x7f5b70153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b6effd700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5b700cd790 0x7f5b70153f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b6effd700 1 -- 172.21.15.111:0/1664813603 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b7015c800 msgr2=0x7f5b7015f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b6effd700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b7015c800 0x7f5b7015f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b6effd700 1 -- 172.21.15.111:0/1664813603 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b68009a70 con 0x7f5b7006bfd0 2023-11-28T12:59:52.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.380+0000 7f5b6effd700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b7006bfd0 0x7f5b700b38e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400e9b0 tx=0x7f5b6400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:52.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.381+0000 7f5b577fe700 1 -- 172.21.15.111:0/1664813603 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b64009710 con 0x7f5b7006bfd0 2023-11-28T12:59:52.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.381+0000 7f5b577fe700 1 -- 172.21.15.111:0/1664813603 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6400ee50 con 0x7f5b7006bfd0 2023-11-28T12:59:52.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.381+0000 7f5b577fe700 1 -- 172.21.15.111:0/1664813603 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6400ca50 con 0x7f5b7006bfd0 2023-11-28T12:59:52.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.381+0000 7f5b756ec700 1 -- 172.21.15.111:0/1664813603 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7006b7d0 con 0x7f5b7006bfd0 2023-11-28T12:59:52.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.381+0000 7f5b756ec700 1 -- 172.21.15.111:0/1664813603 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7006b4f0 con 0x7f5b7006bfd0 2023-11-28T12:59:52.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.382+0000 7f5b756ec700 1 -- 172.21.15.111:0/1664813603 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7006bab0 con 0x7f5b7006bfd0 2023-11-28T12:59:52.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.382+0000 7f5b577fe700 1 -- 172.21.15.111:0/1664813603 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5b64019030 con 0x7f5b7006bfd0 2023-11-28T12:59:52.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.383+0000 7f5b577fe700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5b5c0682f0 0x7f5b5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.383+0000 7f5b577fe700 1 -- 172.21.15.111:0/1664813603 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f5b640865a0 con 0x7f5b7006bfd0 2023-11-28T12:59:52.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.383+0000 7f5b6e7fc700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5b5c0682f0 0x7f5b5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.383+0000 7f5b6e7fc700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5b5c0682f0 0x7f5b5c06a7a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5b700cd1f0 tx=0x7f5b60007f10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:52.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.386+0000 7f5b577fe700 1 -- 172.21.15.111:0/1664813603 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5b640565a0 con 0x7f5b7006bfd0 2023-11-28T12:59:52.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.571+0000 7f5b756ec700 1 -- 172.21.15.111:0/1664813603 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b70067170 con 0x7f5b7006bfd0 2023-11-28T12:59:52.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.572+0000 7f5b577fe700 1 -- 172.21.15.111:0/1664813603 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+19890 (secure 0 0 0) 0x7f5b64056190 con 0x7f5b7006bfd0 2023-11-28T12:59:52.574 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:52.574 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":28,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:41.880678+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:52.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.574+0000 7f5b756ec700 1 -- 172.21.15.111:0/1664813603 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5b5c0682f0 msgr2=0x7f5b5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.574+0000 7f5b756ec700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5b5c0682f0 0x7f5b5c06a7a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5b700cd1f0 tx=0x7f5b60007f10 comp rx=0 tx=0).stop 2023-11-28T12:59:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.574+0000 7f5b756ec700 1 -- 172.21.15.111:0/1664813603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b7006bfd0 msgr2=0x7f5b700b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.574+0000 7f5b756ec700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b7006bfd0 0x7f5b700b38e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400e9b0 tx=0x7f5b6400ecc0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.574+0000 7f5b756ec700 1 -- 172.21.15.111:0/1664813603 shutdown_connections 2023-11-28T12:59:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.574+0000 7f5b756ec700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5b5c0682f0 0x7f5b5c06a7a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.574+0000 7f5b756ec700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5b7015c800 0x7f5b7015f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.574+0000 7f5b756ec700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b7006bfd0 0x7f5b700b38e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.574+0000 7f5b756ec700 1 --2- 172.21.15.111:0/1664813603 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5b700cd790 0x7f5b70153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.574+0000 7f5b756ec700 1 -- 172.21.15.111:0/1664813603 >> 172.21.15.111:0/1664813603 conn(0x7f5b7006db10 msgr2=0x7f5b70156870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.574+0000 7f5b756ec700 1 -- 172.21.15.111:0/1664813603 shutdown_connections 2023-11-28T12:59:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.574+0000 7f5b756ec700 1 -- 172.21.15.111:0/1664813603 wait complete. 2023-11-28T12:59:52.588 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T12:58:53.275206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T12:58:54.588499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T12:58:57.666033+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T12:59:52.588 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:59:52.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.711+0000 7f175fe8f700 1 Processor -- start 2023-11-28T12:59:52.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.711+0000 7f175fe8f700 1 -- start start 2023-11-28T12:59:52.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175fe8f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f175806a980 0x7f17581652a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175fe8f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175806b490 con 0x7f1758154b60 2023-11-28T12:59:52.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175fe8f700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17580c9e80 con 0x7f17580cb830 2023-11-28T12:59:52.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175fe8f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17580c9930 con 0x7f175806a980 2023-11-28T12:59:52.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175dc2b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1758154b60 0x7f175815b350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35348/0 (socket says 172.21.15.111:35348) 2023-11-28T12:59:52.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175dc2b700 1 -- 172.21.15.111:0/3586101635 learned_addr learned my addr 172.21.15.111:0/3586101635 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:52.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175e42c700 1 --2- 172.21.15.111:0/3586101635 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f175806a980 0x7f17581652a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762990641 0 0) 0x7f175806b490 con 0x7f1758154b60 2023-11-28T12:59:52.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17440035b0 con 0x7f1758154b60 2023-11-28T12:59:52.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3648218712 0 0) 0x7f17440035b0 con 0x7f1758154b60 2023-11-28T12:59:52.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f175806a980 msgr2=0x7f17581652a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:52.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175cc29700 1 --2- 172.21.15.111:0/3586101635 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f175806a980 0x7f17581652a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 >> v1:172.21.15.149:6789/0 conn(0x7f17580cb830 legacy=0x7f175806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175806b6e0 con 0x7f1758154b60 2023-11-28T12:59:52.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f1754001670 con 0x7f1758154b60 2023-11-28T12:59:52.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.712+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 >> v1:172.21.15.111:6789/0 conn(0x7f1758154b60 legacy=0x7f175815b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175cc29700 1 --2- 172.21.15.111:0/3586101635 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f175806a980 0x7f1744003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175cc29700 1 --2- 172.21.15.111:0/3586101635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1744003f70 0x7f1744004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175cc29700 1 --2- 172.21.15.111:0/3586101635 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1744004d90 0x7f1744005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17580c9930 con 0x7f1744003f70 2023-11-28T12:59:52.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174400ba40 con 0x7f1744004d90 2023-11-28T12:59:52.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174400bc00 con 0x7f175806a980 2023-11-28T12:59:52.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175d42a700 1 --2- 172.21.15.111:0/3586101635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1744003f70 0x7f1744004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175dc2b700 1 --2- 172.21.15.111:0/3586101635 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f175806a980 0x7f1744003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175e42c700 1 --2- 172.21.15.111:0/3586101635 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1744004d90 0x7f1744005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175d42a700 1 -- 172.21.15.111:0/3586101635 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f175806a980 msgr2=0x7f1744003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175d42a700 1 --2- 172.21.15.111:0/3586101635 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f175806a980 0x7f1744003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175d42a700 1 -- 172.21.15.111:0/3586101635 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1744004d90 msgr2=0x7f1744005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175d42a700 1 --2- 172.21.15.111:0/3586101635 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1744004d90 0x7f1744005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175d42a700 1 -- 172.21.15.111:0/3586101635 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17440035b0 con 0x7f1744003f70 2023-11-28T12:59:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175d42a700 1 --2- 172.21.15.111:0/3586101635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1744003f70 0x7f1744004850 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f17580cf290 tx=0x7f1744005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e2ab465d2c02ef server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174800c660 con 0x7f1744003f70 2023-11-28T12:59:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174800c7c0 con 0x7f1744003f70 2023-11-28T12:59:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.713+0000 7f175cc29700 1 -- 172.21.15.111:0/3586101635 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174800cb70 con 0x7f1744003f70 2023-11-28T12:59:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.714+0000 7f175fe8f700 1 -- 172.21.15.111:0/3586101635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1744003f70 msgr2=0x7f1744004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.714+0000 7f175fe8f700 1 --2- 172.21.15.111:0/3586101635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1744003f70 0x7f1744004850 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f17580cf290 tx=0x7f1744005880 comp rx=0 tx=0).stop 2023-11-28T12:59:52.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.714+0000 7f175e42c700 1 -- 172.21.15.111:0/3586101635 reap_dead start 2023-11-28T12:59:52.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.714+0000 7f175fe8f700 1 -- 172.21.15.111:0/3586101635 shutdown_connections 2023-11-28T12:59:52.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.714+0000 7f175fe8f700 1 -- 172.21.15.111:0/3586101635 >> 172.21.15.111:0/3586101635 conn(0x7f17580cf2e0 msgr2=0x7f17580cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:52.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.714+0000 7f175fe8f700 1 -- 172.21.15.111:0/3586101635 shutdown_connections 2023-11-28T12:59:52.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.714+0000 7f175fe8f700 1 -- 172.21.15.111:0/3586101635 wait complete. 2023-11-28T12:59:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.715+0000 7f175fe8f700 1 Processor -- start 2023-11-28T12:59:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.715+0000 7f175fe8f700 1 -- start start 2023-11-28T12:59:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175fe8f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1744003f70 0x7f1758151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175fe8f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1758151510 0x7f1758150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175fe8f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17581574e0 0x7f1758157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175fe8f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175815ef20 con 0x7f17581574e0 2023-11-28T12:59:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175fe8f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1758151320 con 0x7f1758151510 2023-11-28T12:59:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175fe8f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1758156160 con 0x7f1744003f70 2023-11-28T12:59:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175e42c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17581574e0 0x7f1758157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175e42c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17581574e0 0x7f1758157980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34586/0 (socket says 172.21.15.111:34586) 2023-11-28T12:59:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175e42c700 1 -- 172.21.15.111:0/2252867521 learned_addr learned my addr 172.21.15.111:0/2252867521 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175dc2b700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1744003f70 0x7f1758151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175d42a700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1758151510 0x7f1758150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175e42c700 1 -- 172.21.15.111:0/2252867521 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1744003f70 msgr2=0x7f1758151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175e42c700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1744003f70 0x7f1758151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175e42c700 1 -- 172.21.15.111:0/2252867521 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1758151510 msgr2=0x7f1758150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175e42c700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1758151510 0x7f1758150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175e42c700 1 -- 172.21.15.111:0/2252867521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17580c9b10 con 0x7f17581574e0 2023-11-28T12:59:52.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f175e42c700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17581574e0 0x7f1758157980 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f175806ad70 tx=0x7f1750011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:52.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.716+0000 7f174e7fc700 1 -- 172.21.15.111:0/2252867521 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175000e040 con 0x7f17581574e0 2023-11-28T12:59:52.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.717+0000 7f174e7fc700 1 -- 172.21.15.111:0/2252867521 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175000f040 con 0x7f17581574e0 2023-11-28T12:59:52.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.717+0000 7f174e7fc700 1 -- 172.21.15.111:0/2252867521 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175000fc70 con 0x7f17581574e0 2023-11-28T12:59:52.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.717+0000 7f175fe8f700 1 -- 172.21.15.111:0/2252867521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f175806b9c0 con 0x7f17581574e0 2023-11-28T12:59:52.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.717+0000 7f175fe8f700 1 -- 172.21.15.111:0/2252867521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f175806e990 con 0x7f17581574e0 2023-11-28T12:59:52.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.717+0000 7f175fe8f700 1 -- 172.21.15.111:0/2252867521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1758067170 con 0x7f17581574e0 2023-11-28T12:59:52.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.718+0000 7f174e7fc700 1 -- 172.21.15.111:0/2252867521 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1750022460 con 0x7f17581574e0 2023-11-28T12:59:52.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.718+0000 7f174e7fc700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f173c0682f0 0x7f173c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:52.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.718+0000 7f175dc2b700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f173c0682f0 0x7f173c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:52.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.718+0000 7f174e7fc700 1 -- 172.21.15.111:0/2252867521 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1750086520 con 0x7f17581574e0 2023-11-28T12:59:52.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.719+0000 7f175dc2b700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f173c0682f0 0x7f173c06a7a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f17581f3610 tx=0x7f1754001c40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:52.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.723+0000 7f174e7fc700 1 -- 172.21.15.111:0/2252867521 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f17500529f0 con 0x7f17581574e0 2023-11-28T12:59:52.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.905+0000 7f175fe8f700 1 -- 172.21.15.111:0/2252867521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17581595b0 con 0x7f17581574e0 2023-11-28T12:59:52.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.906+0000 7f174e7fc700 1 -- 172.21.15.111:0/2252867521 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1750056010 con 0x7f17581574e0 2023-11-28T12:59:52.907 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:59:52.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.907+0000 7f175fe8f700 1 -- 172.21.15.111:0/2252867521 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f173c0682f0 msgr2=0x7f173c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.907+0000 7f175fe8f700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f173c0682f0 0x7f173c06a7a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f17581f3610 tx=0x7f1754001c40 comp rx=0 tx=0).stop 2023-11-28T12:59:52.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.908+0000 7f175fe8f700 1 -- 172.21.15.111:0/2252867521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17581574e0 msgr2=0x7f1758157980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:52.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.908+0000 7f175fe8f700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17581574e0 0x7f1758157980 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f175806ad70 tx=0x7f1750011ed0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.908+0000 7f175fe8f700 1 -- 172.21.15.111:0/2252867521 shutdown_connections 2023-11-28T12:59:52.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.908+0000 7f175fe8f700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17581574e0 0x7f1758157980 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.908+0000 7f175fe8f700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1744003f70 0x7f1758151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.908+0000 7f175fe8f700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f173c0682f0 0x7f173c06a7a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.908+0000 7f175fe8f700 1 --2- 172.21.15.111:0/2252867521 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1758151510 0x7f1758150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:52.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.908+0000 7f175fe8f700 1 -- 172.21.15.111:0/2252867521 >> 172.21.15.111:0/2252867521 conn(0x7f17580cf2e0 msgr2=0x7f17580cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:52.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.908+0000 7f175fe8f700 1 -- 172.21.15.111:0/2252867521 shutdown_connections 2023-11-28T12:59:52.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:52.908+0000 7f175fe8f700 1 -- 172.21.15.111:0/2252867521 wait complete. 2023-11-28T12:59:52.921 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:59:53.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.042+0000 7fbb5bc69700 1 Processor -- start 2023-11-28T12:59:53.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.042+0000 7fbb5bc69700 1 -- start start 2023-11-28T12:59:53.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb5bc69700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbb541604b0 0x7fbb540ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb5bc69700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb540cc550 0x7fbb540cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb5bc69700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5415cd90 con 0x7fbb540b0d70 2023-11-28T12:59:53.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb5bc69700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb54160950 con 0x7fbb541604b0 2023-11-28T12:59:53.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb5bc69700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb540ccf10 con 0x7fbb540cc550 2023-11-28T12:59:53.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb59204700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbb541604b0 0x7fbb540ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb59204700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbb541604b0 0x7fbb540ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34592/0 (socket says 172.21.15.111:34592) 2023-11-28T12:59:53.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb59204700 1 -- 172.21.15.111:0/902393877 learned_addr learned my addr 172.21.15.111:0/902393877 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:53.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb5a206700 1 --2- 172.21.15.111:0/902393877 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb540cc550 0x7fbb540cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb59204700 1 -- 172.21.15.111:0/902393877 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb540cc550 msgr2=0x7fbb540cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:53.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb59204700 1 --2- 172.21.15.111:0/902393877 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb540cc550 0x7fbb540cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb59204700 1 -- 172.21.15.111:0/902393877 >> v1:172.21.15.149:6789/0 conn(0x7fbb540b0d70 legacy=0x7fbb540b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb59204700 1 -- 172.21.15.111:0/902393877 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb541613e0 con 0x7fbb541604b0 2023-11-28T12:59:53.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb59204700 1 --2- 172.21.15.111:0/902393877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb541604b0 0x7fbb540ce6c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbb480098f0 tx=0x7fbb48009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a343d958e9f134f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:53.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb58a03700 1 -- 172.21.15.111:0/902393877 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb48011830 con 0x7fbb541604b0 2023-11-28T12:59:53.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.043+0000 7fbb58a03700 1 -- 172.21.15.111:0/902393877 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb48011990 con 0x7fbb541604b0 2023-11-28T12:59:53.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.044+0000 7fbb58a03700 1 -- 172.21.15.111:0/902393877 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb48011d20 con 0x7fbb541604b0 2023-11-28T12:59:53.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.044+0000 7fbb5bc69700 1 -- 172.21.15.111:0/902393877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb541604b0 msgr2=0x7fbb540ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.044+0000 7fbb5bc69700 1 --2- 172.21.15.111:0/902393877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb541604b0 0x7fbb540ce6c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbb480098f0 tx=0x7fbb48009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:53.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.044+0000 7fbb5bc69700 1 -- 172.21.15.111:0/902393877 shutdown_connections 2023-11-28T12:59:53.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.044+0000 7fbb5bc69700 1 --2- 172.21.15.111:0/902393877 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb540cc550 0x7fbb540cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.044+0000 7fbb5bc69700 1 --2- 172.21.15.111:0/902393877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb541604b0 0x7fbb540ce6c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.044+0000 7fbb5bc69700 1 -- 172.21.15.111:0/902393877 >> 172.21.15.111:0/902393877 conn(0x7fbb54098ee0 msgr2=0x7fbb5406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:53.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.044+0000 7fbb5bc69700 1 -- 172.21.15.111:0/902393877 shutdown_connections 2023-11-28T12:59:53.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.044+0000 7fbb5bc69700 1 -- 172.21.15.111:0/902393877 wait complete. 2023-11-28T12:59:53.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.045+0000 7fbb5bc69700 1 Processor -- start 2023-11-28T12:59:53.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.045+0000 7fbb5bc69700 1 -- start start 2023-11-28T12:59:53.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.046+0000 7fbb5bc69700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb540b0d70 0x7fbb541f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.046+0000 7fbb5bc69700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb540cc550 0x7fbb540cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.046+0000 7fbb5bc69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb541604b0 0x7fbb541e6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.046+0000 7fbb5bc69700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb540b5350 con 0x7fbb541604b0 2023-11-28T12:59:53.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.046+0000 7fbb5bc69700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb541f2400 con 0x7fbb540b0d70 2023-11-28T12:59:53.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.046+0000 7fbb5bc69700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb540b4740 con 0x7fbb540cc550 2023-11-28T12:59:53.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.046+0000 7fbb5a206700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb541604b0 0x7fbb541e6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.046+0000 7fbb5a206700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb541604b0 0x7fbb541e6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34600/0 (socket says 172.21.15.111:34600) 2023-11-28T12:59:53.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.046+0000 7fbb5a206700 1 -- 172.21.15.111:0/1525450856 learned_addr learned my addr 172.21.15.111:0/1525450856 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:53.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.046+0000 7fbb59a05700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb540b0d70 0x7fbb541f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.047+0000 7fbb59204700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb540cc550 0x7fbb540cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.047+0000 7fbb5a206700 1 -- 172.21.15.111:0/1525450856 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb540cc550 msgr2=0x7fbb540cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.047+0000 7fbb5a206700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb540cc550 0x7fbb540cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.047+0000 7fbb5a206700 1 -- 172.21.15.111:0/1525450856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb540b0d70 msgr2=0x7fbb541f1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.047+0000 7fbb5a206700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb540b0d70 0x7fbb541f1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.047+0000 7fbb5a206700 1 -- 172.21.15.111:0/1525450856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb48009550 con 0x7fbb541604b0 2023-11-28T12:59:53.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.047+0000 7fbb5a206700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb541604b0 0x7fbb541e6310 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00efc0 tx=0x7fbb4c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:53.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.047+0000 7fbb467fc700 1 -- 172.21.15.111:0/1525450856 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c00cd40 con 0x7fbb541604b0 2023-11-28T12:59:53.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.047+0000 7fbb467fc700 1 -- 172.21.15.111:0/1525450856 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4c00f040 con 0x7fbb541604b0 2023-11-28T12:59:53.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.047+0000 7fbb467fc700 1 -- 172.21.15.111:0/1525450856 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c0146c0 con 0x7fbb541604b0 2023-11-28T12:59:53.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.047+0000 7fbb5bc69700 1 -- 172.21.15.111:0/1525450856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb54160e20 con 0x7fbb541604b0 2023-11-28T12:59:53.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.047+0000 7fbb5bc69700 1 -- 172.21.15.111:0/1525450856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb54160b40 con 0x7fbb541604b0 2023-11-28T12:59:53.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.048+0000 7fbb5bc69700 1 -- 172.21.15.111:0/1525450856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb54161100 con 0x7fbb541604b0 2023-11-28T12:59:53.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.048+0000 7fbb467fc700 1 -- 172.21.15.111:0/1525450856 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbb4c014820 con 0x7fbb541604b0 2023-11-28T12:59:53.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.048+0000 7fbb467fc700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbb380682f0 0x7fbb3806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.049+0000 7fbb467fc700 1 -- 172.21.15.111:0/1525450856 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fbb4c086a30 con 0x7fbb541604b0 2023-11-28T12:59:53.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.049+0000 7fbb59a05700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbb380682f0 0x7fbb3806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.049+0000 7fbb59a05700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbb380682f0 0x7fbb3806a7a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbb5415c500 tx=0x7fbb50009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:53.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.053+0000 7fbb467fc700 1 -- 172.21.15.111:0/1525450856 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbb4c052f60 con 0x7fbb541604b0 2023-11-28T12:59:53.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.234+0000 7fbb5bc69700 1 -- 172.21.15.111:0/1525450856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fbb54067170 con 0x7fbb541604b0 2023-11-28T12:59:53.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.235+0000 7fbb467fc700 1 -- 172.21.15.111:0/1525450856 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbb4c00cea0 con 0x7fbb541604b0 2023-11-28T12:59:53.237 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:59:53.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.236+0000 7fbb5bc69700 1 -- 172.21.15.111:0/1525450856 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbb380682f0 msgr2=0x7fbb3806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.237+0000 7fbb5bc69700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbb380682f0 0x7fbb3806a7a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbb5415c500 tx=0x7fbb50009040 comp rx=0 tx=0).stop 2023-11-28T12:59:53.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.237+0000 7fbb5bc69700 1 -- 172.21.15.111:0/1525450856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb541604b0 msgr2=0x7fbb541e6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.237+0000 7fbb5bc69700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb541604b0 0x7fbb541e6310 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00efc0 tx=0x7fbb4c00c3a0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.237+0000 7fbb5bc69700 1 -- 172.21.15.111:0/1525450856 shutdown_connections 2023-11-28T12:59:53.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.237+0000 7fbb5bc69700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb541604b0 0x7fbb541e6310 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.237+0000 7fbb5bc69700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbb380682f0 0x7fbb3806a7a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.237+0000 7fbb5bc69700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb540b0d70 0x7fbb541f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.237+0000 7fbb5bc69700 1 --2- 172.21.15.111:0/1525450856 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb540cc550 0x7fbb540cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.237+0000 7fbb5bc69700 1 -- 172.21.15.111:0/1525450856 >> 172.21.15.111:0/1525450856 conn(0x7fbb54098ee0 msgr2=0x7fbb5406f470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:53.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.237+0000 7fbb5bc69700 1 -- 172.21.15.111:0/1525450856 shutdown_connections 2023-11-28T12:59:53.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.237+0000 7fbb5bc69700 1 -- 172.21.15.111:0/1525450856 wait complete. 2023-11-28T12:59:53.250 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T12:59:53.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.372+0000 7f56217c7700 1 Processor -- start 2023-11-28T12:59:53.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.372+0000 7f56217c7700 1 -- start start 2023-11-28T12:59:53.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f56217c7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0cf860 0x7f561c06b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f56217c7700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561c0cfdb0 con 0x7f561c15cbf0 2023-11-28T12:59:53.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f56217c7700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561c0cc740 con 0x7f561c0cffd0 2023-11-28T12:59:53.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f56217c7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561c0b35e0 con 0x7f561c0cf860 2023-11-28T12:59:53.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f561affd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f561c15cbf0 0x7f561c0b3b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35382/0 (socket says 172.21.15.111:35382) 2023-11-28T12:59:53.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f561affd700 1 -- 172.21.15.111:0/586930456 learned_addr learned my addr 172.21.15.111:0/586930456 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:53.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f561a7fc700 1 --2- 172.21.15.111:0/586930456 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0cf860 0x7f561c06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229953326 0 0) 0x7f561c0cfdb0 con 0x7f561c15cbf0 2023-11-28T12:59:53.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56040035b0 con 0x7f561c15cbf0 2023-11-28T12:59:53.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278290770 0 0) 0x7f561c0cc740 con 0x7f561c0cffd0 2023-11-28T12:59:53.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f561c0cfdb0 con 0x7f561c0cffd0 2023-11-28T12:59:53.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3726498616 0 0) 0x7f56040035b0 con 0x7f561c15cbf0 2023-11-28T12:59:53.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0cf860 msgr2=0x7f561c06b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:53.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f5619ffb700 1 --2- 172.21.15.111:0/586930456 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0cf860 0x7f561c06b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 >> v1:172.21.15.149:6789/0 conn(0x7f561c0cffd0 legacy=0x7f561c0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.373+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c0cbe10 con 0x7f561c15cbf0 2023-11-28T12:59:53.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f5610001670 con 0x7f561c15cbf0 2023-11-28T12:59:53.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 >> v1:172.21.15.111:6789/0 conn(0x7f561c15cbf0 legacy=0x7f561c0b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f5619ffb700 1 --2- 172.21.15.111:0/586930456 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5604003bc0 0x7f5604003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f5619ffb700 1 --2- 172.21.15.111:0/586930456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5604004850 0x7f5604004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f5619ffb700 1 --2- 172.21.15.111:0/586930456 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0cf860 0x7f5604005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561c0b35e0 con 0x7f5604004850 2023-11-28T12:59:53.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560400baf0 con 0x7f5604003bc0 2023-11-28T12:59:53.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560400bcb0 con 0x7f561c0cf860 2023-11-28T12:59:53.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f561a7fc700 1 --2- 172.21.15.111:0/586930456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5604004850 0x7f5604004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f561affd700 1 --2- 172.21.15.111:0/586930456 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5604003bc0 0x7f5604003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f561b7fe700 1 --2- 172.21.15.111:0/586930456 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0cf860 0x7f5604005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f561a7fc700 1 -- 172.21.15.111:0/586930456 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0cf860 msgr2=0x7f5604005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:53.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f561a7fc700 1 --2- 172.21.15.111:0/586930456 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0cf860 0x7f5604005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f561a7fc700 1 -- 172.21.15.111:0/586930456 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5604003bc0 msgr2=0x7f5604003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:53.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f561a7fc700 1 --2- 172.21.15.111:0/586930456 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5604003bc0 0x7f5604003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f561a7fc700 1 -- 172.21.15.111:0/586930456 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56040035b0 con 0x7f5604004850 2023-11-28T12:59:53.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f561b7fe700 1 --2- 172.21.15.111:0/586930456 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0cf860 0x7f5604005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:53.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f561a7fc700 1 --2- 172.21.15.111:0/586930456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5604004850 0x7f5604004cc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f560c009b40 tx=0x7f5604005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d013e238528802a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:53.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c00e6c0 con 0x7f5604004850 2023-11-28T12:59:53.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560c00e820 con 0x7f5604004850 2023-11-28T12:59:53.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.374+0000 7f5619ffb700 1 -- 172.21.15.111:0/586930456 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c00ebd0 con 0x7f5604004850 2023-11-28T12:59:53.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.375+0000 7f56217c7700 1 -- 172.21.15.111:0/586930456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5604004850 msgr2=0x7f5604004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.375+0000 7f56217c7700 1 --2- 172.21.15.111:0/586930456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5604004850 0x7f5604004cc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f560c009b40 tx=0x7f5604005880 comp rx=0 tx=0).stop 2023-11-28T12:59:53.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.375+0000 7f561b7fe700 1 -- 172.21.15.111:0/586930456 reap_dead start 2023-11-28T12:59:53.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.375+0000 7f56217c7700 1 -- 172.21.15.111:0/586930456 shutdown_connections 2023-11-28T12:59:53.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.375+0000 7f56217c7700 1 -- 172.21.15.111:0/586930456 >> 172.21.15.111:0/586930456 conn(0x7f561c09a040 msgr2=0x7f561c0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:53.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.375+0000 7f56217c7700 1 -- 172.21.15.111:0/586930456 shutdown_connections 2023-11-28T12:59:53.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.375+0000 7f56217c7700 1 -- 172.21.15.111:0/586930456 wait complete. 2023-11-28T12:59:53.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.376+0000 7f56217c7700 1 Processor -- start 2023-11-28T12:59:53.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.376+0000 7f56217c7700 1 -- start start 2023-11-28T12:59:53.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f56217c7700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5604004850 0x7f561c156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f56217c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561c156a00 0x7f561c1f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f56217c7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0c4770 0x7f561c15efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f56217c7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561c1f5da0 con 0x7f561c156a00 2023-11-28T12:59:53.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f56217c7700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561c0c86a0 con 0x7f5604004850 2023-11-28T12:59:53.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f56217c7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561c0c80e0 con 0x7f561c0c4770 2023-11-28T12:59:53.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f561a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561c156a00 0x7f561c1f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f561a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561c156a00 0x7f561c1f6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34618/0 (socket says 172.21.15.111:34618) 2023-11-28T12:59:53.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f561a7fc700 1 -- 172.21.15.111:0/4290421144 learned_addr learned my addr 172.21.15.111:0/4290421144 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:53.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f561affd700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5604004850 0x7f561c156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f561b7fe700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0c4770 0x7f561c15efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f561a7fc700 1 -- 172.21.15.111:0/4290421144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0c4770 msgr2=0x7f561c15efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f561a7fc700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0c4770 0x7f561c15efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f561a7fc700 1 -- 172.21.15.111:0/4290421144 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5604004850 msgr2=0x7f561c156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f561a7fc700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5604004850 0x7f561c156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f561a7fc700 1 -- 172.21.15.111:0/4290421144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c0cc3d0 con 0x7f561c156a00 2023-11-28T12:59:53.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f561b7fe700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0c4770 0x7f561c15efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:53.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.377+0000 7f561a7fc700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561c156a00 0x7f561c1f6230 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f560c0090b0 tx=0x7f560c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:53.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.378+0000 7f56037fe700 1 -- 172.21.15.111:0/4290421144 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c021070 con 0x7f561c156a00 2023-11-28T12:59:53.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.378+0000 7f56037fe700 1 -- 172.21.15.111:0/4290421144 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560c013950 con 0x7f561c156a00 2023-11-28T12:59:53.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.378+0000 7f56217c7700 1 -- 172.21.15.111:0/4290421144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f561c0cc0f0 con 0x7f561c156a00 2023-11-28T12:59:53.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.378+0000 7f56037fe700 1 -- 172.21.15.111:0/4290421144 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c00e6c0 con 0x7f561c156a00 2023-11-28T12:59:53.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.378+0000 7f56217c7700 1 -- 172.21.15.111:0/4290421144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561c0b4270 con 0x7f561c156a00 2023-11-28T12:59:53.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.379+0000 7f56217c7700 1 -- 172.21.15.111:0/4290421144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f561c067880 con 0x7f561c156a00 2023-11-28T12:59:53.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.379+0000 7f56037fe700 1 -- 172.21.15.111:0/4290421144 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f560c02e030 con 0x7f561c156a00 2023-11-28T12:59:53.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.379+0000 7f56037fe700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56080682f0 0x7f560806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.380+0000 7f561affd700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56080682f0 0x7f560806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.380+0000 7f56037fe700 1 -- 172.21.15.111:0/4290421144 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f560c0872e0 con 0x7f561c156a00 2023-11-28T12:59:53.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.380+0000 7f561affd700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56080682f0 0x7f560806a7a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5610001eb0 tx=0x7f5610001e20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:53.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.384+0000 7f56037fe700 1 -- 172.21.15.111:0/4290421144 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f560c0538c0 con 0x7f561c156a00 2023-11-28T12:59:53.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.569+0000 7f56217c7700 1 -- 172.21.15.111:0/4290421144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f561c1600f0 con 0x7f561c156a00 2023-11-28T12:59:53.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.569+0000 7f56037fe700 1 -- 172.21.15.111:0/4290421144 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f560c056ee0 con 0x7f561c156a00 2023-11-28T12:59:53.571 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:53.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.571+0000 7f56217c7700 1 -- 172.21.15.111:0/4290421144 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56080682f0 msgr2=0x7f560806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.571+0000 7f56217c7700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56080682f0 0x7f560806a7a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5610001eb0 tx=0x7f5610001e20 comp rx=0 tx=0).stop 2023-11-28T12:59:53.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.571+0000 7f56217c7700 1 -- 172.21.15.111:0/4290421144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561c156a00 msgr2=0x7f561c1f6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.571+0000 7f56217c7700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561c156a00 0x7f561c1f6230 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f560c0090b0 tx=0x7f560c0047c0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.571+0000 7f56217c7700 1 -- 172.21.15.111:0/4290421144 shutdown_connections 2023-11-28T12:59:53.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.571+0000 7f56217c7700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f561c0c4770 0x7f561c15efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.571+0000 7f56217c7700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56080682f0 0x7f560806a7a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.571+0000 7f56217c7700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5604004850 0x7f561c156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.571+0000 7f56217c7700 1 --2- 172.21.15.111:0/4290421144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f561c156a00 0x7f561c1f6230 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.571+0000 7f56217c7700 1 -- 172.21.15.111:0/4290421144 >> 172.21.15.111:0/4290421144 conn(0x7f561c09a040 msgr2=0x7f561c0b5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:53.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.571+0000 7f56217c7700 1 -- 172.21.15.111:0/4290421144 shutdown_connections 2023-11-28T12:59:53.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.571+0000 7f56217c7700 1 -- 172.21.15.111:0/4290421144 wait complete. 2023-11-28T12:59:53.584 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T12:59:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.706+0000 7f6a1d9c7700 1 Processor -- start 2023-11-28T12:59:53.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.706+0000 7f6a1d9c7700 1 -- start start 2023-11-28T12:59:53.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a1d9c7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6a1815eee0 0x7f6a1806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a1d9c7700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f6a1806a7b0 0x7f6a180c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a1d9c7700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a1806ac10 con 0x7f6a1815a7a0 2023-11-28T12:59:53.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a1d9c7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1806cc10 con 0x7f6a1815eee0 2023-11-28T12:59:53.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a1d9c7700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1806cef0 con 0x7f6a1806a7b0 2023-11-28T12:59:53.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a167fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6a1815eee0 0x7f6a1806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a16ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6a1815a7a0 0x7f6a1806c3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35402/0 (socket says 172.21.15.111:35402) 2023-11-28T12:59:53.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a167fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6a1815eee0 0x7f6a1806a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34626/0 (socket says 172.21.15.111:34626) 2023-11-28T12:59:53.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a16ffd700 1 -- 172.21.15.111:0/2051496570 learned_addr learned my addr 172.21.15.111:0/2051496570 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:53.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a177fe700 1 --2- 172.21.15.111:0/2051496570 >> v2:172.21.15.149:3300/0 conn(0x7f6a1806a7b0 0x7f6a180c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a167fc700 1 -- 172.21.15.111:0/2051496570 >> v2:172.21.15.149:3300/0 conn(0x7f6a1806a7b0 msgr2=0x7f6a180c9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:53.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a167fc700 1 --2- 172.21.15.111:0/2051496570 >> v2:172.21.15.149:3300/0 conn(0x7f6a1806a7b0 0x7f6a180c9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a167fc700 1 -- 172.21.15.111:0/2051496570 >> v1:172.21.15.111:6789/0 conn(0x7f6a1815a7a0 legacy=0x7f6a1806c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a167fc700 1 -- 172.21.15.111:0/2051496570 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a180c9070 con 0x7f6a1815eee0 2023-11-28T12:59:53.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a15ffb700 1 -- 172.21.15.111:0/2051496570 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380427128 0 0) 0x7f6a1806ac10 con 0x7f6a1815a7a0 2023-11-28T12:59:53.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a177fe700 1 --2- 172.21.15.111:0/2051496570 >> v2:172.21.15.149:3300/0 conn(0x7f6a1806a7b0 0x7f6a180c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:53.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a167fc700 1 --2- 172.21.15.111:0/2051496570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a1815eee0 0x7f6a1806a270 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6a1806b3f0 tx=0x7f6a08009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf996e8365ea0e3c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:53.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.707+0000 7f6a15ffb700 1 -- 172.21.15.111:0/2051496570 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08011830 con 0x7f6a1815eee0 2023-11-28T12:59:53.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.708+0000 7f6a15ffb700 1 -- 172.21.15.111:0/2051496570 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a08011990 con 0x7f6a1815eee0 2023-11-28T12:59:53.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.708+0000 7f6a15ffb700 1 -- 172.21.15.111:0/2051496570 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08011d20 con 0x7f6a1815eee0 2023-11-28T12:59:53.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.708+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/2051496570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a1815eee0 msgr2=0x7f6a1806a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.708+0000 7f6a1d9c7700 1 --2- 172.21.15.111:0/2051496570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a1815eee0 0x7f6a1806a270 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6a1806b3f0 tx=0x7f6a08009b20 comp rx=0 tx=0).stop 2023-11-28T12:59:53.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.708+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/2051496570 shutdown_connections 2023-11-28T12:59:53.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.708+0000 7f6a1d9c7700 1 --2- 172.21.15.111:0/2051496570 >> v2:172.21.15.149:3300/0 conn(0x7f6a1806a7b0 0x7f6a180c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.708+0000 7f6a1d9c7700 1 --2- 172.21.15.111:0/2051496570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a1815eee0 0x7f6a1806a270 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.708+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/2051496570 >> 172.21.15.111:0/2051496570 conn(0x7f6a180cb490 msgr2=0x7f6a180cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:53.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.708+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/2051496570 shutdown_connections 2023-11-28T12:59:53.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.708+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/2051496570 wait complete. 2023-11-28T12:59:53.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.709+0000 7f6a1d9c7700 1 Processor -- start 2023-11-28T12:59:53.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.709+0000 7f6a1d9c7700 1 -- start start 2023-11-28T12:59:53.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.710+0000 7f6a1d9c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a1806a7b0 0x7f6a181521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.710+0000 7f6a1d9c7700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6a1815a7a0 0x7f6a18158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.710+0000 7f6a1d9c7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6a1815eee0 0x7f6a18159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.710+0000 7f6a1d9c7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a18155940 con 0x7f6a1806a7b0 2023-11-28T12:59:53.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.710+0000 7f6a1d9c7700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a180cdf40 con 0x7f6a1815a7a0 2023-11-28T12:59:53.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.710+0000 7f6a1d9c7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a180ce220 con 0x7f6a1815eee0 2023-11-28T12:59:53.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.710+0000 7f6a16ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a1806a7b0 0x7f6a181521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.710+0000 7f6a16ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a1806a7b0 0x7f6a181521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34630/0 (socket says 172.21.15.111:34630) 2023-11-28T12:59:53.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.710+0000 7f6a16ffd700 1 -- 172.21.15.111:0/479369040 learned_addr learned my addr 172.21.15.111:0/479369040 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:53.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.710+0000 7f6a167fc700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6a1815a7a0 0x7f6a18158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.710+0000 7f6a177fe700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6a1815eee0 0x7f6a18159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.710+0000 7f6a16ffd700 1 -- 172.21.15.111:0/479369040 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6a1815eee0 msgr2=0x7f6a18159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.710+0000 7f6a16ffd700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6a1815eee0 0x7f6a18159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.711+0000 7f6a16ffd700 1 -- 172.21.15.111:0/479369040 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6a1815a7a0 msgr2=0x7f6a18158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.711+0000 7f6a16ffd700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6a1815a7a0 0x7f6a18158f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.711+0000 7f6a16ffd700 1 -- 172.21.15.111:0/479369040 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a080094a0 con 0x7f6a1806a7b0 2023-11-28T12:59:53.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.711+0000 7f6a177fe700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6a1815eee0 0x7f6a18159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:53.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.711+0000 7f6a16ffd700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a1806a7b0 0x7f6a181521a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0011e0 tx=0x7f6a0c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:53.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.711+0000 7f69fffff700 1 -- 172.21.15.111:0/479369040 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c004c80 con 0x7f6a1806a7b0 2023-11-28T12:59:53.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.711+0000 7f69fffff700 1 -- 172.21.15.111:0/479369040 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c004e00 con 0x7f6a1806a7b0 2023-11-28T12:59:53.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.711+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/479369040 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a180c9630 con 0x7f6a1806a7b0 2023-11-28T12:59:53.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.711+0000 7f69fffff700 1 -- 172.21.15.111:0/479369040 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c007310 con 0x7f6a1806a7b0 2023-11-28T12:59:53.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.711+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/479369040 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a180c9350 con 0x7f6a1806a7b0 2023-11-28T12:59:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.712+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/479369040 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1815ab70 con 0x7f6a1806a7b0 2023-11-28T12:59:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.712+0000 7f69fffff700 1 -- 172.21.15.111:0/479369040 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6a0c01a020 con 0x7f6a1806a7b0 2023-11-28T12:59:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.713+0000 7f69fffff700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6a040682f0 0x7f6a0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.713+0000 7f69fffff700 1 -- 172.21.15.111:0/479369040 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f6a0c085c10 con 0x7f6a1806a7b0 2023-11-28T12:59:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.713+0000 7f6a167fc700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6a040682f0 0x7f6a0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.713+0000 7f6a167fc700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6a040682f0 0x7f6a0406a7a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6a080090b0 tx=0x7f6a08008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.717+0000 7f69fffff700 1 -- 172.21.15.111:0/479369040 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6a0c052140 con 0x7f6a1806a7b0 2023-11-28T12:59:53.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.898+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/479369040 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6a180cba90 con 0x7f6a1806a7b0 2023-11-28T12:59:53.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.898+0000 7f69fffff700 1 -- 172.21.15.111:0/479369040 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6a0c055760 con 0x7f6a1806a7b0 2023-11-28T12:59:53.900 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T12:59:53.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.900+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/479369040 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6a040682f0 msgr2=0x7f6a0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.900+0000 7f6a1d9c7700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6a040682f0 0x7f6a0406a7a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6a080090b0 tx=0x7f6a08008030 comp rx=0 tx=0).stop 2023-11-28T12:59:53.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.900+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/479369040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a1806a7b0 msgr2=0x7f6a181521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:53.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.900+0000 7f6a1d9c7700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a1806a7b0 0x7f6a181521a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0011e0 tx=0x7f6a0c00cae0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.901+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/479369040 shutdown_connections 2023-11-28T12:59:53.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.901+0000 7f6a1d9c7700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6a1815eee0 0x7f6a18159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.901+0000 7f6a1d9c7700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a1806a7b0 0x7f6a181521a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.901+0000 7f6a1d9c7700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6a040682f0 0x7f6a0406a7a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.901+0000 7f6a1d9c7700 1 --2- 172.21.15.111:0/479369040 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6a1815a7a0 0x7f6a18158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:53.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.901+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/479369040 >> 172.21.15.111:0/479369040 conn(0x7f6a180cb490 msgr2=0x7f6a180ca690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:53.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.901+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/479369040 shutdown_connections 2023-11-28T12:59:53.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:53.901+0000 7f6a1d9c7700 1 -- 172.21.15.111:0/479369040 wait complete. 2023-11-28T12:59:53.914 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T12:59:54.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.037+0000 7f19ba1ab700 1 Processor -- start 2023-11-28T12:59:54.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.037+0000 7f19ba1ab700 1 -- start start 2023-11-28T12:59:54.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.038+0000 7f19ba1ab700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f19b415c6c0 0x7f19b40afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.038+0000 7f19ba1ab700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f19b40b0440 0x7f19b40b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.038+0000 7f19ba1ab700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b40cc750 con 0x7f19b41602a0 2023-11-28T12:59:54.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.038+0000 7f19ba1ab700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19b40b0e20 con 0x7f19b415c6c0 2023-11-28T12:59:54.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.038+0000 7f19ba1ab700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19b40cca30 con 0x7f19b40b0440 2023-11-28T12:59:54.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.038+0000 7f19b37fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f19b415c6c0 0x7f19b40afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.038+0000 7f19b37fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f19b415c6c0 0x7f19b40afb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34638/0 (socket says 172.21.15.111:34638) 2023-11-28T12:59:54.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.038+0000 7f19b37fe700 1 -- 172.21.15.111:0/2089135210 learned_addr learned my addr 172.21.15.111:0/2089135210 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:54.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19b2ffd700 1 --2- 172.21.15.111:0/2089135210 >> v2:172.21.15.149:3300/0 conn(0x7f19b40b0440 0x7f19b40b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19b37fe700 1 -- 172.21.15.111:0/2089135210 >> v2:172.21.15.149:3300/0 conn(0x7f19b40b0440 msgr2=0x7f19b40b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:54.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19b37fe700 1 --2- 172.21.15.111:0/2089135210 >> v2:172.21.15.149:3300/0 conn(0x7f19b40b0440 0x7f19b40b08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19b37fe700 1 -- 172.21.15.111:0/2089135210 >> v1:172.21.15.149:6789/0 conn(0x7f19b41602a0 legacy=0x7f19b4160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:54.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19b37fe700 1 -- 172.21.15.111:0/2089135210 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b40ccd10 con 0x7f19b415c6c0 2023-11-28T12:59:54.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19b2ffd700 1 --2- 172.21.15.111:0/2089135210 >> v2:172.21.15.149:3300/0 conn(0x7f19b40b0440 0x7f19b40b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:54.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19b37fe700 1 --2- 172.21.15.111:0/2089135210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19b415c6c0 0x7f19b40afb50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f19b4066980 tx=0x7f199c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d6282fda793c83 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:54.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19b27fc700 1 -- 172.21.15.111:0/2089135210 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c004230 con 0x7f19b415c6c0 2023-11-28T12:59:54.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19b27fc700 1 -- 172.21.15.111:0/2089135210 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199c004390 con 0x7f19b415c6c0 2023-11-28T12:59:54.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19b27fc700 1 -- 172.21.15.111:0/2089135210 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c0036a0 con 0x7f19b415c6c0 2023-11-28T12:59:54.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2089135210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19b415c6c0 msgr2=0x7f19b40afb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19ba1ab700 1 --2- 172.21.15.111:0/2089135210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19b415c6c0 0x7f19b40afb50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f19b4066980 tx=0x7f199c009bd0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2089135210 shutdown_connections 2023-11-28T12:59:54.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19ba1ab700 1 --2- 172.21.15.111:0/2089135210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19b415c6c0 0x7f19b40afb50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19ba1ab700 1 --2- 172.21.15.111:0/2089135210 >> v2:172.21.15.149:3300/0 conn(0x7f19b40b0440 0x7f19b40b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2089135210 >> 172.21.15.111:0/2089135210 conn(0x7f19b4098ee0 msgr2=0x7f19b406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:54.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.039+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2089135210 shutdown_connections 2023-11-28T12:59:54.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.040+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2089135210 wait complete. 2023-11-28T12:59:54.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.041+0000 7f19ba1ab700 1 Processor -- start 2023-11-28T12:59:54.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.041+0000 7f19ba1ab700 1 -- start start 2023-11-28T12:59:54.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19ba1ab700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19b40b0440 0x7f19b415b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19ba1ab700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f19b415c6c0 0x7f19b414b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19ba1ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19b41602a0 0x7f19b415e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19ba1ab700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b40c71e0 con 0x7f19b41602a0 2023-11-28T12:59:54.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19ba1ab700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b40b48a0 con 0x7f19b40b0440 2023-11-28T12:59:54.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19ba1ab700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b40b3f70 con 0x7f19b415c6c0 2023-11-28T12:59:54.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19b3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19b41602a0 0x7f19b415e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19b3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19b41602a0 0x7f19b415e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34646/0 (socket says 172.21.15.111:34646) 2023-11-28T12:59:54.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19b3fff700 1 -- 172.21.15.111:0/2039739592 learned_addr learned my addr 172.21.15.111:0/2039739592 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:54.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19b37fe700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19b40b0440 0x7f19b415b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19b3fff700 1 -- 172.21.15.111:0/2039739592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f19b415c6c0 msgr2=0x7f19b414b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19b3fff700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f19b415c6c0 0x7f19b414b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19b3fff700 1 -- 172.21.15.111:0/2039739592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19b40b0440 msgr2=0x7f19b415b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19b3fff700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19b40b0440 0x7f19b415b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19b3fff700 1 -- 172.21.15.111:0/2039739592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c009550 con 0x7f19b41602a0 2023-11-28T12:59:54.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19b37fe700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19b40b0440 0x7f19b415b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:54.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19b3fff700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19b41602a0 0x7f19b415e560 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00cc10 tx=0x7f19ac00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:54.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19abfff700 1 -- 172.21.15.111:0/2039739592 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac00abb0 con 0x7f19b41602a0 2023-11-28T12:59:54.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19abfff700 1 -- 172.21.15.111:0/2039739592 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ac00ad10 con 0x7f19b41602a0 2023-11-28T12:59:54.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2039739592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b40cd2d0 con 0x7f19b41602a0 2023-11-28T12:59:54.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19abfff700 1 -- 172.21.15.111:0/2039739592 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac012660 con 0x7f19b41602a0 2023-11-28T12:59:54.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.042+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2039739592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b40ccff0 con 0x7f19b41602a0 2023-11-28T12:59:54.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.043+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2039739592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b415ca90 con 0x7f19b41602a0 2023-11-28T12:59:54.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.044+0000 7f19abfff700 1 -- 172.21.15.111:0/2039739592 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f19ac022070 con 0x7f19b41602a0 2023-11-28T12:59:54.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.044+0000 7f19abfff700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f19940682f0 0x7f199406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.044+0000 7f19b37fe700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f19940682f0 0x7f199406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.044+0000 7f19abfff700 1 -- 172.21.15.111:0/2039739592 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f19ac086690 con 0x7f19b41602a0 2023-11-28T12:59:54.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.045+0000 7f19b37fe700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f19940682f0 0x7f199406a7a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f19b40c68e0 tx=0x7f199c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:54.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.048+0000 7f19abfff700 1 -- 172.21.15.111:0/2039739592 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f19ac052bc0 con 0x7f19b41602a0 2023-11-28T12:59:54.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.233+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2039739592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f19b40cdf80 con 0x7f19b41602a0 2023-11-28T12:59:54.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.234+0000 7f19abfff700 1 -- 172.21.15.111:0/2039739592 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f19ac0160a0 con 0x7f19b41602a0 2023-11-28T12:59:54.235 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:54.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.236+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2039739592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f19940682f0 msgr2=0x7f199406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.236+0000 7f19ba1ab700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f19940682f0 0x7f199406a7a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f19b40c68e0 tx=0x7f199c008030 comp rx=0 tx=0).stop 2023-11-28T12:59:54.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.236+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2039739592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19b41602a0 msgr2=0x7f19b415e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.236+0000 7f19ba1ab700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19b41602a0 0x7f19b415e560 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00cc10 tx=0x7f19ac00a060 comp rx=0 tx=0).stop 2023-11-28T12:59:54.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.236+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2039739592 shutdown_connections 2023-11-28T12:59:54.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.236+0000 7f19ba1ab700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19b41602a0 0x7f19b415e560 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.236+0000 7f19ba1ab700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f19940682f0 0x7f199406a7a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.236+0000 7f19ba1ab700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f19b40b0440 0x7f19b415b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.236+0000 7f19ba1ab700 1 --2- 172.21.15.111:0/2039739592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f19b415c6c0 0x7f19b414b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.236+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2039739592 >> 172.21.15.111:0/2039739592 conn(0x7f19b4098ee0 msgr2=0x7f19b40ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:54.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.236+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2039739592 shutdown_connections 2023-11-28T12:59:54.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.236+0000 7f19ba1ab700 1 -- 172.21.15.111:0/2039739592 wait complete. 2023-11-28T12:59:54.250 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T12:59:54.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.370+0000 7fdf09971700 1 Processor -- start 2023-11-28T12:59:54.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.370+0000 7fdf09971700 1 -- start start 2023-11-28T12:59:54.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.371+0000 7fdf09971700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdf041604b0 0x7fdf040ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.371+0000 7fdf09971700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf040cc550 0x7fdf040cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.371+0000 7fdf09971700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0415cd90 con 0x7fdf040b0d70 2023-11-28T12:59:54.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.371+0000 7fdf09971700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf04160950 con 0x7fdf041604b0 2023-11-28T12:59:54.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.371+0000 7fdf09971700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040ccf10 con 0x7fdf040cc550 2023-11-28T12:59:54.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.371+0000 7fdf027fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdf041604b0 0x7fdf040ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.371+0000 7fdf027fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdf041604b0 0x7fdf040ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34654/0 (socket says 172.21.15.111:34654) 2023-11-28T12:59:54.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.371+0000 7fdf027fc700 1 -- 172.21.15.111:0/210712232 learned_addr learned my addr 172.21.15.111:0/210712232 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:54.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.371+0000 7fdf027fc700 1 -- 172.21.15.111:0/210712232 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf040cc550 msgr2=0x7fdf040cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:54.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.371+0000 7fdf027fc700 1 --2- 172.21.15.111:0/210712232 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf040cc550 0x7fdf040cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.371+0000 7fdf027fc700 1 -- 172.21.15.111:0/210712232 >> v1:172.21.15.149:6789/0 conn(0x7fdf040b0d70 legacy=0x7fdf040b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:54.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.371+0000 7fdf027fc700 1 -- 172.21.15.111:0/210712232 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf041613e0 con 0x7fdf041604b0 2023-11-28T12:59:54.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.372+0000 7fdf027fc700 1 --2- 172.21.15.111:0/210712232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf041604b0 0x7fdf040ce6c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdef40098f0 tx=0x7fdef4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9940e77c8385f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:54.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.372+0000 7fdf01ffb700 1 -- 172.21.15.111:0/210712232 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4004230 con 0x7fdf041604b0 2023-11-28T12:59:54.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.372+0000 7fdf01ffb700 1 -- 172.21.15.111:0/210712232 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef4004390 con 0x7fdf041604b0 2023-11-28T12:59:54.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.372+0000 7fdf01ffb700 1 -- 172.21.15.111:0/210712232 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef40036a0 con 0x7fdf041604b0 2023-11-28T12:59:54.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.372+0000 7fdf09971700 1 -- 172.21.15.111:0/210712232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf041604b0 msgr2=0x7fdf040ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.372+0000 7fdf09971700 1 --2- 172.21.15.111:0/210712232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf041604b0 0x7fdf040ce6c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdef40098f0 tx=0x7fdef4009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:54.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.372+0000 7fdf09971700 1 -- 172.21.15.111:0/210712232 shutdown_connections 2023-11-28T12:59:54.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.372+0000 7fdf09971700 1 --2- 172.21.15.111:0/210712232 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf040cc550 0x7fdf040cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.372+0000 7fdf09971700 1 --2- 172.21.15.111:0/210712232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf041604b0 0x7fdf040ce6c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.372+0000 7fdf09971700 1 -- 172.21.15.111:0/210712232 >> 172.21.15.111:0/210712232 conn(0x7fdf04098ee0 msgr2=0x7fdf0406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:54.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.372+0000 7fdf09971700 1 -- 172.21.15.111:0/210712232 shutdown_connections 2023-11-28T12:59:54.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.372+0000 7fdf09971700 1 -- 172.21.15.111:0/210712232 wait complete. 2023-11-28T12:59:54.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.374+0000 7fdf09971700 1 Processor -- start 2023-11-28T12:59:54.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.374+0000 7fdf09971700 1 -- start start 2023-11-28T12:59:54.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf09971700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf040b0d70 0x7fdf04159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf09971700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf040cc550 0x7fdf0406df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf09971700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf041604b0 0x7fdf040b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf09971700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040c64d0 con 0x7fdf040b0d70 2023-11-28T12:59:54.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf09971700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0415a680 con 0x7fdf041604b0 2023-11-28T12:59:54.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf09971700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0415a110 con 0x7fdf040cc550 2023-11-28T12:59:54.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf02ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf040b0d70 0x7fdf04159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf02ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf040b0d70 0x7fdf04159bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34658/0 (socket says 172.21.15.111:34658) 2023-11-28T12:59:54.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf02ffd700 1 -- 172.21.15.111:0/3296363045 learned_addr learned my addr 172.21.15.111:0/3296363045 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:54.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf037fe700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf041604b0 0x7fdf040b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf027fc700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf040cc550 0x7fdf0406df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf02ffd700 1 -- 172.21.15.111:0/3296363045 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf040cc550 msgr2=0x7fdf0406df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf02ffd700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf040cc550 0x7fdf0406df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf02ffd700 1 -- 172.21.15.111:0/3296363045 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf041604b0 msgr2=0x7fdf040b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf02ffd700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf041604b0 0x7fdf040b2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf02ffd700 1 -- 172.21.15.111:0/3296363045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef4009550 con 0x7fdf040b0d70 2023-11-28T12:59:54.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.375+0000 7fdf027fc700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf040cc550 0x7fdf0406df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:54.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.376+0000 7fdf02ffd700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf040b0d70 0x7fdf04159bd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdef800c4b0 tx=0x7fdef800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:54.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.376+0000 7fdeebfff700 1 -- 172.21.15.111:0/3296363045 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef8011790 con 0x7fdf040b0d70 2023-11-28T12:59:54.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.376+0000 7fdeebfff700 1 -- 172.21.15.111:0/3296363045 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef80118f0 con 0x7fdf040b0d70 2023-11-28T12:59:54.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.376+0000 7fdeebfff700 1 -- 172.21.15.111:0/3296363045 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef8011ca0 con 0x7fdf040b0d70 2023-11-28T12:59:54.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.376+0000 7fdf09971700 1 -- 172.21.15.111:0/3296363045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf04160e20 con 0x7fdf040b0d70 2023-11-28T12:59:54.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.376+0000 7fdf09971700 1 -- 172.21.15.111:0/3296363045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf04160b40 con 0x7fdf040b0d70 2023-11-28T12:59:54.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.377+0000 7fdf09971700 1 -- 172.21.15.111:0/3296363045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf04161100 con 0x7fdf040b0d70 2023-11-28T12:59:54.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.377+0000 7fdeebfff700 1 -- 172.21.15.111:0/3296363045 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdef8010050 con 0x7fdf040b0d70 2023-11-28T12:59:54.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.377+0000 7fdeebfff700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdef00682f0 0x7fdef006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.377+0000 7fdeebfff700 1 -- 172.21.15.111:0/3296363045 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fdef8085bc0 con 0x7fdf040b0d70 2023-11-28T12:59:54.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.377+0000 7fdf027fc700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdef00682f0 0x7fdef006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.378+0000 7fdf027fc700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdef00682f0 0x7fdef006a7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009130 tx=0x7fdef4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:54.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.381+0000 7fdeebfff700 1 -- 172.21.15.111:0/3296363045 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdef8052090 con 0x7fdf040b0d70 2023-11-28T12:59:54.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.578+0000 7fdf09971700 1 -- 172.21.15.111:0/3296363045 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdf04067170 con 0x7fdf040b0d70 2023-11-28T12:59:54.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.578+0000 7fdeebfff700 1 -- 172.21.15.111:0/3296363045 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdef80556b0 con 0x7fdf040b0d70 2023-11-28T12:59:54.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.580+0000 7fdf09971700 1 -- 172.21.15.111:0/3296363045 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdef00682f0 msgr2=0x7fdef006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.580+0000 7fdf09971700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdef00682f0 0x7fdef006a7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009130 tx=0x7fdef4008030 comp rx=0 tx=0).stop 2023-11-28T12:59:54.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.580+0000 7fdf09971700 1 -- 172.21.15.111:0/3296363045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf040b0d70 msgr2=0x7fdf04159bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.580+0000 7fdf09971700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf040b0d70 0x7fdf04159bd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdef800c4b0 tx=0x7fdef800c7c0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.580+0000 7fdf09971700 1 -- 172.21.15.111:0/3296363045 shutdown_connections 2023-11-28T12:59:54.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.580+0000 7fdf09971700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdef00682f0 0x7fdef006a7a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.580+0000 7fdf09971700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf041604b0 0x7fdf040b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.580+0000 7fdf09971700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf040b0d70 0x7fdf04159bd0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.580+0000 7fdf09971700 1 --2- 172.21.15.111:0/3296363045 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf040cc550 0x7fdf0406df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.580+0000 7fdf09971700 1 -- 172.21.15.111:0/3296363045 >> 172.21.15.111:0/3296363045 conn(0x7fdf04098ee0 msgr2=0x7fdf0406f470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:54.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.580+0000 7fdf09971700 1 -- 172.21.15.111:0/3296363045 shutdown_connections 2023-11-28T12:59:54.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.580+0000 7fdf09971700 1 -- 172.21.15.111:0/3296363045 wait complete. 2023-11-28T12:59:54.583 INFO:teuthology.orchestra.run.smithi111.stderr:listed 0 entries 2023-11-28T12:59:54.594 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-11-28T12:59:54.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:54.594 DEBUG:teuthology.orchestra.run.smithi149:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-11-28T12:59:54.639 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi149.front.sepia.ceph.com 2023-11-28T12:59:54.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T12:59:54.639 DEBUG:teuthology.orchestra.run.smithi149:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-28T12:59:54.658 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T12:59:54.659 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T12:59:54.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.777+0000 7f94b4499700 1 Processor -- start 2023-11-28T12:59:54.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.777+0000 7f94b4499700 1 -- start start 2023-11-28T12:59:54.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.778+0000 7f94b4499700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f94ac0cd460 0x7f94ac06c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.778+0000 7f94b4499700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac0b88e0 con 0x7f94ac0b8510 2023-11-28T12:59:54.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.778+0000 7f94b4499700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac0cd8f0 con 0x7f94ac0d0d60 2023-11-28T12:59:54.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.778+0000 7f94b4499700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ac0cdbd0 con 0x7f94ac0cd460 2023-11-28T12:59:54.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.779+0000 7f94b1a34700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f94ac0cd460 0x7f94ac06c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.779+0000 7f94b2235700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f94ac0b8510 0x7f94ac06b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35444/0 (socket says 172.21.15.111:35444) 2023-11-28T12:59:54.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.779+0000 7f94b2235700 1 -- 172.21.15.111:0/2417716604 learned_addr learned my addr 172.21.15.111:0/2417716604 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:54.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.779+0000 7f94b1a34700 1 -- 172.21.15.111:0/2417716604 >> v1:172.21.15.149:6789/0 conn(0x7f94ac0d0d60 legacy=0x7f94ac0d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:54.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.779+0000 7f94b1a34700 1 -- 172.21.15.111:0/2417716604 >> v1:172.21.15.111:6789/0 conn(0x7f94ac0b8510 legacy=0x7f94ac06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.779+0000 7f94b1a34700 1 -- 172.21.15.111:0/2417716604 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac0cfd10 con 0x7f94ac0cd460 2023-11-28T12:59:54.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.779+0000 7f94b1233700 1 -- 172.21.15.111:0/2417716604 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622355353 0 0) 0x7f94ac0b88e0 con 0x7f94ac0b8510 2023-11-28T12:59:54.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.779+0000 7f94b1a34700 1 --2- 172.21.15.111:0/2417716604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94ac0cd460 0x7f94ac06c080 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f94a8009840 tx=0x7f94a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7b028303d9795c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:54.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.779+0000 7f94b1233700 1 -- 172.21.15.111:0/2417716604 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8004230 con 0x7f94ac0cd460 2023-11-28T12:59:54.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.779+0000 7f94b1233700 1 -- 172.21.15.111:0/2417716604 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a8004390 con 0x7f94ac0cd460 2023-11-28T12:59:54.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.779+0000 7f94b1233700 1 -- 172.21.15.111:0/2417716604 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a80036a0 con 0x7f94ac0cd460 2023-11-28T12:59:54.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.780+0000 7f94b4499700 1 -- 172.21.15.111:0/2417716604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94ac0cd460 msgr2=0x7f94ac06c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.780+0000 7f94b4499700 1 --2- 172.21.15.111:0/2417716604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94ac0cd460 0x7f94ac06c080 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f94a8009840 tx=0x7f94a8009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:54.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.780+0000 7f94b4499700 1 -- 172.21.15.111:0/2417716604 shutdown_connections 2023-11-28T12:59:54.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.780+0000 7f94b4499700 1 --2- 172.21.15.111:0/2417716604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94ac0cd460 0x7f94ac06c080 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.780+0000 7f94b4499700 1 -- 172.21.15.111:0/2417716604 >> 172.21.15.111:0/2417716604 conn(0x7f94ac0719a0 msgr2=0x7f94ac071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:54.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.780+0000 7f94b4499700 1 -- 172.21.15.111:0/2417716604 shutdown_connections 2023-11-28T12:59:54.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.780+0000 7f94b4499700 1 -- 172.21.15.111:0/2417716604 wait complete. 2023-11-28T12:59:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.781+0000 7f94b4499700 1 Processor -- start 2023-11-28T12:59:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.781+0000 7f94b4499700 1 -- start start 2023-11-28T12:59:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b4499700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f94ac0b8510 0x7f94ac1e7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b4499700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94ac0cd460 0x7f94ac1f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b4499700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f94ac0d0d60 0x7f94ac06b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b4499700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac1e5be0 con 0x7f94ac0cd460 2023-11-28T12:59:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b4499700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac1e5ec0 con 0x7f94ac0b8510 2023-11-28T12:59:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b4499700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac1e61a0 con 0x7f94ac0d0d60 2023-11-28T12:59:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b1a34700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94ac0cd460 0x7f94ac1f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b1a34700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94ac0cd460 0x7f94ac1f22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34674/0 (socket says 172.21.15.111:34674) 2023-11-28T12:59:54.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b1a34700 1 -- 172.21.15.111:0/4215997994 learned_addr learned my addr 172.21.15.111:0/4215997994 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:54.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b2235700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f94ac0b8510 0x7f94ac1e7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b2a36700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f94ac0d0d60 0x7f94ac06b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b1a34700 1 -- 172.21.15.111:0/4215997994 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f94ac0d0d60 msgr2=0x7f94ac06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b1a34700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f94ac0d0d60 0x7f94ac06b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b1a34700 1 -- 172.21.15.111:0/4215997994 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f94ac0b8510 msgr2=0x7f94ac1e7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:54.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b1a34700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f94ac0b8510 0x7f94ac1e7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:54.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b1a34700 1 -- 172.21.15.111:0/4215997994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a80094a0 con 0x7f94ac0cd460 2023-11-28T12:59:54.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b1a34700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94ac0cd460 0x7f94ac1f22c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f94a80090b0 tx=0x7f94a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:54.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94a2ffd700 1 -- 172.21.15.111:0/4215997994 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8011830 con 0x7f94ac0cd460 2023-11-28T12:59:54.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94a2ffd700 1 -- 172.21.15.111:0/4215997994 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a801e070 con 0x7f94ac0cd460 2023-11-28T12:59:54.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b4499700 1 -- 172.21.15.111:0/4215997994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ac0d29c0 con 0x7f94ac0cd460 2023-11-28T12:59:54.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94a2ffd700 1 -- 172.21.15.111:0/4215997994 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8011bb0 con 0x7f94ac0cd460 2023-11-28T12:59:54.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.782+0000 7f94b4499700 1 -- 172.21.15.111:0/4215997994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac0cfff0 con 0x7f94ac0cd460 2023-11-28T12:59:54.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.783+0000 7f94b4499700 1 -- 172.21.15.111:0/4215997994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ac0d2b50 con 0x7f94ac0cd460 2023-11-28T12:59:54.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.784+0000 7f94a2ffd700 1 -- 172.21.15.111:0/4215997994 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f94a8017020 con 0x7f94ac0cd460 2023-11-28T12:59:54.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.784+0000 7f94a2ffd700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f94900682f0 0x7f949006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:54.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.784+0000 7f94a2ffd700 1 -- 172.21.15.111:0/4215997994 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f94a80872e0 con 0x7f94ac0cd460 2023-11-28T12:59:54.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.784+0000 7f94b2235700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f94900682f0 0x7f949006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:54.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.785+0000 7f94b2235700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f94900682f0 0x7f949006a7a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f949c000f60 tx=0x7f949c007be0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:54.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:54.789+0000 7f94a2ffd700 1 -- 172.21.15.111:0/4215997994 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f94a8053810 con 0x7f94ac0cd460 2023-11-28T12:59:55.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.013+0000 7f94b4499700 1 -- 172.21.15.111:0/4215997994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94ac067880 con 0x7f94ac0cd460 2023-11-28T12:59:55.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.014+0000 7f94a2ffd700 1 -- 172.21.15.111:0/4215997994 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4656 (secure 0 0 0) 0x7f94a8056e30 con 0x7f94ac0cd460 2023-11-28T12:59:55.016 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:55.016 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2513950989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2513950989},{"type":"v1","addr":"172.21.15.149:6834","nonce":2513950989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1258891714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1258891714},{"type":"v1","addr":"172.21.15.149:6835","nonce":1258891714}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/3923909288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3923909288},{"type":"v1","addr":"172.21.15.111:6836","nonce":3923909288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T12:58:55.633507+0000","modified":"2023-11-28T12:59:17.702148+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/3180164366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":3180164366},{"type":"v1","addr":"172.21.15.111:6837","nonce":3180164366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T12:59:55.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.016+0000 7f94b4499700 1 -- 172.21.15.111:0/4215997994 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f94900682f0 msgr2=0x7f949006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.016+0000 7f94b4499700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f94900682f0 0x7f949006a7a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f949c000f60 tx=0x7f949c007be0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.016+0000 7f94b4499700 1 -- 172.21.15.111:0/4215997994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94ac0cd460 msgr2=0x7f94ac1f22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.016+0000 7f94b4499700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94ac0cd460 0x7f94ac1f22c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f94a80090b0 tx=0x7f94a8004750 comp rx=0 tx=0).stop 2023-11-28T12:59:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.016+0000 7f94b4499700 1 -- 172.21.15.111:0/4215997994 shutdown_connections 2023-11-28T12:59:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.016+0000 7f94b4499700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f94ac0d0d60 0x7f94ac06b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.016+0000 7f94b4499700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f94900682f0 0x7f949006a7a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.016+0000 7f94b4499700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f94ac0b8510 0x7f94ac1e7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.016+0000 7f94b4499700 1 --2- 172.21.15.111:0/4215997994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94ac0cd460 0x7f94ac1f22c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.016+0000 7f94b4499700 1 -- 172.21.15.111:0/4215997994 >> 172.21.15.111:0/4215997994 conn(0x7f94ac0719a0 msgr2=0x7f94ac0b6850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:55.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.016+0000 7f94b4499700 1 -- 172.21.15.111:0/4215997994 shutdown_connections 2023-11-28T12:59:55.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.016+0000 7f94b4499700 1 -- 172.21.15.111:0/4215997994 wait complete. 2023-11-28T12:59:55.019 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 9 2023-11-28T12:59:55.029 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:55.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.148+0000 7f878eb9a700 1 Processor -- start 2023-11-28T12:59:55.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.148+0000 7f878eb9a700 1 -- start start 2023-11-28T12:59:55.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.148+0000 7f878eb9a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f87880b0d70 0x7f87881604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.148+0000 7f878eb9a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f87880b15b0 0x7f87880cc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f878eb9a700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878815cd90 con 0x7f87880b1140 2023-11-28T12:59:55.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f878eb9a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87880cedd0 con 0x7f87880b15b0 2023-11-28T12:59:55.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f878eb9a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87881613e0 con 0x7f87880b0d70 2023-11-28T12:59:55.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f878d137700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f87880b15b0 0x7f87880cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f878d137700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f87880b15b0 0x7f87880cc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34684/0 (socket says 172.21.15.111:34684) 2023-11-28T12:59:55.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f878d137700 1 -- 172.21.15.111:0/1739137406 learned_addr learned my addr 172.21.15.111:0/1739137406 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:55.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f878c936700 1 --2- 172.21.15.111:0/1739137406 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f87880b0d70 0x7f87881604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f878d137700 1 -- 172.21.15.111:0/1739137406 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f87880b0d70 msgr2=0x7f87881604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:55.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f878d137700 1 --2- 172.21.15.111:0/1739137406 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f87880b0d70 0x7f87881604b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f878d137700 1 -- 172.21.15.111:0/1739137406 >> v1:172.21.15.149:6789/0 conn(0x7f87880b1140 legacy=0x7f87880ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:55.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f878d137700 1 -- 172.21.15.111:0/1739137406 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8788160b40 con 0x7f87880b15b0 2023-11-28T12:59:55.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f878d137700 1 --2- 172.21.15.111:0/1739137406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87880b15b0 0x7f87880cc550 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8780009e50 tx=0x7f878000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a30aca4da88206 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:55.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f87877fe700 1 -- 172.21.15.111:0/1739137406 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878000e070 con 0x7f87880b15b0 2023-11-28T12:59:55.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f87877fe700 1 -- 172.21.15.111:0/1739137406 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8780003a20 con 0x7f87880b15b0 2023-11-28T12:59:55.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.149+0000 7f87877fe700 1 -- 172.21.15.111:0/1739137406 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780003dd0 con 0x7f87880b15b0 2023-11-28T12:59:55.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.150+0000 7f878eb9a700 1 -- 172.21.15.111:0/1739137406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87880b15b0 msgr2=0x7f87880cc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.150+0000 7f878eb9a700 1 --2- 172.21.15.111:0/1739137406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87880b15b0 0x7f87880cc550 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8780009e50 tx=0x7f878000b330 comp rx=0 tx=0).stop 2023-11-28T12:59:55.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.150+0000 7f878eb9a700 1 -- 172.21.15.111:0/1739137406 shutdown_connections 2023-11-28T12:59:55.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.150+0000 7f878eb9a700 1 --2- 172.21.15.111:0/1739137406 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f87880b0d70 0x7f87881604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.150+0000 7f878eb9a700 1 --2- 172.21.15.111:0/1739137406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87880b15b0 0x7f87880cc550 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.150+0000 7f878eb9a700 1 -- 172.21.15.111:0/1739137406 >> 172.21.15.111:0/1739137406 conn(0x7f8788098ee0 msgr2=0x7f878806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:55.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.150+0000 7f878eb9a700 1 -- 172.21.15.111:0/1739137406 shutdown_connections 2023-11-28T12:59:55.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.150+0000 7f878eb9a700 1 -- 172.21.15.111:0/1739137406 wait complete. 2023-11-28T12:59:55.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.151+0000 7f878eb9a700 1 Processor -- start 2023-11-28T12:59:55.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.151+0000 7f878eb9a700 1 -- start start 2023-11-28T12:59:55.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878eb9a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f87880b0d70 0x7f878815b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878eb9a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f87880b1140 0x7f87881e7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878eb9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87880b15b0 0x7f87880b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878eb9a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87881e6fc0 con 0x7f87880b15b0 2023-11-28T12:59:55.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878eb9a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87881e63b0 con 0x7f87880b1140 2023-11-28T12:59:55.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878eb9a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87881e6690 con 0x7f87880b0d70 2023-11-28T12:59:55.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878d137700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87880b15b0 0x7f87880b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878d137700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87880b15b0 0x7f87880b5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34690/0 (socket says 172.21.15.111:34690) 2023-11-28T12:59:55.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878d137700 1 -- 172.21.15.111:0/3153434529 learned_addr learned my addr 172.21.15.111:0/3153434529 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:55.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f8787fff700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f87880b1140 0x7f87881e7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878c936700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f87880b0d70 0x7f878815b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878d137700 1 -- 172.21.15.111:0/3153434529 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f87880b0d70 msgr2=0x7f878815b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878d137700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f87880b0d70 0x7f878815b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878d137700 1 -- 172.21.15.111:0/3153434529 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f87880b1140 msgr2=0x7f87881e7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878d137700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f87880b1140 0x7f87881e7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f878d137700 1 -- 172.21.15.111:0/3153434529 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8780009b30 con 0x7f87880b15b0 2023-11-28T12:59:55.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.152+0000 7f8787fff700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f87880b1140 0x7f87881e7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:55.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.153+0000 7f878d137700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87880b15b0 0x7f87880b5060 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8780015040 tx=0x7f8780004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:55.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.153+0000 7f87857fa700 1 -- 172.21.15.111:0/3153434529 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780004440 con 0x7f87880b15b0 2023-11-28T12:59:55.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.153+0000 7f87857fa700 1 -- 172.21.15.111:0/3153434529 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87800045c0 con 0x7f87880b15b0 2023-11-28T12:59:55.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.153+0000 7f878eb9a700 1 -- 172.21.15.111:0/3153434529 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8788161100 con 0x7f87880b15b0 2023-11-28T12:59:55.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.153+0000 7f87857fa700 1 -- 172.21.15.111:0/3153434529 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780004970 con 0x7f87880b15b0 2023-11-28T12:59:55.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.153+0000 7f878eb9a700 1 -- 172.21.15.111:0/3153434529 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8788160e20 con 0x7f87880b15b0 2023-11-28T12:59:55.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.154+0000 7f878eb9a700 1 -- 172.21.15.111:0/3153434529 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87880af340 con 0x7f87880b15b0 2023-11-28T12:59:55.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.154+0000 7f87857fa700 1 -- 172.21.15.111:0/3153434529 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8780022030 con 0x7f87880b15b0 2023-11-28T12:59:55.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.154+0000 7f87857fa700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f876c0682f0 0x7f876c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.155+0000 7f87857fa700 1 -- 172.21.15.111:0/3153434529 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f8780087150 con 0x7f87880b15b0 2023-11-28T12:59:55.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.155+0000 7f878c936700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f876c0682f0 0x7f876c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.155+0000 7f878c936700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f876c0682f0 0x7f876c06a7a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f877c00aaa0 tx=0x7f877c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:55.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.159+0000 7f87857fa700 1 -- 172.21.15.111:0/3153434529 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8780053680 con 0x7f87880b15b0 2023-11-28T12:59:55.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.340+0000 7f878eb9a700 1 -- 172.21.15.111:0/3153434529 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f878815cd90 con 0x7f87880b15b0 2023-11-28T12:59:55.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.341+0000 7f87857fa700 1 -- 172.21.15.111:0/3153434529 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+19890 (secure 0 0 0) 0x7f878815cd90 con 0x7f87880b15b0 2023-11-28T12:59:55.342 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:55.343 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":28,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:41.880678+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:55.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.343+0000 7f878eb9a700 1 -- 172.21.15.111:0/3153434529 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f876c0682f0 msgr2=0x7f876c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.343+0000 7f878eb9a700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f876c0682f0 0x7f876c06a7a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f877c00aaa0 tx=0x7f877c008040 comp rx=0 tx=0).stop 2023-11-28T12:59:55.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.343+0000 7f878eb9a700 1 -- 172.21.15.111:0/3153434529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87880b15b0 msgr2=0x7f87880b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.343+0000 7f878eb9a700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87880b15b0 0x7f87880b5060 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8780015040 tx=0x7f8780004ad0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.343+0000 7f878eb9a700 1 -- 172.21.15.111:0/3153434529 shutdown_connections 2023-11-28T12:59:55.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.343+0000 7f878eb9a700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87880b15b0 0x7f87880b5060 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.343+0000 7f878eb9a700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f87880b0d70 0x7f878815b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.343+0000 7f878eb9a700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f876c0682f0 0x7f876c06a7a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.343+0000 7f878eb9a700 1 --2- 172.21.15.111:0/3153434529 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f87880b1140 0x7f87881e7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.343+0000 7f878eb9a700 1 -- 172.21.15.111:0/3153434529 >> 172.21.15.111:0/3153434529 conn(0x7f8788098ee0 msgr2=0x7f878806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:55.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.343+0000 7f878eb9a700 1 -- 172.21.15.111:0/3153434529 shutdown_connections 2023-11-28T12:59:55.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.343+0000 7f878eb9a700 1 -- 172.21.15.111:0/3153434529 wait complete. 2023-11-28T12:59:55.357 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T12:58:53.275206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T12:58:54.588499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-28T12:58:57.666033+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-28T12:59:55.358 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T12:59:55.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.479+0000 7efc8cb68700 1 Processor -- start 2023-11-28T12:59:55.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.479+0000 7efc8cb68700 1 -- start start 2023-11-28T12:59:55.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.480+0000 7efc8cb68700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efc880cf010 0x7efc880cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.480+0000 7efc8cb68700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc880cc7b0 con 0x7efc880b2220 2023-11-28T12:59:55.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.480+0000 7efc8cb68700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc880b25f0 con 0x7efc880cb380 2023-11-28T12:59:55.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.480+0000 7efc8cb68700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc8808ae90 con 0x7efc880cf010 2023-11-28T12:59:55.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.480+0000 7efc85d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efc880cf010 0x7efc880cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.480+0000 7efc8659c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efc880b2220 0x7efc880ce900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35472/0 (socket says 172.21.15.111:35472) 2023-11-28T12:59:55.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.480+0000 7efc8659c700 1 -- 172.21.15.111:0/2114105558 learned_addr learned my addr 172.21.15.111:0/2114105558 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:55.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.480+0000 7efc85d9b700 1 -- 172.21.15.111:0/2114105558 >> v1:172.21.15.149:6789/0 conn(0x7efc880cb380 legacy=0x7efc880b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:55.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.480+0000 7efc85d9b700 1 -- 172.21.15.111:0/2114105558 >> v1:172.21.15.111:6789/0 conn(0x7efc880b2220 legacy=0x7efc880ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.480+0000 7efc8559a700 1 -- 172.21.15.111:0/2114105558 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266064771 0 0) 0x7efc880cc7b0 con 0x7efc880b2220 2023-11-28T12:59:55.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.481+0000 7efc85d9b700 1 -- 172.21.15.111:0/2114105558 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc880cca10 con 0x7efc880cf010 2023-11-28T12:59:55.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.481+0000 7efc85d9b700 1 --2- 172.21.15.111:0/2114105558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc880cf010 0x7efc880cc270 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efc78009840 tx=0x7efc78009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0ffa7867cbdae9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:55.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.481+0000 7efc8559a700 1 -- 172.21.15.111:0/2114105558 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc78004230 con 0x7efc880cf010 2023-11-28T12:59:55.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.481+0000 7efc8559a700 1 -- 172.21.15.111:0/2114105558 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc78004390 con 0x7efc880cf010 2023-11-28T12:59:55.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.481+0000 7efc8559a700 1 -- 172.21.15.111:0/2114105558 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc780036a0 con 0x7efc880cf010 2023-11-28T12:59:55.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.481+0000 7efc8cb68700 1 -- 172.21.15.111:0/2114105558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc880cf010 msgr2=0x7efc880cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.481+0000 7efc8cb68700 1 --2- 172.21.15.111:0/2114105558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc880cf010 0x7efc880cc270 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efc78009840 tx=0x7efc78009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:55.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.481+0000 7efc8cb68700 1 -- 172.21.15.111:0/2114105558 shutdown_connections 2023-11-28T12:59:55.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.481+0000 7efc8cb68700 1 --2- 172.21.15.111:0/2114105558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc880cf010 0x7efc880cc270 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.481+0000 7efc8cb68700 1 -- 172.21.15.111:0/2114105558 >> 172.21.15.111:0/2114105558 conn(0x7efc8809a040 msgr2=0x7efc880d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:55.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.481+0000 7efc8cb68700 1 -- 172.21.15.111:0/2114105558 shutdown_connections 2023-11-28T12:59:55.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.481+0000 7efc8cb68700 1 -- 172.21.15.111:0/2114105558 wait complete. 2023-11-28T12:59:55.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.483+0000 7efc8cb68700 1 Processor -- start 2023-11-28T12:59:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.483+0000 7efc8cb68700 1 -- start start 2023-11-28T12:59:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.483+0000 7efc8cb68700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efc880b2220 0x7efc880c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.483+0000 7efc8cb68700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc880cb380 0x7efc8806f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.483+0000 7efc8cb68700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efc880cf010 0x7efc8806fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.483+0000 7efc8cb68700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8814faf0 con 0x7efc880cb380 2023-11-28T12:59:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc8cb68700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc88153580 con 0x7efc880cf010 2023-11-28T12:59:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc8cb68700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc880c7fa0 con 0x7efc880b2220 2023-11-28T12:59:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc85d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc880cb380 0x7efc8806f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc85d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc880cb380 0x7efc8806f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34704/0 (socket says 172.21.15.111:34704) 2023-11-28T12:59:55.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc85d9b700 1 -- 172.21.15.111:0/205151829 learned_addr learned my addr 172.21.15.111:0/205151829 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:55.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc8659c700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efc880b2220 0x7efc880c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc86d9d700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efc880cf010 0x7efc8806fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc85d9b700 1 -- 172.21.15.111:0/205151829 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efc880b2220 msgr2=0x7efc880c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc85d9b700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efc880b2220 0x7efc880c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc85d9b700 1 -- 172.21.15.111:0/205151829 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efc880cf010 msgr2=0x7efc8806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc85d9b700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efc880cf010 0x7efc8806fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc85d9b700 1 -- 172.21.15.111:0/205151829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc780094a0 con 0x7efc880cb380 2023-11-28T12:59:55.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc85d9b700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc880cb380 0x7efc8806f510 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efc780090b0 tx=0x7efc780047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:55.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc76ffd700 1 -- 172.21.15.111:0/205151829 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc78011630 con 0x7efc880cb380 2023-11-28T12:59:55.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc76ffd700 1 -- 172.21.15.111:0/205151829 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc78011790 con 0x7efc880cb380 2023-11-28T12:59:55.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc76ffd700 1 -- 172.21.15.111:0/205151829 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc78011b40 con 0x7efc880cb380 2023-11-28T12:59:55.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc8cb68700 1 -- 172.21.15.111:0/205151829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc880ccfd0 con 0x7efc880cb380 2023-11-28T12:59:55.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.484+0000 7efc8cb68700 1 -- 172.21.15.111:0/205151829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc880cccf0 con 0x7efc880cb380 2023-11-28T12:59:55.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.485+0000 7efc8cb68700 1 -- 172.21.15.111:0/205151829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc880cd2b0 con 0x7efc880cb380 2023-11-28T12:59:55.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.486+0000 7efc76ffd700 1 -- 172.21.15.111:0/205151829 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efc78017020 con 0x7efc880cb380 2023-11-28T12:59:55.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.486+0000 7efc76ffd700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efc680682f0 0x7efc6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.486+0000 7efc76ffd700 1 -- 172.21.15.111:0/205151829 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7efc78087810 con 0x7efc880cb380 2023-11-28T12:59:55.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.486+0000 7efc8659c700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efc680682f0 0x7efc6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.487+0000 7efc8659c700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efc680682f0 0x7efc6806a7a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efc7c000f60 tx=0x7efc7c007be0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:55.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.491+0000 7efc76ffd700 1 -- 172.21.15.111:0/205151829 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efc78053d40 con 0x7efc880cb380 2023-11-28T12:59:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.672+0000 7efc8cb68700 1 -- 172.21.15.111:0/205151829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efc88067170 con 0x7efc880cb380 2023-11-28T12:59:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.673+0000 7efc76ffd700 1 -- 172.21.15.111:0/205151829 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7efc78057360 con 0x7efc880cb380 2023-11-28T12:59:55.675 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T12:59:55.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.674+0000 7efc8cb68700 1 -- 172.21.15.111:0/205151829 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efc680682f0 msgr2=0x7efc6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.674+0000 7efc8cb68700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efc680682f0 0x7efc6806a7a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efc7c000f60 tx=0x7efc7c007be0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.675+0000 7efc8cb68700 1 -- 172.21.15.111:0/205151829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc880cb380 msgr2=0x7efc8806f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.675+0000 7efc8cb68700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc880cb380 0x7efc8806f510 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efc780090b0 tx=0x7efc780047c0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.675+0000 7efc8cb68700 1 -- 172.21.15.111:0/205151829 shutdown_connections 2023-11-28T12:59:55.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.675+0000 7efc8cb68700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efc680682f0 0x7efc6806a7a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.675+0000 7efc8cb68700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efc880cf010 0x7efc8806fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.675+0000 7efc8cb68700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efc880b2220 0x7efc880c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.675+0000 7efc8cb68700 1 --2- 172.21.15.111:0/205151829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc880cb380 0x7efc8806f510 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.675+0000 7efc8cb68700 1 -- 172.21.15.111:0/205151829 >> 172.21.15.111:0/205151829 conn(0x7efc8809a040 msgr2=0x7efc880b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:55.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.675+0000 7efc8cb68700 1 -- 172.21.15.111:0/205151829 shutdown_connections 2023-11-28T12:59:55.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.675+0000 7efc8cb68700 1 -- 172.21.15.111:0/205151829 wait complete. 2023-11-28T12:59:55.690 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T12:59:55.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.811+0000 7efd70365700 1 Processor -- start 2023-11-28T12:59:55.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.811+0000 7efd70365700 1 -- start start 2023-11-28T12:59:55.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.812+0000 7efd70365700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efd680b15c0 0x7efd680ac670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.812+0000 7efd70365700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd680af650 con 0x7efd6815a9e0 2023-11-28T12:59:55.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.812+0000 7efd70365700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd680cb6d0 con 0x7efd680b0250 2023-11-28T12:59:55.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.812+0000 7efd70365700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd680cb9b0 con 0x7efd680b15c0 2023-11-28T12:59:55.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.812+0000 7efd6d900700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efd680b15c0 0x7efd680ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.812+0000 7efd6e101700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efd6815a9e0 0x7efd680caef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35486/0 (socket says 172.21.15.111:35486) 2023-11-28T12:59:55.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.812+0000 7efd6e101700 1 -- 172.21.15.111:0/238214717 learned_addr learned my addr 172.21.15.111:0/238214717 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:55.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.812+0000 7efd6d900700 1 -- 172.21.15.111:0/238214717 >> v1:172.21.15.149:6789/0 conn(0x7efd680b0250 legacy=0x7efd680cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:55.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.812+0000 7efd6d900700 1 -- 172.21.15.111:0/238214717 >> v1:172.21.15.111:6789/0 conn(0x7efd6815a9e0 legacy=0x7efd680caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.812+0000 7efd6d900700 1 -- 172.21.15.111:0/238214717 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd680cbc90 con 0x7efd680b15c0 2023-11-28T12:59:55.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.812+0000 7efd6d900700 1 --2- 172.21.15.111:0/238214717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd680b15c0 0x7efd680ac670 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efd64009840 tx=0x7efd64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dada3920ee5b05b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:55.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.813+0000 7efd6d0ff700 1 -- 172.21.15.111:0/238214717 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64004230 con 0x7efd680b15c0 2023-11-28T12:59:55.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.813+0000 7efd6d0ff700 1 -- 172.21.15.111:0/238214717 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd64004390 con 0x7efd680b15c0 2023-11-28T12:59:55.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.813+0000 7efd6d0ff700 1 -- 172.21.15.111:0/238214717 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd640036a0 con 0x7efd680b15c0 2023-11-28T12:59:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.813+0000 7efd70365700 1 -- 172.21.15.111:0/238214717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd680b15c0 msgr2=0x7efd680ac670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.813+0000 7efd70365700 1 --2- 172.21.15.111:0/238214717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd680b15c0 0x7efd680ac670 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efd64009840 tx=0x7efd64009b50 comp rx=0 tx=0).stop 2023-11-28T12:59:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.813+0000 7efd70365700 1 -- 172.21.15.111:0/238214717 shutdown_connections 2023-11-28T12:59:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.813+0000 7efd70365700 1 --2- 172.21.15.111:0/238214717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd680b15c0 0x7efd680ac670 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.813+0000 7efd70365700 1 -- 172.21.15.111:0/238214717 >> 172.21.15.111:0/238214717 conn(0x7efd68098ee0 msgr2=0x7efd6806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.813+0000 7efd70365700 1 -- 172.21.15.111:0/238214717 shutdown_connections 2023-11-28T12:59:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.813+0000 7efd70365700 1 -- 172.21.15.111:0/238214717 wait complete. 2023-11-28T12:59:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.815+0000 7efd70365700 1 Processor -- start 2023-11-28T12:59:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.815+0000 7efd70365700 1 -- start start 2023-11-28T12:59:55.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.815+0000 7efd70365700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd680b0250 0x7efd68154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.815+0000 7efd70365700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd680b15c0 0x7efd68158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.815+0000 7efd70365700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd6815a9e0 0x7efd680b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.815+0000 7efd70365700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd681571f0 con 0x7efd680b0250 2023-11-28T12:59:55.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.815+0000 7efd70365700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd681574d0 con 0x7efd680b15c0 2023-11-28T12:59:55.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.815+0000 7efd70365700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd681577b0 con 0x7efd6815a9e0 2023-11-28T12:59:55.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.815+0000 7efd6e101700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd680b0250 0x7efd68154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.815+0000 7efd6e101700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd680b0250 0x7efd68154450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34714/0 (socket says 172.21.15.111:34714) 2023-11-28T12:59:55.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.815+0000 7efd6e101700 1 -- 172.21.15.111:0/932717513 learned_addr learned my addr 172.21.15.111:0/932717513 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:55.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.816+0000 7efd6e101700 1 -- 172.21.15.111:0/932717513 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd6815a9e0 msgr2=0x7efd680b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.816+0000 7efd6e101700 1 --2- 172.21.15.111:0/932717513 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd6815a9e0 0x7efd680b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.816+0000 7efd6d900700 1 --2- 172.21.15.111:0/932717513 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd680b15c0 0x7efd68158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.816+0000 7efd6e101700 1 -- 172.21.15.111:0/932717513 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd680b15c0 msgr2=0x7efd68158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:55.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.816+0000 7efd6e101700 1 --2- 172.21.15.111:0/932717513 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd680b15c0 0x7efd68158290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:55.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.816+0000 7efd6e101700 1 -- 172.21.15.111:0/932717513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd640094a0 con 0x7efd680b0250 2023-11-28T12:59:55.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.816+0000 7efd6e101700 1 --2- 172.21.15.111:0/932717513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd680b0250 0x7efd68154450 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7efd58007fd0 tx=0x7efd5800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:55.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.816+0000 7efd5effd700 1 -- 172.21.15.111:0/932717513 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd580074e0 con 0x7efd680b0250 2023-11-28T12:59:55.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.816+0000 7efd5effd700 1 -- 172.21.15.111:0/932717513 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd58007640 con 0x7efd680b0250 2023-11-28T12:59:55.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.816+0000 7efd70365700 1 -- 172.21.15.111:0/932717513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd680cc250 con 0x7efd680b0250 2023-11-28T12:59:55.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.816+0000 7efd5effd700 1 -- 172.21.15.111:0/932717513 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58005a50 con 0x7efd680b0250 2023-11-28T12:59:55.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.816+0000 7efd70365700 1 -- 172.21.15.111:0/932717513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd680cbf70 con 0x7efd680b0250 2023-11-28T12:59:55.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.817+0000 7efd70365700 1 -- 172.21.15.111:0/932717513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd680b06d0 con 0x7efd680b0250 2023-11-28T12:59:55.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.817+0000 7efd5effd700 1 -- 172.21.15.111:0/932717513 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efd58019030 con 0x7efd680b0250 2023-11-28T12:59:55.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.818+0000 7efd5effd700 1 --2- 172.21.15.111:0/932717513 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd4c0682f0 0x7efd4c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:55.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.818+0000 7efd5effd700 1 -- 172.21.15.111:0/932717513 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7efd580852a0 con 0x7efd680b0250 2023-11-28T12:59:55.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.818+0000 7efd6d900700 1 --2- 172.21.15.111:0/932717513 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd4c0682f0 0x7efd4c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:55.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.818+0000 7efd6d900700 1 --2- 172.21.15.111:0/932717513 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd4c0682f0 0x7efd4c06a7a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efd6809d9a0 tx=0x7efd64008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:55.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:55.822+0000 7efd5effd700 1 -- 172.21.15.111:0/932717513 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efd58051890 con 0x7efd680b0250 2023-11-28T12:59:56.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.006+0000 7efd70365700 1 -- 172.21.15.111:0/932717513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7efd68067170 con 0x7efd680b0250 2023-11-28T12:59:56.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.007+0000 7efd5effd700 1 -- 172.21.15.111:0/932717513 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7efd58017020 con 0x7efd680b0250 2023-11-28T12:59:56.008 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T12:59:56.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.009+0000 7efd70365700 1 -- 172.21.15.111:0/932717513 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd4c0682f0 msgr2=0x7efd4c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.009+0000 7efd70365700 1 --2- 172.21.15.111:0/932717513 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd4c0682f0 0x7efd4c06a7a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efd6809d9a0 tx=0x7efd64008030 comp rx=0 tx=0).stop 2023-11-28T12:59:56.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.009+0000 7efd70365700 1 -- 172.21.15.111:0/932717513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd680b0250 msgr2=0x7efd68154450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.009+0000 7efd70365700 1 --2- 172.21.15.111:0/932717513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd680b0250 0x7efd68154450 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7efd58007fd0 tx=0x7efd5800c790 comp rx=0 tx=0).stop 2023-11-28T12:59:56.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.009+0000 7efd70365700 1 -- 172.21.15.111:0/932717513 shutdown_connections 2023-11-28T12:59:56.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.009+0000 7efd70365700 1 --2- 172.21.15.111:0/932717513 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efd6815a9e0 0x7efd680b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.009+0000 7efd70365700 1 --2- 172.21.15.111:0/932717513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efd680b0250 0x7efd68154450 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.009+0000 7efd70365700 1 --2- 172.21.15.111:0/932717513 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efd4c0682f0 0x7efd4c06a7a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.009+0000 7efd70365700 1 --2- 172.21.15.111:0/932717513 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efd680b15c0 0x7efd68158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.009+0000 7efd70365700 1 -- 172.21.15.111:0/932717513 >> 172.21.15.111:0/932717513 conn(0x7efd68098ee0 msgr2=0x7efd6806e650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:56.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.009+0000 7efd70365700 1 -- 172.21.15.111:0/932717513 shutdown_connections 2023-11-28T12:59:56.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.009+0000 7efd70365700 1 -- 172.21.15.111:0/932717513 wait complete. 2023-11-28T12:59:56.022 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T12:59:56.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.142+0000 7fa9de55e700 1 Processor -- start 2023-11-28T12:59:56.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.142+0000 7fa9de55e700 1 -- start start 2023-11-28T12:59:56.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.143+0000 7fa9de55e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fa9d815a5d0 0x7fa9d80ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.143+0000 7fa9de55e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d80b0160 0x7fa9d80ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.143+0000 7fa9de55e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa9d80cb2d0 0x7fa9d815e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.143+0000 7fa9de55e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9d80cc1b0 con 0x7fa9d80cb2d0 2023-11-28T12:59:56.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.143+0000 7fa9de55e700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9d80cc390 con 0x7fa9d815a5d0 2023-11-28T12:59:56.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.143+0000 7fa9de55e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80cc670 con 0x7fa9d80b0160 2023-11-28T12:59:56.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.143+0000 7fa9dcafb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa9d80cb2d0 0x7fa9d815e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.143+0000 7fa9dcafb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa9d80cb2d0 0x7fa9d815e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34726/0 (socket says 172.21.15.111:34726) 2023-11-28T12:59:56.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.143+0000 7fa9dcafb700 1 -- 172.21.15.111:0/440373255 learned_addr learned my addr 172.21.15.111:0/440373255 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:56.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9dcafb700 1 -- 172.21.15.111:0/440373255 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d80b0160 msgr2=0x7fa9d80ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:56.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9d77fe700 1 --2- 172.21.15.111:0/440373255 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d80b0160 0x7fa9d80ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9d7fff700 1 --2- 172.21.15.111:0/440373255 >> v2:172.21.15.149:3300/0 conn(0x7fa9d815a5d0 0x7fa9d80ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9dcafb700 1 --2- 172.21.15.111:0/440373255 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d80b0160 0x7fa9d80ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9dcafb700 1 -- 172.21.15.111:0/440373255 >> v2:172.21.15.149:3300/0 conn(0x7fa9d815a5d0 msgr2=0x7fa9d80ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:56.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9dcafb700 1 --2- 172.21.15.111:0/440373255 >> v2:172.21.15.149:3300/0 conn(0x7fa9d815a5d0 0x7fa9d80ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9dcafb700 1 -- 172.21.15.111:0/440373255 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d80cc950 con 0x7fa9d80cb2d0 2023-11-28T12:59:56.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9dcafb700 1 --2- 172.21.15.111:0/440373255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d80cb2d0 0x7fa9d815e120 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8067d30 tx=0x7fa9d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c8b18508dd2820 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:56.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9d6ffd700 1 -- 172.21.15.111:0/440373255 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d000e070 con 0x7fa9d80cb2d0 2023-11-28T12:59:56.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9d6ffd700 1 -- 172.21.15.111:0/440373255 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d000b9c0 con 0x7fa9d80cb2d0 2023-11-28T12:59:56.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9d6ffd700 1 -- 172.21.15.111:0/440373255 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d000bd70 con 0x7fa9d80cb2d0 2023-11-28T12:59:56.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9de55e700 1 -- 172.21.15.111:0/440373255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d80cb2d0 msgr2=0x7fa9d815e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9de55e700 1 --2- 172.21.15.111:0/440373255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d80cb2d0 0x7fa9d815e120 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8067d30 tx=0x7fa9d000b330 comp rx=0 tx=0).stop 2023-11-28T12:59:56.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9de55e700 1 -- 172.21.15.111:0/440373255 shutdown_connections 2023-11-28T12:59:56.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9de55e700 1 --2- 172.21.15.111:0/440373255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d80cb2d0 0x7fa9d815e120 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9de55e700 1 --2- 172.21.15.111:0/440373255 >> v2:172.21.15.149:3300/0 conn(0x7fa9d815a5d0 0x7fa9d80ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9de55e700 1 --2- 172.21.15.111:0/440373255 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d80b0160 0x7fa9d80ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.144+0000 7fa9de55e700 1 -- 172.21.15.111:0/440373255 >> 172.21.15.111:0/440373255 conn(0x7fa9d809a040 msgr2=0x7fa9d80d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:56.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.145+0000 7fa9de55e700 1 -- 172.21.15.111:0/440373255 shutdown_connections 2023-11-28T12:59:56.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.145+0000 7fa9de55e700 1 -- 172.21.15.111:0/440373255 wait complete. 2023-11-28T12:59:56.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.146+0000 7fa9de55e700 1 Processor -- start 2023-11-28T12:59:56.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.146+0000 7fa9de55e700 1 -- start start 2023-11-28T12:59:56.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9de55e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d80b0160 0x7fa9d80ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9de55e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d80cb2d0 0x7fa9d80ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9de55e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d815a5d0 0x7fa9d80aef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9de55e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80ca8b0 con 0x7fa9d80b0160 2023-11-28T12:59:56.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9de55e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d815cab0 con 0x7fa9d815a5d0 2023-11-28T12:59:56.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9de55e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d815cd90 con 0x7fa9d80cb2d0 2023-11-28T12:59:56.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9d7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d80b0160 0x7fa9d80ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9d7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d80b0160 0x7fa9d80ca370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34728/0 (socket says 172.21.15.111:34728) 2023-11-28T12:59:56.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9d7fff700 1 -- 172.21.15.111:0/3070692513 learned_addr learned my addr 172.21.15.111:0/3070692513 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:56.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9dcafb700 1 --2- 172.21.15.111:0/3070692513 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d815a5d0 0x7fa9d80aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9d7fff700 1 -- 172.21.15.111:0/3070692513 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d80cb2d0 msgr2=0x7fa9d80ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9d7fff700 1 --2- 172.21.15.111:0/3070692513 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d80cb2d0 0x7fa9d80ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9d7fff700 1 -- 172.21.15.111:0/3070692513 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d815a5d0 msgr2=0x7fa9d80aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9d7fff700 1 --2- 172.21.15.111:0/3070692513 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d815a5d0 0x7fa9d80aef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9d7fff700 1 -- 172.21.15.111:0/3070692513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d0009b30 con 0x7fa9d80b0160 2023-11-28T12:59:56.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9d7fff700 1 --2- 172.21.15.111:0/3070692513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d80b0160 0x7fa9d80ca370 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00c7f0 tx=0x7fa9cc00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:56.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9d4ff9700 1 -- 172.21.15.111:0/3070692513 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc011800 con 0x7fa9d80b0160 2023-11-28T12:59:56.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9d4ff9700 1 -- 172.21.15.111:0/3070692513 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9cc011960 con 0x7fa9d80b0160 2023-11-28T12:59:56.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9de55e700 1 -- 172.21.15.111:0/3070692513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d80ccf10 con 0x7fa9d80b0160 2023-11-28T12:59:56.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9d4ff9700 1 -- 172.21.15.111:0/3070692513 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc011cf0 con 0x7fa9d80b0160 2023-11-28T12:59:56.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.147+0000 7fa9de55e700 1 -- 172.21.15.111:0/3070692513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d80ccc30 con 0x7fa9d80b0160 2023-11-28T12:59:56.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.148+0000 7fa9de55e700 1 -- 172.21.15.111:0/3070692513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d815a9a0 con 0x7fa9d80b0160 2023-11-28T12:59:56.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.149+0000 7fa9d4ff9700 1 -- 172.21.15.111:0/3070692513 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa9cc021070 con 0x7fa9d80b0160 2023-11-28T12:59:56.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.149+0000 7fa9d4ff9700 1 --2- 172.21.15.111:0/3070692513 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9c40682f0 0x7fa9c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.149+0000 7fa9d4ff9700 1 -- 172.21.15.111:0/3070692513 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fa9cc085d70 con 0x7fa9d80b0160 2023-11-28T12:59:56.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.149+0000 7fa9d77fe700 1 --2- 172.21.15.111:0/3070692513 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9c40682f0 0x7fa9c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.149+0000 7fa9d77fe700 1 --2- 172.21.15.111:0/3070692513 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9c40682f0 0x7fa9c406a7a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8006ba0 tx=0x7fa9c8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:56.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.153+0000 7fa9d4ff9700 1 -- 172.21.15.111:0/3070692513 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa9cc0522a0 con 0x7fa9d80b0160 2023-11-28T12:59:56.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.333+0000 7fa9de55e700 1 -- 172.21.15.111:0/3070692513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa9d815d2b0 con 0x7fa9d80b0160 2023-11-28T12:59:56.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.334+0000 7fa9d4ff9700 1 -- 172.21.15.111:0/3070692513 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa9d815d2b0 con 0x7fa9d80b0160 2023-11-28T12:59:56.335 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:56.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.335+0000 7fa9de55e700 1 -- 172.21.15.111:0/3070692513 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9c40682f0 msgr2=0x7fa9c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.335+0000 7fa9de55e700 1 --2- 172.21.15.111:0/3070692513 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9c40682f0 0x7fa9c406a7a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8006ba0 tx=0x7fa9c8008040 comp rx=0 tx=0).stop 2023-11-28T12:59:56.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.336+0000 7fa9de55e700 1 -- 172.21.15.111:0/3070692513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d80b0160 msgr2=0x7fa9d80ca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.336+0000 7fa9de55e700 1 --2- 172.21.15.111:0/3070692513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d80b0160 0x7fa9d80ca370 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00c7f0 tx=0x7fa9cc00cbb0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.336+0000 7fa9de55e700 1 -- 172.21.15.111:0/3070692513 shutdown_connections 2023-11-28T12:59:56.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.336+0000 7fa9de55e700 1 --2- 172.21.15.111:0/3070692513 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9c40682f0 0x7fa9c406a7a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.336+0000 7fa9de55e700 1 --2- 172.21.15.111:0/3070692513 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d815a5d0 0x7fa9d80aef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.336+0000 7fa9de55e700 1 --2- 172.21.15.111:0/3070692513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d80b0160 0x7fa9d80ca370 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.336+0000 7fa9de55e700 1 --2- 172.21.15.111:0/3070692513 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d80cb2d0 0x7fa9d80ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.336+0000 7fa9de55e700 1 -- 172.21.15.111:0/3070692513 >> 172.21.15.111:0/3070692513 conn(0x7fa9d809a040 msgr2=0x7fa9d80701d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:56.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.336+0000 7fa9de55e700 1 -- 172.21.15.111:0/3070692513 shutdown_connections 2023-11-28T12:59:56.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.336+0000 7fa9de55e700 1 -- 172.21.15.111:0/3070692513 wait complete. 2023-11-28T12:59:56.349 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T12:59:56.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.471+0000 7f13bc4ee700 1 Processor -- start 2023-11-28T12:59:56.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.471+0000 7f13bc4ee700 1 -- start start 2023-11-28T12:59:56.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.472+0000 7f13bc4ee700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b40cdd90 0x7f13b40b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.472+0000 7f13bc4ee700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f13b40aecc0 0x7f13b40ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.472+0000 7f13bc4ee700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b40ad420 con 0x7f13b4159960 2023-11-28T12:59:56.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.472+0000 7f13bc4ee700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b4159d30 con 0x7f13b40aecc0 2023-11-28T12:59:56.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.472+0000 7f13bc4ee700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b40ac810 con 0x7f13b40cdd90 2023-11-28T12:59:56.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.472+0000 7f13baa8b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f13b40aecc0 0x7f13b40ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.472+0000 7f13baa8b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f13b40aecc0 0x7f13b40ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34740/0 (socket says 172.21.15.111:34740) 2023-11-28T12:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.472+0000 7f13baa8b700 1 -- 172.21.15.111:0/2871828208 learned_addr learned my addr 172.21.15.111:0/2871828208 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.472+0000 7f13b9a89700 1 --2- 172.21.15.111:0/2871828208 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b40cdd90 0x7f13b40b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13baa8b700 1 -- 172.21.15.111:0/2871828208 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b40cdd90 msgr2=0x7f13b40b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13baa8b700 1 --2- 172.21.15.111:0/2871828208 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b40cdd90 0x7f13b40b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13baa8b700 1 -- 172.21.15.111:0/2871828208 >> v1:172.21.15.149:6789/0 conn(0x7f13b4159960 legacy=0x7f13b40cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13baa8b700 1 -- 172.21.15.111:0/2871828208 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b40acaf0 con 0x7f13b40aecc0 2023-11-28T12:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13baa8b700 1 --2- 172.21.15.111:0/2871828208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b40aecc0 0x7f13b40ad8b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f13ac009e50 tx=0x7f13ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c26db8b459499d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13b9288700 1 -- 172.21.15.111:0/2871828208 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac00e070 con 0x7f13b40aecc0 2023-11-28T12:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13b9288700 1 -- 172.21.15.111:0/2871828208 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ac003a20 con 0x7f13b40aecc0 2023-11-28T12:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13b9288700 1 -- 172.21.15.111:0/2871828208 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac003dd0 con 0x7f13b40aecc0 2023-11-28T12:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2871828208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b40aecc0 msgr2=0x7f13b40ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13bc4ee700 1 --2- 172.21.15.111:0/2871828208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b40aecc0 0x7f13b40ad8b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f13ac009e50 tx=0x7f13ac00b330 comp rx=0 tx=0).stop 2023-11-28T12:59:56.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2871828208 shutdown_connections 2023-11-28T12:59:56.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13bc4ee700 1 --2- 172.21.15.111:0/2871828208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b40aecc0 0x7f13b40ad8b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13bc4ee700 1 --2- 172.21.15.111:0/2871828208 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b40cdd90 0x7f13b40b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2871828208 >> 172.21.15.111:0/2871828208 conn(0x7f13b4098ee0 msgr2=0x7f13b406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:56.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2871828208 shutdown_connections 2023-11-28T12:59:56.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.473+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2871828208 wait complete. 2023-11-28T12:59:56.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.475+0000 7f13bc4ee700 1 Processor -- start 2023-11-28T12:59:56.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.475+0000 7f13bc4ee700 1 -- start start 2023-11-28T12:59:56.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.475+0000 7f13bc4ee700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b40aecc0 0x7f13b40c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.475+0000 7f13bc4ee700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f13b40cdd90 0x7f13b40b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.475+0000 7f13bc4ee700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b4159960 0x7f13b414ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.475+0000 7f13bc4ee700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b40c5c60 con 0x7f13b40aecc0 2023-11-28T12:59:56.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.475+0000 7f13bc4ee700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b40c5050 con 0x7f13b40cdd90 2023-11-28T12:59:56.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.475+0000 7f13bc4ee700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b40c5330 con 0x7f13b4159960 2023-11-28T12:59:56.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.475+0000 7f13ba28a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b40aecc0 0x7f13b40c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.475+0000 7f13ba28a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b40aecc0 0x7f13b40c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34742/0 (socket says 172.21.15.111:34742) 2023-11-28T12:59:56.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.475+0000 7f13ba28a700 1 -- 172.21.15.111:0/2373801903 learned_addr learned my addr 172.21.15.111:0/2373801903 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.476+0000 7f13ba28a700 1 -- 172.21.15.111:0/2373801903 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b4159960 msgr2=0x7f13b414ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.476+0000 7f13ba28a700 1 --2- 172.21.15.111:0/2373801903 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b4159960 0x7f13b414ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.476+0000 7f13ba28a700 1 -- 172.21.15.111:0/2373801903 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f13b40cdd90 msgr2=0x7f13b40b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.476+0000 7f13b9a89700 1 --2- 172.21.15.111:0/2373801903 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f13b40cdd90 0x7f13b40b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.476+0000 7f13ba28a700 1 --2- 172.21.15.111:0/2373801903 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f13b40cdd90 0x7f13b40b2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.476+0000 7f13ba28a700 1 -- 172.21.15.111:0/2373801903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac009b30 con 0x7f13b40aecc0 2023-11-28T12:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.476+0000 7f13ba28a700 1 --2- 172.21.15.111:0/2373801903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b40aecc0 0x7f13b40c60f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f13a400b450 tx=0x7f13a400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.476+0000 7f13aaffd700 1 -- 172.21.15.111:0/2373801903 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a4004020 con 0x7f13b40aecc0 2023-11-28T12:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.476+0000 7f13aaffd700 1 -- 172.21.15.111:0/2373801903 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a4004180 con 0x7f13b40aecc0 2023-11-28T12:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.476+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2373801903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b40ad0b0 con 0x7f13b40aecc0 2023-11-28T12:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.476+0000 7f13aaffd700 1 -- 172.21.15.111:0/2373801903 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a4004530 con 0x7f13b40aecc0 2023-11-28T12:59:56.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.476+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2373801903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b40acdd0 con 0x7f13b40aecc0 2023-11-28T12:59:56.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.477+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2373801903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b40addf0 con 0x7f13b40aecc0 2023-11-28T12:59:56.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.477+0000 7f13aaffd700 1 -- 172.21.15.111:0/2373801903 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f13a400f050 con 0x7f13b40aecc0 2023-11-28T12:59:56.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.478+0000 7f13aaffd700 1 --2- 172.21.15.111:0/2373801903 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13980682f0 0x7f139806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.478+0000 7f13b9a89700 1 --2- 172.21.15.111:0/2373801903 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13980682f0 0x7f139806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.478+0000 7f13aaffd700 1 -- 172.21.15.111:0/2373801903 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f13a40856b0 con 0x7f13b40aecc0 2023-11-28T12:59:56.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.478+0000 7f13b9a89700 1 --2- 172.21.15.111:0/2373801903 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13980682f0 0x7f139806a7a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f13b40c4a30 tx=0x7f13b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:56.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.482+0000 7f13aaffd700 1 -- 172.21.15.111:0/2373801903 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f13a4051b80 con 0x7f13b40aecc0 2023-11-28T12:59:56.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.664+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2373801903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f13b40c5f40 con 0x7f13b40aecc0 2023-11-28T12:59:56.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.664+0000 7f13aaffd700 1 -- 172.21.15.111:0/2373801903 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f13a40150a0 con 0x7f13b40aecc0 2023-11-28T12:59:56.666 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T12:59:56.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.666+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2373801903 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13980682f0 msgr2=0x7f139806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.666+0000 7f13bc4ee700 1 --2- 172.21.15.111:0/2373801903 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13980682f0 0x7f139806a7a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f13b40c4a30 tx=0x7f13b0008040 comp rx=0 tx=0).stop 2023-11-28T12:59:56.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.666+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2373801903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b40aecc0 msgr2=0x7f13b40c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.666+0000 7f13bc4ee700 1 --2- 172.21.15.111:0/2373801903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b40aecc0 0x7f13b40c60f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f13a400b450 tx=0x7f13a400b810 comp rx=0 tx=0).stop 2023-11-28T12:59:56.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.666+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2373801903 shutdown_connections 2023-11-28T12:59:56.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.666+0000 7f13bc4ee700 1 --2- 172.21.15.111:0/2373801903 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b4159960 0x7f13b414ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.666+0000 7f13bc4ee700 1 --2- 172.21.15.111:0/2373801903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b40aecc0 0x7f13b40c60f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.666+0000 7f13bc4ee700 1 --2- 172.21.15.111:0/2373801903 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13980682f0 0x7f139806a7a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.666+0000 7f13bc4ee700 1 --2- 172.21.15.111:0/2373801903 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f13b40cdd90 0x7f13b40b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.666+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2373801903 >> 172.21.15.111:0/2373801903 conn(0x7f13b4098ee0 msgr2=0x7f13b406e070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:56.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.666+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2373801903 shutdown_connections 2023-11-28T12:59:56.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.666+0000 7f13bc4ee700 1 -- 172.21.15.111:0/2373801903 wait complete. 2023-11-28T12:59:56.680 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-28T12:59:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.801+0000 7f7466fa5700 1 Processor -- start 2023-11-28T12:59:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.802+0000 7f7466fa5700 1 -- start start 2023-11-28T12:59:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.802+0000 7f7466fa5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74600b0440 0x7f74600b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.802+0000 7f7466fa5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74600cc750 con 0x7f74600ca2f0 2023-11-28T12:59:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.802+0000 7f7466fa5700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74600cca30 con 0x7f746015c6c0 2023-11-28T12:59:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.802+0000 7f7466fa5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74600b0260 con 0x7f74600b0440 2023-11-28T12:59:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f7465542700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f74600ca2f0 0x7f74600ca770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35532/0 (socket says 172.21.15.111:35532) 2023-11-28T12:59:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f7465542700 1 -- 172.21.15.111:0/776263812 learned_addr learned my addr 172.21.15.111:0/776263812 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988543251 0 0) 0x7f74600cc750 con 0x7f74600ca2f0 2023-11-28T12:59:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74500035b0 con 0x7f74600ca2f0 2023-11-28T12:59:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745ffff700 1 --2- 172.21.15.111:0/776263812 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74600b0440 0x7f74600b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4052767143 0 0) 0x7f74500035b0 con 0x7f74600ca2f0 2023-11-28T12:59:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74600b0440 msgr2=0x7f74600b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 --2- 172.21.15.111:0/776263812 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74600b0440 0x7f74600b08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 >> v1:172.21.15.149:6789/0 conn(0x7f746015c6c0 legacy=0x7f74600afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74600ccd10 con 0x7f74600ca2f0 2023-11-28T12:59:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f74580019f0 con 0x7f74600ca2f0 2023-11-28T12:59:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 >> v1:172.21.15.111:6789/0 conn(0x7f74600ca2f0 legacy=0x7f74600ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 --2- 172.21.15.111:0/776263812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7450003a30 0x7f7450003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 --2- 172.21.15.111:0/776263812 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7450004850 0x7f7450004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 --2- 172.21.15.111:0/776263812 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74600b0440 0x7f7450005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74600b0260 con 0x7f7450003a30 2023-11-28T12:59:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745000bac0 con 0x7f7450004850 2023-11-28T12:59:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745000bc50 con 0x7f74600b0440 2023-11-28T12:59:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f7464d41700 1 --2- 172.21.15.111:0/776263812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7450003a30 0x7f7450003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745ffff700 1 --2- 172.21.15.111:0/776263812 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7450004850 0x7f7450004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f7465542700 1 --2- 172.21.15.111:0/776263812 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74600b0440 0x7f7450005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f7464d41700 1 -- 172.21.15.111:0/776263812 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74600b0440 msgr2=0x7f7450005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f7464d41700 1 --2- 172.21.15.111:0/776263812 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74600b0440 0x7f7450005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f7464d41700 1 -- 172.21.15.111:0/776263812 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7450004850 msgr2=0x7f7450004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f7464d41700 1 --2- 172.21.15.111:0/776263812 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7450004850 0x7f7450004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f7464d41700 1 -- 172.21.15.111:0/776263812 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74500035b0 con 0x7f7450003a30 2023-11-28T12:59:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.803+0000 7f745ffff700 1 --2- 172.21.15.111:0/776263812 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7450004850 0x7f7450004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.804+0000 7f7464d41700 1 --2- 172.21.15.111:0/776263812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7450003a30 0x7f7450003e40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f74540017c0 tx=0x7f7450005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c29acc8d7e312 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.804+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74540052d0 con 0x7f7450003a30 2023-11-28T12:59:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.804+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7454005430 con 0x7f7450003a30 2023-11-28T12:59:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.804+0000 7f745f7fe700 1 -- 172.21.15.111:0/776263812 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74540038c0 con 0x7f7450003a30 2023-11-28T12:59:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.804+0000 7f7466fa5700 1 -- 172.21.15.111:0/776263812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7450003a30 msgr2=0x7f7450003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.804+0000 7f7466fa5700 1 --2- 172.21.15.111:0/776263812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7450003a30 0x7f7450003e40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f74540017c0 tx=0x7f7450005880 comp rx=0 tx=0).stop 2023-11-28T12:59:56.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.804+0000 7f7465542700 1 -- 172.21.15.111:0/776263812 reap_dead start 2023-11-28T12:59:56.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.804+0000 7f7466fa5700 1 -- 172.21.15.111:0/776263812 shutdown_connections 2023-11-28T12:59:56.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.804+0000 7f7466fa5700 1 -- 172.21.15.111:0/776263812 >> 172.21.15.111:0/776263812 conn(0x7f7460098ee0 msgr2=0x7f746006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:56.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.804+0000 7f7466fa5700 1 -- 172.21.15.111:0/776263812 shutdown_connections 2023-11-28T12:59:56.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.804+0000 7f7466fa5700 1 -- 172.21.15.111:0/776263812 wait complete. 2023-11-28T12:59:56.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.805+0000 7f7466fa5700 1 Processor -- start 2023-11-28T12:59:56.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.806+0000 7f7466fa5700 1 -- start start 2023-11-28T12:59:56.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.806+0000 7f7466fa5700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7450003a30 0x7f7460157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.806+0000 7f7466fa5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74600c8eb0 0x7f74600c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.806+0000 7f7466fa5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74600b4570 0x7f74600b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.806+0000 7f7466fa5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7460159590 con 0x7f74600b4570 2023-11-28T12:59:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.806+0000 7f7466fa5700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7460157ff0 con 0x7f7450003a30 2023-11-28T12:59:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.806+0000 7f7466fa5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74601589e0 con 0x7f74600c8eb0 2023-11-28T12:59:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f7465542700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74600b4570 0x7f74600b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f7465542700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74600b4570 0x7f74600b4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34766/0 (socket says 172.21.15.111:34766) 2023-11-28T12:59:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f7465542700 1 -- 172.21.15.111:0/2664968726 learned_addr learned my addr 172.21.15.111:0/2664968726 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f7464d41700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7450003a30 0x7f7460157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f745ffff700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74600c8eb0 0x7f74600c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f7465542700 1 -- 172.21.15.111:0/2664968726 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74600c8eb0 msgr2=0x7f74600c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f7465542700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74600c8eb0 0x7f74600c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f7465542700 1 -- 172.21.15.111:0/2664968726 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7450003a30 msgr2=0x7f7460157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f7465542700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7450003a30 0x7f7460157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f7465542700 1 -- 172.21.15.111:0/2664968726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74600cd2d0 con 0x7f74600b4570 2023-11-28T12:59:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f7465542700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74600b4570 0x7f74600b4a10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f74600d0d10 tx=0x7f745800e4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f745d7fa700 1 -- 172.21.15.111:0/2664968726 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745800ed80 con 0x7f74600b4570 2023-11-28T12:59:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f745d7fa700 1 -- 172.21.15.111:0/2664968726 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7458013420 con 0x7f74600b4570 2023-11-28T12:59:56.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f7466fa5700 1 -- 172.21.15.111:0/2664968726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74600ccff0 con 0x7f74600b4570 2023-11-28T12:59:56.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f745d7fa700 1 -- 172.21.15.111:0/2664968726 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745800eee0 con 0x7f74600b4570 2023-11-28T12:59:56.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.807+0000 7f7466fa5700 1 -- 172.21.15.111:0/2664968726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746015ca90 con 0x7f74600b4570 2023-11-28T12:59:56.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.808+0000 7f7466fa5700 1 -- 172.21.15.111:0/2664968726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7460158490 con 0x7f74600b4570 2023-11-28T12:59:56.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.808+0000 7f745d7fa700 1 -- 172.21.15.111:0/2664968726 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f74580137d0 con 0x7f74600b4570 2023-11-28T12:59:56.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.809+0000 7f745d7fa700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74440682f0 0x7f744406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:56.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.809+0000 7f745d7fa700 1 -- 172.21.15.111:0/2664968726 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f7458086790 con 0x7f74600b4570 2023-11-28T12:59:56.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.809+0000 7f7464d41700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74440682f0 0x7f744406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:56.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.809+0000 7f7464d41700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74440682f0 0x7f744406a7a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7454005040 tx=0x7f7454009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:56.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.813+0000 7f745d7fa700 1 -- 172.21.15.111:0/2664968726 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7458052cc0 con 0x7f74600b4570 2023-11-28T12:59:56.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.997+0000 7f7466fa5700 1 -- 172.21.15.111:0/2664968726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f74601596d0 con 0x7f74600b4570 2023-11-28T12:59:56.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:56.998+0000 7f745d7fa700 1 -- 172.21.15.111:0/2664968726 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f74601596d0 con 0x7f74600b4570 2023-11-28T12:59:57.000 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T12:59:57.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.000+0000 7f7466fa5700 1 -- 172.21.15.111:0/2664968726 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74440682f0 msgr2=0x7f744406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.000+0000 7f7466fa5700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74440682f0 0x7f744406a7a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7454005040 tx=0x7f7454009030 comp rx=0 tx=0).stop 2023-11-28T12:59:57.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.000+0000 7f7466fa5700 1 -- 172.21.15.111:0/2664968726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74600b4570 msgr2=0x7f74600b4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.000+0000 7f7466fa5700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74600b4570 0x7f74600b4a10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f74600d0d10 tx=0x7f745800e4f0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.000+0000 7f7466fa5700 1 -- 172.21.15.111:0/2664968726 shutdown_connections 2023-11-28T12:59:57.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.000+0000 7f7466fa5700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74600b4570 0x7f74600b4a10 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.000+0000 7f7466fa5700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74440682f0 0x7f744406a7a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.000+0000 7f7466fa5700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7450003a30 0x7f7460157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.000+0000 7f7466fa5700 1 --2- 172.21.15.111:0/2664968726 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74600c8eb0 0x7f74600c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.000+0000 7f7466fa5700 1 -- 172.21.15.111:0/2664968726 >> 172.21.15.111:0/2664968726 conn(0x7f7460098ee0 msgr2=0x7f746006a630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:57.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.000+0000 7f7466fa5700 1 -- 172.21.15.111:0/2664968726 shutdown_connections 2023-11-28T12:59:57.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.000+0000 7f7466fa5700 1 -- 172.21.15.111:0/2664968726 wait complete. 2023-11-28T12:59:57.013 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T12:59:57.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.130+0000 7f82490e2700 1 Processor -- start 2023-11-28T12:59:57.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.130+0000 7f82490e2700 1 -- start start 2023-11-28T12:59:57.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.131+0000 7f82490e2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f824415c6c0 0x7f82440afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.131+0000 7f82490e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f82440b0440 0x7f82440b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.131+0000 7f82490e2700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82440cc750 con 0x7f82441602a0 2023-11-28T12:59:57.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.131+0000 7f82490e2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82440b0df0 con 0x7f82440b0440 2023-11-28T12:59:57.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.131+0000 7f82490e2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82440cca30 con 0x7f824415c6c0 2023-11-28T12:59:57.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.131+0000 7f824259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f82440b0440 0x7f82440b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.131+0000 7f824259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f82440b0440 0x7f82440b08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34770/0 (socket says 172.21.15.111:34770) 2023-11-28T12:59:57.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.131+0000 7f824259c700 1 -- 172.21.15.111:0/2712580869 learned_addr learned my addr 172.21.15.111:0/2712580869 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:57.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f824259c700 1 -- 172.21.15.111:0/2712580869 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f824415c6c0 msgr2=0x7f82440afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:57.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f824259c700 1 --2- 172.21.15.111:0/2712580869 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f824415c6c0 0x7f82440afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f824259c700 1 -- 172.21.15.111:0/2712580869 >> v1:172.21.15.149:6789/0 conn(0x7f82441602a0 legacy=0x7f8244160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:57.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f824259c700 1 -- 172.21.15.111:0/2712580869 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82440ccd10 con 0x7f82440b0440 2023-11-28T12:59:57.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f824259c700 1 --2- 172.21.15.111:0/2712580869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82440b0440 0x7f82440b08b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f82340098f0 tx=0x7f8234009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6abea7546c8d73 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:57.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f8241d9b700 1 -- 172.21.15.111:0/2712580869 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8234004230 con 0x7f82440b0440 2023-11-28T12:59:57.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f8241d9b700 1 -- 172.21.15.111:0/2712580869 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8234004390 con 0x7f82440b0440 2023-11-28T12:59:57.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f8241d9b700 1 -- 172.21.15.111:0/2712580869 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82340036a0 con 0x7f82440b0440 2023-11-28T12:59:57.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f82490e2700 1 -- 172.21.15.111:0/2712580869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82440b0440 msgr2=0x7f82440b08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f82490e2700 1 --2- 172.21.15.111:0/2712580869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82440b0440 0x7f82440b08b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f82340098f0 tx=0x7f8234009c00 comp rx=0 tx=0).stop 2023-11-28T12:59:57.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f82490e2700 1 -- 172.21.15.111:0/2712580869 shutdown_connections 2023-11-28T12:59:57.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f82490e2700 1 --2- 172.21.15.111:0/2712580869 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f824415c6c0 0x7f82440afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f82490e2700 1 --2- 172.21.15.111:0/2712580869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82440b0440 0x7f82440b08b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f82490e2700 1 -- 172.21.15.111:0/2712580869 >> 172.21.15.111:0/2712580869 conn(0x7f8244098ee0 msgr2=0x7f824406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:57.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f82490e2700 1 -- 172.21.15.111:0/2712580869 shutdown_connections 2023-11-28T12:59:57.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.132+0000 7f82490e2700 1 -- 172.21.15.111:0/2712580869 wait complete. 2023-11-28T12:59:57.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.134+0000 7f82490e2700 1 Processor -- start 2023-11-28T12:59:57.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.134+0000 7f82490e2700 1 -- start start 2023-11-28T12:59:57.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f82490e2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f82440b0440 0x7f8244154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f82490e2700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f824415c6c0 0x7f82440ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f82490e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82441602a0 0x7f82440ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f82490e2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824415bd30 con 0x7f82441602a0 2023-11-28T12:59:57.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f82490e2700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82440b59a0 con 0x7f824415c6c0 2023-11-28T12:59:57.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f82490e2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824415b0b0 con 0x7f82440b0440 2023-11-28T12:59:57.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f824359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82441602a0 0x7f82440ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f824359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82441602a0 0x7f82440ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34778/0 (socket says 172.21.15.111:34778) 2023-11-28T12:59:57.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f824359e700 1 -- 172.21.15.111:0/979221721 learned_addr learned my addr 172.21.15.111:0/979221721 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:57.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f824259c700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f824415c6c0 0x7f82440ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f8242d9d700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f82440b0440 0x7f8244154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f824359e700 1 -- 172.21.15.111:0/979221721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f82440b0440 msgr2=0x7f8244154cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f824359e700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f82440b0440 0x7f8244154cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f824359e700 1 -- 172.21.15.111:0/979221721 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f824415c6c0 msgr2=0x7f82440ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f824359e700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f824415c6c0 0x7f82440ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f824359e700 1 -- 172.21.15.111:0/979221721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8234009550 con 0x7f82441602a0 2023-11-28T12:59:57.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f824359e700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82441602a0 0x7f82440ca8c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f823c00cc10 tx=0x7f823c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:57.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f822b7fe700 1 -- 172.21.15.111:0/979221721 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c00abb0 con 0x7f82441602a0 2023-11-28T12:59:57.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f822b7fe700 1 -- 172.21.15.111:0/979221721 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823c00ad10 con 0x7f82441602a0 2023-11-28T12:59:57.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f82490e2700 1 -- 172.21.15.111:0/979221721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82440cd2d0 con 0x7f82441602a0 2023-11-28T12:59:57.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f822b7fe700 1 -- 172.21.15.111:0/979221721 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c012660 con 0x7f82441602a0 2023-11-28T12:59:57.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.135+0000 7f82490e2700 1 -- 172.21.15.111:0/979221721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82440ccff0 con 0x7f82441602a0 2023-11-28T12:59:57.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.136+0000 7f82490e2700 1 -- 172.21.15.111:0/979221721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824415ca90 con 0x7f82441602a0 2023-11-28T12:59:57.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.137+0000 7f822b7fe700 1 -- 172.21.15.111:0/979221721 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f823c022070 con 0x7f82441602a0 2023-11-28T12:59:57.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.137+0000 7f822b7fe700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f82300682f0 0x7f823006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.137+0000 7f822b7fe700 1 -- 172.21.15.111:0/979221721 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f823c086600 con 0x7f82441602a0 2023-11-28T12:59:57.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.137+0000 7f8242d9d700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f82300682f0 0x7f823006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.137+0000 7f8242d9d700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f82300682f0 0x7f823006a7a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8238006fd0 tx=0x7f8238008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:57.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.141+0000 7f822b7fe700 1 -- 172.21.15.111:0/979221721 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f823c052b30 con 0x7f82441602a0 2023-11-28T12:59:57.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.362+0000 7f82490e2700 1 -- 172.21.15.111:0/979221721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8244067170 con 0x7f82441602a0 2023-11-28T12:59:57.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.362+0000 7f822b7fe700 1 -- 172.21.15.111:0/979221721 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4656 (secure 0 0 0) 0x7f823c016090 con 0x7f82441602a0 2023-11-28T12:59:57.364 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:57.364 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2513950989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2513950989},{"type":"v1","addr":"172.21.15.149:6834","nonce":2513950989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1258891714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1258891714},{"type":"v1","addr":"172.21.15.149:6835","nonce":1258891714}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/3923909288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3923909288},{"type":"v1","addr":"172.21.15.111:6836","nonce":3923909288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T12:58:55.633507+0000","modified":"2023-11-28T12:59:17.702148+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/3180164366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":3180164366},{"type":"v1","addr":"172.21.15.111:6837","nonce":3180164366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T12:59:57.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.364+0000 7f82490e2700 1 -- 172.21.15.111:0/979221721 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f82300682f0 msgr2=0x7f823006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.364+0000 7f82490e2700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f82300682f0 0x7f823006a7a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8238006fd0 tx=0x7f8238008040 comp rx=0 tx=0).stop 2023-11-28T12:59:57.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.364+0000 7f82490e2700 1 -- 172.21.15.111:0/979221721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82441602a0 msgr2=0x7f82440ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.364+0000 7f82490e2700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82441602a0 0x7f82440ca8c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f823c00cc10 tx=0x7f823c00a060 comp rx=0 tx=0).stop 2023-11-28T12:59:57.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.364+0000 7f82490e2700 1 -- 172.21.15.111:0/979221721 shutdown_connections 2023-11-28T12:59:57.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.364+0000 7f82490e2700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82441602a0 0x7f82440ca8c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.364+0000 7f82490e2700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f82440b0440 0x7f8244154cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.364+0000 7f82490e2700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f82300682f0 0x7f823006a7a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.364+0000 7f82490e2700 1 --2- 172.21.15.111:0/979221721 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f824415c6c0 0x7f82440ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.364+0000 7f82490e2700 1 -- 172.21.15.111:0/979221721 >> 172.21.15.111:0/979221721 conn(0x7f8244098ee0 msgr2=0x7f82440af700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:57.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.364+0000 7f82490e2700 1 -- 172.21.15.111:0/979221721 shutdown_connections 2023-11-28T12:59:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.364+0000 7f82490e2700 1 -- 172.21.15.111:0/979221721 wait complete. 2023-11-28T12:59:57.367 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 9 2023-11-28T12:59:57.378 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:57.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.495+0000 7f16fc4c0700 1 Processor -- start 2023-11-28T12:59:57.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.495+0000 7f16fc4c0700 1 -- start start 2023-11-28T12:59:57.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.496+0000 7f16fc4c0700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f16f415c5e0 0x7f16f406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.496+0000 7f16fc4c0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16f406b720 0x7f16f406a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.496+0000 7f16fc4c0700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f415c9b0 con 0x7f16f406acf0 2023-11-28T12:59:57.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.496+0000 7f16fc4c0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f40cce90 con 0x7f16f406b720 2023-11-28T12:59:57.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.496+0000 7f16fc4c0700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f4160e60 con 0x7f16f415c5e0 2023-11-28T12:59:57.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.496+0000 7f16f9a5b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16f406b720 0x7f16f406a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.497+0000 7f16f9a5b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16f406b720 0x7f16f406a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34782/0 (socket says 172.21.15.111:34782) 2023-11-28T12:59:57.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.497+0000 7f16f9a5b700 1 -- 172.21.15.111:0/2179208220 learned_addr learned my addr 172.21.15.111:0/2179208220 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:57.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.497+0000 7f16f9a5b700 1 -- 172.21.15.111:0/2179208220 >> v2:172.21.15.149:3300/0 conn(0x7f16f415c5e0 msgr2=0x7f16f406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.497+0000 7f16f9a5b700 1 --2- 172.21.15.111:0/2179208220 >> v2:172.21.15.149:3300/0 conn(0x7f16f415c5e0 0x7f16f406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.497+0000 7f16f9a5b700 1 -- 172.21.15.111:0/2179208220 >> v1:172.21.15.149:6789/0 conn(0x7f16f406acf0 legacy=0x7f16f40712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.497+0000 7f16f9a5b700 1 -- 172.21.15.111:0/2179208220 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f40cd0a0 con 0x7f16f406b720 2023-11-28T12:59:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.497+0000 7f16f9a5b700 1 --2- 172.21.15.111:0/2179208220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f406b720 0x7f16f406a7b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f16f40cbd10 tx=0x7f16f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79dad5aa34c24c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.497+0000 7f16f925a700 1 -- 172.21.15.111:0/2179208220 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f0004230 con 0x7f16f406b720 2023-11-28T12:59:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.497+0000 7f16f925a700 1 -- 172.21.15.111:0/2179208220 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f0004390 con 0x7f16f406b720 2023-11-28T12:59:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.497+0000 7f16f925a700 1 -- 172.21.15.111:0/2179208220 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f00036a0 con 0x7f16f406b720 2023-11-28T12:59:57.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.498+0000 7f16fc4c0700 1 -- 172.21.15.111:0/2179208220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f406b720 msgr2=0x7f16f406a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.498+0000 7f16fc4c0700 1 --2- 172.21.15.111:0/2179208220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f406b720 0x7f16f406a7b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f16f40cbd10 tx=0x7f16f0009bd0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.498+0000 7f16fc4c0700 1 -- 172.21.15.111:0/2179208220 shutdown_connections 2023-11-28T12:59:57.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.498+0000 7f16fc4c0700 1 --2- 172.21.15.111:0/2179208220 >> v2:172.21.15.149:3300/0 conn(0x7f16f415c5e0 0x7f16f406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.498+0000 7f16fc4c0700 1 --2- 172.21.15.111:0/2179208220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f406b720 0x7f16f406a7b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.498+0000 7f16fc4c0700 1 -- 172.21.15.111:0/2179208220 >> 172.21.15.111:0/2179208220 conn(0x7f16f40b0db0 msgr2=0x7f16f40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:57.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.498+0000 7f16fc4c0700 1 -- 172.21.15.111:0/2179208220 shutdown_connections 2023-11-28T12:59:57.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.498+0000 7f16fc4c0700 1 -- 172.21.15.111:0/2179208220 wait complete. 2023-11-28T12:59:57.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.499+0000 7f16fc4c0700 1 Processor -- start 2023-11-28T12:59:57.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.499+0000 7f16fc4c0700 1 -- start start 2023-11-28T12:59:57.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fc4c0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f406acf0 0x7f16f4152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fc4c0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f16f406b720 0x7f16f41589e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fc4c0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f16f415c5e0 0x7f16f406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fc4c0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f4152c70 con 0x7f16f406acf0 2023-11-28T12:59:57.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fc4c0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f40c3880 con 0x7f16f415c5e0 2023-11-28T12:59:57.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fc4c0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f41593f0 con 0x7f16f406b720 2023-11-28T12:59:57.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fa25c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f406acf0 0x7f16f4152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fa25c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f406acf0 0x7f16f4152730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34788/0 (socket says 172.21.15.111:34788) 2023-11-28T12:59:57.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fa25c700 1 -- 172.21.15.111:0/3633443995 learned_addr learned my addr 172.21.15.111:0/3633443995 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:57.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16faa5d700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f16f415c5e0 0x7f16f406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16f9a5b700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f16f406b720 0x7f16f41589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fa25c700 1 -- 172.21.15.111:0/3633443995 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f16f406b720 msgr2=0x7f16f41589e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fa25c700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f16f406b720 0x7f16f41589e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fa25c700 1 -- 172.21.15.111:0/3633443995 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f16f415c5e0 msgr2=0x7f16f406ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fa25c700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f16f415c5e0 0x7f16f406ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fa25c700 1 -- 172.21.15.111:0/3633443995 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f0009550 con 0x7f16f406acf0 2023-11-28T12:59:57.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16faa5d700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f16f415c5e0 0x7f16f406ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:57.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fa25c700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f406acf0 0x7f16f4152730 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f16e400b560 tx=0x7f16e400b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:57.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16eaffd700 1 -- 172.21.15.111:0/3633443995 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4011830 con 0x7f16f406acf0 2023-11-28T12:59:57.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16eaffd700 1 -- 172.21.15.111:0/3633443995 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e4011990 con 0x7f16f406acf0 2023-11-28T12:59:57.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fc4c0700 1 -- 172.21.15.111:0/3633443995 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f40cd660 con 0x7f16f406acf0 2023-11-28T12:59:57.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16eaffd700 1 -- 172.21.15.111:0/3633443995 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4011d40 con 0x7f16f406acf0 2023-11-28T12:59:57.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.500+0000 7f16fc4c0700 1 -- 172.21.15.111:0/3633443995 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f40cd380 con 0x7f16f406acf0 2023-11-28T12:59:57.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.501+0000 7f16fc4c0700 1 -- 172.21.15.111:0/3633443995 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f4160af0 con 0x7f16f406acf0 2023-11-28T12:59:57.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.502+0000 7f16eaffd700 1 -- 172.21.15.111:0/3633443995 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f16e4021070 con 0x7f16f406acf0 2023-11-28T12:59:57.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.502+0000 7f16eaffd700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f16d80682f0 0x7f16d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.502+0000 7f16eaffd700 1 -- 172.21.15.111:0/3633443995 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f16e4085f00 con 0x7f16f406acf0 2023-11-28T12:59:57.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.502+0000 7f16f9a5b700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f16d80682f0 0x7f16d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.502+0000 7f16f9a5b700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f16d80682f0 0x7f16d806a7a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f16f40cbd10 tx=0x7f16f0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:57.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.506+0000 7f16eaffd700 1 -- 172.21.15.111:0/3633443995 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f16e4052430 con 0x7f16f406acf0 2023-11-28T12:59:57.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.693+0000 7f16fc4c0700 1 -- 172.21.15.111:0/3633443995 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16f4067170 con 0x7f16f406acf0 2023-11-28T12:59:57.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.694+0000 7f16eaffd700 1 -- 172.21.15.111:0/3633443995 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+19890 (secure 0 0 0) 0x7f16e4015020 con 0x7f16f406acf0 2023-11-28T12:59:57.696 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:57.696 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":28,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:41.880678+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:57.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.697+0000 7f16fc4c0700 1 -- 172.21.15.111:0/3633443995 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f16d80682f0 msgr2=0x7f16d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.697+0000 7f16fc4c0700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f16d80682f0 0x7f16d806a7a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f16f40cbd10 tx=0x7f16f0008030 comp rx=0 tx=0).stop 2023-11-28T12:59:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.697+0000 7f16fc4c0700 1 -- 172.21.15.111:0/3633443995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f406acf0 msgr2=0x7f16f4152730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.697+0000 7f16fc4c0700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f406acf0 0x7f16f4152730 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f16e400b560 tx=0x7f16e400b920 comp rx=0 tx=0).stop 2023-11-28T12:59:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.697+0000 7f16fc4c0700 1 -- 172.21.15.111:0/3633443995 shutdown_connections 2023-11-28T12:59:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.697+0000 7f16fc4c0700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f16d80682f0 0x7f16d806a7a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.697+0000 7f16fc4c0700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f16f415c5e0 0x7f16f406ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.697+0000 7f16fc4c0700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f406acf0 0x7f16f4152730 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.697+0000 7f16fc4c0700 1 --2- 172.21.15.111:0/3633443995 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f16f406b720 0x7f16f41589e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.697+0000 7f16fc4c0700 1 -- 172.21.15.111:0/3633443995 >> 172.21.15.111:0/3633443995 conn(0x7f16f40b0db0 msgr2=0x7f16f40af010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:57.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.697+0000 7f16fc4c0700 1 -- 172.21.15.111:0/3633443995 shutdown_connections 2023-11-28T12:59:57.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.697+0000 7f16fc4c0700 1 -- 172.21.15.111:0/3633443995 wait complete. 2023-11-28T12:59:57.711 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T12:59:57.711 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T12:59:57.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.830+0000 7f5226f57700 1 Processor -- start 2023-11-28T12:59:57.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.830+0000 7f5226f57700 1 -- start start 2023-11-28T12:59:57.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.830+0000 7f5226f57700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f5220159440 0x7f5220072460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.830+0000 7f5226f57700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f522006b0d0 0x7f52200cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.830+0000 7f5226f57700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52200712e0 0x7f5220071760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.830+0000 7f5226f57700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5220071ca0 con 0x7f522006b0d0 2023-11-28T12:59:57.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.830+0000 7f5226f57700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52200cb830 con 0x7f5220159440 2023-11-28T12:59:57.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.830+0000 7f5226f57700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522006a310 con 0x7f52200712e0 2023-11-28T12:59:57.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.830+0000 7f521ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f522006b0d0 0x7f52200cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.831+0000 7f521ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f522006b0d0 0x7f52200cafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34796/0 (socket says 172.21.15.111:34796) 2023-11-28T12:59:57.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.831+0000 7f521ffff700 1 -- 172.21.15.111:0/2549039716 learned_addr learned my addr 172.21.15.111:0/2549039716 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:57.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.831+0000 7f5224cf3700 1 --2- 172.21.15.111:0/2549039716 >> v2:172.21.15.149:3300/0 conn(0x7f5220159440 0x7f5220072460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.831+0000 7f52254f4700 1 --2- 172.21.15.111:0/2549039716 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52200712e0 0x7f5220071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.831+0000 7f521ffff700 1 -- 172.21.15.111:0/2549039716 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52200712e0 msgr2=0x7f5220071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:57.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.831+0000 7f521ffff700 1 --2- 172.21.15.111:0/2549039716 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52200712e0 0x7f5220071760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.831+0000 7f521ffff700 1 -- 172.21.15.111:0/2549039716 >> v2:172.21.15.149:3300/0 conn(0x7f5220159440 msgr2=0x7f5220072460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:57.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.831+0000 7f521ffff700 1 --2- 172.21.15.111:0/2549039716 >> v2:172.21.15.149:3300/0 conn(0x7f5220159440 0x7f5220072460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.831+0000 7f521ffff700 1 -- 172.21.15.111:0/2549039716 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522006a5f0 con 0x7f522006b0d0 2023-11-28T12:59:57.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.831+0000 7f521ffff700 1 --2- 172.21.15.111:0/2549039716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f522006b0d0 0x7f52200cafb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f52200cf220 tx=0x7f520c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a0bdfe1fb1eb0c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:57.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.831+0000 7f521f7fe700 1 -- 172.21.15.111:0/2549039716 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c00b870 con 0x7f522006b0d0 2023-11-28T12:59:57.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.831+0000 7f521f7fe700 1 -- 172.21.15.111:0/2549039716 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520c00b9d0 con 0x7f522006b0d0 2023-11-28T12:59:57.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.831+0000 7f521f7fe700 1 -- 172.21.15.111:0/2549039716 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c00bd60 con 0x7f522006b0d0 2023-11-28T12:59:57.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.832+0000 7f5226f57700 1 -- 172.21.15.111:0/2549039716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f522006b0d0 msgr2=0x7f52200cafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.832+0000 7f5226f57700 1 --2- 172.21.15.111:0/2549039716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f522006b0d0 0x7f52200cafb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f52200cf220 tx=0x7f520c009bd0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.832+0000 7f5226f57700 1 -- 172.21.15.111:0/2549039716 shutdown_connections 2023-11-28T12:59:57.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.832+0000 7f5226f57700 1 --2- 172.21.15.111:0/2549039716 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52200712e0 0x7f5220071760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.832+0000 7f5226f57700 1 --2- 172.21.15.111:0/2549039716 >> v2:172.21.15.149:3300/0 conn(0x7f5220159440 0x7f5220072460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.832+0000 7f5226f57700 1 --2- 172.21.15.111:0/2549039716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f522006b0d0 0x7f52200cafb0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.832+0000 7f5226f57700 1 -- 172.21.15.111:0/2549039716 >> 172.21.15.111:0/2549039716 conn(0x7f52200b0db0 msgr2=0x7f52200b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:57.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.832+0000 7f5226f57700 1 -- 172.21.15.111:0/2549039716 shutdown_connections 2023-11-28T12:59:57.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.832+0000 7f5226f57700 1 -- 172.21.15.111:0/2549039716 wait complete. 2023-11-28T12:59:57.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.833+0000 7f5226f57700 1 Processor -- start 2023-11-28T12:59:57.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.833+0000 7f5226f57700 1 -- start start 2023-11-28T12:59:57.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f5226f57700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f522006b0d0 0x7f522015c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f5226f57700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52200712e0 0x7f52200c4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f5226f57700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5220159440 0x7f52201e60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f5226f57700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522015bf60 con 0x7f52200712e0 2023-11-28T12:59:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f5226f57700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522015c930 con 0x7f522006b0d0 2023-11-28T12:59:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f5226f57700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5220159f00 con 0x7f5220159440 2023-11-28T12:59:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f521ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52200712e0 0x7f52200c4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f521ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52200712e0 0x7f52200c4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34804/0 (socket says 172.21.15.111:34804) 2023-11-28T12:59:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f521ffff700 1 -- 172.21.15.111:0/2829613549 learned_addr learned my addr 172.21.15.111:0/2829613549 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f5224cf3700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f522006b0d0 0x7f522015c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f52254f4700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5220159440 0x7f52201e60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f521ffff700 1 -- 172.21.15.111:0/2829613549 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5220159440 msgr2=0x7f52201e60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f521ffff700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5220159440 0x7f52201e60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f521ffff700 1 -- 172.21.15.111:0/2829613549 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f522006b0d0 msgr2=0x7f522015c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f521ffff700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f522006b0d0 0x7f522015c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f521ffff700 1 -- 172.21.15.111:0/2829613549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c009550 con 0x7f52200712e0 2023-11-28T12:59:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f521ffff700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52200712e0 0x7f52200c4780 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f52200b1ac0 tx=0x7f520c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f521d7fa700 1 -- 172.21.15.111:0/2829613549 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c004540 con 0x7f52200712e0 2023-11-28T12:59:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f521d7fa700 1 -- 172.21.15.111:0/2829613549 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520c0036a0 con 0x7f52200712e0 2023-11-28T12:59:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f5226f57700 1 -- 172.21.15.111:0/2829613549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522006abb0 con 0x7f52200712e0 2023-11-28T12:59:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.834+0000 7f521d7fa700 1 -- 172.21.15.111:0/2829613549 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c003a50 con 0x7f52200712e0 2023-11-28T12:59:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.835+0000 7f5226f57700 1 -- 172.21.15.111:0/2829613549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522006a8d0 con 0x7f52200712e0 2023-11-28T12:59:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.835+0000 7f5226f57700 1 -- 172.21.15.111:0/2829613549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f522006b540 con 0x7f52200712e0 2023-11-28T12:59:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.836+0000 7f521d7fa700 1 -- 172.21.15.111:0/2829613549 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f520c00f050 con 0x7f52200712e0 2023-11-28T12:59:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.836+0000 7f521d7fa700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52040682f0 0x7f520406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.836+0000 7f521d7fa700 1 -- 172.21.15.111:0/2829613549 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f520c087890 con 0x7f52200712e0 2023-11-28T12:59:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.836+0000 7f5224cf3700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52040682f0 0x7f520406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.837+0000 7f5224cf3700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52040682f0 0x7f520406a7a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f521400b930 tx=0x7f5214008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:57.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:57.840+0000 7f521d7fa700 1 -- 172.21.15.111:0/2829613549 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f520c053dc0 con 0x7f52200712e0 2023-11-28T12:59:58.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.064+0000 7f5226f57700 1 -- 172.21.15.111:0/2829613549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f522015c110 con 0x7f52200712e0 2023-11-28T12:59:58.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.065+0000 7f521d7fa700 1 -- 172.21.15.111:0/2829613549 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4656 (secure 0 0 0) 0x7f522015c110 con 0x7f52200712e0 2023-11-28T12:59:58.066 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:58.067 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2513950989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2513950989},{"type":"v1","addr":"172.21.15.149:6834","nonce":2513950989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1258891714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1258891714},{"type":"v1","addr":"172.21.15.149:6835","nonce":1258891714}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/3923909288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3923909288},{"type":"v1","addr":"172.21.15.111:6836","nonce":3923909288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T12:58:55.633507+0000","modified":"2023-11-28T12:59:17.702148+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/3180164366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":3180164366},{"type":"v1","addr":"172.21.15.111:6837","nonce":3180164366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T12:59:58.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.067+0000 7f5226f57700 1 -- 172.21.15.111:0/2829613549 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52040682f0 msgr2=0x7f520406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.067+0000 7f5226f57700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52040682f0 0x7f520406a7a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f521400b930 tx=0x7f5214008040 comp rx=0 tx=0).stop 2023-11-28T12:59:58.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.067+0000 7f5226f57700 1 -- 172.21.15.111:0/2829613549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52200712e0 msgr2=0x7f52200c4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.067+0000 7f5226f57700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52200712e0 0x7f52200c4780 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f52200b1ac0 tx=0x7f520c004750 comp rx=0 tx=0).stop 2023-11-28T12:59:58.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.067+0000 7f5226f57700 1 -- 172.21.15.111:0/2829613549 shutdown_connections 2023-11-28T12:59:58.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.067+0000 7f5226f57700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5220159440 0x7f52201e60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.067+0000 7f5226f57700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52040682f0 0x7f520406a7a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.067+0000 7f5226f57700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f522006b0d0 0x7f522015c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.067+0000 7f5226f57700 1 --2- 172.21.15.111:0/2829613549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52200712e0 0x7f52200c4780 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.067+0000 7f5226f57700 1 -- 172.21.15.111:0/2829613549 >> 172.21.15.111:0/2829613549 conn(0x7f52200b0db0 msgr2=0x7f522006e730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:58.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.067+0000 7f5226f57700 1 -- 172.21.15.111:0/2829613549 shutdown_connections 2023-11-28T12:59:58.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.067+0000 7f5226f57700 1 -- 172.21.15.111:0/2829613549 wait complete. 2023-11-28T12:59:58.070 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 9 2023-11-28T12:59:58.080 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T12:59:58.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.199+0000 7fbeca88a700 1 Processor -- start 2023-11-28T12:59:58.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.199+0000 7fbeca88a700 1 -- start start 2023-11-28T12:59:58.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.199+0000 7fbeca88a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec40b0d70 0x7fbec41604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.199+0000 7fbeca88a700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fbec40b15b0 0x7fbec40cc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.199+0000 7fbeca88a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec415cd90 con 0x7fbec40b1140 2023-11-28T12:59:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.199+0000 7fbeca88a700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec40cedd0 con 0x7fbec40b15b0 2023-11-28T12:59:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.199+0000 7fbeca88a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec41613e0 con 0x7fbec40b0d70 2023-11-28T12:59:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec37fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbec40b1140 0x7fbec40ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35588/0 (socket says 172.21.15.111:35588) 2023-11-28T12:59:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec37fe700 1 -- 172.21.15.111:0/4033242739 learned_addr learned my addr 172.21.15.111:0/4033242739 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec3fff700 1 --2- 172.21.15.111:0/4033242739 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec40b0d70 0x7fbec41604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec8e27700 1 --2- 172.21.15.111:0/4033242739 >> v2:172.21.15.149:3300/0 conn(0x7fbec40b15b0 0x7fbec40cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917655336 0 0) 0x7fbeac001110 con 0x7fbec40b1140 2023-11-28T12:59:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeb00035b0 con 0x7fbec40b1140 2023-11-28T12:59:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 286286312 0 0) 0x7fbeb00035b0 con 0x7fbec40b1140 2023-11-28T12:59:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec40b0d70 msgr2=0x7fbec41604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 --2- 172.21.15.111:0/4033242739 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec40b0d70 0x7fbec41604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 >> v2:172.21.15.149:3300/0 conn(0x7fbec40b15b0 msgr2=0x7fbec40cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 --2- 172.21.15.111:0/4033242739 >> v2:172.21.15.149:3300/0 conn(0x7fbec40b15b0 0x7fbec40cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec4160b40 con 0x7fbec40b1140 2023-11-28T12:59:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fbeac001800 con 0x7fbec40b1140 2023-11-28T12:59:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 >> v1:172.21.15.111:6789/0 conn(0x7fbec40b1140 legacy=0x7fbec40ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 --2- 172.21.15.111:0/4033242739 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbeb0003a00 0x7fbeb0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 --2- 172.21.15.111:0/4033242739 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec40b0d70 0x7fbeb0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 --2- 172.21.15.111:0/4033242739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb0004d90 0x7fbeb0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec40cedd0 con 0x7fbeb0004d90 2023-11-28T12:59:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec41613e0 con 0x7fbeb0003a00 2023-11-28T12:59:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb000ba10 con 0x7fbec40b0d70 2023-11-28T12:59:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec8e27700 1 --2- 172.21.15.111:0/4033242739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb0004d90 0x7fbeb0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec3fff700 1 --2- 172.21.15.111:0/4033242739 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbeb0003a00 0x7fbeb0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec37fe700 1 --2- 172.21.15.111:0/4033242739 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec40b0d70 0x7fbeb0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec8e27700 1 -- 172.21.15.111:0/4033242739 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec40b0d70 msgr2=0x7fbeb0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:58.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec8e27700 1 --2- 172.21.15.111:0/4033242739 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec40b0d70 0x7fbeb0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec8e27700 1 -- 172.21.15.111:0/4033242739 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbeb0003a00 msgr2=0x7fbeb0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:58.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec8e27700 1 --2- 172.21.15.111:0/4033242739 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbeb0003a00 0x7fbeb0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec8e27700 1 -- 172.21.15.111:0/4033242739 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb00035b0 con 0x7fbeb0004d90 2023-11-28T12:59:58.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.200+0000 7fbec37fe700 1 --2- 172.21.15.111:0/4033242739 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec40b0d70 0x7fbeb0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T12:59:58.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.201+0000 7fbec8e27700 1 --2- 172.21.15.111:0/4033242739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb0004d90 0x7fbeb0005200 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbec4071fd0 tx=0x7fbeb0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b01557ffd5f12d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:58.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.201+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc00eb70 con 0x7fbeb0004d90 2023-11-28T12:59:58.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.201+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbebc00f040 con 0x7fbeb0004d90 2023-11-28T12:59:58.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.201+0000 7fbec2ffd700 1 -- 172.21.15.111:0/4033242739 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc00eed0 con 0x7fbeb0004d90 2023-11-28T12:59:58.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.201+0000 7fbeca88a700 1 -- 172.21.15.111:0/4033242739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb0004d90 msgr2=0x7fbeb0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.201+0000 7fbeca88a700 1 --2- 172.21.15.111:0/4033242739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbeb0004d90 0x7fbeb0005200 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbec4071fd0 tx=0x7fbeb0005880 comp rx=0 tx=0).stop 2023-11-28T12:59:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.201+0000 7fbec8e27700 1 -- 172.21.15.111:0/4033242739 reap_dead start 2023-11-28T12:59:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.201+0000 7fbeca88a700 1 -- 172.21.15.111:0/4033242739 shutdown_connections 2023-11-28T12:59:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.201+0000 7fbeca88a700 1 -- 172.21.15.111:0/4033242739 >> 172.21.15.111:0/4033242739 conn(0x7fbec4098ee0 msgr2=0x7fbec406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.201+0000 7fbeca88a700 1 -- 172.21.15.111:0/4033242739 shutdown_connections 2023-11-28T12:59:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.201+0000 7fbeca88a700 1 -- 172.21.15.111:0/4033242739 wait complete. 2023-11-28T12:59:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.203+0000 7fbeca88a700 1 Processor -- start 2023-11-28T12:59:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.203+0000 7fbeca88a700 1 -- start start 2023-11-28T12:59:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.203+0000 7fbeca88a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec415b2a0 0x7fbec415b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.203+0000 7fbeca88a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbec40c5b00 0x7fbec40c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.203+0000 7fbeca88a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbec40c92e0 0x7fbec40b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.203+0000 7fbeca88a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec40c9760 con 0x7fbec40c5b00 2023-11-28T12:59:58.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.203+0000 7fbeca88a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec415c190 con 0x7fbec40c92e0 2023-11-28T12:59:58.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.203+0000 7fbeca88a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec40c5f70 con 0x7fbec415b2a0 2023-11-28T12:59:58.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbec40c5b00 0x7fbec40c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbec40c5b00 0x7fbec40c8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34820/0 (socket says 172.21.15.111:34820) 2023-11-28T12:59:58.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec37fe700 1 -- 172.21.15.111:0/2730351408 learned_addr learned my addr 172.21.15.111:0/2730351408 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:58.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec8e27700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbec40c92e0 0x7fbec40b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec3fff700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec415b2a0 0x7fbec415b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec37fe700 1 -- 172.21.15.111:0/2730351408 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec415b2a0 msgr2=0x7fbec415b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec37fe700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec415b2a0 0x7fbec415b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec37fe700 1 -- 172.21.15.111:0/2730351408 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbec40c92e0 msgr2=0x7fbec40b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec37fe700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbec40c92e0 0x7fbec40b2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec37fe700 1 -- 172.21.15.111:0/2730351408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec4161100 con 0x7fbec40c5b00 2023-11-28T12:59:58.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec8e27700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbec40c92e0 0x7fbec40b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:58.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec3fff700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec415b2a0 0x7fbec415b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T12:59:58.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec37fe700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbec40c5b00 0x7fbec40c8da0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbeac007360 tx=0x7fbeac007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:58.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec0ff9700 1 -- 172.21.15.111:0/2730351408 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac007840 con 0x7fbec40c5b00 2023-11-28T12:59:58.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec0ff9700 1 -- 172.21.15.111:0/2730351408 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeac00e860 con 0x7fbec40c5b00 2023-11-28T12:59:58.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbeca88a700 1 -- 172.21.15.111:0/2730351408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec4160e20 con 0x7fbec40c5b00 2023-11-28T12:59:58.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbec0ff9700 1 -- 172.21.15.111:0/2730351408 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac00ebd0 con 0x7fbec40c5b00 2023-11-28T12:59:58.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.204+0000 7fbeca88a700 1 -- 172.21.15.111:0/2730351408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec40af340 con 0x7fbec40c5b00 2023-11-28T12:59:58.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.205+0000 7fbeca88a700 1 -- 172.21.15.111:0/2730351408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec4067170 con 0x7fbec40c5b00 2023-11-28T12:59:58.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.206+0000 7fbec0ff9700 1 -- 172.21.15.111:0/2730351408 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbeac00ed30 con 0x7fbec40c5b00 2023-11-28T12:59:58.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.206+0000 7fbec0ff9700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbea40682f0 0x7fbea406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.206+0000 7fbec0ff9700 1 -- 172.21.15.111:0/2730351408 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fbeac085fd0 con 0x7fbec40c5b00 2023-11-28T12:59:58.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.206+0000 7fbec3fff700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbea40682f0 0x7fbea406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.207+0000 7fbec3fff700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbea40682f0 0x7fbea406a7a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009400 tx=0x7fbeb4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:58.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.210+0000 7fbec0ff9700 1 -- 172.21.15.111:0/2730351408 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbeac052500 con 0x7fbec40c5b00 2023-11-28T12:59:58.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.435+0000 7fbeca88a700 1 -- 172.21.15.111:0/2730351408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbec415cd90 con 0x7fbec40c5b00 2023-11-28T12:59:58.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.436+0000 7fbec0ff9700 1 -- 172.21.15.111:0/2730351408 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4656 (secure 0 0 0) 0x7fbec415cd90 con 0x7fbec40c5b00 2023-11-28T12:59:58.437 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:58.437 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2513950989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2513950989},{"type":"v1","addr":"172.21.15.149:6834","nonce":2513950989}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1258891714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1258891714},{"type":"v1","addr":"172.21.15.149:6835","nonce":1258891714}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/3923909288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3923909288},{"type":"v1","addr":"172.21.15.111:6836","nonce":3923909288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T12:58:55.633507+0000","modified":"2023-11-28T12:59:17.702148+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/3180164366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":3180164366},{"type":"v1","addr":"172.21.15.111:6837","nonce":3180164366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T12:59:58.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.438+0000 7fbeca88a700 1 -- 172.21.15.111:0/2730351408 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbea40682f0 msgr2=0x7fbea406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.438+0000 7fbeca88a700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbea40682f0 0x7fbea406a7a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009400 tx=0x7fbeb4008040 comp rx=0 tx=0).stop 2023-11-28T12:59:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.438+0000 7fbeca88a700 1 -- 172.21.15.111:0/2730351408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbec40c5b00 msgr2=0x7fbec40c8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.438+0000 7fbeca88a700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbec40c5b00 0x7fbec40c8da0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbeac007360 tx=0x7fbeac007720 comp rx=0 tx=0).stop 2023-11-28T12:59:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.438+0000 7fbeca88a700 1 -- 172.21.15.111:0/2730351408 shutdown_connections 2023-11-28T12:59:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.438+0000 7fbeca88a700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbea40682f0 0x7fbea406a7a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.438+0000 7fbeca88a700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbec40c92e0 0x7fbec40b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.438+0000 7fbeca88a700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbec415b2a0 0x7fbec415b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.438+0000 7fbeca88a700 1 --2- 172.21.15.111:0/2730351408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbec40c5b00 0x7fbec40c8da0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.438+0000 7fbeca88a700 1 -- 172.21.15.111:0/2730351408 >> 172.21.15.111:0/2730351408 conn(0x7fbec4098ee0 msgr2=0x7fbec406e950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.438+0000 7fbeca88a700 1 -- 172.21.15.111:0/2730351408 shutdown_connections 2023-11-28T12:59:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.438+0000 7fbeca88a700 1 -- 172.21.15.111:0/2730351408 wait complete. 2023-11-28T12:59:58.441 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 9 2023-11-28T12:59:58.451 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T12:59:58.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.569+0000 7fc5d9e98700 1 Processor -- start 2023-11-28T12:59:58.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.569+0000 7fc5d9e98700 1 -- start start 2023-11-28T12:59:58.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.570+0000 7fc5d9e98700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc5d40b0d70 0x7fc5d41604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.570+0000 7fc5d9e98700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc5d40cc550 0x7fc5d40cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.570+0000 7fc5d9e98700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d415cd90 con 0x7fc5d40b1140 2023-11-28T12:59:58.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.570+0000 7fc5d9e98700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d40cedd0 con 0x7fc5d40cc550 2023-11-28T12:59:58.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.570+0000 7fc5d9e98700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d40cced0 con 0x7fc5d40b0d70 2023-11-28T12:59:58.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.570+0000 7fc5d3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc5d40cc550 0x7fc5d40cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.570+0000 7fc5d3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc5d40cc550 0x7fc5d40cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34832/0 (socket says 172.21.15.111:34832) 2023-11-28T12:59:58.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.570+0000 7fc5d3fff700 1 -- 172.21.15.111:0/87410384 learned_addr learned my addr 172.21.15.111:0/87410384 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.570+0000 7fc5d37fe700 1 --2- 172.21.15.111:0/87410384 >> v2:172.21.15.149:3300/0 conn(0x7fc5d40b0d70 0x7fc5d41604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.570+0000 7fc5d3fff700 1 -- 172.21.15.111:0/87410384 >> v2:172.21.15.149:3300/0 conn(0x7fc5d40b0d70 msgr2=0x7fc5d41604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.570+0000 7fc5d3fff700 1 --2- 172.21.15.111:0/87410384 >> v2:172.21.15.149:3300/0 conn(0x7fc5d40b0d70 0x7fc5d41604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.570+0000 7fc5d3fff700 1 -- 172.21.15.111:0/87410384 >> v1:172.21.15.149:6789/0 conn(0x7fc5d40b1140 legacy=0x7fc5d40ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.570+0000 7fc5d3fff700 1 -- 172.21.15.111:0/87410384 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d41613e0 con 0x7fc5d40cc550 2023-11-28T12:59:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.571+0000 7fc5d3fff700 1 --2- 172.21.15.111:0/87410384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5d40cc550 0x7fc5d40cc990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4071960 tx=0x7fc5cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d3526d0043ed6b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.571+0000 7fc5d27fc700 1 -- 172.21.15.111:0/87410384 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc00e070 con 0x7fc5d40cc550 2023-11-28T12:59:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.571+0000 7fc5d27fc700 1 -- 172.21.15.111:0/87410384 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc00b910 con 0x7fc5d40cc550 2023-11-28T12:59:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.571+0000 7fc5d27fc700 1 -- 172.21.15.111:0/87410384 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc00bcc0 con 0x7fc5d40cc550 2023-11-28T12:59:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.571+0000 7fc5d9e98700 1 -- 172.21.15.111:0/87410384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5d40cc550 msgr2=0x7fc5d40cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.571+0000 7fc5d9e98700 1 --2- 172.21.15.111:0/87410384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5d40cc550 0x7fc5d40cc990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4071960 tx=0x7fc5cc00b330 comp rx=0 tx=0).stop 2023-11-28T12:59:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.571+0000 7fc5d9e98700 1 -- 172.21.15.111:0/87410384 shutdown_connections 2023-11-28T12:59:58.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.571+0000 7fc5d9e98700 1 --2- 172.21.15.111:0/87410384 >> v2:172.21.15.149:3300/0 conn(0x7fc5d40b0d70 0x7fc5d41604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.571+0000 7fc5d9e98700 1 --2- 172.21.15.111:0/87410384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5d40cc550 0x7fc5d40cc990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.571+0000 7fc5d9e98700 1 -- 172.21.15.111:0/87410384 >> 172.21.15.111:0/87410384 conn(0x7fc5d4098ee0 msgr2=0x7fc5d406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:58.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.571+0000 7fc5d9e98700 1 -- 172.21.15.111:0/87410384 shutdown_connections 2023-11-28T12:59:58.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.571+0000 7fc5d9e98700 1 -- 172.21.15.111:0/87410384 wait complete. 2023-11-28T12:59:58.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.573+0000 7fc5d9e98700 1 Processor -- start 2023-11-28T12:59:58.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.573+0000 7fc5d9e98700 1 -- start start 2023-11-28T12:59:58.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d9e98700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5d40b0d70 0x7fc5d414b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d9e98700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5d40b1140 0x7fc5d40c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d9e98700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc5d40cc550 0x7fc5d40cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d9e98700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d40c4c20 con 0x7fc5d40b1140 2023-11-28T12:59:58.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d9e98700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d40c4910 con 0x7fc5d40b0d70 2023-11-28T12:59:58.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d9e98700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d40c3fa0 con 0x7fc5d40cc550 2023-11-28T12:59:58.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5d40b1140 0x7fc5d40c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5d40b1140 0x7fc5d40c35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34836/0 (socket says 172.21.15.111:34836) 2023-11-28T12:59:58.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d2ffd700 1 -- 172.21.15.111:0/743191596 learned_addr learned my addr 172.21.15.111:0/743191596 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:58.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d37fe700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5d40b0d70 0x7fc5d414b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d3fff700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc5d40cc550 0x7fc5d40cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d2ffd700 1 -- 172.21.15.111:0/743191596 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc5d40cc550 msgr2=0x7fc5d40cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d2ffd700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc5d40cc550 0x7fc5d40cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d2ffd700 1 -- 172.21.15.111:0/743191596 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5d40b0d70 msgr2=0x7fc5d414b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d2ffd700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5d40b0d70 0x7fc5d414b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.574+0000 7fc5d2ffd700 1 -- 172.21.15.111:0/743191596 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc009b30 con 0x7fc5d40b1140 2023-11-28T12:59:58.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.575+0000 7fc5d2ffd700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5d40b1140 0x7fc5d40c35c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400c4e0 tx=0x7fc5c400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:58.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.575+0000 7fc5cbfff700 1 -- 172.21.15.111:0/743191596 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c40052d0 con 0x7fc5d40b1140 2023-11-28T12:59:58.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.575+0000 7fc5cbfff700 1 -- 172.21.15.111:0/743191596 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c4005430 con 0x7fc5d40b1140 2023-11-28T12:59:58.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.575+0000 7fc5d9e98700 1 -- 172.21.15.111:0/743191596 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d4160e20 con 0x7fc5d40b1140 2023-11-28T12:59:58.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.575+0000 7fc5cbfff700 1 -- 172.21.15.111:0/743191596 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c40038c0 con 0x7fc5d40b1140 2023-11-28T12:59:58.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.575+0000 7fc5d9e98700 1 -- 172.21.15.111:0/743191596 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d4160b40 con 0x7fc5d40b1140 2023-11-28T12:59:58.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.576+0000 7fc5d9e98700 1 -- 172.21.15.111:0/743191596 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d4161100 con 0x7fc5d40b1140 2023-11-28T12:59:58.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.576+0000 7fc5cbfff700 1 -- 172.21.15.111:0/743191596 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc5c4019030 con 0x7fc5d40b1140 2023-11-28T12:59:58.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.576+0000 7fc5cbfff700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc5b40682f0 0x7fc5b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.576+0000 7fc5cbfff700 1 -- 172.21.15.111:0/743191596 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fc5c4085030 con 0x7fc5d40b1140 2023-11-28T12:59:58.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.577+0000 7fc5d37fe700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc5b40682f0 0x7fc5b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.577+0000 7fc5d37fe700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc5b40682f0 0x7fc5b406a7a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc009bc0 tx=0x7fc5bc0091c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:58.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.581+0000 7fc5cbfff700 1 -- 172.21.15.111:0/743191596 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc5c4051560 con 0x7fc5d40b1140 2023-11-28T12:59:58.600 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T12:59:58.603 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T12:59:58.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.773+0000 7fc5d9e98700 1 -- 172.21.15.111:0/743191596 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5d40c4d60 con 0x7fc5d40b1140 2023-11-28T12:59:58.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.774+0000 7fc5cbfff700 1 -- 172.21.15.111:0/743191596 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+19890 (secure 0 0 0) 0x7fc5d40c4d60 con 0x7fc5d40b1140 2023-11-28T12:59:58.776 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T12:59:58.776 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":28,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T12:59:41.880678+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-11-28T12:58:53.275206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-11-28T12:58:54.588499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-11-28T12:58:57.666033+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T12:59:58.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.776+0000 7fc5d9e98700 1 -- 172.21.15.111:0/743191596 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc5b40682f0 msgr2=0x7fc5b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.776+0000 7fc5d9e98700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc5b40682f0 0x7fc5b406a7a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc009bc0 tx=0x7fc5bc0091c0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.776+0000 7fc5d9e98700 1 -- 172.21.15.111:0/743191596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5d40b1140 msgr2=0x7fc5d40c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.776+0000 7fc5d9e98700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5d40b1140 0x7fc5d40c35c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400c4e0 tx=0x7fc5c400c7f0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.777+0000 7fc5d9e98700 1 -- 172.21.15.111:0/743191596 shutdown_connections 2023-11-28T12:59:58.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.777+0000 7fc5d9e98700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc5d40cc550 0x7fc5d40cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.777+0000 7fc5d9e98700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc5b40682f0 0x7fc5b406a7a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.777+0000 7fc5d9e98700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc5d40b0d70 0x7fc5d414b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.777+0000 7fc5d9e98700 1 --2- 172.21.15.111:0/743191596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5d40b1140 0x7fc5d40c35c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.777+0000 7fc5d9e98700 1 -- 172.21.15.111:0/743191596 >> 172.21.15.111:0/743191596 conn(0x7fc5d4098ee0 msgr2=0x7fc5d4072cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:58.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.777+0000 7fc5d9e98700 1 -- 172.21.15.111:0/743191596 shutdown_connections 2023-11-28T12:59:58.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.777+0000 7fc5d9e98700 1 -- 172.21.15.111:0/743191596 wait complete. 2023-11-28T12:59:58.791 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T12:59:58.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.910+0000 7f3ecd19e700 1 Processor -- start 2023-11-28T12:59:58.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.910+0000 7f3ecd19e700 1 -- start start 2023-11-28T12:59:58.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.911+0000 7f3ecd19e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f3ec80b0d70 0x7f3ec81604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.911+0000 7f3ecd19e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3ec80cc550 0x7f3ec80cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.911+0000 7f3ecd19e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec815cd90 con 0x7f3ec80b1140 2023-11-28T12:59:58.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.911+0000 7f3ecd19e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec80cedd0 con 0x7f3ec80cc550 2023-11-28T12:59:58.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.911+0000 7f3ecd19e700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec80cced0 con 0x7f3ec80b0d70 2023-11-28T12:59:58.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.911+0000 7f3ec759e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3ec80cc550 0x7f3ec80cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.911+0000 7f3ec659c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3ec80b1140 0x7f3ec80ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35620/0 (socket says 172.21.15.111:35620) 2023-11-28T12:59:58.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.911+0000 7f3ec659c700 1 -- 172.21.15.111:0/495796944 learned_addr learned my addr 172.21.15.111:0/495796944 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T12:59:58.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ec6d9d700 1 --2- 172.21.15.111:0/495796944 >> v2:172.21.15.149:3300/0 conn(0x7f3ec80b0d70 0x7f3ec81604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ec759e700 1 -- 172.21.15.111:0/495796944 >> v2:172.21.15.149:3300/0 conn(0x7f3ec80b0d70 msgr2=0x7f3ec81604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T12:59:58.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ec759e700 1 --2- 172.21.15.111:0/495796944 >> v2:172.21.15.149:3300/0 conn(0x7f3ec80b0d70 0x7f3ec81604b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ec759e700 1 -- 172.21.15.111:0/495796944 >> v1:172.21.15.111:6789/0 conn(0x7f3ec80b1140 legacy=0x7f3ec80ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ec759e700 1 -- 172.21.15.111:0/495796944 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec81613e0 con 0x7f3ec80cc550 2023-11-28T12:59:58.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ec5d9b700 1 -- 172.21.15.111:0/495796944 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578526176 0 0) 0x7f3eb8001110 con 0x7f3ec80b1140 2023-11-28T12:59:58.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ec759e700 1 --2- 172.21.15.111:0/495796944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec80cc550 0x7f3ec80cc990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8071960 tx=0x7f3ec000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=724ad955d6db9257 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:58.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ec5d9b700 1 -- 172.21.15.111:0/495796944 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec000e070 con 0x7f3ec80cc550 2023-11-28T12:59:58.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ec5d9b700 1 -- 172.21.15.111:0/495796944 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec000bbd0 con 0x7f3ec80cc550 2023-11-28T12:59:58.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ec5d9b700 1 -- 172.21.15.111:0/495796944 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0012430 con 0x7f3ec80cc550 2023-11-28T12:59:58.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ecd19e700 1 -- 172.21.15.111:0/495796944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec80cc550 msgr2=0x7f3ec80cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ecd19e700 1 --2- 172.21.15.111:0/495796944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec80cc550 0x7f3ec80cc990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8071960 tx=0x7f3ec000b330 comp rx=0 tx=0).stop 2023-11-28T12:59:58.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ecd19e700 1 -- 172.21.15.111:0/495796944 shutdown_connections 2023-11-28T12:59:58.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ecd19e700 1 --2- 172.21.15.111:0/495796944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec80cc550 0x7f3ec80cc990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ecd19e700 1 --2- 172.21.15.111:0/495796944 >> v2:172.21.15.149:3300/0 conn(0x7f3ec80b0d70 0x7f3ec81604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.912+0000 7f3ecd19e700 1 -- 172.21.15.111:0/495796944 >> 172.21.15.111:0/495796944 conn(0x7f3ec8098ee0 msgr2=0x7f3ec806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:58.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.913+0000 7f3ecd19e700 1 -- 172.21.15.111:0/495796944 shutdown_connections 2023-11-28T12:59:58.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.913+0000 7f3ecd19e700 1 -- 172.21.15.111:0/495796944 wait complete. 2023-11-28T12:59:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.914+0000 7f3ecd19e700 1 Processor -- start 2023-11-28T12:59:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.914+0000 7f3ecd19e700 1 -- start start 2023-11-28T12:59:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ecd19e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec80b0d70 0x7f3ec8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ecd19e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ec80b1140 0x7f3ec80c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ecd19e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ec80cc550 0x7f3ec815e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ecd19e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80c50e0 con 0x7f3ec80b0d70 2023-11-28T12:59:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ecd19e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec815ecf0 con 0x7f3ec80b1140 2023-11-28T12:59:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ecd19e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec815efd0 con 0x7f3ec80cc550 2023-11-28T12:59:58.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ec6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec80b0d70 0x7f3ec8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ec6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec80b0d70 0x7f3ec8159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34846/0 (socket says 172.21.15.111:34846) 2023-11-28T12:59:58.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ec6d9d700 1 -- 172.21.15.111:0/1699870024 learned_addr learned my addr 172.21.15.111:0/1699870024 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T12:59:58.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ec659c700 1 --2- 172.21.15.111:0/1699870024 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ec80b1140 0x7f3ec80c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ec6d9d700 1 -- 172.21.15.111:0/1699870024 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ec80cc550 msgr2=0x7f3ec815e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ec6d9d700 1 --2- 172.21.15.111:0/1699870024 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ec80cc550 0x7f3ec815e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ec6d9d700 1 -- 172.21.15.111:0/1699870024 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ec80b1140 msgr2=0x7f3ec80c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ec6d9d700 1 --2- 172.21.15.111:0/1699870024 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ec80b1140 0x7f3ec80c4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ec6d9d700 1 -- 172.21.15.111:0/1699870024 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec0009a70 con 0x7f3ec80b0d70 2023-11-28T12:59:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ec6d9d700 1 --2- 172.21.15.111:0/1699870024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec80b0d70 0x7f3ec8159fb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc00c7f0 tx=0x7f3ebc00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3eaf7fe700 1 -- 172.21.15.111:0/1699870024 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc011800 con 0x7f3ec80b0d70 2023-11-28T12:59:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3eaf7fe700 1 -- 172.21.15.111:0/1699870024 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ebc011960 con 0x7f3ec80b0d70 2023-11-28T12:59:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3eaf7fe700 1 -- 172.21.15.111:0/1699870024 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc011d10 con 0x7f3ec80b0d70 2023-11-28T12:59:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.915+0000 7f3ecd19e700 1 -- 172.21.15.111:0/1699870024 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec8160e20 con 0x7f3ec80b0d70 2023-11-28T12:59:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.916+0000 7f3ecd19e700 1 -- 172.21.15.111:0/1699870024 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec8160b40 con 0x7f3ec80b0d70 2023-11-28T12:59:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.916+0000 7f3ecd19e700 1 -- 172.21.15.111:0/1699870024 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec8161100 con 0x7f3ec80b0d70 2023-11-28T12:59:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.917+0000 7f3eaf7fe700 1 -- 172.21.15.111:0/1699870024 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3ebc021070 con 0x7f3ec80b0d70 2023-11-28T12:59:58.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.917+0000 7f3eaf7fe700 1 --2- 172.21.15.111:0/1699870024 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3eb40682f0 0x7f3eb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T12:59:58.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.917+0000 7f3ec659c700 1 --2- 172.21.15.111:0/1699870024 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3eb40682f0 0x7f3eb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T12:59:58.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.917+0000 7f3eaf7fe700 1 -- 172.21.15.111:0/1699870024 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f3ebc085ec0 con 0x7f3ec80b0d70 2023-11-28T12:59:58.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.918+0000 7f3ec659c700 1 --2- 172.21.15.111:0/1699870024 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3eb40682f0 0x7f3eb406a7a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3ec81f55b0 tx=0x7f3eb8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T12:59:58.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:58.922+0000 7f3eaf7fe700 1 -- 172.21.15.111:0/1699870024 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3ebc052370 con 0x7f3ec80b0d70 2023-11-28T12:59:59.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.148+0000 7f3ecd19e700 1 -- 172.21.15.111:0/1699870024 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3ec8067170 con 0x7f3ec80b0d70 2023-11-28T12:59:59.957 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-11-28T12:59:59.956+0000 7fb379352700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T12:59:59.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.962+0000 7f3eaf7fe700 1 -- 172.21.15.111:0/1699870024 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3ebc055990 con 0x7f3ec80b0d70 2023-11-28T12:59:59.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.965+0000 7f3ecd19e700 1 -- 172.21.15.111:0/1699870024 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3eb40682f0 msgr2=0x7f3eb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:59.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.965+0000 7f3ecd19e700 1 --2- 172.21.15.111:0/1699870024 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3eb40682f0 0x7f3eb406a7a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3ec81f55b0 tx=0x7f3eb8009040 comp rx=0 tx=0).stop 2023-11-28T12:59:59.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.965+0000 7f3ecd19e700 1 -- 172.21.15.111:0/1699870024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec80b0d70 msgr2=0x7f3ec8159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T12:59:59.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.965+0000 7f3ecd19e700 1 --2- 172.21.15.111:0/1699870024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec80b0d70 0x7f3ec8159fb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc00c7f0 tx=0x7f3ebc00cbb0 comp rx=0 tx=0).stop 2023-11-28T12:59:59.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.965+0000 7f3ecd19e700 1 -- 172.21.15.111:0/1699870024 shutdown_connections 2023-11-28T12:59:59.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.965+0000 7f3ecd19e700 1 --2- 172.21.15.111:0/1699870024 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ec80cc550 0x7f3ec815e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:59.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.965+0000 7f3ecd19e700 1 --2- 172.21.15.111:0/1699870024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec80b0d70 0x7f3ec8159fb0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:59.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.965+0000 7f3ecd19e700 1 --2- 172.21.15.111:0/1699870024 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3eb40682f0 0x7f3eb406a7a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:59.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.965+0000 7f3ecd19e700 1 --2- 172.21.15.111:0/1699870024 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ec80b1140 0x7f3ec80c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T12:59:59.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.966+0000 7f3ecd19e700 1 -- 172.21.15.111:0/1699870024 >> 172.21.15.111:0/1699870024 conn(0x7f3ec8098ee0 msgr2=0x7f3ec806dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T12:59:59.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.966+0000 7f3ecd19e700 1 -- 172.21.15.111:0/1699870024 shutdown_connections 2023-11-28T12:59:59.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T12:59:59.966+0000 7f3ecd19e700 1 -- 172.21.15.111:0/1699870024 wait complete. 2023-11-28T12:59:59.968 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T12:59:59.969 INFO:tasks.ceph.mds.c.smithi111.stderr: -5436> 2023-11-28T12:59:00.319+0000 7f1d180c4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T12:59:59.986 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T13:00:00.000 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-11-28T12:59:59.998+0000 7fb379352700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2023-11-28T13:00:00.007 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-11-28T13:00:00.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.113+0000 7f725038b700 1 Processor -- start 2023-11-28T13:00:00.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.113+0000 7f725038b700 1 -- start start 2023-11-28T13:00:00.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f725038b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72481596e0 0x7f724815cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:00.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f725038b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724815de70 con 0x7f7248159310 2023-11-28T13:00:00.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f725038b700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72480ced10 con 0x7f724815d510 2023-11-28T13:00:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f725038b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724815d990 con 0x7f72481596e0 2023-11-28T13:00:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f724e127700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7248159310 0x7f72480ce1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35644/0 (socket says 172.21.15.111:35644) 2023-11-28T13:00:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f724e127700 1 -- 172.21.15.111:0/3930387219 learned_addr learned my addr 172.21.15.111:0/3930387219 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f724d926700 1 --2- 172.21.15.111:0/3930387219 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72481596e0 0x7f724815cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029851381 0 0) 0x7f724815de70 con 0x7f7248159310 2023-11-28T13:00:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72340035b0 con 0x7f7248159310 2023-11-28T13:00:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2090568720 0 0) 0x7f72340035b0 con 0x7f7248159310 2023-11-28T13:00:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72481596e0 msgr2=0x7f724815cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f724d125700 1 --2- 172.21.15.111:0/3930387219 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72481596e0 0x7f724815cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 >> v1:172.21.15.149:6789/0 conn(0x7f724815d510 legacy=0x7f72480cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72480ceff0 con 0x7f7248159310 2023-11-28T13:00:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.114+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f7244001670 con 0x7f7248159310 2023-11-28T13:00:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 >> v1:172.21.15.111:6789/0 conn(0x7f7248159310 legacy=0x7f72480ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724d125700 1 --2- 172.21.15.111:0/3930387219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7234003a30 0x7f7234003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724d125700 1 --2- 172.21.15.111:0/3930387219 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72481596e0 0x7f7234004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724d125700 1 --2- 172.21.15.111:0/3930387219 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7234004d90 0x7f7234005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724815d990 con 0x7f7234003a30 2023-11-28T13:00:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723400bac0 con 0x7f7234004d90 2023-11-28T13:00:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723400bc50 con 0x7f72481596e0 2023-11-28T13:00:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724e127700 1 --2- 172.21.15.111:0/3930387219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7234003a30 0x7f7234003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724d926700 1 --2- 172.21.15.111:0/3930387219 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72481596e0 0x7f7234004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724e928700 1 --2- 172.21.15.111:0/3930387219 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7234004d90 0x7f7234005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724e127700 1 -- 172.21.15.111:0/3930387219 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72481596e0 msgr2=0x7f7234004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724e127700 1 --2- 172.21.15.111:0/3930387219 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72481596e0 0x7f7234004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724e127700 1 -- 172.21.15.111:0/3930387219 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7234004d90 msgr2=0x7f7234005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724e127700 1 --2- 172.21.15.111:0/3930387219 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7234004d90 0x7f7234005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724e127700 1 -- 172.21.15.111:0/3930387219 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72340035b0 con 0x7f7234003a30 2023-11-28T13:00:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724e928700 1 --2- 172.21.15.111:0/3930387219 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7234004d90 0x7f7234005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724e127700 1 --2- 172.21.15.111:0/3930387219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7234003a30 0x7f7234003e40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f724400c2a0 tx=0x7f7234005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9c6ada4fe82571 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7244007c10 con 0x7f7234003a30 2023-11-28T13:00:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724400cdc0 con 0x7f7234003a30 2023-11-28T13:00:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.115+0000 7f724d125700 1 -- 172.21.15.111:0/3930387219 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7244005ba0 con 0x7f7234003a30 2023-11-28T13:00:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.116+0000 7f725038b700 1 -- 172.21.15.111:0/3930387219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7234003a30 msgr2=0x7f7234003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.116+0000 7f725038b700 1 --2- 172.21.15.111:0/3930387219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7234003a30 0x7f7234003e40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f724400c2a0 tx=0x7f7234005880 comp rx=0 tx=0).stop 2023-11-28T13:00:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.116+0000 7f724e928700 1 -- 172.21.15.111:0/3930387219 reap_dead start 2023-11-28T13:00:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.116+0000 7f725038b700 1 -- 172.21.15.111:0/3930387219 shutdown_connections 2023-11-28T13:00:00.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.116+0000 7f725038b700 1 -- 172.21.15.111:0/3930387219 >> 172.21.15.111:0/3930387219 conn(0x7f724806db10 msgr2=0x7f724806df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:00.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.116+0000 7f725038b700 1 -- 172.21.15.111:0/3930387219 shutdown_connections 2023-11-28T13:00:00.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.116+0000 7f725038b700 1 -- 172.21.15.111:0/3930387219 wait complete. 2023-11-28T13:00:00.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.117+0000 7f725038b700 1 Processor -- start 2023-11-28T13:00:00.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.118+0000 7f725038b700 1 -- start start 2023-11-28T13:00:00.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.118+0000 7f725038b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7234003a30 0x7f72480c35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:00.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.118+0000 7f725038b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72480c6520 0x7f72480c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:00.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.118+0000 7f725038b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72480ae900 0x7f72480aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:00.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.118+0000 7f725038b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72480af2e0 con 0x7f72480ae900 2023-11-28T13:00:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.118+0000 7f725038b700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7248146860 con 0x7f72480c6520 2023-11-28T13:00:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.118+0000 7f725038b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72480c5760 con 0x7f7234003a30 2023-11-28T13:00:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.118+0000 7f724e928700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72480ae900 0x7f72480aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.118+0000 7f724e928700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72480ae900 0x7f72480aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34878/0 (socket says 172.21.15.111:34878) 2023-11-28T13:00:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.118+0000 7f724e928700 1 -- 172.21.15.111:0/3595957869 learned_addr learned my addr 172.21.15.111:0/3595957869 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.118+0000 7f724d926700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72480c6520 0x7f72480c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.118+0000 7f724e127700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7234003a30 0x7f72480c35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.119+0000 7f724e928700 1 -- 172.21.15.111:0/3595957869 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7234003a30 msgr2=0x7f72480c35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.119+0000 7f724e928700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7234003a30 0x7f72480c35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.119+0000 7f724e928700 1 -- 172.21.15.111:0/3595957869 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72480c6520 msgr2=0x7f72480c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.119+0000 7f724e928700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72480c6520 0x7f72480c3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:00.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.119+0000 7f724e928700 1 -- 172.21.15.111:0/3595957869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72480cb0f0 con 0x7f72480ae900 2023-11-28T13:00:00.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.119+0000 7f724e127700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7234003a30 0x7f72480c35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:00.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.119+0000 7f724e928700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72480ae900 0x7f72480aeda0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f724815d8e0 tx=0x7f724000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:00.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.119+0000 7f723effd700 1 -- 172.21.15.111:0/3595957869 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240003c10 con 0x7f72480ae900 2023-11-28T13:00:00.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.119+0000 7f723effd700 1 -- 172.21.15.111:0/3595957869 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7240003d90 con 0x7f72480ae900 2023-11-28T13:00:00.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.119+0000 7f725038b700 1 -- 172.21.15.111:0/3595957869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72480cf2d0 con 0x7f72480ae900 2023-11-28T13:00:00.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.119+0000 7f723effd700 1 -- 172.21.15.111:0/3595957869 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240005310 con 0x7f72480ae900 2023-11-28T13:00:00.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.119+0000 7f725038b700 1 -- 172.21.15.111:0/3595957869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72480cb3d0 con 0x7f72480ae900 2023-11-28T13:00:00.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.120+0000 7f725038b700 1 -- 172.21.15.111:0/3595957869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f724806c4d0 con 0x7f72480ae900 2023-11-28T13:00:00.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.121+0000 7f723effd700 1 -- 172.21.15.111:0/3595957869 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7240005470 con 0x7f72480ae900 2023-11-28T13:00:00.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.121+0000 7f723effd700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f722c0682f0 0x7f722c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:00.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.121+0000 7f723effd700 1 -- 172.21.15.111:0/3595957869 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f7240086190 con 0x7f72480ae900 2023-11-28T13:00:00.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.121+0000 7f724e127700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f722c0682f0 0x7f722c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:00.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.121+0000 7f724e127700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f722c0682f0 0x7f722c06a7a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7244001930 tx=0x7f7244007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:00.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.125+0000 7f723effd700 1 -- 172.21.15.111:0/3595957869 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7240052670 con 0x7f72480ae900 2023-11-28T13:00:00.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.352+0000 7f725038b700 1 -- 172.21.15.111:0/3595957869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7248067170 con 0x7f72480ae900 2023-11-28T13:00:00.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.963+0000 7f723effd700 1 -- 172.21.15.111:0/3595957869 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7240055c90 con 0x7f72480ae900 2023-11-28T13:00:00.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.966+0000 7f725038b700 1 -- 172.21.15.111:0/3595957869 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f722c0682f0 msgr2=0x7f722c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:00.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.966+0000 7f725038b700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f722c0682f0 0x7f722c06a7a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7244001930 tx=0x7f7244007000 comp rx=0 tx=0).stop 2023-11-28T13:00:00.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.966+0000 7f725038b700 1 -- 172.21.15.111:0/3595957869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72480ae900 msgr2=0x7f72480aeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:00.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.966+0000 7f725038b700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72480ae900 0x7f72480aeda0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f724815d8e0 tx=0x7f724000ee70 comp rx=0 tx=0).stop 2023-11-28T13:00:00.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.966+0000 7f725038b700 1 -- 172.21.15.111:0/3595957869 shutdown_connections 2023-11-28T13:00:00.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.966+0000 7f725038b700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72480ae900 0x7f72480aeda0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:00.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.966+0000 7f725038b700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7234003a30 0x7f72480c35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:00.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.966+0000 7f725038b700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f722c0682f0 0x7f722c06a7a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:00.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.966+0000 7f725038b700 1 --2- 172.21.15.111:0/3595957869 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72480c6520 0x7f72480c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:00.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.966+0000 7f725038b700 1 -- 172.21.15.111:0/3595957869 >> 172.21.15.111:0/3595957869 conn(0x7f724806db10 msgr2=0x7f72480cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:00.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.966+0000 7f725038b700 1 -- 172.21.15.111:0/3595957869 shutdown_connections 2023-11-28T13:00:00.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:00.966+0000 7f725038b700 1 -- 172.21.15.111:0/3595957869 wait complete. 2023-11-28T13:00:00.982 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T13:00:01.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.150+0000 7f285aecb700 1 Processor -- start 2023-11-28T13:00:01.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.150+0000 7f285aecb700 1 -- start start 2023-11-28T13:00:01.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.151+0000 7f285aecb700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f28540b0d70 0x7f28541604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:01.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.151+0000 7f285aecb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28540b1140 0x7f28540ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:01.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.151+0000 7f285aecb700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285415cd90 con 0x7f28540cc550 2023-11-28T13:00:01.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.151+0000 7f285aecb700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28541613e0 con 0x7f28540b1140 2023-11-28T13:00:01.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.151+0000 7f285aecb700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2854160b40 con 0x7f28540b0d70 2023-11-28T13:00:01.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.151+0000 7f2853fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28540b1140 0x7f28540ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:01.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.151+0000 7f2853fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28540b1140 0x7f28540ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34886/0 (socket says 172.21.15.111:34886) 2023-11-28T13:00:01.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.151+0000 7f2853fff700 1 -- 172.21.15.111:0/438261272 learned_addr learned my addr 172.21.15.111:0/438261272 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:01.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.152+0000 7f2858c67700 1 --2- 172.21.15.111:0/438261272 >> v2:172.21.15.149:3300/0 conn(0x7f28540b0d70 0x7f28541604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:01.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.152+0000 7f2853fff700 1 -- 172.21.15.111:0/438261272 >> v2:172.21.15.149:3300/0 conn(0x7f28540b0d70 msgr2=0x7f28541604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:01.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.152+0000 7f2853fff700 1 --2- 172.21.15.111:0/438261272 >> v2:172.21.15.149:3300/0 conn(0x7f28540b0d70 0x7f28541604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:01.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.152+0000 7f2853fff700 1 -- 172.21.15.111:0/438261272 >> v1:172.21.15.149:6789/0 conn(0x7f28540cc550 legacy=0x7f28540af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:01.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.152+0000 7f2853fff700 1 -- 172.21.15.111:0/438261272 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2854160e20 con 0x7f28540b1140 2023-11-28T13:00:01.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.152+0000 7f2858c67700 1 --2- 172.21.15.111:0/438261272 >> v2:172.21.15.149:3300/0 conn(0x7f28540b0d70 0x7f28541604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:01.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.152+0000 7f2853fff700 1 --2- 172.21.15.111:0/438261272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b1140 0x7f28540ce6c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2854071960 tx=0x7f2840009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2feeee131f0f4d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:01.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.152+0000 7f28537fe700 1 -- 172.21.15.111:0/438261272 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840004230 con 0x7f28540b1140 2023-11-28T13:00:01.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.152+0000 7f28537fe700 1 -- 172.21.15.111:0/438261272 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2840004390 con 0x7f28540b1140 2023-11-28T13:00:01.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.152+0000 7f28537fe700 1 -- 172.21.15.111:0/438261272 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28400036a0 con 0x7f28540b1140 2023-11-28T13:00:01.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.153+0000 7f285aecb700 1 -- 172.21.15.111:0/438261272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b1140 msgr2=0x7f28540ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:01.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.153+0000 7f285aecb700 1 --2- 172.21.15.111:0/438261272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b1140 0x7f28540ce6c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2854071960 tx=0x7f2840009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:01.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.153+0000 7f285aecb700 1 -- 172.21.15.111:0/438261272 shutdown_connections 2023-11-28T13:00:01.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.153+0000 7f285aecb700 1 --2- 172.21.15.111:0/438261272 >> v2:172.21.15.149:3300/0 conn(0x7f28540b0d70 0x7f28541604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:01.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.153+0000 7f285aecb700 1 --2- 172.21.15.111:0/438261272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b1140 0x7f28540ce6c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:01.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.153+0000 7f285aecb700 1 -- 172.21.15.111:0/438261272 >> 172.21.15.111:0/438261272 conn(0x7f2854098ee0 msgr2=0x7f285406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:01.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.153+0000 7f285aecb700 1 -- 172.21.15.111:0/438261272 shutdown_connections 2023-11-28T13:00:01.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.153+0000 7f285aecb700 1 -- 172.21.15.111:0/438261272 wait complete. 2023-11-28T13:00:01.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.154+0000 7f285aecb700 1 Processor -- start 2023-11-28T13:00:01.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.154+0000 7f285aecb700 1 -- start start 2023-11-28T13:00:01.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f285aecb700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28540b0d70 0x7f285415be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:01.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f285aecb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b1140 0x7f285406f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:01.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f285aecb700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28540cc550 0x7f28540c57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:01.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f285aecb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28540b56e0 con 0x7f28540b1140 2023-11-28T13:00:01.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f285aecb700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2854160000 con 0x7f28540b0d70 2023-11-28T13:00:01.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f285aecb700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285415f480 con 0x7f28540cc550 2023-11-28T13:00:01.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f2853fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b1140 0x7f285406f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:01.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f2853fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b1140 0x7f285406f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34892/0 (socket says 172.21.15.111:34892) 2023-11-28T13:00:01.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f2853fff700 1 -- 172.21.15.111:0/2938507382 learned_addr learned my addr 172.21.15.111:0/2938507382 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:01.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f2858c67700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28540b0d70 0x7f285415be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:01.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f2859468700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28540cc550 0x7f28540c57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:01.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f2853fff700 1 -- 172.21.15.111:0/2938507382 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28540cc550 msgr2=0x7f28540c57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:01.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f2853fff700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28540cc550 0x7f28540c57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:01.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f2853fff700 1 -- 172.21.15.111:0/2938507382 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28540b0d70 msgr2=0x7f285415be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:01.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f2853fff700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28540b0d70 0x7f285415be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:01.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f2853fff700 1 -- 172.21.15.111:0/2938507382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2840009550 con 0x7f28540b1140 2023-11-28T13:00:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f2858c67700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28540b0d70 0x7f285415be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f2859468700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28540cc550 0x7f28540c57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f2853fff700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b1140 0x7f285406f030 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2840009fd0 tx=0x7f2840004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f28517fa700 1 -- 172.21.15.111:0/2938507382 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840003d70 con 0x7f28540b1140 2023-11-28T13:00:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f28517fa700 1 -- 172.21.15.111:0/2938507382 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284001f070 con 0x7f28540b1140 2023-11-28T13:00:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f28517fa700 1 -- 172.21.15.111:0/2938507382 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2840011870 con 0x7f28540b1140 2023-11-28T13:00:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f285aecb700 1 -- 172.21.15.111:0/2938507382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28540af340 con 0x7f28540b1140 2023-11-28T13:00:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.155+0000 7f285aecb700 1 -- 172.21.15.111:0/2938507382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2854161100 con 0x7f28540b1140 2023-11-28T13:00:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.156+0000 7f285aecb700 1 -- 172.21.15.111:0/2938507382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28540cec00 con 0x7f28540b1140 2023-11-28T13:00:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.157+0000 7f28517fa700 1 -- 172.21.15.111:0/2938507382 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2840017020 con 0x7f28540b1140 2023-11-28T13:00:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.157+0000 7f28517fa700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28380682f0 0x7f283806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:01.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.157+0000 7f28517fa700 1 -- 172.21.15.111:0/2938507382 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f2840087660 con 0x7f28540b1140 2023-11-28T13:00:01.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.157+0000 7f2858c67700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28380682f0 0x7f283806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:01.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.158+0000 7f2858c67700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28380682f0 0x7f283806a7a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f284800aaf0 tx=0x7f2848008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:01.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.161+0000 7f28517fa700 1 -- 172.21.15.111:0/2938507382 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2840053ae0 con 0x7f28540b1140 2023-11-28T13:00:01.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:01.344+0000 7f285aecb700 1 -- 172.21.15.111:0/2938507382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f285415cd90 con 0x7f28540b1140 2023-11-28T13:00:02.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.034+0000 7f28517fa700 1 -- 172.21.15.111:0/2938507382 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v30) v1 ==== 181+0+0 (secure 0 0 0) 0x7f285415cd90 con 0x7f28540b1140 2023-11-28T13:00:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.036+0000 7f285aecb700 1 -- 172.21.15.111:0/2938507382 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28380682f0 msgr2=0x7f283806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.036+0000 7f285aecb700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28380682f0 0x7f283806a7a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f284800aaf0 tx=0x7f2848008040 comp rx=0 tx=0).stop 2023-11-28T13:00:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.036+0000 7f285aecb700 1 -- 172.21.15.111:0/2938507382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b1140 msgr2=0x7f285406f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.036+0000 7f285aecb700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b1140 0x7f285406f030 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2840009fd0 tx=0x7f2840004750 comp rx=0 tx=0).stop 2023-11-28T13:00:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.037+0000 7f285aecb700 1 -- 172.21.15.111:0/2938507382 shutdown_connections 2023-11-28T13:00:02.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.037+0000 7f285aecb700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f28540cc550 0x7f28540c57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.037+0000 7f285aecb700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28380682f0 0x7f283806a7a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.037+0000 7f285aecb700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28540b0d70 0x7f285415be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.037+0000 7f285aecb700 1 --2- 172.21.15.111:0/2938507382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28540b1140 0x7f285406f030 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.037+0000 7f285aecb700 1 -- 172.21.15.111:0/2938507382 >> 172.21.15.111:0/2938507382 conn(0x7f2854098ee0 msgr2=0x7f28540cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.037+0000 7f285aecb700 1 -- 172.21.15.111:0/2938507382 shutdown_connections 2023-11-28T13:00:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.037+0000 7f285aecb700 1 -- 172.21.15.111:0/2938507382 wait complete. 2023-11-28T13:00:02.039 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-11-28T13:00:02.052 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T13:00:02.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.180+0000 7fe88c7e7700 1 Processor -- start 2023-11-28T13:00:02.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.180+0000 7fe88c7e7700 1 -- start start 2023-11-28T13:00:02.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe88c7e7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe884142e80 0x7fe88406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:02.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe88c7e7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8840ca600 0x7fe8840cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:02.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe88c7e7700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88406ac00 con 0x7fe88406a7b0 2023-11-28T13:00:02.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe88c7e7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8840c9380 con 0x7fe8840ca600 2023-11-28T13:00:02.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe88c7e7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe884159b10 con 0x7fe884142e80 2023-11-28T13:00:02.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe88ad84700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8840ca600 0x7fe8840cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:02.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe88ad84700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8840ca600 0x7fe8840cd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34904/0 (socket says 172.21.15.111:34904) 2023-11-28T13:00:02.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe88ad84700 1 -- 172.21.15.111:0/1571887159 learned_addr learned my addr 172.21.15.111:0/1571887159 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:02.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe88ad84700 1 -- 172.21.15.111:0/1571887159 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe884142e80 msgr2=0x7fe88406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:02.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe88ad84700 1 --2- 172.21.15.111:0/1571887159 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe884142e80 0x7fe88406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:02.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe88ad84700 1 -- 172.21.15.111:0/1571887159 >> v1:172.21.15.111:6789/0 conn(0x7fe88406a7b0 legacy=0x7fe8840c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:02.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe88ad84700 1 -- 172.21.15.111:0/1571887159 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe884143250 con 0x7fe8840ca600 2023-11-28T13:00:02.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe889581700 1 -- 172.21.15.111:0/1571887159 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847924681 0 0) 0x7fe874001110 con 0x7fe88406a7b0 2023-11-28T13:00:02.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.181+0000 7fe88ad84700 1 --2- 172.21.15.111:0/1571887159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840ca600 0x7fe8840cd0d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009e50 tx=0x7fe87c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d64504c34237d7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:02.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.182+0000 7fe889581700 1 -- 172.21.15.111:0/1571887159 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c00e070 con 0x7fe8840ca600 2023-11-28T13:00:02.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.182+0000 7fe889581700 1 -- 172.21.15.111:0/1571887159 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c00b9c0 con 0x7fe8840ca600 2023-11-28T13:00:02.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.182+0000 7fe889581700 1 -- 172.21.15.111:0/1571887159 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c00bd70 con 0x7fe8840ca600 2023-11-28T13:00:02.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.182+0000 7fe88c7e7700 1 -- 172.21.15.111:0/1571887159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840ca600 msgr2=0x7fe8840cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:02.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.182+0000 7fe88c7e7700 1 --2- 172.21.15.111:0/1571887159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840ca600 0x7fe8840cd0d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009e50 tx=0x7fe87c00b330 comp rx=0 tx=0).stop 2023-11-28T13:00:02.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.182+0000 7fe88c7e7700 1 -- 172.21.15.111:0/1571887159 shutdown_connections 2023-11-28T13:00:02.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.182+0000 7fe88c7e7700 1 --2- 172.21.15.111:0/1571887159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840ca600 0x7fe8840cd0d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:02.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.182+0000 7fe88c7e7700 1 --2- 172.21.15.111:0/1571887159 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe884142e80 0x7fe88406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:02.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.182+0000 7fe88c7e7700 1 -- 172.21.15.111:0/1571887159 >> 172.21.15.111:0/1571887159 conn(0x7fe8840cb490 msgr2=0x7fe8840cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:02.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.182+0000 7fe88c7e7700 1 -- 172.21.15.111:0/1571887159 shutdown_connections 2023-11-28T13:00:02.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.182+0000 7fe88c7e7700 1 -- 172.21.15.111:0/1571887159 wait complete. 2023-11-28T13:00:02.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.183+0000 7fe88c7e7700 1 Processor -- start 2023-11-28T13:00:02.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.183+0000 7fe88c7e7700 1 -- start start 2023-11-28T13:00:02.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.184+0000 7fe88c7e7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88406a7b0 0x7fe884157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:02.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.184+0000 7fe88c7e7700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8840ca600 0x7fe8840c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:02.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.184+0000 7fe88c7e7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe884142e80 0x7fe8840cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:02.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.184+0000 7fe88c7e7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840c63a0 con 0x7fe88406a7b0 2023-11-28T13:00:02.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.184+0000 7fe88c7e7700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840c6680 con 0x7fe8840ca600 2023-11-28T13:00:02.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.184+0000 7fe88c7e7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840c6960 con 0x7fe884142e80 2023-11-28T13:00:02.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.184+0000 7fe88a583700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88406a7b0 0x7fe884157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:02.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.184+0000 7fe88a583700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88406a7b0 0x7fe884157b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34906/0 (socket says 172.21.15.111:34906) 2023-11-28T13:00:02.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.184+0000 7fe88a583700 1 -- 172.21.15.111:0/3547548324 learned_addr learned my addr 172.21.15.111:0/3547548324 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:02.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.184+0000 7fe88ad84700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe884142e80 0x7fe8840cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:02.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.184+0000 7fe889d82700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8840ca600 0x7fe8840c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:02.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.185+0000 7fe88a583700 1 -- 172.21.15.111:0/3547548324 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe884142e80 msgr2=0x7fe8840cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:02.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.185+0000 7fe88a583700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe884142e80 0x7fe8840cd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:02.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.185+0000 7fe88a583700 1 -- 172.21.15.111:0/3547548324 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8840ca600 msgr2=0x7fe8840c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:02.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.185+0000 7fe88a583700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8840ca600 0x7fe8840c5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:02.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.185+0000 7fe88a583700 1 -- 172.21.15.111:0/3547548324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c009b30 con 0x7fe88406a7b0 2023-11-28T13:00:02.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.185+0000 7fe88a583700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88406a7b0 0x7fe884157b60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe88000b4a0 tx=0x7fe88000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:02.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.185+0000 7fe87affd700 1 -- 172.21.15.111:0/3547548324 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880011830 con 0x7fe88406a7b0 2023-11-28T13:00:02.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.185+0000 7fe87affd700 1 -- 172.21.15.111:0/3547548324 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe880011990 con 0x7fe88406a7b0 2023-11-28T13:00:02.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.185+0000 7fe88c7e7700 1 -- 172.21.15.111:0/3547548324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8840cc5f0 con 0x7fe88406a7b0 2023-11-28T13:00:02.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.185+0000 7fe87affd700 1 -- 172.21.15.111:0/3547548324 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880011d40 con 0x7fe88406a7b0 2023-11-28T13:00:02.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.185+0000 7fe88c7e7700 1 -- 172.21.15.111:0/3547548324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8840cab00 con 0x7fe88406a7b0 2023-11-28T13:00:02.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.186+0000 7fe88c7e7700 1 -- 172.21.15.111:0/3547548324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8840cc8d0 con 0x7fe88406a7b0 2023-11-28T13:00:02.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.186+0000 7fe87affd700 1 -- 172.21.15.111:0/3547548324 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe880021070 con 0x7fe88406a7b0 2023-11-28T13:00:02.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.187+0000 7fe87affd700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8680682f0 0x7fe86806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:02.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.187+0000 7fe87affd700 1 -- 172.21.15.111:0/3547548324 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6519+0+0 (secure 0 0 0) 0x7fe880085eb0 con 0x7fe88406a7b0 2023-11-28T13:00:02.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.187+0000 7fe889d82700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8680682f0 0x7fe86806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:02.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.187+0000 7fe889d82700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8680682f0 0x7fe86806a7a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe874007fd0 tx=0x7fe874009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:02.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.191+0000 7fe87affd700 1 -- 172.21.15.111:0/3547548324 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8800523d0 con 0x7fe88406a7b0 2023-11-28T13:00:02.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:02.381+0000 7fe88c7e7700 1 -- 172.21.15.111:0/3547548324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe88406bab0 con 0x7fe88406a7b0 2023-11-28T13:00:03.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.034+0000 7fe87affd700 1 -- 172.21.15.111:0/3547548324 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v31) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe880015020 con 0x7fe88406a7b0 2023-11-28T13:00:03.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.038+0000 7fe88c7e7700 1 -- 172.21.15.111:0/3547548324 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8680682f0 msgr2=0x7fe86806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:03.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.038+0000 7fe88c7e7700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8680682f0 0x7fe86806a7a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe874007fd0 tx=0x7fe874009040 comp rx=0 tx=0).stop 2023-11-28T13:00:03.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.038+0000 7fe88c7e7700 1 -- 172.21.15.111:0/3547548324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88406a7b0 msgr2=0x7fe884157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:03.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.038+0000 7fe88c7e7700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88406a7b0 0x7fe884157b60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe88000b4a0 tx=0x7fe88000b860 comp rx=0 tx=0).stop 2023-11-28T13:00:03.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.038+0000 7fe88c7e7700 1 -- 172.21.15.111:0/3547548324 shutdown_connections 2023-11-28T13:00:03.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.038+0000 7fe88c7e7700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe884142e80 0x7fe8840cd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:03.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.038+0000 7fe88c7e7700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe88406a7b0 0x7fe884157b60 secure :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe88000b4a0 tx=0x7fe88000b860 comp rx=0 tx=0).stop 2023-11-28T13:00:03.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.038+0000 7fe88c7e7700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8680682f0 0x7fe86806a7a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:03.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.038+0000 7fe88c7e7700 1 --2- 172.21.15.111:0/3547548324 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8840ca600 0x7fe8840c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:03.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.038+0000 7fe88c7e7700 1 -- 172.21.15.111:0/3547548324 >> 172.21.15.111:0/3547548324 conn(0x7fe8840cb490 msgr2=0x7fe8840b3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:03.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.038+0000 7fe88c7e7700 1 -- 172.21.15.111:0/3547548324 shutdown_connections 2023-11-28T13:00:03.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.038+0000 7fe88c7e7700 1 -- 172.21.15.111:0/3547548324 wait complete. 2023-11-28T13:00:03.044 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-11-28T13:00:03.055 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-11-28T13:00:03.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.181+0000 7f4315fcc700 1 Processor -- start 2023-11-28T13:00:03.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.181+0000 7f4315fcc700 1 -- start start 2023-11-28T13:00:03.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f4315fcc700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f431015a5d0 0x7f43100ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f4315fcc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43100cb2d0 0x7f43100ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f4315fcc700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43100ae8b0 con 0x7f43100b0160 2023-11-28T13:00:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f4315fcc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43100cc1b0 con 0x7f43100cb2d0 2023-11-28T13:00:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f4315fcc700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43100cc390 con 0x7f431015a5d0 2023-11-28T13:00:03.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f430ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43100cb2d0 0x7f43100ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:03.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f430ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43100cb2d0 0x7f43100ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34916/0 (socket says 172.21.15.111:34916) 2023-11-28T13:00:03.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f430ffff700 1 -- 172.21.15.111:0/1510727664 learned_addr learned my addr 172.21.15.111:0/1510727664 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:03.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f430f7fe700 1 --2- 172.21.15.111:0/1510727664 >> v2:172.21.15.149:3300/0 conn(0x7f431015a5d0 0x7f43100ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:03.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f430ffff700 1 -- 172.21.15.111:0/1510727664 >> v2:172.21.15.149:3300/0 conn(0x7f431015a5d0 msgr2=0x7f43100ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:03.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f430ffff700 1 --2- 172.21.15.111:0/1510727664 >> v2:172.21.15.149:3300/0 conn(0x7f431015a5d0 0x7f43100ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:03.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f430ffff700 1 -- 172.21.15.111:0/1510727664 >> v1:172.21.15.149:6789/0 conn(0x7f43100b0160 legacy=0x7f431015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:03.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f430ffff700 1 -- 172.21.15.111:0/1510727664 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43100cc670 con 0x7f43100cb2d0 2023-11-28T13:00:03.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f430f7fe700 1 --2- 172.21.15.111:0/1510727664 >> v2:172.21.15.149:3300/0 conn(0x7f431015a5d0 0x7f43100ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:03.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.182+0000 7f430ffff700 1 --2- 172.21.15.111:0/1510727664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100cb2d0 0x7f43100ced90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4310067d30 tx=0x7f430800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcdbcd6011a48e4f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:03.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.183+0000 7f430e7fc700 1 -- 172.21.15.111:0/1510727664 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430800e070 con 0x7f43100cb2d0 2023-11-28T13:00:03.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.183+0000 7f430e7fc700 1 -- 172.21.15.111:0/1510727664 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4308003a20 con 0x7f43100cb2d0 2023-11-28T13:00:03.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.183+0000 7f430e7fc700 1 -- 172.21.15.111:0/1510727664 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308003dd0 con 0x7f43100cb2d0 2023-11-28T13:00:03.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.183+0000 7f4315fcc700 1 -- 172.21.15.111:0/1510727664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100cb2d0 msgr2=0x7f43100ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:03.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.183+0000 7f4315fcc700 1 --2- 172.21.15.111:0/1510727664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100cb2d0 0x7f43100ced90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4310067d30 tx=0x7f430800b330 comp rx=0 tx=0).stop 2023-11-28T13:00:03.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.183+0000 7f4315fcc700 1 -- 172.21.15.111:0/1510727664 shutdown_connections 2023-11-28T13:00:03.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.183+0000 7f4315fcc700 1 --2- 172.21.15.111:0/1510727664 >> v2:172.21.15.149:3300/0 conn(0x7f431015a5d0 0x7f43100ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:03.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.183+0000 7f4315fcc700 1 --2- 172.21.15.111:0/1510727664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100cb2d0 0x7f43100ced90 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:03.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.183+0000 7f4315fcc700 1 -- 172.21.15.111:0/1510727664 >> 172.21.15.111:0/1510727664 conn(0x7f431009a040 msgr2=0x7f43100d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:03.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.183+0000 7f4315fcc700 1 -- 172.21.15.111:0/1510727664 shutdown_connections 2023-11-28T13:00:03.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.183+0000 7f4315fcc700 1 -- 172.21.15.111:0/1510727664 wait complete. 2023-11-28T13:00:03.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.184+0000 7f4315fcc700 1 Processor -- start 2023-11-28T13:00:03.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.184+0000 7f4315fcc700 1 -- start start 2023-11-28T13:00:03.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.185+0000 7f4315fcc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100b0160 0x7f43100c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:03.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.185+0000 7f4315fcc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f43100cb2d0 0x7f431014bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:03.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.185+0000 7f4315fcc700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f431015a5d0 0x7f43100c9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:03.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.185+0000 7f4315fcc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4310154800 con 0x7f43100b0160 2023-11-28T13:00:03.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.185+0000 7f4315fcc700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43100c9670 con 0x7f431015a5d0 2023-11-28T13:00:03.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.185+0000 7f4315fcc700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4310154a50 con 0x7f43100cb2d0 2023-11-28T13:00:03.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.185+0000 7f430f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100b0160 0x7f43100c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:03.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.185+0000 7f430f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100b0160 0x7f43100c9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34918/0 (socket says 172.21.15.111:34918) 2023-11-28T13:00:03.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.185+0000 7f430f7fe700 1 -- 172.21.15.111:0/2230921839 learned_addr learned my addr 172.21.15.111:0/2230921839 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:03.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.185+0000 7f430effd700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f43100cb2d0 0x7f431014bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:03.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.185+0000 7f430ffff700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f431015a5d0 0x7f43100c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:03.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.186+0000 7f430f7fe700 1 -- 172.21.15.111:0/2230921839 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f43100cb2d0 msgr2=0x7f431014bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:03.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.186+0000 7f430f7fe700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f43100cb2d0 0x7f431014bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:03.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.186+0000 7f430f7fe700 1 -- 172.21.15.111:0/2230921839 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f431015a5d0 msgr2=0x7f43100c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:03.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.186+0000 7f430f7fe700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f431015a5d0 0x7f43100c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:03.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.186+0000 7f430f7fe700 1 -- 172.21.15.111:0/2230921839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4308009b30 con 0x7f43100b0160 2023-11-28T13:00:03.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.186+0000 7f430ffff700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f431015a5d0 0x7f43100c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:03.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.186+0000 7f430f7fe700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100b0160 0x7f43100c9ac0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f430400c7f0 tx=0x7f430400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:03.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.186+0000 7f42f7fff700 1 -- 172.21.15.111:0/2230921839 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4304005c20 con 0x7f43100b0160 2023-11-28T13:00:03.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.186+0000 7f42f7fff700 1 -- 172.21.15.111:0/2230921839 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4304005d80 con 0x7f43100b0160 2023-11-28T13:00:03.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.186+0000 7f4315fcc700 1 -- 172.21.15.111:0/2230921839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43100ccc30 con 0x7f43100b0160 2023-11-28T13:00:03.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.186+0000 7f42f7fff700 1 -- 172.21.15.111:0/2230921839 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4304011820 con 0x7f43100b0160 2023-11-28T13:00:03.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.186+0000 7f4315fcc700 1 -- 172.21.15.111:0/2230921839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43100cc950 con 0x7f43100b0160 2023-11-28T13:00:03.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.187+0000 7f4315fcc700 1 -- 172.21.15.111:0/2230921839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43100ccf10 con 0x7f43100b0160 2023-11-28T13:00:03.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.187+0000 7f42f7fff700 1 -- 172.21.15.111:0/2230921839 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4304020070 con 0x7f43100b0160 2023-11-28T13:00:03.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.188+0000 7f42f7fff700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42fc0682f0 0x7f42fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:03.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.188+0000 7f42f7fff700 1 -- 172.21.15.111:0/2230921839 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6047+0+0 (secure 0 0 0) 0x7f4304085e40 con 0x7f43100b0160 2023-11-28T13:00:03.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.188+0000 7f430effd700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42fc0682f0 0x7f42fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:03.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.188+0000 7f430effd700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42fc0682f0 0x7f42fc06a7a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f43100ce6b0 tx=0x7f43000090e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:03.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.192+0000 7f42f7fff700 1 -- 172.21.15.111:0/2230921839 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4304052630 con 0x7f43100b0160 2023-11-28T13:00:03.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:03.374+0000 7f4315fcc700 1 -- 172.21.15.111:0/2230921839 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4310067170 con 0x7f43100b0160 2023-11-28T13:00:04.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.045+0000 7f42f7fff700 1 -- 172.21.15.111:0/2230921839 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v32) v1 ==== 178+0+0 (secure 0 0 0) 0x7f4304055c50 con 0x7f43100b0160 2023-11-28T13:00:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.049+0000 7f4315fcc700 1 -- 172.21.15.111:0/2230921839 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42fc0682f0 msgr2=0x7f42fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.049+0000 7f4315fcc700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42fc0682f0 0x7f42fc06a7a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f43100ce6b0 tx=0x7f43000090e0 comp rx=0 tx=0).stop 2023-11-28T13:00:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.049+0000 7f4315fcc700 1 -- 172.21.15.111:0/2230921839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100b0160 msgr2=0x7f43100c9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.049+0000 7f4315fcc700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100b0160 0x7f43100c9ac0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f430400c7f0 tx=0x7f430400cb00 comp rx=0 tx=0).stop 2023-11-28T13:00:04.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.049+0000 7f4315fcc700 1 -- 172.21.15.111:0/2230921839 shutdown_connections 2023-11-28T13:00:04.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.049+0000 7f4315fcc700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42fc0682f0 0x7f42fc06a7a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:04.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.049+0000 7f4315fcc700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f431015a5d0 0x7f43100c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:04.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.049+0000 7f4315fcc700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100b0160 0x7f43100c9ac0 secure :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f430400c7f0 tx=0x7f430400cb00 comp rx=0 tx=0).stop 2023-11-28T13:00:04.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.049+0000 7f4315fcc700 1 --2- 172.21.15.111:0/2230921839 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f43100cb2d0 0x7f431014bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:04.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.049+0000 7f4315fcc700 1 -- 172.21.15.111:0/2230921839 >> 172.21.15.111:0/2230921839 conn(0x7f431009a040 msgr2=0x7f43100aea40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:04.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.049+0000 7f4315fcc700 1 -- 172.21.15.111:0/2230921839 shutdown_connections 2023-11-28T13:00:04.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.049+0000 7f4315fcc700 1 -- 172.21.15.111:0/2230921839 wait complete. 2023-11-28T13:00:04.052 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data_ec' removed 2023-11-28T13:00:04.066 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T13:00:04.066 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T13:00:04.066 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T13:00:04.066 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:00:04.067 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T13:00:04.067 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T13:00:04.067 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T13:00:04.067 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:00:04.067 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T13:00:04.067 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T13:00:04.068 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T13:00:04.068 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:00:04.068 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T13:00:04.068 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T13:00:04.068 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T13:00:04.068 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:00:04.077 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T13:00:04.077 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T13:00:04.078 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T13:00:04.078 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T13:00:04.079 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T13:00:04.080 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T13:00:04.080 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T13:00:04.080 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T13:00:04.120 INFO:tasks.ceph.mds.a:Started 2023-11-28T13:00:04.120 DEBUG:teuthology.parallel:result is None 2023-11-28T13:00:04.121 INFO:tasks.ceph.mds.c:Started 2023-11-28T13:00:04.121 DEBUG:teuthology.parallel:result is None 2023-11-28T13:00:04.122 INFO:tasks.ceph.mds.b:Started 2023-11-28T13:00:04.122 DEBUG:teuthology.parallel:result is None 2023-11-28T13:00:04.123 INFO:tasks.ceph.mds.d:Started 2023-11-28T13:00:04.123 DEBUG:teuthology.parallel:result is None 2023-11-28T13:00:04.123 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T13:00:04.202 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-11-28T13:00:04.202 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-11-28T13:00:04.208 INFO:tasks.ceph.mds.d.smithi149.stdout:starting mds.d at 2023-11-28T13:00:04.209 INFO:tasks.ceph.mds.b.smithi149.stdout:starting mds.b at 2023-11-28T13:00:04.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.257+0000 7f4a4962f700 1 Processor -- start 2023-11-28T13:00:04.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.257+0000 7f4a4962f700 1 -- start start 2023-11-28T13:00:04.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a4962f700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4a440b8510 0x7f4a4406b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:04.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a4962f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4a440d0d60 0x7f4a440d1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:04.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a4962f700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a440b88e0 con 0x7f4a440cd460 2023-11-28T13:00:04.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a4962f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a440cdbd0 con 0x7f4a440d0d60 2023-11-28T13:00:04.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a4962f700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a440d02d0 con 0x7f4a440b8510 2023-11-28T13:00:04.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a437fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4a440d0d60 0x7f4a440d1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:04.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a437fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4a440d0d60 0x7f4a440d1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34954/0 (socket says 172.21.15.111:34954) 2023-11-28T13:00:04.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a437fe700 1 -- 172.21.15.111:0/3263257948 learned_addr learned my addr 172.21.15.111:0/3263257948 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:04.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a437fe700 1 -- 172.21.15.111:0/3263257948 >> v2:172.21.15.149:3300/0 conn(0x7f4a440b8510 msgr2=0x7f4a4406b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:04.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a437fe700 1 --2- 172.21.15.111:0/3263257948 >> v2:172.21.15.149:3300/0 conn(0x7f4a440b8510 0x7f4a4406b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:04.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a437fe700 1 -- 172.21.15.111:0/3263257948 >> v1:172.21.15.149:6789/0 conn(0x7f4a440cd460 legacy=0x7f4a4406beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:04.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a437fe700 1 -- 172.21.15.111:0/3263257948 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a440cd8f0 con 0x7f4a440d0d60 2023-11-28T13:00:04.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a437fe700 1 --2- 172.21.15.111:0/3263257948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a440d0d60 0x7f4a440d1200 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4a440b5a50 tx=0x7f4a3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7537c1756a1eccf2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:04.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a41ffb700 1 -- 172.21.15.111:0/3263257948 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c00e070 con 0x7f4a440d0d60 2023-11-28T13:00:04.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a41ffb700 1 -- 172.21.15.111:0/3263257948 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3c003a20 con 0x7f4a440d0d60 2023-11-28T13:00:04.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.258+0000 7f4a41ffb700 1 -- 172.21.15.111:0/3263257948 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c003dd0 con 0x7f4a440d0d60 2023-11-28T13:00:04.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.259+0000 7f4a4962f700 1 -- 172.21.15.111:0/3263257948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a440d0d60 msgr2=0x7f4a440d1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:04.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.259+0000 7f4a4962f700 1 --2- 172.21.15.111:0/3263257948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a440d0d60 0x7f4a440d1200 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4a440b5a50 tx=0x7f4a3c00b330 comp rx=0 tx=0).stop 2023-11-28T13:00:04.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.259+0000 7f4a4962f700 1 -- 172.21.15.111:0/3263257948 shutdown_connections 2023-11-28T13:00:04.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.259+0000 7f4a4962f700 1 --2- 172.21.15.111:0/3263257948 >> v2:172.21.15.149:3300/0 conn(0x7f4a440b8510 0x7f4a4406b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:04.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.259+0000 7f4a4962f700 1 --2- 172.21.15.111:0/3263257948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a440d0d60 0x7f4a440d1200 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:04.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.259+0000 7f4a4962f700 1 -- 172.21.15.111:0/3263257948 >> 172.21.15.111:0/3263257948 conn(0x7f4a440719a0 msgr2=0x7f4a44071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:04.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.259+0000 7f4a4962f700 1 -- 172.21.15.111:0/3263257948 shutdown_connections 2023-11-28T13:00:04.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.259+0000 7f4a4962f700 1 -- 172.21.15.111:0/3263257948 wait complete. 2023-11-28T13:00:04.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.260+0000 7f4a4962f700 1 Processor -- start 2023-11-28T13:00:04.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.260+0000 7f4a4962f700 1 -- start start 2023-11-28T13:00:04.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a4962f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a440b8510 0x7f4a441e1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:04.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a4962f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4a440cd460 0x7f4a441e3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:04.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a4962f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4a440d0d60 0x7f4a44159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:04.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a4962f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a441e1f90 con 0x7f4a440b8510 2023-11-28T13:00:04.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a4962f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a441e2270 con 0x7f4a440cd460 2023-11-28T13:00:04.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a4962f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a441e2550 con 0x7f4a440d0d60 2023-11-28T13:00:04.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a42ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a440b8510 0x7f4a441e1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:04.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a42ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a440b8510 0x7f4a441e1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34956/0 (socket says 172.21.15.111:34956) 2023-11-28T13:00:04.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a42ffd700 1 -- 172.21.15.111:0/1964530355 learned_addr learned my addr 172.21.15.111:0/1964530355 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:04.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a427fc700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4a440cd460 0x7f4a441e3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:04.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a437fe700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4a440d0d60 0x7f4a44159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:04.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a42ffd700 1 -- 172.21.15.111:0/1964530355 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4a440d0d60 msgr2=0x7f4a44159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:04.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a42ffd700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4a440d0d60 0x7f4a44159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:04.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a42ffd700 1 -- 172.21.15.111:0/1964530355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4a440cd460 msgr2=0x7f4a441e3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:04.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a42ffd700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4a440cd460 0x7f4a441e3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:04.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a42ffd700 1 -- 172.21.15.111:0/1964530355 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c009b30 con 0x7f4a440b8510 2023-11-28T13:00:04.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.261+0000 7f4a42ffd700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a440b8510 0x7f4a441e1900 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400b530 tx=0x7f4a3400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:04.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.262+0000 7f4a2b7fe700 1 -- 172.21.15.111:0/1964530355 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34011830 con 0x7f4a440b8510 2023-11-28T13:00:04.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.262+0000 7f4a2b7fe700 1 -- 172.21.15.111:0/1964530355 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a34011990 con 0x7f4a440b8510 2023-11-28T13:00:04.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.262+0000 7f4a4962f700 1 -- 172.21.15.111:0/1964530355 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a440cfff0 con 0x7f4a440b8510 2023-11-28T13:00:04.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.262+0000 7f4a2b7fe700 1 -- 172.21.15.111:0/1964530355 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34011d40 con 0x7f4a440b8510 2023-11-28T13:00:04.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.262+0000 7f4a4962f700 1 -- 172.21.15.111:0/1964530355 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a440cfd10 con 0x7f4a440b8510 2023-11-28T13:00:04.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.263+0000 7f4a4962f700 1 -- 172.21.15.111:0/1964530355 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a44067880 con 0x7f4a440b8510 2023-11-28T13:00:04.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.263+0000 7f4a2b7fe700 1 -- 172.21.15.111:0/1964530355 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4a34021070 con 0x7f4a440b8510 2023-11-28T13:00:04.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.263+0000 7f4a2b7fe700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4a300682f0 0x7f4a3006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:04.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.263+0000 7f4a2b7fe700 1 -- 172.21.15.111:0/1964530355 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f4a34085960 con 0x7f4a440b8510 2023-11-28T13:00:04.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.263+0000 7f4a427fc700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4a300682f0 0x7f4a3006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:04.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.264+0000 7f4a427fc700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4a300682f0 0x7f4a3006a7a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4a38007fd0 tx=0x7f4a38009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:04.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.267+0000 7f4a2b7fe700 1 -- 172.21.15.111:0/1964530355 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4a34052370 con 0x7f4a440b8510 2023-11-28T13:00:04.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:04.465+0000 7f4a4962f700 1 -- 172.21.15.111:0/1964530355 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4a440d2940 con 0x7f4a440b8510 2023-11-28T13:00:05.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.250+0000 7f4a2b7fe700 1 -- 172.21.15.111:0/1964530355 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v37) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4a34015080 con 0x7f4a440b8510 2023-11-28T13:00:05.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.253+0000 7f4a4962f700 1 -- 172.21.15.111:0/1964530355 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4a300682f0 msgr2=0x7f4a3006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:05.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.253+0000 7f4a4962f700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4a300682f0 0x7f4a3006a7a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4a38007fd0 tx=0x7f4a38009040 comp rx=0 tx=0).stop 2023-11-28T13:00:05.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.253+0000 7f4a4962f700 1 -- 172.21.15.111:0/1964530355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a440b8510 msgr2=0x7f4a441e1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:05.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.253+0000 7f4a4962f700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a440b8510 0x7f4a441e1900 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400b530 tx=0x7f4a3400b840 comp rx=0 tx=0).stop 2023-11-28T13:00:05.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.253+0000 7f4a4962f700 1 -- 172.21.15.111:0/1964530355 shutdown_connections 2023-11-28T13:00:05.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.253+0000 7f4a4962f700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4a440d0d60 0x7f4a44159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.253+0000 7f4a4962f700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a440b8510 0x7f4a441e1900 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.253+0000 7f4a4962f700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4a300682f0 0x7f4a3006a7a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.253+0000 7f4a4962f700 1 --2- 172.21.15.111:0/1964530355 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4a440cd460 0x7f4a441e3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.253+0000 7f4a4962f700 1 -- 172.21.15.111:0/1964530355 >> 172.21.15.111:0/1964530355 conn(0x7f4a440719a0 msgr2=0x7f4a440b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:05.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.253+0000 7f4a4962f700 1 -- 172.21.15.111:0/1964530355 shutdown_connections 2023-11-28T13:00:05.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.253+0000 7f4a4962f700 1 -- 172.21.15.111:0/1964530355 wait complete. 2023-11-28T13:00:05.257 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-11-28T13:00:05.272 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T13:00:05.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.398+0000 7f75ef503700 1 Processor -- start 2023-11-28T13:00:05.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.399+0000 7f75ef503700 1 -- start start 2023-11-28T13:00:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75ef503700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75e80b0d70 0x7f75e81604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75ef503700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f75e80b1140 0x7f75e80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75ef503700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f75e80b15b0 0x7f75e80cc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75ef503700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e815cd90 con 0x7f75e80b1140 2023-11-28T13:00:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75ef503700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e81613e0 con 0x7f75e80b15b0 2023-11-28T13:00:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75ef503700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e8160b40 con 0x7f75e80b0d70 2023-11-28T13:00:05.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75eca9e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f75e80b1140 0x7f75e80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75eca9e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f75e80b1140 0x7f75e80ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34970/0 (socket says 172.21.15.111:34970) 2023-11-28T13:00:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75eca9e700 1 -- 172.21.15.111:0/2659583918 learned_addr learned my addr 172.21.15.111:0/2659583918 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75edaa0700 1 --2- 172.21.15.111:0/2659583918 >> v2:172.21.15.149:3300/0 conn(0x7f75e80b15b0 0x7f75e80cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75ed29f700 1 --2- 172.21.15.111:0/2659583918 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75e80b0d70 0x7f75e81604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75eca9e700 1 -- 172.21.15.111:0/2659583918 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75e80b0d70 msgr2=0x7f75e81604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75eca9e700 1 --2- 172.21.15.111:0/2659583918 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75e80b0d70 0x7f75e81604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75eca9e700 1 -- 172.21.15.111:0/2659583918 >> v2:172.21.15.149:3300/0 conn(0x7f75e80b15b0 msgr2=0x7f75e80cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75eca9e700 1 --2- 172.21.15.111:0/2659583918 >> v2:172.21.15.149:3300/0 conn(0x7f75e80b15b0 0x7f75e80cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75eca9e700 1 -- 172.21.15.111:0/2659583918 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e8160e20 con 0x7f75e80b1140 2023-11-28T13:00:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75ed29f700 1 --2- 172.21.15.111:0/2659583918 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75e80b0d70 0x7f75e81604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75eca9e700 1 --2- 172.21.15.111:0/2659583918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e80b1140 0x7f75e80ce6c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f75e80cfb10 tx=0x7f75e4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d47ed51653bffce server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:05.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75dffff700 1 -- 172.21.15.111:0/2659583918 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e400b870 con 0x7f75e80b1140 2023-11-28T13:00:05.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75dffff700 1 -- 172.21.15.111:0/2659583918 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e400b9d0 con 0x7f75e80b1140 2023-11-28T13:00:05.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.400+0000 7f75dffff700 1 -- 172.21.15.111:0/2659583918 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e400bd60 con 0x7f75e80b1140 2023-11-28T13:00:05.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.401+0000 7f75ef503700 1 -- 172.21.15.111:0/2659583918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e80b1140 msgr2=0x7f75e80ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:05.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.401+0000 7f75ef503700 1 --2- 172.21.15.111:0/2659583918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e80b1140 0x7f75e80ce6c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f75e80cfb10 tx=0x7f75e4009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.401+0000 7f75ef503700 1 -- 172.21.15.111:0/2659583918 shutdown_connections 2023-11-28T13:00:05.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.401+0000 7f75ef503700 1 --2- 172.21.15.111:0/2659583918 >> v2:172.21.15.149:3300/0 conn(0x7f75e80b15b0 0x7f75e80cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.401+0000 7f75ef503700 1 --2- 172.21.15.111:0/2659583918 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75e80b0d70 0x7f75e81604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.401+0000 7f75ef503700 1 --2- 172.21.15.111:0/2659583918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e80b1140 0x7f75e80ce6c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.401+0000 7f75ef503700 1 -- 172.21.15.111:0/2659583918 >> 172.21.15.111:0/2659583918 conn(0x7f75e8098ee0 msgr2=0x7f75e806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:05.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.401+0000 7f75ef503700 1 -- 172.21.15.111:0/2659583918 shutdown_connections 2023-11-28T13:00:05.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.401+0000 7f75ef503700 1 -- 172.21.15.111:0/2659583918 wait complete. 2023-11-28T13:00:05.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.402+0000 7f75ef503700 1 Processor -- start 2023-11-28T13:00:05.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.402+0000 7f75ef503700 1 -- start start 2023-11-28T13:00:05.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75ef503700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75e80b0d70 0x7f75e81e7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:05.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75ef503700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75e80b1140 0x7f75e81f6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:05.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75ef503700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e80b15b0 0x7f75e81f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:05.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75ef503700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e815bb50 con 0x7f75e80b15b0 2023-11-28T13:00:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75ef503700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e80c8190 con 0x7f75e80b0d70 2023-11-28T13:00:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75ef503700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e81e8c80 con 0x7f75e80b1140 2023-11-28T13:00:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75edaa0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e80b15b0 0x7f75e81f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75edaa0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e80b15b0 0x7f75e81f6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34978/0 (socket says 172.21.15.111:34978) 2023-11-28T13:00:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75edaa0700 1 -- 172.21.15.111:0/3861720182 learned_addr learned my addr 172.21.15.111:0/3861720182 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75ed29f700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75e80b0d70 0x7f75e81e7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75eca9e700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75e80b1140 0x7f75e81f6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75edaa0700 1 -- 172.21.15.111:0/3861720182 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75e80b1140 msgr2=0x7f75e81f6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75edaa0700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75e80b1140 0x7f75e81f6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75edaa0700 1 -- 172.21.15.111:0/3861720182 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75e80b0d70 msgr2=0x7f75e81e7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75edaa0700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75e80b0d70 0x7f75e81e7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.403+0000 7f75edaa0700 1 -- 172.21.15.111:0/3861720182 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e4009550 con 0x7f75e80b15b0 2023-11-28T13:00:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.404+0000 7f75edaa0700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e80b15b0 0x7f75e81f6b80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f75e0007e10 tx=0x7f75e000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.404+0000 7f75ddffb700 1 -- 172.21.15.111:0/3861720182 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e000ed30 con 0x7f75e80b15b0 2023-11-28T13:00:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.404+0000 7f75ddffb700 1 -- 172.21.15.111:0/3861720182 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e000eeb0 con 0x7f75e80b15b0 2023-11-28T13:00:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.404+0000 7f75ef503700 1 -- 172.21.15.111:0/3861720182 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e80af340 con 0x7f75e80b15b0 2023-11-28T13:00:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.404+0000 7f75ddffb700 1 -- 172.21.15.111:0/3861720182 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e0014640 con 0x7f75e80b15b0 2023-11-28T13:00:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.404+0000 7f75ef503700 1 -- 172.21.15.111:0/3861720182 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e8161100 con 0x7f75e80b15b0 2023-11-28T13:00:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.405+0000 7f75ef503700 1 -- 172.21.15.111:0/3861720182 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e80cec00 con 0x7f75e80b15b0 2023-11-28T13:00:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.405+0000 7f75ddffb700 1 -- 172.21.15.111:0/3861720182 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f75e0012050 con 0x7f75e80b15b0 2023-11-28T13:00:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.405+0000 7f75ddffb700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75cc0682f0 0x7f75cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.406+0000 7f75ed29f700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75cc0682f0 0x7f75cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.406+0000 7f75ddffb700 1 -- 172.21.15.111:0/3861720182 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f75e0085f40 con 0x7f75e80b15b0 2023-11-28T13:00:05.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.406+0000 7f75ed29f700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75cc0682f0 0x7f75cc06a7a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f75e809d960 tx=0x7f75d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:05.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.410+0000 7f75ddffb700 1 -- 172.21.15.111:0/3861720182 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f75e00529b0 con 0x7f75e80b15b0 2023-11-28T13:00:05.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.663+0000 7f75ef503700 1 -- 172.21.15.111:0/3861720182 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f75e80c9c40 con 0x7f75e80b15b0 2023-11-28T13:00:05.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.664+0000 7f75ddffb700 1 -- 172.21.15.111:0/3861720182 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f75e0055fd0 con 0x7f75e80b15b0 2023-11-28T13:00:05.665 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:05.665 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:00:05.665 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-11-28T13:00:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-11-28T13:00:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlH823BRAArK8kE5qUgPoqFZPxIudN+w==", 2023-11-28T13:00:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:00:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:00:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-11-28T13:00:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlIo9ZEhAA5NmyiPE68OmMcsG1hTiTgw==", 2023-11-28T13:00:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:00:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:00:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-11-28T13:00:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlQqHxDBAA8RDJ2PETpTHNjnn1i1T3Ow==", 2023-11-28T13:00:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:00:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:00:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-11-28T13:00:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42Vl8RDmGRAAkiGKEQMr9rHahKJ/3lw9BA==", 2023-11-28T13:00:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:00:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:00:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-11-28T13:00:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDc42VlfB0eNRAAJEa0AL3iDLArHc4n6CWK9g==", 2023-11-28T13:00:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-11-28T13:00:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDf42VlnrbLNBAAU+sVJ/6EsadNo3BXpyrbSw==", 2023-11-28T13:00:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-11-28T13:00:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDi42VlwCUsNRAAwZKJ/o1qTBtWwR35B1r2zQ==", 2023-11-28T13:00:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-11-28T13:00:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDl42Vl3/t2NBAAMPuhlpUi37Fpd/6NclsErw==", 2023-11-28T13:00:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-11-28T13:00:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDr42VlkrWgNxAAuu2dXZZ3U/XRrq3cWCEa8w==", 2023-11-28T13:00:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-11-28T13:00:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDu42VldDFENxAAYKD2Dqgka5G+XxGjOR3M6w==", 2023-11-28T13:00:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-11-28T13:00:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDx42VlvjotNhAAWD2E1sgAZLdxb5CG1Rg26g==", 2023-11-28T13:00:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-11-28T13:00:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQD042VlZheVNBAAlMIKaDmgUMmt/hA+vx2/PA==", 2023-11-28T13:00:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-11-28T13:00:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42Vl3MswHhAA/oDIgj9lzhcnU7f0ksZ9cw==", 2023-11-28T13:00:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-11-28T13:00:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-11-28T13:00:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-11-28T13:00:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-11-28T13:00:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlV3dQJxAAdzQk1q+DoxFDEr/TYJi/6A==", 2023-11-28T13:00:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-11-28T13:00:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-11-28T13:00:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-11-28T13:00:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-11-28T13:00:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlJiXlIxAAYuFMFLUqIDN5f0E008GKZw==", 2023-11-28T13:00:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-11-28T13:00:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-11-28T13:00:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-11-28T13:00:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-11-28T13:00:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlagSZKhAAkLS8RwTZZTDyO2slb0MzIQ==", 2023-11-28T13:00:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-11-28T13:00:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-11-28T13:00:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-11-28T13:00:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-11-28T13:00:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDY42Vl74R5IhAADPhHAlk7DX23esg8UFYW/Q==", 2023-11-28T13:00:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-11-28T13:00:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:00:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:00:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.bootstrap-mds", 2023-11-28T13:00:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQD642VlZybcMRAAYsXlXT9nW5WBU6U4BuGI9A==", 2023-11-28T13:00:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile bootstrap-mds" 2023-11-28T13:00:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.bootstrap-mgr", 2023-11-28T13:00:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQD642Vl6i7cMRAAhOSpGxD1Mk4cvFBlqHkNwQ==", 2023-11-28T13:00:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile bootstrap-mgr" 2023-11-28T13:00:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.bootstrap-osd", 2023-11-28T13:00:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQD642Vl9zXcMRAA+lMUs7tHSyotM2WIeAtKdA==", 2023-11-28T13:00:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile bootstrap-osd" 2023-11-28T13:00:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.bootstrap-rbd", 2023-11-28T13:00:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQD642VlVz/cMRAAxsPFmR5nE5LWWigpMqlEnw==", 2023-11-28T13:00:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile bootstrap-rbd" 2023-11-28T13:00:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-11-28T13:00:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQD642Vl8EbcMRAAgEBHCson/piJ6RI1AUsF6g==", 2023-11-28T13:00:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-11-28T13:00:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.bootstrap-rgw", 2023-11-28T13:00:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQD642VleU7cMRAAMXzg92cbC1nWodHIxXhHdw==", 2023-11-28T13:00:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile bootstrap-rgw" 2023-11-28T13:00:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-11-28T13:00:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42Vl+AxXAhAADTJ8Jt/WNo8qr5zAyQJ4rw==", 2023-11-28T13:00:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-11-28T13:00:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-11-28T13:00:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-11-28T13:00:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDY42Vl6WCiOBAALRDd8ZjOOlxxLKQsgXP9Jg==", 2023-11-28T13:00:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-11-28T13:00:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-11-28T13:00:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-11-28T13:00:05.684 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:00:05.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.665+0000 7f75ef503700 1 -- 172.21.15.111:0/3861720182 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75cc0682f0 msgr2=0x7f75cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:05.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.665+0000 7f75ef503700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75cc0682f0 0x7f75cc06a7a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f75e809d960 tx=0x7f75d8008040 comp rx=0 tx=0).stop 2023-11-28T13:00:05.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.665+0000 7f75ef503700 1 -- 172.21.15.111:0/3861720182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e80b15b0 msgr2=0x7f75e81f6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:05.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.665+0000 7f75ef503700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e80b15b0 0x7f75e81f6b80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f75e0007e10 tx=0x7f75e000e3c0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.666+0000 7f75ef503700 1 -- 172.21.15.111:0/3861720182 shutdown_connections 2023-11-28T13:00:05.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.666+0000 7f75ef503700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75e80b15b0 0x7f75e81f6b80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.666+0000 7f75ef503700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f75cc0682f0 0x7f75cc06a7a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.666+0000 7f75ef503700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f75e80b0d70 0x7f75e81e7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.666+0000 7f75ef503700 1 --2- 172.21.15.111:0/3861720182 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f75e80b1140 0x7f75e81f6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.666+0000 7f75ef503700 1 -- 172.21.15.111:0/3861720182 >> 172.21.15.111:0/3861720182 conn(0x7f75e8098ee0 msgr2=0x7f75e80cc320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:05.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.666+0000 7f75ef503700 1 -- 172.21.15.111:0/3861720182 shutdown_connections 2023-11-28T13:00:05.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.666+0000 7f75ef503700 1 -- 172.21.15.111:0/3861720182 wait complete. 2023-11-28T13:00:05.687 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-11-28T13:00:05.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.807+0000 7fc6daa70700 1 Processor -- start 2023-11-28T13:00:05.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.807+0000 7fc6daa70700 1 -- start start 2023-11-28T13:00:05.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6daa70700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc6d415fed0 0x7fc6d40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:05.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6daa70700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc6d40af5a0 0x7fc6d40ca220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:05.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6daa70700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d406ac80 con 0x7fc6d40cf7d0 2023-11-28T13:00:05.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6daa70700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d40cee00 con 0x7fc6d40af5a0 2023-11-28T13:00:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6daa70700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d406aed0 con 0x7fc6d415fed0 2023-11-28T13:00:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6d900d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc6d40af5a0 0x7fc6d40ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6d900d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc6d40af5a0 0x7fc6d40ca220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34986/0 (socket says 172.21.15.111:34986) 2023-11-28T13:00:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6d900d700 1 -- 172.21.15.111:0/2457392311 learned_addr learned my addr 172.21.15.111:0/2457392311 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6d880c700 1 --2- 172.21.15.111:0/2457392311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc6d415fed0 0x7fc6d40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6d900d700 1 -- 172.21.15.111:0/2457392311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc6d415fed0 msgr2=0x7fc6d40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6d900d700 1 --2- 172.21.15.111:0/2457392311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc6d415fed0 0x7fc6d40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6d900d700 1 -- 172.21.15.111:0/2457392311 >> v1:172.21.15.149:6789/0 conn(0x7fc6d40cf7d0 legacy=0x7fc6d40b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6d900d700 1 -- 172.21.15.111:0/2457392311 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d40cea90 con 0x7fc6d40af5a0 2023-11-28T13:00:05.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.808+0000 7fc6d900d700 1 --2- 172.21.15.111:0/2457392311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d40af5a0 0x7fc6d40ca220 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009e50 tx=0x7fc6cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef78a05af2f0b314 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.809+0000 7fc6d37fe700 1 -- 172.21.15.111:0/2457392311 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc00e070 con 0x7fc6d40af5a0 2023-11-28T13:00:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.809+0000 7fc6d37fe700 1 -- 172.21.15.111:0/2457392311 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc00b910 con 0x7fc6d40af5a0 2023-11-28T13:00:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.809+0000 7fc6d37fe700 1 -- 172.21.15.111:0/2457392311 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc00bcc0 con 0x7fc6d40af5a0 2023-11-28T13:00:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.809+0000 7fc6daa70700 1 -- 172.21.15.111:0/2457392311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d40af5a0 msgr2=0x7fc6d40ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.809+0000 7fc6daa70700 1 --2- 172.21.15.111:0/2457392311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d40af5a0 0x7fc6d40ca220 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009e50 tx=0x7fc6cc00b330 comp rx=0 tx=0).stop 2023-11-28T13:00:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.809+0000 7fc6daa70700 1 -- 172.21.15.111:0/2457392311 shutdown_connections 2023-11-28T13:00:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.809+0000 7fc6daa70700 1 --2- 172.21.15.111:0/2457392311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc6d415fed0 0x7fc6d40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.809+0000 7fc6daa70700 1 --2- 172.21.15.111:0/2457392311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d40af5a0 0x7fc6d40ca220 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.809+0000 7fc6daa70700 1 -- 172.21.15.111:0/2457392311 >> 172.21.15.111:0/2457392311 conn(0x7fc6d406bca0 msgr2=0x7fc6d40cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.809+0000 7fc6daa70700 1 -- 172.21.15.111:0/2457392311 shutdown_connections 2023-11-28T13:00:05.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.809+0000 7fc6daa70700 1 -- 172.21.15.111:0/2457392311 wait complete. 2023-11-28T13:00:05.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.810+0000 7fc6daa70700 1 Processor -- start 2023-11-28T13:00:05.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.811+0000 7fc6daa70700 1 -- start start 2023-11-28T13:00:05.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.811+0000 7fc6daa70700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d40af5a0 0x7fc6d4159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:05.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.811+0000 7fc6daa70700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc6d40cf7d0 0x7fc6d40b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:05.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.811+0000 7fc6daa70700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc6d415fed0 0x7fc6d40c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:05.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.811+0000 7fc6daa70700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d40b3f80 con 0x7fc6d40af5a0 2023-11-28T13:00:05.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.811+0000 7fc6daa70700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d40b43a0 con 0x7fc6d415fed0 2023-11-28T13:00:05.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.811+0000 7fc6daa70700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d40b3300 con 0x7fc6d40cf7d0 2023-11-28T13:00:05.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.811+0000 7fc6d880c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d40af5a0 0x7fc6d4159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:05.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6d880c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d40af5a0 0x7fc6d4159090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34988/0 (socket says 172.21.15.111:34988) 2023-11-28T13:00:05.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6d880c700 1 -- 172.21.15.111:0/3658972204 learned_addr learned my addr 172.21.15.111:0/3658972204 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:05.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6d900d700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc6d415fed0 0x7fc6d40c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:05.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6d3fff700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc6d40cf7d0 0x7fc6d40b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:05.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6d880c700 1 -- 172.21.15.111:0/3658972204 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc6d40cf7d0 msgr2=0x7fc6d40b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:05.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6d880c700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc6d40cf7d0 0x7fc6d40b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6d880c700 1 -- 172.21.15.111:0/3658972204 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc6d415fed0 msgr2=0x7fc6d40c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:05.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6d880c700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc6d415fed0 0x7fc6d40c6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:05.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6d880c700 1 -- 172.21.15.111:0/3658972204 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc009b30 con 0x7fc6d40af5a0 2023-11-28T13:00:05.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6d880c700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d40af5a0 0x7fc6d4159090 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800e870 tx=0x7fc6c800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:05.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6d17fa700 1 -- 172.21.15.111:0/3658972204 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8009590 con 0x7fc6d40af5a0 2023-11-28T13:00:05.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6d17fa700 1 -- 172.21.15.111:0/3658972204 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c800ee70 con 0x7fc6d40af5a0 2023-11-28T13:00:05.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6daa70700 1 -- 172.21.15.111:0/3658972204 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d40ce6b0 con 0x7fc6d40af5a0 2023-11-28T13:00:05.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6d17fa700 1 -- 172.21.15.111:0/3658972204 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c800ca50 con 0x7fc6d40af5a0 2023-11-28T13:00:05.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.812+0000 7fc6daa70700 1 -- 172.21.15.111:0/3658972204 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d41602a0 con 0x7fc6d40af5a0 2023-11-28T13:00:05.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.813+0000 7fc6daa70700 1 -- 172.21.15.111:0/3658972204 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d40ce840 con 0x7fc6d40af5a0 2023-11-28T13:00:05.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.813+0000 7fc6d17fa700 1 -- 172.21.15.111:0/3658972204 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc6c8019030 con 0x7fc6d40af5a0 2023-11-28T13:00:05.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.814+0000 7fc6d17fa700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc6b80682f0 0x7fc6b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:05.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.814+0000 7fc6d17fa700 1 -- 172.21.15.111:0/3658972204 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fc6c8085880 con 0x7fc6d40af5a0 2023-11-28T13:00:05.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.814+0000 7fc6d3fff700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc6b80682f0 0x7fc6b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:05.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.814+0000 7fc6d3fff700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc6b80682f0 0x7fc6b806a7a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40b45b0 tx=0x7fc6c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:05.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:05.818+0000 7fc6d17fa700 1 -- 172.21.15.111:0/3658972204 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc6c80522f0 con 0x7fc6d40af5a0 2023-11-28T13:00:06.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.077+0000 7fc6daa70700 1 -- 172.21.15.111:0/3658972204 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fc6d40b41f0 con 0x7fc6d40af5a0 2023-11-28T13:00:06.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.082+0000 7fc6d17fa700 1 -- 172.21.15.111:0/3658972204 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fc6c8055910 con 0x7fc6d40af5a0 2023-11-28T13:00:06.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.084+0000 7fc6daa70700 1 -- 172.21.15.111:0/3658972204 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc6b80682f0 msgr2=0x7fc6b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.084+0000 7fc6daa70700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc6b80682f0 0x7fc6b806a7a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40b45b0 tx=0x7fc6c0009040 comp rx=0 tx=0).stop 2023-11-28T13:00:06.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.084+0000 7fc6daa70700 1 -- 172.21.15.111:0/3658972204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d40af5a0 msgr2=0x7fc6d4159090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.084+0000 7fc6daa70700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d40af5a0 0x7fc6d4159090 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800e870 tx=0x7fc6c800ec30 comp rx=0 tx=0).stop 2023-11-28T13:00:06.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.084+0000 7fc6daa70700 1 -- 172.21.15.111:0/3658972204 shutdown_connections 2023-11-28T13:00:06.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.084+0000 7fc6daa70700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc6b80682f0 0x7fc6b806a7a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.084+0000 7fc6daa70700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc6d415fed0 0x7fc6d40c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.084+0000 7fc6daa70700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6d40af5a0 0x7fc6d4159090 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.084+0000 7fc6daa70700 1 --2- 172.21.15.111:0/3658972204 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc6d40cf7d0 0x7fc6d40b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.084+0000 7fc6daa70700 1 -- 172.21.15.111:0/3658972204 >> 172.21.15.111:0/3658972204 conn(0x7fc6d406bca0 msgr2=0x7fc6d40aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:06.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.084+0000 7fc6daa70700 1 -- 172.21.15.111:0/3658972204 shutdown_connections 2023-11-28T13:00:06.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.084+0000 7fc6daa70700 1 -- 172.21.15.111:0/3658972204 wait complete. 2023-11-28T13:00:06.097 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-11-28T13:00:06.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.217+0000 7f3843f0e700 1 Processor -- start 2023-11-28T13:00:06.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.217+0000 7f3843f0e700 1 -- start start 2023-11-28T13:00:06.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f3843f0e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f383c159960 0x7f383c0b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:06.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f3843f0e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0cdd90 0x7f383c0ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:06.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f3843f0e700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c0ad420 con 0x7f383c0aecc0 2023-11-28T13:00:06.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f3843f0e700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383c159d30 con 0x7f383c159960 2023-11-28T13:00:06.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f3843f0e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0ac810 con 0x7f383c0cdd90 2023-11-28T13:00:06.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f38424ab700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0cdd90 0x7f383c0ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:06.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f3841caa700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f383c159960 0x7f383c0b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:06.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f38414a9700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f383c0aecc0 0x7f383c0cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:44318/0 (socket says 172.21.15.111:44318) 2023-11-28T13:00:06.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f38424ab700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0cdd90 0x7f383c0ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:56748/0 (socket says 172.21.15.111:56748) 2023-11-28T13:00:06.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f38414a9700 1 -- 172.21.15.111:0/1247515940 learned_addr learned my addr 172.21.15.111:0/1247515940 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:06.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f3841caa700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f383c159960 0x7f383c0b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:52896/0 (socket says 172.21.15.111:52896) 2023-11-28T13:00:06.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f3840ca8700 1 -- 172.21.15.111:0/1247515940 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978307475 0 0) 0x7f383c0ad420 con 0x7f383c0aecc0 2023-11-28T13:00:06.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f3840ca8700 1 -- 172.21.15.111:0/1247515940 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38280035b0 con 0x7f383c0aecc0 2023-11-28T13:00:06.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f38424ab700 1 -- 172.21.15.111:0/1247515940 >> v2:172.21.15.149:3300/0 conn(0x7f383c159960 msgr2=0x7f383c0b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:06.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f38424ab700 1 --2- 172.21.15.111:0/1247515940 >> v2:172.21.15.149:3300/0 conn(0x7f383c159960 0x7f383c0b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f38424ab700 1 -- 172.21.15.111:0/1247515940 >> v1:172.21.15.149:6789/0 conn(0x7f383c0aecc0 legacy=0x7f383c0cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f38424ab700 1 -- 172.21.15.111:0/1247515940 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c0acaf0 con 0x7f383c0cdd90 2023-11-28T13:00:06.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.218+0000 7f3841caa700 1 --2- 172.21.15.111:0/1247515940 >> v2:172.21.15.149:3300/0 conn(0x7f383c159960 0x7f383c0b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:06.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.219+0000 7f38424ab700 1 --2- 172.21.15.111:0/1247515940 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0cdd90 0x7f383c0ad8b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3834009d90 tx=0x7f383400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d254828a8b3e03c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:06.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.219+0000 7f3840ca8700 1 -- 172.21.15.111:0/1247515940 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383400e070 con 0x7f383c0cdd90 2023-11-28T13:00:06.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.219+0000 7f3840ca8700 1 -- 172.21.15.111:0/1247515940 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383400bb20 con 0x7f383c0cdd90 2023-11-28T13:00:06.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.219+0000 7f3840ca8700 1 -- 172.21.15.111:0/1247515940 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834012430 con 0x7f383c0cdd90 2023-11-28T13:00:06.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.219+0000 7f3843f0e700 1 -- 172.21.15.111:0/1247515940 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0cdd90 msgr2=0x7f383c0ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.219+0000 7f3843f0e700 1 --2- 172.21.15.111:0/1247515940 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0cdd90 0x7f383c0ad8b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3834009d90 tx=0x7f383400b330 comp rx=0 tx=0).stop 2023-11-28T13:00:06.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.219+0000 7f3843f0e700 1 -- 172.21.15.111:0/1247515940 shutdown_connections 2023-11-28T13:00:06.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.219+0000 7f3843f0e700 1 --2- 172.21.15.111:0/1247515940 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0cdd90 0x7f383c0ad8b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.219+0000 7f3843f0e700 1 --2- 172.21.15.111:0/1247515940 >> v2:172.21.15.149:3300/0 conn(0x7f383c159960 0x7f383c0b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.219+0000 7f3843f0e700 1 -- 172.21.15.111:0/1247515940 >> 172.21.15.111:0/1247515940 conn(0x7f383c098ee0 msgr2=0x7f383c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:06.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.219+0000 7f3843f0e700 1 -- 172.21.15.111:0/1247515940 shutdown_connections 2023-11-28T13:00:06.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.219+0000 7f3843f0e700 1 -- 172.21.15.111:0/1247515940 wait complete. 2023-11-28T13:00:06.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.221+0000 7f3843f0e700 1 Processor -- start 2023-11-28T13:00:06.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.221+0000 7f3843f0e700 1 -- start start 2023-11-28T13:00:06.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.221+0000 7f3843f0e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c0aecc0 0x7f383c1f2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:06.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.221+0000 7f3843f0e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0cdd90 0x7f383c157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:06.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.221+0000 7f3843f0e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c159960 0x7f383c1efed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:06.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.221+0000 7f3843f0e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0b2d10 con 0x7f383c159960 2023-11-28T13:00:06.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.221+0000 7f3843f0e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0b2970 con 0x7f383c0aecc0 2023-11-28T13:00:06.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.221+0000 7f3843f0e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c1f2b50 con 0x7f383c0cdd90 2023-11-28T13:00:06.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.221+0000 7f38424ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c159960 0x7f383c1efed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:06.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.221+0000 7f38424ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c159960 0x7f383c1efed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35006/0 (socket says 172.21.15.111:35006) 2023-11-28T13:00:06.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.221+0000 7f38424ab700 1 -- 172.21.15.111:0/2595588820 learned_addr learned my addr 172.21.15.111:0/2595588820 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:06.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f38414a9700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0cdd90 0x7f383c157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:06.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f3841caa700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c0aecc0 0x7f383c1f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:06.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f38424ab700 1 -- 172.21.15.111:0/2595588820 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0cdd90 msgr2=0x7f383c157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f38424ab700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0cdd90 0x7f383c157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f38424ab700 1 -- 172.21.15.111:0/2595588820 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c0aecc0 msgr2=0x7f383c1f2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f38424ab700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c0aecc0 0x7f383c1f2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f38424ab700 1 -- 172.21.15.111:0/2595588820 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3838009430 con 0x7f383c159960 2023-11-28T13:00:06.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f38414a9700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0cdd90 0x7f383c157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f3841caa700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c0aecc0 0x7f383c1f2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f38424ab700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c159960 0x7f383c1efed0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3834014040 tx=0x7f3834004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f38327fc700 1 -- 172.21.15.111:0/2595588820 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834012430 con 0x7f383c159960 2023-11-28T13:00:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f38327fc700 1 -- 172.21.15.111:0/2595588820 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3834020070 con 0x7f383c159960 2023-11-28T13:00:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f3843f0e700 1 -- 172.21.15.111:0/2595588820 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3834009a70 con 0x7f383c159960 2023-11-28T13:00:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f38327fc700 1 -- 172.21.15.111:0/2595588820 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834004280 con 0x7f383c159960 2023-11-28T13:00:06.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.222+0000 7f3843f0e700 1 -- 172.21.15.111:0/2595588820 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383c0ad0b0 con 0x7f383c159960 2023-11-28T13:00:06.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.223+0000 7f3843f0e700 1 -- 172.21.15.111:0/2595588820 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383c0acdd0 con 0x7f383c159960 2023-11-28T13:00:06.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.223+0000 7f38327fc700 1 -- 172.21.15.111:0/2595588820 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f38340043e0 con 0x7f383c159960 2023-11-28T13:00:06.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.224+0000 7f38327fc700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38200682f0 0x7f382006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:06.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.224+0000 7f38327fc700 1 -- 172.21.15.111:0/2595588820 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f3834022070 con 0x7f383c159960 2023-11-28T13:00:06.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.224+0000 7f3841caa700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38200682f0 0x7f382006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:06.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.224+0000 7f3841caa700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38200682f0 0x7f382006a7a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f383c06f7e0 tx=0x7f3838008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:06.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.228+0000 7f38327fc700 1 -- 172.21.15.111:0/2595588820 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f38340545d0 con 0x7f383c159960 2023-11-28T13:00:06.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.484+0000 7f3843f0e700 1 -- 172.21.15.111:0/2595588820 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f383c0addf0 con 0x7f383c159960 2023-11-28T13:00:06.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.489+0000 7f38327fc700 1 -- 172.21.15.111:0/2595588820 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3834057bf0 con 0x7f383c159960 2023-11-28T13:00:06.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.492+0000 7f3843f0e700 1 -- 172.21.15.111:0/2595588820 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38200682f0 msgr2=0x7f382006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.492+0000 7f3843f0e700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38200682f0 0x7f382006a7a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f383c06f7e0 tx=0x7f3838008040 comp rx=0 tx=0).stop 2023-11-28T13:00:06.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.492+0000 7f3843f0e700 1 -- 172.21.15.111:0/2595588820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c159960 msgr2=0x7f383c1efed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.492+0000 7f3843f0e700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c159960 0x7f383c1efed0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3834014040 tx=0x7f3834004ad0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.492+0000 7f3843f0e700 1 -- 172.21.15.111:0/2595588820 shutdown_connections 2023-11-28T13:00:06.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.492+0000 7f3843f0e700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c159960 0x7f383c1efed0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.492+0000 7f3843f0e700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38200682f0 0x7f382006a7a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.492+0000 7f3843f0e700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c0aecc0 0x7f383c1f2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.492+0000 7f3843f0e700 1 --2- 172.21.15.111:0/2595588820 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0cdd90 0x7f383c157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.492+0000 7f3843f0e700 1 -- 172.21.15.111:0/2595588820 >> 172.21.15.111:0/2595588820 conn(0x7f383c098ee0 msgr2=0x7f383c0ae230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:06.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.492+0000 7f3843f0e700 1 -- 172.21.15.111:0/2595588820 shutdown_connections 2023-11-28T13:00:06.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.492+0000 7f3843f0e700 1 -- 172.21.15.111:0/2595588820 wait complete. 2023-11-28T13:00:06.505 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-11-28T13:00:06.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.624+0000 7ff7ed3a2700 1 Processor -- start 2023-11-28T13:00:06.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.624+0000 7ff7ed3a2700 1 -- start start 2023-11-28T13:00:06.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.625+0000 7ff7ed3a2700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff7e80ad980 0x7ff7e80d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:06.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.625+0000 7ff7ed3a2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7e80b0980 0x7ff7e80b0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:06.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.625+0000 7ff7ed3a2700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e80b0210 con 0x7ff7e80b1330 2023-11-28T13:00:06.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.625+0000 7ff7ed3a2700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7e80ce6b0 con 0x7ff7e80ad980 2023-11-28T13:00:06.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.625+0000 7ff7ed3a2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e80b0080 con 0x7ff7e80b0980 2023-11-28T13:00:06.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.625+0000 7ff7e6ffd700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff7e80ad980 0x7ff7e80d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:06.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.625+0000 7ff7e67fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7e80b0980 0x7ff7e80b0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:06.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.626+0000 7ff7e77fe700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7ff7e80b1330 0x7ff7e80add50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:44330/0 (socket says 172.21.15.111:44330) 2023-11-28T13:00:06.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.626+0000 7ff7e77fe700 1 -- 172.21.15.111:0/2872149843 learned_addr learned my addr 172.21.15.111:0/2872149843 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:06.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.626+0000 7ff7e6ffd700 1 -- 172.21.15.111:0/2872149843 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7e80b0980 msgr2=0x7ff7e80b0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:06.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.626+0000 7ff7e6ffd700 1 --2- 172.21.15.111:0/2872149843 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7e80b0980 0x7ff7e80b0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.626+0000 7ff7e6ffd700 1 -- 172.21.15.111:0/2872149843 >> v1:172.21.15.149:6789/0 conn(0x7ff7e80b1330 legacy=0x7ff7e80add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.626+0000 7ff7e5ffb700 1 -- 172.21.15.111:0/2872149843 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183630197 0 0) 0x7ff7e80b0210 con 0x7ff7e80b1330 2023-11-28T13:00:06.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.626+0000 7ff7e6ffd700 1 -- 172.21.15.111:0/2872149843 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e80b04f0 con 0x7ff7e80ad980 2023-11-28T13:00:06.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.626+0000 7ff7e67fc700 1 --2- 172.21.15.111:0/2872149843 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7e80b0980 0x7ff7e80b0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:06.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.626+0000 7ff7e6ffd700 1 --2- 172.21.15.111:0/2872149843 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7e80ad980 0x7ff7e80d06d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009840 tx=0x7ff7dc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6f25bda67122582 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:06.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.626+0000 7ff7e5ffb700 1 -- 172.21.15.111:0/2872149843 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc011830 con 0x7ff7e80ad980 2023-11-28T13:00:06.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.626+0000 7ff7e5ffb700 1 -- 172.21.15.111:0/2872149843 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7dc011990 con 0x7ff7e80ad980 2023-11-28T13:00:06.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.626+0000 7ff7e5ffb700 1 -- 172.21.15.111:0/2872149843 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc011d20 con 0x7ff7e80ad980 2023-11-28T13:00:06.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.627+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/2872149843 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7e80ad980 msgr2=0x7ff7e80d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.627+0000 7ff7ed3a2700 1 --2- 172.21.15.111:0/2872149843 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7e80ad980 0x7ff7e80d06d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009840 tx=0x7ff7dc009b50 comp rx=0 tx=0).stop 2023-11-28T13:00:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.627+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/2872149843 shutdown_connections 2023-11-28T13:00:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.627+0000 7ff7ed3a2700 1 --2- 172.21.15.111:0/2872149843 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7e80ad980 0x7ff7e80d06d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.627+0000 7ff7ed3a2700 1 --2- 172.21.15.111:0/2872149843 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7e80b0980 0x7ff7e80b0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.627+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/2872149843 >> 172.21.15.111:0/2872149843 conn(0x7ff7e809a020 msgr2=0x7ff7e80d0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.627+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/2872149843 shutdown_connections 2023-11-28T13:00:06.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.627+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/2872149843 wait complete. 2023-11-28T13:00:06.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.628+0000 7ff7ed3a2700 1 Processor -- start 2023-11-28T13:00:06.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.628+0000 7ff7ed3a2700 1 -- start start 2023-11-28T13:00:06.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7ed3a2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7e80ad980 0x7ff7e8158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:06.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7ed3a2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7e80b0980 0x7ff7e814e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:06.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7ed3a2700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7e80b1330 0x7ff7e8150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7ed3a2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e81589f0 con 0x7ff7e80ad980 2023-11-28T13:00:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7ed3a2700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e8157de0 con 0x7ff7e80b1330 2023-11-28T13:00:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7ed3a2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e81580c0 con 0x7ff7e80b0980 2023-11-28T13:00:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7e6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7e80ad980 0x7ff7e8158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7e6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7e80ad980 0x7ff7e8158e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35016/0 (socket says 172.21.15.111:35016) 2023-11-28T13:00:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7e6ffd700 1 -- 172.21.15.111:0/3199767421 learned_addr learned my addr 172.21.15.111:0/3199767421 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7e77fe700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7e80b1330 0x7ff7e8150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7e67fc700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7e80b0980 0x7ff7e814e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7e6ffd700 1 -- 172.21.15.111:0/3199767421 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7e80b0980 msgr2=0x7ff7e814e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7e6ffd700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7e80b0980 0x7ff7e814e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7e6ffd700 1 -- 172.21.15.111:0/3199767421 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7e80b1330 msgr2=0x7ff7e8150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7e6ffd700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7e80b1330 0x7ff7e8150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7e6ffd700 1 -- 172.21.15.111:0/3199767421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d80094a0 con 0x7ff7e80ad980 2023-11-28T13:00:06.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7e6ffd700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7e80ad980 0x7ff7e8158e80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009fd0 tx=0x7ff7dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:06.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.629+0000 7ff7cf7fe700 1 -- 172.21.15.111:0/3199767421 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc004550 con 0x7ff7e80ad980 2023-11-28T13:00:06.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.630+0000 7ff7cf7fe700 1 -- 172.21.15.111:0/3199767421 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7dc01f070 con 0x7ff7e80ad980 2023-11-28T13:00:06.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.630+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/3199767421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7dc0094a0 con 0x7ff7e80ad980 2023-11-28T13:00:06.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.630+0000 7ff7cf7fe700 1 -- 172.21.15.111:0/3199767421 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc0038e0 con 0x7ff7e80ad980 2023-11-28T13:00:06.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.630+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/3199767421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e80af940 con 0x7ff7e80ad980 2023-11-28T13:00:06.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.630+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/3199767421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e80af660 con 0x7ff7e80ad980 2023-11-28T13:00:06.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.631+0000 7ff7cf7fe700 1 -- 172.21.15.111:0/3199767421 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff7dc00f050 con 0x7ff7e80ad980 2023-11-28T13:00:06.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.631+0000 7ff7cf7fe700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7d40682f0 0x7ff7d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:06.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.631+0000 7ff7cf7fe700 1 -- 172.21.15.111:0/3199767421 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5530+0+0 (secure 0 0 0) 0x7ff7dc021070 con 0x7ff7e80ad980 2023-11-28T13:00:06.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.631+0000 7ff7e67fc700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7d40682f0 0x7ff7d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:06.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.632+0000 7ff7e67fc700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7d40682f0 0x7ff7d406a7a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff7e815a480 tx=0x7ff7d8009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:06.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.635+0000 7ff7cf7fe700 1 -- 172.21.15.111:0/3199767421 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff7dc053c90 con 0x7ff7e80ad980 2023-11-28T13:00:06.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.895+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/3199767421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7ff7e80afcc0 con 0x7ff7e80ad980 2023-11-28T13:00:06.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.900+0000 7ff7cf7fe700 1 -- 172.21.15.111:0/3199767421 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff7dc0572b0 con 0x7ff7e80ad980 2023-11-28T13:00:06.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.902+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/3199767421 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7d40682f0 msgr2=0x7ff7d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.902+0000 7ff7ed3a2700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7d40682f0 0x7ff7d406a7a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff7e815a480 tx=0x7ff7d8009210 comp rx=0 tx=0).stop 2023-11-28T13:00:06.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.902+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/3199767421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7e80ad980 msgr2=0x7ff7e8158e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.902+0000 7ff7ed3a2700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7e80ad980 0x7ff7e8158e80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009fd0 tx=0x7ff7dc004750 comp rx=0 tx=0).stop 2023-11-28T13:00:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.902+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/3199767421 shutdown_connections 2023-11-28T13:00:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.902+0000 7ff7ed3a2700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7d40682f0 0x7ff7d406a7a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.902+0000 7ff7ed3a2700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7e80b1330 0x7ff7e8150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.902+0000 7ff7ed3a2700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7e80ad980 0x7ff7e8158e80 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.902+0000 7ff7ed3a2700 1 --2- 172.21.15.111:0/3199767421 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7e80b0980 0x7ff7e814e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.902+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/3199767421 >> 172.21.15.111:0/3199767421 conn(0x7ff7e809a020 msgr2=0x7ff7e80b17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.902+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/3199767421 shutdown_connections 2023-11-28T13:00:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:06.902+0000 7ff7ed3a2700 1 -- 172.21.15.111:0/3199767421 wait complete. 2023-11-28T13:00:06.915 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-11-28T13:00:07.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.034+0000 7f74d3806700 1 Processor -- start 2023-11-28T13:00:07.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.034+0000 7f74d3806700 1 -- start start 2023-11-28T13:00:07.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d3806700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74cc0b0d70 0x7f74cc1604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d3806700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f74cc0b1140 0x7f74cc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d3806700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74cc15cd90 con 0x7f74cc0b15b0 2023-11-28T13:00:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d3806700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74cc1613e0 con 0x7f74cc0b1140 2023-11-28T13:00:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d3806700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc160b40 con 0x7f74cc0b0d70 2023-11-28T13:00:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d0da1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f74cc0b1140 0x7f74cc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d0da1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f74cc0b1140 0x7f74cc0ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35024/0 (socket says 172.21.15.111:35024) 2023-11-28T13:00:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d0da1700 1 -- 172.21.15.111:0/193844936 learned_addr learned my addr 172.21.15.111:0/193844936 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d15a2700 1 --2- 172.21.15.111:0/193844936 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74cc0b0d70 0x7f74cc1604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d0da1700 1 -- 172.21.15.111:0/193844936 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74cc0b0d70 msgr2=0x7f74cc1604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d0da1700 1 --2- 172.21.15.111:0/193844936 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74cc0b0d70 0x7f74cc1604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d0da1700 1 -- 172.21.15.111:0/193844936 >> v1:172.21.15.149:6789/0 conn(0x7f74cc0b15b0 legacy=0x7f74cc0cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d0da1700 1 -- 172.21.15.111:0/193844936 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc160e20 con 0x7f74cc0b1140 2023-11-28T13:00:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d15a2700 1 --2- 172.21.15.111:0/193844936 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74cc0b0d70 0x7f74cc1604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74d0da1700 1 --2- 172.21.15.111:0/193844936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc0b1140 0x7f74cc0ce6c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f74bc0098f0 tx=0x7f74bc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f0e9023ee00acb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74c3fff700 1 -- 172.21.15.111:0/193844936 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc004230 con 0x7f74cc0b1140 2023-11-28T13:00:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74c3fff700 1 -- 172.21.15.111:0/193844936 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74bc004390 con 0x7f74cc0b1140 2023-11-28T13:00:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.035+0000 7f74c3fff700 1 -- 172.21.15.111:0/193844936 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc0036a0 con 0x7f74cc0b1140 2023-11-28T13:00:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.036+0000 7f74d3806700 1 -- 172.21.15.111:0/193844936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc0b1140 msgr2=0x7f74cc0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.036+0000 7f74d3806700 1 --2- 172.21.15.111:0/193844936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc0b1140 0x7f74cc0ce6c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f74bc0098f0 tx=0x7f74bc009c00 comp rx=0 tx=0).stop 2023-11-28T13:00:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.036+0000 7f74d3806700 1 -- 172.21.15.111:0/193844936 shutdown_connections 2023-11-28T13:00:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.036+0000 7f74d3806700 1 --2- 172.21.15.111:0/193844936 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74cc0b0d70 0x7f74cc1604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.036+0000 7f74d3806700 1 --2- 172.21.15.111:0/193844936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc0b1140 0x7f74cc0ce6c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.036+0000 7f74d3806700 1 -- 172.21.15.111:0/193844936 >> 172.21.15.111:0/193844936 conn(0x7f74cc098ee0 msgr2=0x7f74cc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.036+0000 7f74d3806700 1 -- 172.21.15.111:0/193844936 shutdown_connections 2023-11-28T13:00:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.036+0000 7f74d3806700 1 -- 172.21.15.111:0/193844936 wait complete. 2023-11-28T13:00:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.037+0000 7f74d3806700 1 Processor -- start 2023-11-28T13:00:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.037+0000 7f74d3806700 1 -- start start 2023-11-28T13:00:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d3806700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f74cc0b0d70 0x7f74cc0b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d3806700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74cc0b1140 0x7f74cc0b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d3806700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc0b15b0 0x7f74cc0cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d3806700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc15dd40 con 0x7f74cc0b15b0 2023-11-28T13:00:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d3806700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc0b5490 con 0x7f74cc0b0d70 2023-11-28T13:00:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d3806700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc0b5770 con 0x7f74cc0b1140 2023-11-28T13:00:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d1da3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc0b15b0 0x7f74cc0cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d1da3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc0b15b0 0x7f74cc0cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35034/0 (socket says 172.21.15.111:35034) 2023-11-28T13:00:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d1da3700 1 -- 172.21.15.111:0/3744526630 learned_addr learned my addr 172.21.15.111:0/3744526630 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d15a2700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f74cc0b0d70 0x7f74cc0b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d0da1700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74cc0b1140 0x7f74cc0b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d1da3700 1 -- 172.21.15.111:0/3744526630 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74cc0b1140 msgr2=0x7f74cc0b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d1da3700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74cc0b1140 0x7f74cc0b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d1da3700 1 -- 172.21.15.111:0/3744526630 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f74cc0b0d70 msgr2=0x7f74cc0b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d1da3700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f74cc0b0d70 0x7f74cc0b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d1da3700 1 -- 172.21.15.111:0/3744526630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc009550 con 0x7f74cc0b15b0 2023-11-28T13:00:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d15a2700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f74cc0b0d70 0x7f74cc0b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d0da1700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74cc0b1140 0x7f74cc0b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.038+0000 7f74d1da3700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc0b15b0 0x7f74cc0cb7b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f74c400be60 tx=0x7f74c400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.039+0000 7f74c1ffb700 1 -- 172.21.15.111:0/3744526630 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c400cd90 con 0x7f74cc0b15b0 2023-11-28T13:00:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.039+0000 7f74c1ffb700 1 -- 172.21.15.111:0/3744526630 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c4003a00 con 0x7f74cc0b15b0 2023-11-28T13:00:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.039+0000 7f74d3806700 1 -- 172.21.15.111:0/3744526630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74cc0af340 con 0x7f74cc0b15b0 2023-11-28T13:00:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.039+0000 7f74c1ffb700 1 -- 172.21.15.111:0/3744526630 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4003df0 con 0x7f74cc0b15b0 2023-11-28T13:00:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.039+0000 7f74d3806700 1 -- 172.21.15.111:0/3744526630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74cc161100 con 0x7f74cc0b15b0 2023-11-28T13:00:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.040+0000 7f74d3806700 1 -- 172.21.15.111:0/3744526630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74cc0cec00 con 0x7f74cc0b15b0 2023-11-28T13:00:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.040+0000 7f74c1ffb700 1 -- 172.21.15.111:0/3744526630 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f74c4003f50 con 0x7f74cc0b15b0 2023-11-28T13:00:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.041+0000 7f74c1ffb700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74b00682f0 0x7f74b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.041+0000 7f74c1ffb700 1 -- 172.21.15.111:0/3744526630 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f74c4085ad0 con 0x7f74cc0b15b0 2023-11-28T13:00:07.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.041+0000 7f74d15a2700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74b00682f0 0x7f74b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.041+0000 7f74d15a2700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74b00682f0 0x7f74b006a7a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f74c800aaf0 tx=0x7f74c8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:07.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.045+0000 7f74c1ffb700 1 -- 172.21.15.111:0/3744526630 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f74c404c5e0 con 0x7f74cc0b15b0 2023-11-28T13:00:07.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.301+0000 7f74d3806700 1 -- 172.21.15.111:0/3744526630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f74cc15d830 con 0x7f74cc0b15b0 2023-11-28T13:00:07.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.306+0000 7f74c1ffb700 1 -- 172.21.15.111:0/3744526630 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f74cc15d830 con 0x7f74cc0b15b0 2023-11-28T13:00:07.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.309+0000 7f74d3806700 1 -- 172.21.15.111:0/3744526630 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74b00682f0 msgr2=0x7f74b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.309+0000 7f74d3806700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74b00682f0 0x7f74b006a7a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f74c800aaf0 tx=0x7f74c8008040 comp rx=0 tx=0).stop 2023-11-28T13:00:07.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.309+0000 7f74d3806700 1 -- 172.21.15.111:0/3744526630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc0b15b0 msgr2=0x7f74cc0cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.309+0000 7f74d3806700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc0b15b0 0x7f74cc0cb7b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f74c400be60 tx=0x7f74c400bf40 comp rx=0 tx=0).stop 2023-11-28T13:00:07.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.309+0000 7f74d3806700 1 -- 172.21.15.111:0/3744526630 shutdown_connections 2023-11-28T13:00:07.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.309+0000 7f74d3806700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74cc0b15b0 0x7f74cc0cb7b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.309+0000 7f74d3806700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f74b00682f0 0x7f74b006a7a0 secure :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f74c800aaf0 tx=0x7f74c8008040 comp rx=0 tx=0).stop 2023-11-28T13:00:07.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.309+0000 7f74d3806700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f74cc0b0d70 0x7f74cc0b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.309+0000 7f74d3806700 1 --2- 172.21.15.111:0/3744526630 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74cc0b1140 0x7f74cc0b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.309+0000 7f74d3806700 1 -- 172.21.15.111:0/3744526630 >> 172.21.15.111:0/3744526630 conn(0x7f74cc098ee0 msgr2=0x7f74cc15c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:07.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.309+0000 7f74d3806700 1 -- 172.21.15.111:0/3744526630 shutdown_connections 2023-11-28T13:00:07.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.309+0000 7f74d3806700 1 -- 172.21.15.111:0/3744526630 wait complete. 2023-11-28T13:00:07.322 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-11-28T13:00:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.441+0000 7f4dcee8e700 1 Processor -- start 2023-11-28T13:00:07.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.441+0000 7f4dcee8e700 1 -- start start 2023-11-28T13:00:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.442+0000 7f4dcee8e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4dc80cfc90 0x7f4dc80d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.442+0000 7f4dcee8e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4dc80cc5a0 0x7f4dc80cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.442+0000 7f4dcee8e700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc80d05a0 con 0x7f4dc80cdd20 2023-11-28T13:00:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.442+0000 7f4dcee8e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc80ce1c0 con 0x7f4dc80cfc90 2023-11-28T13:00:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.442+0000 7f4dcee8e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc80ca260 con 0x7f4dc80cc5a0 2023-11-28T13:00:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.442+0000 7f4dccc2a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4dc80cfc90 0x7f4dc80d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.442+0000 7f4dccc2a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4dc80cfc90 0x7f4dc80d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35038/0 (socket says 172.21.15.111:35038) 2023-11-28T13:00:07.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.442+0000 7f4dccc2a700 1 -- 172.21.15.111:0/897970556 learned_addr learned my addr 172.21.15.111:0/897970556 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.442+0000 7f4dc7fff700 1 --2- 172.21.15.111:0/897970556 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4dc80cc5a0 0x7f4dc80cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dccc2a700 1 -- 172.21.15.111:0/897970556 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4dc80cc5a0 msgr2=0x7f4dc80cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dccc2a700 1 --2- 172.21.15.111:0/897970556 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4dc80cc5a0 0x7f4dc80cd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dccc2a700 1 -- 172.21.15.111:0/897970556 >> v1:172.21.15.149:6789/0 conn(0x7f4dc80cdd20 legacy=0x7f4dc815d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dccc2a700 1 -- 172.21.15.111:0/897970556 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc80ccaa0 con 0x7f4dc80cfc90 2023-11-28T13:00:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dccc2a700 1 --2- 172.21.15.111:0/897970556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dc80cfc90 0x7f4dc80d0060 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00bb90 tx=0x7f4dbc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4615895b478493ca server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dc77fe700 1 -- 172.21.15.111:0/897970556 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc00dd50 con 0x7f4dc80cfc90 2023-11-28T13:00:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dc77fe700 1 -- 172.21.15.111:0/897970556 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc010470 con 0x7f4dc80cfc90 2023-11-28T13:00:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dc77fe700 1 -- 172.21.15.111:0/897970556 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc010800 con 0x7f4dc80cfc90 2023-11-28T13:00:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dcee8e700 1 -- 172.21.15.111:0/897970556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dc80cfc90 msgr2=0x7f4dc80d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dcee8e700 1 --2- 172.21.15.111:0/897970556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dc80cfc90 0x7f4dc80d0060 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00bb90 tx=0x7f4dbc00bea0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dcee8e700 1 -- 172.21.15.111:0/897970556 shutdown_connections 2023-11-28T13:00:07.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dcee8e700 1 --2- 172.21.15.111:0/897970556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dc80cfc90 0x7f4dc80d0060 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dcee8e700 1 --2- 172.21.15.111:0/897970556 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4dc80cc5a0 0x7f4dc80cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.443+0000 7f4dcee8e700 1 -- 172.21.15.111:0/897970556 >> 172.21.15.111:0/897970556 conn(0x7f4dc8098ee0 msgr2=0x7f4dc806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:07.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.444+0000 7f4dcee8e700 1 -- 172.21.15.111:0/897970556 shutdown_connections 2023-11-28T13:00:07.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.444+0000 7f4dcee8e700 1 -- 172.21.15.111:0/897970556 wait complete. 2023-11-28T13:00:07.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.445+0000 7f4dcee8e700 1 Processor -- start 2023-11-28T13:00:07.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.445+0000 7f4dcee8e700 1 -- start start 2023-11-28T13:00:07.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcee8e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4dc80cc5a0 0x7f4dc81eefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcee8e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4dc80cdd20 0x7f4dc81ed600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcee8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dc80cfc90 0x7f4dc80b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcee8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc815a980 con 0x7f4dc80cfc90 2023-11-28T13:00:07.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcee8e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc815ac60 con 0x7f4dc80cdd20 2023-11-28T13:00:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcee8e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc815af40 con 0x7f4dc80cc5a0 2023-11-28T13:00:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcd42b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dc80cfc90 0x7f4dc80b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcd42b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dc80cfc90 0x7f4dc80b47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35048/0 (socket says 172.21.15.111:35048) 2023-11-28T13:00:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcd42b700 1 -- 172.21.15.111:0/3242714304 learned_addr learned my addr 172.21.15.111:0/3242714304 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcd42b700 1 -- 172.21.15.111:0/3242714304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4dc80cc5a0 msgr2=0x7f4dc81eefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcd42b700 1 --2- 172.21.15.111:0/3242714304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4dc80cc5a0 0x7f4dc81eefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcd42b700 1 -- 172.21.15.111:0/3242714304 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4dc80cdd20 msgr2=0x7f4dc81ed600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dc7fff700 1 --2- 172.21.15.111:0/3242714304 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4dc80cdd20 0x7f4dc81ed600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcd42b700 1 --2- 172.21.15.111:0/3242714304 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4dc80cdd20 0x7f4dc81ed600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcd42b700 1 -- 172.21.15.111:0/3242714304 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc00b7f0 con 0x7f4dc80cfc90 2023-11-28T13:00:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcd42b700 1 --2- 172.21.15.111:0/3242714304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dc80cfc90 0x7f4dc80b47c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000cc50 tx=0x7f4dc000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:07.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dc57fa700 1 -- 172.21.15.111:0/3242714304 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc000ad80 con 0x7f4dc80cfc90 2023-11-28T13:00:07.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dc57fa700 1 -- 172.21.15.111:0/3242714304 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc000ce50 con 0x7f4dc80cfc90 2023-11-28T13:00:07.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dc57fa700 1 -- 172.21.15.111:0/3242714304 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0003c40 con 0x7f4dc80cfc90 2023-11-28T13:00:07.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcee8e700 1 -- 172.21.15.111:0/3242714304 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc80cd060 con 0x7f4dc80cfc90 2023-11-28T13:00:07.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.446+0000 7f4dcee8e700 1 -- 172.21.15.111:0/3242714304 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc80ccd80 con 0x7f4dc80cfc90 2023-11-28T13:00:07.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.447+0000 7f4dcee8e700 1 -- 172.21.15.111:0/3242714304 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc80cd340 con 0x7f4dc80cfc90 2023-11-28T13:00:07.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.448+0000 7f4dc57fa700 1 -- 172.21.15.111:0/3242714304 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4dc0010050 con 0x7f4dc80cfc90 2023-11-28T13:00:07.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.448+0000 7f4dc57fa700 1 --2- 172.21.15.111:0/3242714304 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4dac0682f0 0x7f4dac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.448+0000 7f4dc57fa700 1 -- 172.21.15.111:0/3242714304 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f4dc00858a0 con 0x7f4dc80cfc90 2023-11-28T13:00:07.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.448+0000 7f4dccc2a700 1 --2- 172.21.15.111:0/3242714304 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4dac0682f0 0x7f4dac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.449+0000 7f4dccc2a700 1 --2- 172.21.15.111:0/3242714304 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4dac0682f0 0x7f4dac06a7a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00b340 tx=0x7f4dbc012000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:07.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.452+0000 7f4dc57fa700 1 -- 172.21.15.111:0/3242714304 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4dc0052310 con 0x7f4dc80cfc90 2023-11-28T13:00:07.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.710+0000 7f4dcee8e700 1 -- 172.21.15.111:0/3242714304 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f4dc80d05a0 con 0x7f4dc80cfc90 2023-11-28T13:00:07.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.715+0000 7f4dc57fa700 1 -- 172.21.15.111:0/3242714304 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4dc80d05a0 con 0x7f4dc80cfc90 2023-11-28T13:00:07.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.717+0000 7f4dcee8e700 1 -- 172.21.15.111:0/3242714304 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4dac0682f0 msgr2=0x7f4dac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.717+0000 7f4dcee8e700 1 --2- 172.21.15.111:0/3242714304 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4dac0682f0 0x7f4dac06a7a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00b340 tx=0x7f4dbc012000 comp rx=0 tx=0).stop 2023-11-28T13:00:07.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.717+0000 7f4dcee8e700 1 -- 172.21.15.111:0/3242714304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dc80cfc90 msgr2=0x7f4dc80b47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.717+0000 7f4dcee8e700 1 --2- 172.21.15.111:0/3242714304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dc80cfc90 0x7f4dc80b47c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000cc50 tx=0x7f4dc000a340 comp rx=0 tx=0).stop 2023-11-28T13:00:07.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.717+0000 7f4dcee8e700 1 -- 172.21.15.111:0/3242714304 shutdown_connections 2023-11-28T13:00:07.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.717+0000 7f4dcee8e700 1 --2- 172.21.15.111:0/3242714304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dc80cfc90 0x7f4dc80b47c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.717+0000 7f4dcee8e700 1 --2- 172.21.15.111:0/3242714304 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4dc80cc5a0 0x7f4dc81eefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.717+0000 7f4dcee8e700 1 --2- 172.21.15.111:0/3242714304 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4dac0682f0 0x7f4dac06a7a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.717+0000 7f4dcee8e700 1 --2- 172.21.15.111:0/3242714304 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4dc80cdd20 0x7f4dc81ed600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.717+0000 7f4dcee8e700 1 -- 172.21.15.111:0/3242714304 >> 172.21.15.111:0/3242714304 conn(0x7f4dc8098ee0 msgr2=0x7f4dc80acc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:07.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.717+0000 7f4dcee8e700 1 -- 172.21.15.111:0/3242714304 shutdown_connections 2023-11-28T13:00:07.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.717+0000 7f4dcee8e700 1 -- 172.21.15.111:0/3242714304 wait complete. 2023-11-28T13:00:07.731 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-11-28T13:00:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.849+0000 7f2dbc457700 1 Processor -- start 2023-11-28T13:00:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.849+0000 7f2dbc457700 1 -- start start 2023-11-28T13:00:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.850+0000 7f2dbc457700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2db40b2130 0x7f2db4160b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.850+0000 7f2dbc457700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2db40b5ca0 0x7f2db40cf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.850+0000 7f2dbc457700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f2db40cfda0 0x7f2db40d3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.850+0000 7f2dbc457700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db40d0220 con 0x7f2db40b5ca0 2023-11-28T13:00:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.850+0000 7f2dbc457700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db4161070 con 0x7f2db40cfda0 2023-11-28T13:00:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.850+0000 7f2dbc457700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db40cdb20 con 0x7f2db40b2130 2023-11-28T13:00:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.850+0000 7f2db99f2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2db40b5ca0 0x7f2db40cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.850+0000 7f2db99f2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2db40b5ca0 0x7f2db40cf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35052/0 (socket says 172.21.15.111:35052) 2023-11-28T13:00:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.850+0000 7f2db99f2700 1 -- 172.21.15.111:0/1789471685 learned_addr learned my addr 172.21.15.111:0/1789471685 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.851+0000 7f2dba9f4700 1 --2- 172.21.15.111:0/1789471685 >> v2:172.21.15.149:3300/0 conn(0x7f2db40cfda0 0x7f2db40d3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.851+0000 7f2dba1f3700 1 --2- 172.21.15.111:0/1789471685 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2db40b2130 0x7f2db4160b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.851+0000 7f2db99f2700 1 -- 172.21.15.111:0/1789471685 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2db40b2130 msgr2=0x7f2db4160b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.851+0000 7f2db99f2700 1 --2- 172.21.15.111:0/1789471685 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2db40b2130 0x7f2db4160b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.851+0000 7f2db99f2700 1 -- 172.21.15.111:0/1789471685 >> v2:172.21.15.149:3300/0 conn(0x7f2db40cfda0 msgr2=0x7f2db40d3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.851+0000 7f2db99f2700 1 --2- 172.21.15.111:0/1789471685 >> v2:172.21.15.149:3300/0 conn(0x7f2db40cfda0 0x7f2db40d3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.851+0000 7f2db99f2700 1 -- 172.21.15.111:0/1789471685 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db40cf320 con 0x7f2db40b5ca0 2023-11-28T13:00:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.851+0000 7f2db99f2700 1 --2- 172.21.15.111:0/1789471685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40b5ca0 0x7f2db40cf860 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2db408c140 tx=0x7f2db0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da918eafaa1c498 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.851+0000 7f2db91f1700 1 -- 172.21.15.111:0/1789471685 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db000b870 con 0x7f2db40b5ca0 2023-11-28T13:00:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.851+0000 7f2db91f1700 1 -- 172.21.15.111:0/1789471685 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db000b9d0 con 0x7f2db40b5ca0 2023-11-28T13:00:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.851+0000 7f2db91f1700 1 -- 172.21.15.111:0/1789471685 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db000bd60 con 0x7f2db40b5ca0 2023-11-28T13:00:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.851+0000 7f2dbc457700 1 -- 172.21.15.111:0/1789471685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40b5ca0 msgr2=0x7f2db40cf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.851+0000 7f2dbc457700 1 --2- 172.21.15.111:0/1789471685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40b5ca0 0x7f2db40cf860 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2db408c140 tx=0x7f2db0009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.852+0000 7f2dbc457700 1 -- 172.21.15.111:0/1789471685 shutdown_connections 2023-11-28T13:00:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.852+0000 7f2dbc457700 1 --2- 172.21.15.111:0/1789471685 >> v2:172.21.15.149:3300/0 conn(0x7f2db40cfda0 0x7f2db40d3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.852+0000 7f2dbc457700 1 --2- 172.21.15.111:0/1789471685 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2db40b2130 0x7f2db4160b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.852+0000 7f2dbc457700 1 --2- 172.21.15.111:0/1789471685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40b5ca0 0x7f2db40cf860 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.852+0000 7f2dbc457700 1 -- 172.21.15.111:0/1789471685 >> 172.21.15.111:0/1789471685 conn(0x7f2db40719a0 msgr2=0x7f2db4071da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.852+0000 7f2dbc457700 1 -- 172.21.15.111:0/1789471685 shutdown_connections 2023-11-28T13:00:07.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.852+0000 7f2dbc457700 1 -- 172.21.15.111:0/1789471685 wait complete. 2023-11-28T13:00:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.853+0000 7f2dbc457700 1 Processor -- start 2023-11-28T13:00:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.853+0000 7f2dbc457700 1 -- start start 2023-11-28T13:00:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dbc457700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40b2130 0x7f2db415a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dbc457700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2db40b5ca0 0x7f2db40b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dbc457700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2db40cfda0 0x7f2db41f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dbc457700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db41e7f30 con 0x7f2db40b2130 2023-11-28T13:00:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dbc457700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db415f700 con 0x7f2db40cfda0 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dbc457700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db41e75d0 con 0x7f2db40b5ca0 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dba1f3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40b2130 0x7f2db415a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dba1f3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40b2130 0x7f2db415a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35058/0 (socket says 172.21.15.111:35058) 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dba1f3700 1 -- 172.21.15.111:0/1127084239 learned_addr learned my addr 172.21.15.111:0/1127084239 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dba9f4700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2db40cfda0 0x7f2db41f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2db99f2700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2db40b5ca0 0x7f2db40b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dba1f3700 1 -- 172.21.15.111:0/1127084239 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2db40b5ca0 msgr2=0x7f2db40b10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dba1f3700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2db40b5ca0 0x7f2db40b10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dba1f3700 1 -- 172.21.15.111:0/1127084239 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2db40cfda0 msgr2=0x7f2db41f2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dba1f3700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2db40cfda0 0x7f2db41f2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dba1f3700 1 -- 172.21.15.111:0/1127084239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db0009550 con 0x7f2db40b2130 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dba1f3700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40b2130 0x7f2db415a370 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2da400c820 tx=0x7f2da400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2daaffd700 1 -- 172.21.15.111:0/1127084239 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4011800 con 0x7f2db40b2130 2023-11-28T13:00:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2daaffd700 1 -- 172.21.15.111:0/1127084239 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da4011960 con 0x7f2db40b2130 2023-11-28T13:00:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dbc457700 1 -- 172.21.15.111:0/1127084239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db40d2630 con 0x7f2db40b2130 2023-11-28T13:00:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2daaffd700 1 -- 172.21.15.111:0/1127084239 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4011d10 con 0x7f2db40b2130 2023-11-28T13:00:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.854+0000 7f2dbc457700 1 -- 172.21.15.111:0/1127084239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db40cd840 con 0x7f2db40b2130 2023-11-28T13:00:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.855+0000 7f2dbc457700 1 -- 172.21.15.111:0/1127084239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db40d2910 con 0x7f2db40b2130 2023-11-28T13:00:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.856+0000 7f2daaffd700 1 -- 172.21.15.111:0/1127084239 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2da4017030 con 0x7f2db40b2130 2023-11-28T13:00:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.856+0000 7f2daaffd700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2d980682f0 0x7f2d9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.856+0000 7f2daaffd700 1 -- 172.21.15.111:0/1127084239 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f2da4085400 con 0x7f2db40b2130 2023-11-28T13:00:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.856+0000 7f2db99f2700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2d980682f0 0x7f2d9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:07.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.856+0000 7f2db99f2700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2d980682f0 0x7f2d9806a7a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2db415a320 tx=0x7f2db0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:07.860+0000 7f2daaffd700 1 -- 172.21.15.111:0/1127084239 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2da4051e70 con 0x7f2db40b2130 2023-11-28T13:00:08.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.116+0000 7f2dbc457700 1 -- 172.21.15.111:0/1127084239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f2db4067170 con 0x7f2db40b2130 2023-11-28T13:00:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.121+0000 7f2daaffd700 1 -- 172.21.15.111:0/1127084239 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f2da4055490 con 0x7f2db40b2130 2023-11-28T13:00:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.123+0000 7f2dbc457700 1 -- 172.21.15.111:0/1127084239 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2d980682f0 msgr2=0x7f2d9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.123+0000 7f2dbc457700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2d980682f0 0x7f2d9806a7a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2db415a320 tx=0x7f2db0008030 comp rx=0 tx=0).stop 2023-11-28T13:00:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.123+0000 7f2dbc457700 1 -- 172.21.15.111:0/1127084239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40b2130 msgr2=0x7f2db415a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.123+0000 7f2dbc457700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40b2130 0x7f2db415a370 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2da400c820 tx=0x7f2da400cbe0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.123+0000 7f2dbc457700 1 -- 172.21.15.111:0/1127084239 shutdown_connections 2023-11-28T13:00:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.123+0000 7f2dbc457700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2d980682f0 0x7f2d9806a7a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.123+0000 7f2dbc457700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2db40cfda0 0x7f2db41f2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.123+0000 7f2dbc457700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40b2130 0x7f2db415a370 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.123+0000 7f2dbc457700 1 --2- 172.21.15.111:0/1127084239 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2db40b5ca0 0x7f2db40b10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.123+0000 7f2dbc457700 1 -- 172.21.15.111:0/1127084239 >> 172.21.15.111:0/1127084239 conn(0x7f2db40719a0 msgr2=0x7f2db40d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.123+0000 7f2dbc457700 1 -- 172.21.15.111:0/1127084239 shutdown_connections 2023-11-28T13:00:08.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.123+0000 7f2dbc457700 1 -- 172.21.15.111:0/1127084239 wait complete. 2023-11-28T13:00:08.136 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:00:08.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.254+0000 7f3ed8fdd700 1 Processor -- start 2023-11-28T13:00:08.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.254+0000 7f3ed8fdd700 1 -- start start 2023-11-28T13:00:08.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.255+0000 7f3ed8fdd700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f3ed415a6a0 0x7f3ed406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.255+0000 7f3ed8fdd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ed40af160 0x7f3ed406a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.255+0000 7f3ed8fdd700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed415aa70 con 0x7f3ed40cfb30 2023-11-28T13:00:08.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.255+0000 7f3ed8fdd700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ed40ad280 con 0x7f3ed415a6a0 2023-11-28T13:00:08.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.255+0000 7f3ed8fdd700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed40ac950 con 0x7f3ed40af160 2023-11-28T13:00:08.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.256+0000 7f3ed1d9b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ed40af160 0x7f3ed406a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.256+0000 7f3ed259c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f3ed415a6a0 0x7f3ed406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.256+0000 7f3ed2d9d700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f3ed40cfb30 0x7f3ed40cffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:44390/0 (socket says 172.21.15.111:44390) 2023-11-28T13:00:08.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.256+0000 7f3ed259c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f3ed415a6a0 0x7f3ed406a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:52966/0 (socket says 172.21.15.111:52966) 2023-11-28T13:00:08.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.256+0000 7f3ed1d9b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ed40af160 0x7f3ed406a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:56816/0 (socket says 172.21.15.111:56816) 2023-11-28T13:00:08.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.256+0000 7f3ed259c700 1 -- 172.21.15.111:0/2410326311 learned_addr learned my addr 172.21.15.111:0/2410326311 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:08.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.256+0000 7f3ed259c700 1 -- 172.21.15.111:0/2410326311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ed40af160 msgr2=0x7f3ed406a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:08.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.256+0000 7f3ed259c700 1 --2- 172.21.15.111:0/2410326311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ed40af160 0x7f3ed406a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.256+0000 7f3ed259c700 1 -- 172.21.15.111:0/2410326311 >> v1:172.21.15.149:6789/0 conn(0x7f3ed40cfb30 legacy=0x7f3ed40cffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.256+0000 7f3ed259c700 1 -- 172.21.15.111:0/2410326311 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed40acc30 con 0x7f3ed415a6a0 2023-11-28T13:00:08.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.256+0000 7f3ed159a700 1 -- 172.21.15.111:0/2410326311 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393966893 0 0) 0x7f3ecc001490 con 0x7f3ed40cfb30 2023-11-28T13:00:08.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.256+0000 7f3ed1d9b700 1 --2- 172.21.15.111:0/2410326311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ed40af160 0x7f3ed406a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:08.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.256+0000 7f3ed259c700 1 --2- 172.21.15.111:0/2410326311 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ed415a6a0 0x7f3ed406a270 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc0098f0 tx=0x7f3ebc009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=891881f9ab9a96ca server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:08.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.257+0000 7f3ed159a700 1 -- 172.21.15.111:0/2410326311 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc011830 con 0x7f3ed415a6a0 2023-11-28T13:00:08.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.257+0000 7f3ed159a700 1 -- 172.21.15.111:0/2410326311 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ebc011990 con 0x7f3ed415a6a0 2023-11-28T13:00:08.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.257+0000 7f3ed159a700 1 -- 172.21.15.111:0/2410326311 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc011d20 con 0x7f3ed415a6a0 2023-11-28T13:00:08.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.257+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/2410326311 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ed415a6a0 msgr2=0x7f3ed406a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.257+0000 7f3ed8fdd700 1 --2- 172.21.15.111:0/2410326311 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ed415a6a0 0x7f3ed406a270 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc0098f0 tx=0x7f3ebc009c00 comp rx=0 tx=0).stop 2023-11-28T13:00:08.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.257+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/2410326311 shutdown_connections 2023-11-28T13:00:08.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.257+0000 7f3ed8fdd700 1 --2- 172.21.15.111:0/2410326311 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ed415a6a0 0x7f3ed406a270 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.257+0000 7f3ed8fdd700 1 --2- 172.21.15.111:0/2410326311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ed40af160 0x7f3ed406a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.257+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/2410326311 >> 172.21.15.111:0/2410326311 conn(0x7f3ed4098ee0 msgr2=0x7f3ed406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:08.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.257+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/2410326311 shutdown_connections 2023-11-28T13:00:08.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.257+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/2410326311 wait complete. 2023-11-28T13:00:08.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.258+0000 7f3ed8fdd700 1 Processor -- start 2023-11-28T13:00:08.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.259+0000 7f3ed8fdd700 1 -- start start 2023-11-28T13:00:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.259+0000 7f3ed8fdd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ed40af160 0x7f3ed40b55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.259+0000 7f3ed8fdd700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ed40cfb30 0x7f3ed4153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.259+0000 7f3ed8fdd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ed415a6a0 0x7f3ed40c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.259+0000 7f3ed8fdd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed415bb90 con 0x7f3ed40af160 2023-11-28T13:00:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.259+0000 7f3ed8fdd700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed415c810 con 0x7f3ed40cfb30 2023-11-28T13:00:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.259+0000 7f3ed8fdd700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed40b5b10 con 0x7f3ed415a6a0 2023-11-28T13:00:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.259+0000 7f3ed259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ed40af160 0x7f3ed40b55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.259+0000 7f3ed259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ed40af160 0x7f3ed40b55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35072/0 (socket says 172.21.15.111:35072) 2023-11-28T13:00:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.259+0000 7f3ed259c700 1 -- 172.21.15.111:0/3283682042 learned_addr learned my addr 172.21.15.111:0/3283682042 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.260+0000 7f3ed2d9d700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ed415a6a0 0x7f3ed40c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.260+0000 7f3ed1d9b700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ed40cfb30 0x7f3ed4153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.260+0000 7f3ed259c700 1 -- 172.21.15.111:0/3283682042 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ed415a6a0 msgr2=0x7f3ed40c6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.260+0000 7f3ed259c700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ed415a6a0 0x7f3ed40c6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.260+0000 7f3ed259c700 1 -- 172.21.15.111:0/3283682042 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ed40cfb30 msgr2=0x7f3ed4153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.260+0000 7f3ed259c700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ed40cfb30 0x7f3ed4153300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.260+0000 7f3ed259c700 1 -- 172.21.15.111:0/3283682042 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec40094a0 con 0x7f3ed40af160 2023-11-28T13:00:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.260+0000 7f3ed259c700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ed40af160 0x7f3ed40b55d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009130 tx=0x7f3ebc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.260+0000 7f3ecaffd700 1 -- 172.21.15.111:0/3283682042 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc0036a0 con 0x7f3ed40af160 2023-11-28T13:00:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.260+0000 7f3ecaffd700 1 -- 172.21.15.111:0/3283682042 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ebc01e070 con 0x7f3ed40af160 2023-11-28T13:00:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.260+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/3283682042 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ebc009550 con 0x7f3ed40af160 2023-11-28T13:00:08.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.260+0000 7f3ecaffd700 1 -- 172.21.15.111:0/3283682042 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc003a20 con 0x7f3ed40af160 2023-11-28T13:00:08.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.260+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/3283682042 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed40992e0 con 0x7f3ed40af160 2023-11-28T13:00:08.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.261+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/3283682042 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed40acf10 con 0x7f3ed40af160 2023-11-28T13:00:08.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.261+0000 7f3ecaffd700 1 -- 172.21.15.111:0/3283682042 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3ebc00f050 con 0x7f3ed40af160 2023-11-28T13:00:08.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.262+0000 7f3ecaffd700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3eb40682f0 0x7f3eb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.262+0000 7f3ed1d9b700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3eb40682f0 0x7f3eb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.262+0000 7f3ecaffd700 1 -- 172.21.15.111:0/3283682042 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f3ebc086a50 con 0x7f3ed40af160 2023-11-28T13:00:08.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.262+0000 7f3ed1d9b700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3eb40682f0 0x7f3eb406a7a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3ed40ca100 tx=0x7f3ec4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:08.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.266+0000 7f3ecaffd700 1 -- 172.21.15.111:0/3283682042 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3ebc0534c0 con 0x7f3ed40af160 2023-11-28T13:00:08.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.449+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/3283682042 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ed40aec10 con 0x7f3ed40af160 2023-11-28T13:00:08.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.450+0000 7f3ecaffd700 1 -- 172.21.15.111:0/3283682042 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+14520 (secure 0 0 0) 0x7f3ebc015090 con 0x7f3ed40af160 2023-11-28T13:00:08.451 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:08.451 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":37,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:05.246979+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:00:08.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.452+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/3283682042 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3eb40682f0 msgr2=0x7f3eb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.452+0000 7f3ed8fdd700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3eb40682f0 0x7f3eb406a7a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3ed40ca100 tx=0x7f3ec4008040 comp rx=0 tx=0).stop 2023-11-28T13:00:08.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.452+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/3283682042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ed40af160 msgr2=0x7f3ed40b55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.452+0000 7f3ed8fdd700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ed40af160 0x7f3ed40b55d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009130 tx=0x7f3ebc004750 comp rx=0 tx=0).stop 2023-11-28T13:00:08.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.452+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/3283682042 shutdown_connections 2023-11-28T13:00:08.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.452+0000 7f3ed8fdd700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3ed415a6a0 0x7f3ed40c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.452+0000 7f3ed8fdd700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ed40af160 0x7f3ed40b55d0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.452+0000 7f3ed8fdd700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3eb40682f0 0x7f3eb406a7a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.452+0000 7f3ed8fdd700 1 --2- 172.21.15.111:0/3283682042 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3ed40cfb30 0x7f3ed4153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.452+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/3283682042 >> 172.21.15.111:0/3283682042 conn(0x7f3ed4098ee0 msgr2=0x7f3ed40b0370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:08.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.452+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/3283682042 shutdown_connections 2023-11-28T13:00:08.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.452+0000 7f3ed8fdd700 1 -- 172.21.15.111:0/3283682042 wait complete. 2023-11-28T13:00:08.465 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T13:00:08.466 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T13:00:08.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.587+0000 7f3f67682700 1 Processor -- start 2023-11-28T13:00:08.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.587+0000 7f3f67682700 1 -- start start 2023-11-28T13:00:08.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.587+0000 7f3f67682700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f3f60159960 0x7f3f600b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f67682700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3f600aecc0 0x7f3f600cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f67682700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f600ce020 con 0x7f3f600cdbc0 2023-11-28T13:00:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f67682700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f60159d30 con 0x7f3f600aecc0 2023-11-28T13:00:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f67682700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f600ad420 con 0x7f3f60159960 2023-11-28T13:00:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f64c1d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3f600aecc0 0x7f3f600cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f65c1f700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3f600cdbc0 0x7f3f600afa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35862/0 (socket says 172.21.15.111:35862) 2023-11-28T13:00:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f65c1f700 1 -- 172.21.15.111:0/2654065192 learned_addr learned my addr 172.21.15.111:0/2654065192 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f6541e700 1 --2- 172.21.15.111:0/2654065192 >> v2:172.21.15.149:3300/0 conn(0x7f3f60159960 0x7f3f600b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f64c1d700 1 -- 172.21.15.111:0/2654065192 >> v2:172.21.15.149:3300/0 conn(0x7f3f60159960 msgr2=0x7f3f600b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f64c1d700 1 --2- 172.21.15.111:0/2654065192 >> v2:172.21.15.149:3300/0 conn(0x7f3f60159960 0x7f3f600b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f64c1d700 1 -- 172.21.15.111:0/2654065192 >> v1:172.21.15.111:6789/0 conn(0x7f3f600cdbc0 legacy=0x7f3f600afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f64c1d700 1 -- 172.21.15.111:0/2654065192 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f600ac810 con 0x7f3f600aecc0 2023-11-28T13:00:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f57fff700 1 -- 172.21.15.111:0/2654065192 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240759355 0 0) 0x7f3f600ce020 con 0x7f3f600cdbc0 2023-11-28T13:00:08.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f64c1d700 1 --2- 172.21.15.111:0/2654065192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f600aecc0 0x7f3f600cd680 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3f6006c6a0 tx=0x7f3f5c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9240298c1bd10807 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f57fff700 1 -- 172.21.15.111:0/2654065192 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c011830 con 0x7f3f600aecc0 2023-11-28T13:00:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f57fff700 1 -- 172.21.15.111:0/2654065192 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5c011990 con 0x7f3f600aecc0 2023-11-28T13:00:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.588+0000 7f3f57fff700 1 -- 172.21.15.111:0/2654065192 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c011d20 con 0x7f3f600aecc0 2023-11-28T13:00:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.589+0000 7f3f67682700 1 -- 172.21.15.111:0/2654065192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f600aecc0 msgr2=0x7f3f600cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.589+0000 7f3f67682700 1 --2- 172.21.15.111:0/2654065192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f600aecc0 0x7f3f600cd680 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3f6006c6a0 tx=0x7f3f5c009b20 comp rx=0 tx=0).stop 2023-11-28T13:00:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.589+0000 7f3f67682700 1 -- 172.21.15.111:0/2654065192 shutdown_connections 2023-11-28T13:00:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.589+0000 7f3f67682700 1 --2- 172.21.15.111:0/2654065192 >> v2:172.21.15.149:3300/0 conn(0x7f3f60159960 0x7f3f600b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.589+0000 7f3f67682700 1 --2- 172.21.15.111:0/2654065192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f600aecc0 0x7f3f600cd680 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.589+0000 7f3f67682700 1 -- 172.21.15.111:0/2654065192 >> 172.21.15.111:0/2654065192 conn(0x7f3f60098ee0 msgr2=0x7f3f6006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.589+0000 7f3f67682700 1 -- 172.21.15.111:0/2654065192 shutdown_connections 2023-11-28T13:00:08.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.589+0000 7f3f67682700 1 -- 172.21.15.111:0/2654065192 wait complete. 2023-11-28T13:00:08.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.590+0000 7f3f67682700 1 Processor -- start 2023-11-28T13:00:08.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.590+0000 7f3f67682700 1 -- start start 2023-11-28T13:00:08.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f67682700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3f600aecc0 0x7f3f60147c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f67682700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f600cdbc0 0x7f3f60148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f67682700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3f60159960 0x7f3f600c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f67682700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f601494b0 con 0x7f3f600cdbc0 2023-11-28T13:00:08.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f67682700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f60148300 con 0x7f3f600aecc0 2023-11-28T13:00:08.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f67682700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f601485e0 con 0x7f3f60159960 2023-11-28T13:00:08.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f64c1d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f600cdbc0 0x7f3f60148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f64c1d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f600cdbc0 0x7f3f60148f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35088/0 (socket says 172.21.15.111:35088) 2023-11-28T13:00:08.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f64c1d700 1 -- 172.21.15.111:0/1085255419 learned_addr learned my addr 172.21.15.111:0/1085255419 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:08.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f6541e700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3f600aecc0 0x7f3f60147c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f65c1f700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3f60159960 0x7f3f600c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f64c1d700 1 -- 172.21.15.111:0/1085255419 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3f60159960 msgr2=0x7f3f600c5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f64c1d700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3f60159960 0x7f3f600c5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f64c1d700 1 -- 172.21.15.111:0/1085255419 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3f600aecc0 msgr2=0x7f3f60147c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f64c1d700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3f600aecc0 0x7f3f60147c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f64c1d700 1 -- 172.21.15.111:0/1085255419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c0094a0 con 0x7f3f600cdbc0 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.591+0000 7f3f64c1d700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f600cdbc0 0x7f3f60148f70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c0090b0 tx=0x7f3f5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.592+0000 7f3f55ffb700 1 -- 172.21.15.111:0/1085255419 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c0038a0 con 0x7f3f600cdbc0 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.592+0000 7f3f55ffb700 1 -- 172.21.15.111:0/1085255419 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5c01f070 con 0x7f3f600cdbc0 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.592+0000 7f3f67682700 1 -- 172.21.15.111:0/1085255419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f600acdd0 con 0x7f3f600cdbc0 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.592+0000 7f3f55ffb700 1 -- 172.21.15.111:0/1085255419 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c003c20 con 0x7f3f600cdbc0 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.592+0000 7f3f67682700 1 -- 172.21.15.111:0/1085255419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f600acaf0 con 0x7f3f600cdbc0 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.593+0000 7f3f67682700 1 -- 172.21.15.111:0/1085255419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f600ad0b0 con 0x7f3f600cdbc0 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.593+0000 7f3f55ffb700 1 -- 172.21.15.111:0/1085255419 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3f5c00f050 con 0x7f3f600cdbc0 2023-11-28T13:00:08.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.593+0000 7f3f55ffb700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3f440682f0 0x7f3f4406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.594+0000 7f3f55ffb700 1 -- 172.21.15.111:0/1085255419 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f3f5c021070 con 0x7f3f600cdbc0 2023-11-28T13:00:08.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.594+0000 7f3f6541e700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3f440682f0 0x7f3f4406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.594+0000 7f3f6541e700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3f440682f0 0x7f3f4406a7a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3f600ae290 tx=0x7f3f50008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:08.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.598+0000 7f3f55ffb700 1 -- 172.21.15.111:0/1085255419 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3f5c053ee0 con 0x7f3f600cdbc0 2023-11-28T13:00:08.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.777+0000 7f3f67682700 1 -- 172.21.15.111:0/1085255419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f6006c4f0 con 0x7f3f600cdbc0 2023-11-28T13:00:08.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.777+0000 7f3f55ffb700 1 -- 172.21.15.111:0/1085255419 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f5c057500 con 0x7f3f600cdbc0 2023-11-28T13:00:08.779 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T13:00:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.779+0000 7f3f67682700 1 -- 172.21.15.111:0/1085255419 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3f440682f0 msgr2=0x7f3f4406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.779+0000 7f3f67682700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3f440682f0 0x7f3f4406a7a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3f600ae290 tx=0x7f3f50008040 comp rx=0 tx=0).stop 2023-11-28T13:00:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.779+0000 7f3f67682700 1 -- 172.21.15.111:0/1085255419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f600cdbc0 msgr2=0x7f3f60148f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.779+0000 7f3f67682700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f600cdbc0 0x7f3f60148f70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c0090b0 tx=0x7f3f5c004750 comp rx=0 tx=0).stop 2023-11-28T13:00:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.779+0000 7f3f67682700 1 -- 172.21.15.111:0/1085255419 shutdown_connections 2023-11-28T13:00:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.779+0000 7f3f67682700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3f60159960 0x7f3f600c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.779+0000 7f3f67682700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3f440682f0 0x7f3f4406a7a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.779+0000 7f3f67682700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3f600aecc0 0x7f3f60147c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.779+0000 7f3f67682700 1 --2- 172.21.15.111:0/1085255419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f600cdbc0 0x7f3f60148f70 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.779+0000 7f3f67682700 1 -- 172.21.15.111:0/1085255419 >> 172.21.15.111:0/1085255419 conn(0x7f3f60098ee0 msgr2=0x7f3f600ae4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.779+0000 7f3f67682700 1 -- 172.21.15.111:0/1085255419 shutdown_connections 2023-11-28T13:00:08.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.779+0000 7f3f67682700 1 -- 172.21.15.111:0/1085255419 wait complete. 2023-11-28T13:00:08.793 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T13:00:08.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.914+0000 7f9cd4788700 1 Processor -- start 2023-11-28T13:00:08.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.914+0000 7f9cd4788700 1 -- start start 2023-11-28T13:00:08.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.915+0000 7f9cd4788700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9ccc15c6c0 0x7f9ccc0afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.915+0000 7f9cd4788700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9ccc0cb300 0x7f9ccc0b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.915+0000 7f9cd4788700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ccc0cc750 con 0x7f9ccc0b0440 2023-11-28T13:00:08.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.915+0000 7f9cd4788700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ccc0b0e20 con 0x7f9ccc0cb300 2023-11-28T13:00:08.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.915+0000 7f9cd4788700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ccc0cca30 con 0x7f9ccc15c6c0 2023-11-28T13:00:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.915+0000 7f9cd2d25700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9ccc0cb300 0x7f9ccc0b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.915+0000 7f9cd1d23700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9ccc0b0440 0x7f9ccc1602a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35872/0 (socket says 172.21.15.111:35872) 2023-11-28T13:00:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.915+0000 7f9cd1d23700 1 -- 172.21.15.111:0/155186718 learned_addr learned my addr 172.21.15.111:0/155186718 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.916+0000 7f9cd2524700 1 --2- 172.21.15.111:0/155186718 >> v2:172.21.15.149:3300/0 conn(0x7f9ccc15c6c0 0x7f9ccc0afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.916+0000 7f9cd2d25700 1 -- 172.21.15.111:0/155186718 >> v2:172.21.15.149:3300/0 conn(0x7f9ccc15c6c0 msgr2=0x7f9ccc0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.916+0000 7f9cd2d25700 1 --2- 172.21.15.111:0/155186718 >> v2:172.21.15.149:3300/0 conn(0x7f9ccc15c6c0 0x7f9ccc0afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.916+0000 7f9cd2d25700 1 -- 172.21.15.111:0/155186718 >> v1:172.21.15.111:6789/0 conn(0x7f9ccc0b0440 legacy=0x7f9ccc1602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.916+0000 7f9cd1522700 1 -- 172.21.15.111:0/155186718 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576822092 0 0) 0x7f9ccc0cc750 con 0x7f9ccc0b0440 2023-11-28T13:00:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.916+0000 7f9cd2d25700 1 -- 172.21.15.111:0/155186718 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc0ccd10 con 0x7f9ccc0cb300 2023-11-28T13:00:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.916+0000 7f9cd2d25700 1 --2- 172.21.15.111:0/155186718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ccc0cb300 0x7f9ccc0b08e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc071950 tx=0x7f9cc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5bf33b160f439c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.916+0000 7f9cd1522700 1 -- 172.21.15.111:0/155186718 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc400e070 con 0x7f9ccc0cb300 2023-11-28T13:00:08.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.916+0000 7f9cd1522700 1 -- 172.21.15.111:0/155186718 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc400bbd0 con 0x7f9ccc0cb300 2023-11-28T13:00:08.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.916+0000 7f9cd1522700 1 -- 172.21.15.111:0/155186718 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4012430 con 0x7f9ccc0cb300 2023-11-28T13:00:08.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.916+0000 7f9cd4788700 1 -- 172.21.15.111:0/155186718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ccc0cb300 msgr2=0x7f9ccc0b08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.916+0000 7f9cd4788700 1 --2- 172.21.15.111:0/155186718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ccc0cb300 0x7f9ccc0b08e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc071950 tx=0x7f9cc400b330 comp rx=0 tx=0).stop 2023-11-28T13:00:08.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.917+0000 7f9cd4788700 1 -- 172.21.15.111:0/155186718 shutdown_connections 2023-11-28T13:00:08.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.917+0000 7f9cd4788700 1 --2- 172.21.15.111:0/155186718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ccc0cb300 0x7f9ccc0b08e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.917+0000 7f9cd4788700 1 --2- 172.21.15.111:0/155186718 >> v2:172.21.15.149:3300/0 conn(0x7f9ccc15c6c0 0x7f9ccc0afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.917+0000 7f9cd4788700 1 -- 172.21.15.111:0/155186718 >> 172.21.15.111:0/155186718 conn(0x7f9ccc098ee0 msgr2=0x7f9ccc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:08.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.917+0000 7f9cd4788700 1 -- 172.21.15.111:0/155186718 shutdown_connections 2023-11-28T13:00:08.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.917+0000 7f9cd4788700 1 -- 172.21.15.111:0/155186718 wait complete. 2023-11-28T13:00:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.918+0000 7f9cd4788700 1 Processor -- start 2023-11-28T13:00:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.918+0000 7f9cd4788700 1 -- start start 2023-11-28T13:00:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.918+0000 7f9cd4788700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9ccc0b0440 0x7f9ccc0d0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.918+0000 7f9cd4788700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ccc0cb300 0x7f9ccc154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.918+0000 7f9cd4788700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ccc15c6c0 0x7f9ccc156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd4788700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ccc0b4e20 con 0x7f9ccc0cb300 2023-11-28T13:00:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd4788700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ccc0cea20 con 0x7f9ccc15c6c0 2023-11-28T13:00:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd4788700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ccc0ced00 con 0x7f9ccc0b0440 2023-11-28T13:00:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd1d23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ccc0cb300 0x7f9ccc154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd1d23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ccc0cb300 0x7f9ccc154cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35100/0 (socket says 172.21.15.111:35100) 2023-11-28T13:00:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd1d23700 1 -- 172.21.15.111:0/3791473674 learned_addr learned my addr 172.21.15.111:0/3791473674 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd2524700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9ccc0b0440 0x7f9ccc0d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd2d25700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ccc15c6c0 0x7f9ccc156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd1d23700 1 -- 172.21.15.111:0/3791473674 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9ccc0b0440 msgr2=0x7f9ccc0d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd1d23700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9ccc0b0440 0x7f9ccc0d0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd1d23700 1 -- 172.21.15.111:0/3791473674 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ccc15c6c0 msgr2=0x7f9ccc156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd1d23700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ccc15c6c0 0x7f9ccc156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd1d23700 1 -- 172.21.15.111:0/3791473674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc4009a70 con 0x7f9ccc0cb300 2023-11-28T13:00:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd1d23700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ccc0cb300 0x7f9ccc154cb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800c550 tx=0x7f9cc800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cc2ffd700 1 -- 172.21.15.111:0/3791473674 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8004c80 con 0x7f9ccc0cb300 2023-11-28T13:00:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cc2ffd700 1 -- 172.21.15.111:0/3791473674 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc8004e00 con 0x7f9ccc0cb300 2023-11-28T13:00:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd4788700 1 -- 172.21.15.111:0/3791473674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ccc0cd2d0 con 0x7f9ccc0cb300 2023-11-28T13:00:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cc2ffd700 1 -- 172.21.15.111:0/3791473674 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8007310 con 0x7f9ccc0cb300 2023-11-28T13:00:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.919+0000 7f9cd4788700 1 -- 172.21.15.111:0/3791473674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ccc0ccff0 con 0x7f9ccc0cb300 2023-11-28T13:00:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.920+0000 7f9cd4788700 1 -- 172.21.15.111:0/3791473674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ccc15ca90 con 0x7f9ccc0cb300 2023-11-28T13:00:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.920+0000 7f9cc2ffd700 1 -- 172.21.15.111:0/3791473674 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9cc801a020 con 0x7f9ccc0cb300 2023-11-28T13:00:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.921+0000 7f9cc2ffd700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9cb00682f0 0x7f9cb006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.921+0000 7f9cc2ffd700 1 -- 172.21.15.111:0/3791473674 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f9cc80856b0 con 0x7f9ccc0cb300 2023-11-28T13:00:08.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.921+0000 7f9cd2524700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9cb00682f0 0x7f9cb006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:08.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.921+0000 7f9cd2524700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9cb00682f0 0x7f9cb006a7a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00b9b0 tx=0x7f9cbc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:08.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:08.925+0000 7f9cc2ffd700 1 -- 172.21.15.111:0/3791473674 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9cc8052170 con 0x7f9ccc0cb300 2023-11-28T13:00:09.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.108+0000 7f9cd4788700 1 -- 172.21.15.111:0/3791473674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9ccc0cdf80 con 0x7f9ccc0cb300 2023-11-28T13:00:09.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.109+0000 7f9cc2ffd700 1 -- 172.21.15.111:0/3791473674 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v37) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9cc8055790 con 0x7f9ccc0cb300 2023-11-28T13:00:09.110 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T13:00:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.111+0000 7f9cd4788700 1 -- 172.21.15.111:0/3791473674 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9cb00682f0 msgr2=0x7f9cb006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.111+0000 7f9cd4788700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9cb00682f0 0x7f9cb006a7a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00b9b0 tx=0x7f9cbc008040 comp rx=0 tx=0).stop 2023-11-28T13:00:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.111+0000 7f9cd4788700 1 -- 172.21.15.111:0/3791473674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ccc0cb300 msgr2=0x7f9ccc154cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.111+0000 7f9cd4788700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ccc0cb300 0x7f9ccc154cb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800c550 tx=0x7f9cc800c910 comp rx=0 tx=0).stop 2023-11-28T13:00:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.111+0000 7f9cd4788700 1 -- 172.21.15.111:0/3791473674 shutdown_connections 2023-11-28T13:00:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.111+0000 7f9cd4788700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9cb00682f0 0x7f9cb006a7a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.111+0000 7f9cd4788700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9ccc15c6c0 0x7f9ccc156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.111+0000 7f9cd4788700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9ccc0b0440 0x7f9ccc0d0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.111+0000 7f9cd4788700 1 --2- 172.21.15.111:0/3791473674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ccc0cb300 0x7f9ccc154cb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.111+0000 7f9cd4788700 1 -- 172.21.15.111:0/3791473674 >> 172.21.15.111:0/3791473674 conn(0x7f9ccc098ee0 msgr2=0x7f9ccc06f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.111+0000 7f9cd4788700 1 -- 172.21.15.111:0/3791473674 shutdown_connections 2023-11-28T13:00:09.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.111+0000 7f9cd4788700 1 -- 172.21.15.111:0/3791473674 wait complete. 2023-11-28T13:00:09.125 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T13:00:09.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.243+0000 7f7d41be9700 1 Processor -- start 2023-11-28T13:00:09.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.243+0000 7f7d41be9700 1 -- start start 2023-11-28T13:00:09.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.243+0000 7f7d41be9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7d3c0af9f0 0x7f7d3c0ca220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:09.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.243+0000 7f7d41be9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3c06ac80 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.243+0000 7f7d41be9700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3c0cee00 con 0x7f7d3c15fed0 2023-11-28T13:00:09.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.243+0000 7f7d41be9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3c0cf9a0 con 0x7f7d3c0af9f0 2023-11-28T13:00:09.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.244+0000 7f7d3bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7d3c0af9f0 0x7f7d3c0ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:09.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.244+0000 7f7d3affd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7d3c0af5a0 0x7f7d3c0b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35886/0 (socket says 172.21.15.111:35886) 2023-11-28T13:00:09.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.244+0000 7f7d3affd700 1 -- 172.21.15.111:0/2432637034 learned_addr learned my addr 172.21.15.111:0/2432637034 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:09.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.244+0000 7f7d3bfff700 1 -- 172.21.15.111:0/2432637034 >> v1:172.21.15.149:6789/0 conn(0x7f7d3c15fed0 legacy=0x7f7d3c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:09.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.244+0000 7f7d3bfff700 1 -- 172.21.15.111:0/2432637034 >> v1:172.21.15.111:6789/0 conn(0x7f7d3c0af5a0 legacy=0x7f7d3c0b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:09.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.244+0000 7f7d3bfff700 1 -- 172.21.15.111:0/2432637034 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c06aed0 con 0x7f7d3c0af9f0 2023-11-28T13:00:09.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.244+0000 7f7d3bfff700 1 --2- 172.21.15.111:0/2432637034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d3c0af9f0 0x7f7d3c0ca220 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009d90 tx=0x7f7d3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0773d1686d6931 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:09.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.244+0000 7f7d3a7fc700 1 -- 172.21.15.111:0/2432637034 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3400e070 con 0x7f7d3c0af9f0 2023-11-28T13:00:09.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.244+0000 7f7d3a7fc700 1 -- 172.21.15.111:0/2432637034 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3400bb90 con 0x7f7d3c0af9f0 2023-11-28T13:00:09.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.244+0000 7f7d3a7fc700 1 -- 172.21.15.111:0/2432637034 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34003a20 con 0x7f7d3c0af9f0 2023-11-28T13:00:09.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.245+0000 7f7d41be9700 1 -- 172.21.15.111:0/2432637034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d3c0af9f0 msgr2=0x7f7d3c0ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:09.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.245+0000 7f7d41be9700 1 --2- 172.21.15.111:0/2432637034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d3c0af9f0 0x7f7d3c0ca220 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009d90 tx=0x7f7d3400b330 comp rx=0 tx=0).stop 2023-11-28T13:00:09.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.245+0000 7f7d41be9700 1 -- 172.21.15.111:0/2432637034 shutdown_connections 2023-11-28T13:00:09.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.245+0000 7f7d41be9700 1 --2- 172.21.15.111:0/2432637034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d3c0af9f0 0x7f7d3c0ca220 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.245+0000 7f7d41be9700 1 -- 172.21.15.111:0/2432637034 >> 172.21.15.111:0/2432637034 conn(0x7f7d3c06bca0 msgr2=0x7f7d3c0cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:09.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.245+0000 7f7d41be9700 1 -- 172.21.15.111:0/2432637034 shutdown_connections 2023-11-28T13:00:09.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.245+0000 7f7d41be9700 1 -- 172.21.15.111:0/2432637034 wait complete. 2023-11-28T13:00:09.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.246+0000 7f7d41be9700 1 Processor -- start 2023-11-28T13:00:09.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.246+0000 7f7d41be9700 1 -- start start 2023-11-28T13:00:09.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d41be9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d3c0af5a0 0x7f7d3c0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:09.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d41be9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d3c0af9f0 0x7f7d3c1551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d41be9700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d3c15fed0 0x7f7d3c0b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d41be9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c0b4040 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d41be9700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c1556e0 con 0x7f7d3c15fed0 2023-11-28T13:00:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d41be9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c0b3430 con 0x7f7d3c0af9f0 2023-11-28T13:00:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d3b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d3c0af5a0 0x7f7d3c0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d3b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d3c0af5a0 0x7f7d3c0b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35112/0 (socket says 172.21.15.111:35112) 2023-11-28T13:00:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d3b7fe700 1 -- 172.21.15.111:0/2750208204 learned_addr learned my addr 172.21.15.111:0/2750208204 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d3b7fe700 1 -- 172.21.15.111:0/2750208204 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d3c0af9f0 msgr2=0x7f7d3c1551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:09.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d3b7fe700 1 --2- 172.21.15.111:0/2750208204 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d3c0af9f0 0x7f7d3c1551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d3b7fe700 1 -- 172.21.15.111:0/2750208204 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d3c15fed0 msgr2=0x7f7d3c0b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d3b7fe700 1 --2- 172.21.15.111:0/2750208204 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d3c15fed0 0x7f7d3c0b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d3b7fe700 1 -- 172.21.15.111:0/2750208204 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d34009a70 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d3b7fe700 1 --2- 172.21.15.111:0/2750208204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d3c0af5a0 0x7f7d3c0b44d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00b4a0 tx=0x7f7d2c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d33fff700 1 -- 172.21.15.111:0/2750208204 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c011830 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d33fff700 1 -- 172.21.15.111:0/2750208204 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c011990 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d33fff700 1 -- 172.21.15.111:0/2750208204 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c011d40 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d41be9700 1 -- 172.21.15.111:0/2750208204 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3c1602a0 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.247+0000 7f7d41be9700 1 -- 172.21.15.111:0/2750208204 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3c0cea90 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.248+0000 7f7d41be9700 1 -- 172.21.15.111:0/2750208204 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3c0ce6b0 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.249+0000 7f7d33fff700 1 -- 172.21.15.111:0/2750208204 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7d2c00f050 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.249+0000 7f7d33fff700 1 --2- 172.21.15.111:0/2750208204 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7d1c0682f0 0x7f7d1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:09.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.249+0000 7f7d33fff700 1 -- 172.21.15.111:0/2750208204 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f7d2c085670 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.249+0000 7f7d3affd700 1 --2- 172.21.15.111:0/2750208204 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7d1c0682f0 0x7f7d1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:09.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.250+0000 7f7d3affd700 1 --2- 172.21.15.111:0/2750208204 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7d1c0682f0 0x7f7d1c06a7a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7d24000f60 tx=0x7f7d24007b10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:09.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.253+0000 7f7d33fff700 1 -- 172.21.15.111:0/2750208204 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7d2c0520e0 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.443+0000 7f7d41be9700 1 -- 172.21.15.111:0/2750208204 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7d3c067170 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.444+0000 7f7d33fff700 1 -- 172.21.15.111:0/2750208204 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v37) v1 ==== 84+0+126 (secure 0 0 0) 0x7f7d2c015090 con 0x7f7d3c0af5a0 2023-11-28T13:00:09.445 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:09.445 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-11-28T13:00:09.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:09.446 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-11-28T13:00:09.446 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-11-28T13:00:09.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:09.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:09.446 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 2, 2023-11-28T13:00:09.446 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": "rbd" 2023-11-28T13:00:09.446 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:09.446 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-11-28T13:00:09.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.446+0000 7f7d41be9700 1 -- 172.21.15.111:0/2750208204 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7d1c0682f0 msgr2=0x7f7d1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:09.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.446+0000 7f7d41be9700 1 --2- 172.21.15.111:0/2750208204 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7d1c0682f0 0x7f7d1c06a7a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7d24000f60 tx=0x7f7d24007b10 comp rx=0 tx=0).stop 2023-11-28T13:00:09.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.446+0000 7f7d41be9700 1 -- 172.21.15.111:0/2750208204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d3c0af5a0 msgr2=0x7f7d3c0b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:09.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.446+0000 7f7d41be9700 1 --2- 172.21.15.111:0/2750208204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d3c0af5a0 0x7f7d3c0b44d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00b4a0 tx=0x7f7d2c00b860 comp rx=0 tx=0).stop 2023-11-28T13:00:09.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.446+0000 7f7d41be9700 1 -- 172.21.15.111:0/2750208204 shutdown_connections 2023-11-28T13:00:09.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.446+0000 7f7d41be9700 1 --2- 172.21.15.111:0/2750208204 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7d1c0682f0 0x7f7d1c06a7a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.446+0000 7f7d41be9700 1 --2- 172.21.15.111:0/2750208204 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d3c15fed0 0x7f7d3c0b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.446+0000 7f7d41be9700 1 --2- 172.21.15.111:0/2750208204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d3c0af5a0 0x7f7d3c0b44d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.446+0000 7f7d41be9700 1 --2- 172.21.15.111:0/2750208204 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d3c0af9f0 0x7f7d3c1551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.446+0000 7f7d41be9700 1 -- 172.21.15.111:0/2750208204 >> 172.21.15.111:0/2750208204 conn(0x7f7d3c06bca0 msgr2=0x7f7d3c0cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:09.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.446+0000 7f7d41be9700 1 -- 172.21.15.111:0/2750208204 shutdown_connections 2023-11-28T13:00:09.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.446+0000 7f7d41be9700 1 -- 172.21.15.111:0/2750208204 wait complete. 2023-11-28T13:00:09.459 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T13:00:09.459 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T13:00:09.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.581+0000 7ff253298700 1 Processor -- start 2023-11-28T13:00:09.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.581+0000 7ff253298700 1 -- start start 2023-11-28T13:00:09.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.582+0000 7ff253298700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff24c0b5250 0x7ff24c0c6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:09.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.582+0000 7ff253298700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c0d0b90 con 0x7ff24c0cf010 2023-11-28T13:00:09.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.582+0000 7ff253298700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c0b3db0 con 0x7ff24c0b2360 2023-11-28T13:00:09.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.582+0000 7ff253298700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24c0c6cc0 con 0x7ff24c0b5250 2023-11-28T13:00:09.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.582+0000 7ff251835700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff24c0b5250 0x7ff24c0c6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:09.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.582+0000 7ff250833700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff24c0cf010 0x7ff24c0c4880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35900/0 (socket says 172.21.15.111:35900) 2023-11-28T13:00:09.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.582+0000 7ff250833700 1 -- 172.21.15.111:0/3894489050 learned_addr learned my addr 172.21.15.111:0/3894489050 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:09.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.582+0000 7ff251835700 1 -- 172.21.15.111:0/3894489050 >> v1:172.21.15.149:6789/0 conn(0x7ff24c0b2360 legacy=0x7ff24c0ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:09.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.582+0000 7ff251835700 1 -- 172.21.15.111:0/3894489050 >> v1:172.21.15.111:6789/0 conn(0x7ff24c0cf010 legacy=0x7ff24c0c4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:09.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.582+0000 7ff251835700 1 -- 172.21.15.111:0/3894489050 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0c6eb0 con 0x7ff24c0b5250 2023-11-28T13:00:09.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.582+0000 7ff251835700 1 --2- 172.21.15.111:0/3894489050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff24c0b5250 0x7ff24c0c6780 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff244009d90 tx=0x7ff24400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9881d5448688a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:09.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.583+0000 7ff243fff700 1 -- 172.21.15.111:0/3894489050 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24400e070 con 0x7ff24c0b5250 2023-11-28T13:00:09.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.583+0000 7ff243fff700 1 -- 172.21.15.111:0/3894489050 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24400bbd0 con 0x7ff24c0b5250 2023-11-28T13:00:09.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.583+0000 7ff243fff700 1 -- 172.21.15.111:0/3894489050 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244012430 con 0x7ff24c0b5250 2023-11-28T13:00:09.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.583+0000 7ff253298700 1 -- 172.21.15.111:0/3894489050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff24c0b5250 msgr2=0x7ff24c0c6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:09.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.583+0000 7ff253298700 1 --2- 172.21.15.111:0/3894489050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff24c0b5250 0x7ff24c0c6780 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff244009d90 tx=0x7ff24400b330 comp rx=0 tx=0).stop 2023-11-28T13:00:09.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.583+0000 7ff253298700 1 -- 172.21.15.111:0/3894489050 shutdown_connections 2023-11-28T13:00:09.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.583+0000 7ff253298700 1 --2- 172.21.15.111:0/3894489050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff24c0b5250 0x7ff24c0c6780 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.583+0000 7ff253298700 1 -- 172.21.15.111:0/3894489050 >> 172.21.15.111:0/3894489050 conn(0x7ff24c09a040 msgr2=0x7ff24c0d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:09.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.583+0000 7ff253298700 1 -- 172.21.15.111:0/3894489050 shutdown_connections 2023-11-28T13:00:09.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.583+0000 7ff253298700 1 -- 172.21.15.111:0/3894489050 wait complete. 2023-11-28T13:00:09.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.584+0000 7ff253298700 1 Processor -- start 2023-11-28T13:00:09.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.584+0000 7ff253298700 1 -- start start 2023-11-28T13:00:09.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.585+0000 7ff253298700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff24c0b2360 0x7ff24c0cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:09.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.585+0000 7ff253298700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff24c0b5250 0x7ff24c1552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:09.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.585+0000 7ff253298700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff24c0cf010 0x7ff24c0ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:09.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.585+0000 7ff253298700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c0ccd80 con 0x7ff24c0cf010 2023-11-28T13:00:09.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.585+0000 7ff253298700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c155830 con 0x7ff24c0b2360 2023-11-28T13:00:09.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.585+0000 7ff253298700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c155a20 con 0x7ff24c0b5250 2023-11-28T13:00:09.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.585+0000 7ff251835700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff24c0cf010 0x7ff24c0ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:09.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.585+0000 7ff251835700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff24c0cf010 0x7ff24c0ca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35130/0 (socket says 172.21.15.111:35130) 2023-11-28T13:00:09.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.585+0000 7ff251835700 1 -- 172.21.15.111:0/3366470155 learned_addr learned my addr 172.21.15.111:0/3366470155 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:09.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff250833700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff24c0b5250 0x7ff24c1552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:09.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff251034700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff24c0b2360 0x7ff24c0cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:09.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff251835700 1 -- 172.21.15.111:0/3366470155 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff24c0b5250 msgr2=0x7ff24c1552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:09.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff251835700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff24c0b5250 0x7ff24c1552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff251835700 1 -- 172.21.15.111:0/3366470155 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff24c0b2360 msgr2=0x7ff24c0cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:09.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff251835700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff24c0b2360 0x7ff24c0cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:09.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff251835700 1 -- 172.21.15.111:0/3366470155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff244009a70 con 0x7ff24c0cf010 2023-11-28T13:00:09.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff251034700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff24c0b2360 0x7ff24c0cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:09.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff250833700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff24c0b5250 0x7ff24c1552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:09.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff251835700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff24c0cf010 0x7ff24c0ca080 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff24c067d30 tx=0x7ff244004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:09.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff241ffb700 1 -- 172.21.15.111:0/3366470155 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2440098f0 con 0x7ff24c0cf010 2023-11-28T13:00:09.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff241ffb700 1 -- 172.21.15.111:0/3366470155 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff244020070 con 0x7ff24c0cf010 2023-11-28T13:00:09.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff241ffb700 1 -- 172.21.15.111:0/3366470155 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244004280 con 0x7ff24c0cf010 2023-11-28T13:00:09.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff253298700 1 -- 172.21.15.111:0/3366470155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24c0c7470 con 0x7ff24c0cf010 2023-11-28T13:00:09.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.586+0000 7ff253298700 1 -- 172.21.15.111:0/3366470155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c0c7190 con 0x7ff24c0cf010 2023-11-28T13:00:09.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.587+0000 7ff253298700 1 -- 172.21.15.111:0/3366470155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff24c0c4f90 con 0x7ff24c0cf010 2023-11-28T13:00:09.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.588+0000 7ff241ffb700 1 -- 172.21.15.111:0/3366470155 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff24402e070 con 0x7ff24c0cf010 2023-11-28T13:00:09.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.588+0000 7ff241ffb700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff2300682f0 0x7ff23006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:09.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.588+0000 7ff241ffb700 1 -- 172.21.15.111:0/3366470155 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5530+0+0 (secure 0 0 0) 0x7ff244022070 con 0x7ff24c0cf010 2023-11-28T13:00:09.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.588+0000 7ff251034700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff2300682f0 0x7ff23006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:09.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.589+0000 7ff251034700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff2300682f0 0x7ff23006a7a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff23c000fb0 tx=0x7ff23c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:09.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.592+0000 7ff241ffb700 1 -- 172.21.15.111:0/3366470155 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff244054400 con 0x7ff24c0cf010 2023-11-28T13:00:09.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:09.774+0000 7ff253298700 1 -- 172.21.15.111:0/3366470155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff24c067170 con 0x7ff24c0cf010 2023-11-28T13:00:10.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.616+0000 7ff241ffb700 1 -- 172.21.15.111:0/3366470155 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v38) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff24402c020 con 0x7ff24c0cf010 2023-11-28T13:00:10.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.621+0000 7ff253298700 1 -- 172.21.15.111:0/3366470155 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff2300682f0 msgr2=0x7ff23006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:10.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.621+0000 7ff253298700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff2300682f0 0x7ff23006a7a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff23c000fb0 tx=0x7ff23c009040 comp rx=0 tx=0).stop 2023-11-28T13:00:10.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.621+0000 7ff253298700 1 -- 172.21.15.111:0/3366470155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff24c0cf010 msgr2=0x7ff24c0ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:10.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.621+0000 7ff253298700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff24c0cf010 0x7ff24c0ca080 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff24c067d30 tx=0x7ff244004ad0 comp rx=0 tx=0).stop 2023-11-28T13:00:10.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.621+0000 7ff253298700 1 -- 172.21.15.111:0/3366470155 shutdown_connections 2023-11-28T13:00:10.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.621+0000 7ff253298700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff24c0cf010 0x7ff24c0ca080 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:10.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.621+0000 7ff253298700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff2300682f0 0x7ff23006a7a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:10.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.621+0000 7ff253298700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff24c0b2360 0x7ff24c0cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:10.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.621+0000 7ff253298700 1 --2- 172.21.15.111:0/3366470155 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff24c0b5250 0x7ff24c1552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:10.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.621+0000 7ff253298700 1 -- 172.21.15.111:0/3366470155 >> 172.21.15.111:0/3366470155 conn(0x7ff24c09a040 msgr2=0x7ff24c0c4860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:10.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.621+0000 7ff253298700 1 -- 172.21.15.111:0/3366470155 shutdown_connections 2023-11-28T13:00:10.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.622+0000 7ff253298700 1 -- 172.21.15.111:0/3366470155 wait complete. 2023-11-28T13:00:10.627 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-11-28T13:00:10.638 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T13:00:10.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.765+0000 7f52d7f35700 1 Processor -- start 2023-11-28T13:00:10.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.765+0000 7f52d7f35700 1 -- start start 2023-11-28T13:00:10.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.766+0000 7f52d7f35700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f52d006bff0 0x7f52d015d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:10.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.766+0000 7f52d7f35700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52d0160ca0 0x7f52d00b1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:10.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.766+0000 7f52d7f35700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52d00cd750 0x7f52d00cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:10.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.766+0000 7f52d7f35700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d00b1b40 con 0x7f52d0160ca0 2023-11-28T13:00:10.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.766+0000 7f52d7f35700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d0161140 con 0x7f52d006bff0 2023-11-28T13:00:10.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.766+0000 7f52d7f35700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00ce0f0 con 0x7f52d00cd750 2023-11-28T13:00:10.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.766+0000 7f52d54d0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52d0160ca0 0x7f52d00b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:10.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.766+0000 7f52d54d0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52d0160ca0 0x7f52d00b1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35136/0 (socket says 172.21.15.111:35136) 2023-11-28T13:00:10.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.766+0000 7f52d54d0700 1 -- 172.21.15.111:0/1371213949 learned_addr learned my addr 172.21.15.111:0/1371213949 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:10.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.766+0000 7f52d5cd1700 1 --2- 172.21.15.111:0/1371213949 >> v2:172.21.15.149:3300/0 conn(0x7f52d006bff0 0x7f52d015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:10.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.766+0000 7f52d64d2700 1 --2- 172.21.15.111:0/1371213949 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52d00cd750 0x7f52d00cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:10.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d54d0700 1 -- 172.21.15.111:0/1371213949 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52d00cd750 msgr2=0x7f52d00cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:10.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d54d0700 1 --2- 172.21.15.111:0/1371213949 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52d00cd750 0x7f52d00cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:10.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d54d0700 1 -- 172.21.15.111:0/1371213949 >> v2:172.21.15.149:3300/0 conn(0x7f52d006bff0 msgr2=0x7f52d015d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:10.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d54d0700 1 --2- 172.21.15.111:0/1371213949 >> v2:172.21.15.149:3300/0 conn(0x7f52d006bff0 0x7f52d015d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:10.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d54d0700 1 -- 172.21.15.111:0/1371213949 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d00b1d50 con 0x7f52d0160ca0 2023-11-28T13:00:10.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d54d0700 1 --2- 172.21.15.111:0/1371213949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d0160ca0 0x7f52d00b1600 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f52d006d6c0 tx=0x7f52cc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c7766ad342bbf6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:10.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d4ccf700 1 -- 172.21.15.111:0/1371213949 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc011830 con 0x7f52d0160ca0 2023-11-28T13:00:10.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d4ccf700 1 -- 172.21.15.111:0/1371213949 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52cc011990 con 0x7f52d0160ca0 2023-11-28T13:00:10.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d4ccf700 1 -- 172.21.15.111:0/1371213949 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc011d20 con 0x7f52d0160ca0 2023-11-28T13:00:10.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d7f35700 1 -- 172.21.15.111:0/1371213949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d0160ca0 msgr2=0x7f52d00b1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:10.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d7f35700 1 --2- 172.21.15.111:0/1371213949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d0160ca0 0x7f52d00b1600 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f52d006d6c0 tx=0x7f52cc009c00 comp rx=0 tx=0).stop 2023-11-28T13:00:10.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d7f35700 1 -- 172.21.15.111:0/1371213949 shutdown_connections 2023-11-28T13:00:10.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d7f35700 1 --2- 172.21.15.111:0/1371213949 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52d00cd750 0x7f52d00cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:10.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d7f35700 1 --2- 172.21.15.111:0/1371213949 >> v2:172.21.15.149:3300/0 conn(0x7f52d006bff0 0x7f52d015d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:10.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.767+0000 7f52d7f35700 1 --2- 172.21.15.111:0/1371213949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d0160ca0 0x7f52d00b1600 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:10.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.768+0000 7f52d7f35700 1 -- 172.21.15.111:0/1371213949 >> 172.21.15.111:0/1371213949 conn(0x7f52d006e790 msgr2=0x7f52d00cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:10.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.768+0000 7f52d7f35700 1 -- 172.21.15.111:0/1371213949 shutdown_connections 2023-11-28T13:00:10.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.768+0000 7f52d7f35700 1 -- 172.21.15.111:0/1371213949 wait complete. 2023-11-28T13:00:10.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.769+0000 7f52d7f35700 1 Processor -- start 2023-11-28T13:00:10.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.769+0000 7f52d7f35700 1 -- start start 2023-11-28T13:00:10.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.769+0000 7f52d7f35700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d006bff0 0x7f52d00c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:10.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.769+0000 7f52d7f35700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52d00cd750 0x7f52d01e7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:10.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.769+0000 7f52d7f35700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52d0160ca0 0x7f52d01e75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:10.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.769+0000 7f52d7f35700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00ac2e0 con 0x7f52d006bff0 2023-11-28T13:00:10.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d7f35700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d01e8af0 con 0x7f52d0160ca0 2023-11-28T13:00:10.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d7f35700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d01e8660 con 0x7f52d00cd750 2023-11-28T13:00:10.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d5cd1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d006bff0 0x7f52d00c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:10.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d5cd1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d006bff0 0x7f52d00c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35140/0 (socket says 172.21.15.111:35140) 2023-11-28T13:00:10.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d5cd1700 1 -- 172.21.15.111:0/2526333360 learned_addr learned my addr 172.21.15.111:0/2526333360 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:10.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d54d0700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52d00cd750 0x7f52d01e7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:10.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d64d2700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52d0160ca0 0x7f52d01e75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:10.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d5cd1700 1 -- 172.21.15.111:0/2526333360 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52d00cd750 msgr2=0x7f52d01e7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:10.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d5cd1700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52d00cd750 0x7f52d01e7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:10.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d5cd1700 1 -- 172.21.15.111:0/2526333360 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52d0160ca0 msgr2=0x7f52d01e75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:10.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d5cd1700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52d0160ca0 0x7f52d01e75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:10.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d5cd1700 1 -- 172.21.15.111:0/2526333360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc009550 con 0x7f52d006bff0 2023-11-28T13:00:10.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d5cd1700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d006bff0 0x7f52d00c7b40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f52c000e870 tx=0x7f52c000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:10.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52c67fc700 1 -- 172.21.15.111:0/2526333360 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0013680 con 0x7f52d006bff0 2023-11-28T13:00:10.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52c67fc700 1 -- 172.21.15.111:0/2526333360 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c00137e0 con 0x7f52d006bff0 2023-11-28T13:00:10.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d7f35700 1 -- 172.21.15.111:0/2526333360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d015d020 con 0x7f52d006bff0 2023-11-28T13:00:10.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52c67fc700 1 -- 172.21.15.111:0/2526333360 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c0013b90 con 0x7f52d006bff0 2023-11-28T13:00:10.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.770+0000 7f52d7f35700 1 -- 172.21.15.111:0/2526333360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d015cd40 con 0x7f52d006bff0 2023-11-28T13:00:10.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.771+0000 7f52d7f35700 1 -- 172.21.15.111:0/2526333360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d015d300 con 0x7f52d006bff0 2023-11-28T13:00:10.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.772+0000 7f52c67fc700 1 -- 172.21.15.111:0/2526333360 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f52c0019030 con 0x7f52d006bff0 2023-11-28T13:00:10.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.772+0000 7f52c67fc700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52b40682f0 0x7f52b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:10.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.772+0000 7f52c67fc700 1 -- 172.21.15.111:0/2526333360 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f52c0085910 con 0x7f52d006bff0 2023-11-28T13:00:10.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.772+0000 7f52d54d0700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52b40682f0 0x7f52b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:10.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.772+0000 7f52d54d0700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52b40682f0 0x7f52b406a7a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009870 tx=0x7f52cc01d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:10.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.776+0000 7f52c67fc700 1 -- 172.21.15.111:0/2526333360 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f52c0052210 con 0x7f52d006bff0 2023-11-28T13:00:10.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:10.960+0000 7f52d7f35700 1 -- 172.21.15.111:0/2526333360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f52d0067170 con 0x7f52d006bff0 2023-11-28T13:00:11.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.621+0000 7f52c67fc700 1 -- 172.21.15.111:0/2526333360 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v39) v1 ==== 170+0+0 (secure 0 0 0) 0x7f52c0055830 con 0x7f52d006bff0 2023-11-28T13:00:11.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.626+0000 7f52d7f35700 1 -- 172.21.15.111:0/2526333360 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52b40682f0 msgr2=0x7f52b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:11.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.626+0000 7f52d7f35700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52b40682f0 0x7f52b406a7a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009870 tx=0x7f52cc01d040 comp rx=0 tx=0).stop 2023-11-28T13:00:11.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.626+0000 7f52d7f35700 1 -- 172.21.15.111:0/2526333360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d006bff0 msgr2=0x7f52d00c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:11.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.626+0000 7f52d7f35700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d006bff0 0x7f52d00c7b40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f52c000e870 tx=0x7f52c000eb80 comp rx=0 tx=0).stop 2023-11-28T13:00:11.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.630+0000 7f52d7f35700 1 -- 172.21.15.111:0/2526333360 shutdown_connections 2023-11-28T13:00:11.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.630+0000 7f52d7f35700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52b40682f0 0x7f52b406a7a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:11.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.630+0000 7f52d7f35700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52d0160ca0 0x7f52d01e75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:11.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.630+0000 7f52d7f35700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d006bff0 0x7f52d00c7b40 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:11.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.630+0000 7f52d7f35700 1 --2- 172.21.15.111:0/2526333360 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52d00cd750 0x7f52d01e7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:11.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.630+0000 7f52d7f35700 1 -- 172.21.15.111:0/2526333360 >> 172.21.15.111:0/2526333360 conn(0x7f52d006e790 msgr2=0x7f52d00af9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:11.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.631+0000 7f52d7f35700 1 -- 172.21.15.111:0/2526333360 shutdown_connections 2023-11-28T13:00:11.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.631+0000 7f52d7f35700 1 -- 172.21.15.111:0/2526333360 wait complete. 2023-11-28T13:00:11.634 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-11-28T13:00:11.651 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T13:00:11.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.774+0000 7f7040752700 1 Processor -- start 2023-11-28T13:00:11.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.774+0000 7f7040752700 1 -- start start 2023-11-28T13:00:11.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.775+0000 7f7040752700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70380cfc90 0x7f70380d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:11.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.775+0000 7f7040752700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f70380cc5a0 0x7f70380cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:11.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.775+0000 7f7040752700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380d05a0 con 0x7f70380cdd20 2023-11-28T13:00:11.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.775+0000 7f7040752700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70380ce1c0 con 0x7f70380cc5a0 2023-11-28T13:00:11.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.775+0000 7f7040752700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70380ca260 con 0x7f70380cfc90 2023-11-28T13:00:11.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.776+0000 7f703dced700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f70380cc5a0 0x7f70380cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.776+0000 7f703e4ee700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70380cfc90 0x7f70380d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.776+0000 7f703ecef700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f70380cdd20 0x7f703815d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:44472/0 (socket says 172.21.15.111:44472) 2023-11-28T13:00:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.776+0000 7f703ecef700 1 -- 172.21.15.111:0/1163799530 learned_addr learned my addr 172.21.15.111:0/1163799530 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.776+0000 7f703dced700 1 -- 172.21.15.111:0/1163799530 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70380cfc90 msgr2=0x7f70380d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.776+0000 7f703dced700 1 --2- 172.21.15.111:0/1163799530 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70380cfc90 0x7f70380d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.776+0000 7f703dced700 1 -- 172.21.15.111:0/1163799530 >> v1:172.21.15.149:6789/0 conn(0x7f70380cdd20 legacy=0x7f703815d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.776+0000 7f703d4ec700 1 -- 172.21.15.111:0/1163799530 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807771348 0 0) 0x7f7030001490 con 0x7f70380cdd20 2023-11-28T13:00:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.776+0000 7f703dced700 1 -- 172.21.15.111:0/1163799530 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70380ccaa0 con 0x7f70380cc5a0 2023-11-28T13:00:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.776+0000 7f703e4ee700 1 --2- 172.21.15.111:0/1163799530 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70380cfc90 0x7f70380d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.776+0000 7f703dced700 1 --2- 172.21.15.111:0/1163799530 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70380cc5a0 0x7f70380cd7e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7034009840 tx=0x7f7034009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d57e3b0b4eb75f4e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.776+0000 7f703d4ec700 1 -- 172.21.15.111:0/1163799530 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034011830 con 0x7f70380cc5a0 2023-11-28T13:00:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.777+0000 7f703d4ec700 1 -- 172.21.15.111:0/1163799530 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7034011990 con 0x7f70380cc5a0 2023-11-28T13:00:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.777+0000 7f703d4ec700 1 -- 172.21.15.111:0/1163799530 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034011d20 con 0x7f70380cc5a0 2023-11-28T13:00:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.777+0000 7f7040752700 1 -- 172.21.15.111:0/1163799530 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70380cc5a0 msgr2=0x7f70380cd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.777+0000 7f7040752700 1 --2- 172.21.15.111:0/1163799530 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70380cc5a0 0x7f70380cd7e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7034009840 tx=0x7f7034009b50 comp rx=0 tx=0).stop 2023-11-28T13:00:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.777+0000 7f7040752700 1 -- 172.21.15.111:0/1163799530 shutdown_connections 2023-11-28T13:00:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.777+0000 7f7040752700 1 --2- 172.21.15.111:0/1163799530 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70380cfc90 0x7f70380d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.777+0000 7f7040752700 1 --2- 172.21.15.111:0/1163799530 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70380cc5a0 0x7f70380cd7e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.777+0000 7f7040752700 1 -- 172.21.15.111:0/1163799530 >> 172.21.15.111:0/1163799530 conn(0x7f7038098ee0 msgr2=0x7f703806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.777+0000 7f7040752700 1 -- 172.21.15.111:0/1163799530 shutdown_connections 2023-11-28T13:00:11.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.777+0000 7f7040752700 1 -- 172.21.15.111:0/1163799530 wait complete. 2023-11-28T13:00:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.778+0000 7f7040752700 1 Processor -- start 2023-11-28T13:00:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.778+0000 7f7040752700 1 -- start start 2023-11-28T13:00:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f7040752700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70380cc5a0 0x7f70380c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f7040752700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70380cdd20 0x7f70380b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f7040752700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70380cfc90 0x7f70380c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f7040752700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70380b4130 con 0x7f70380cfc90 2023-11-28T13:00:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f7040752700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70380b4410 con 0x7f70380cc5a0 2023-11-28T13:00:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f7040752700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70380b46f0 con 0x7f70380cdd20 2023-11-28T13:00:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f703ecef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70380cfc90 0x7f70380c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f703dced700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70380cdd20 0x7f70380b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f703ecef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70380cfc90 0x7f70380c1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35158/0 (socket says 172.21.15.111:35158) 2023-11-28T13:00:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f703e4ee700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70380cc5a0 0x7f70380c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f703ecef700 1 -- 172.21.15.111:0/601349158 learned_addr learned my addr 172.21.15.111:0/601349158 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f703ecef700 1 -- 172.21.15.111:0/601349158 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70380cdd20 msgr2=0x7f70380b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f703ecef700 1 --2- 172.21.15.111:0/601349158 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70380cdd20 0x7f70380b51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f703ecef700 1 -- 172.21.15.111:0/601349158 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70380cc5a0 msgr2=0x7f70380c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f703ecef700 1 --2- 172.21.15.111:0/601349158 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70380cc5a0 0x7f70380c5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.779+0000 7f703ecef700 1 -- 172.21.15.111:0/601349158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70280094a0 con 0x7f70380cfc90 2023-11-28T13:00:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.780+0000 7f703ecef700 1 --2- 172.21.15.111:0/601349158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70380cfc90 0x7f70380c1930 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f703000cca0 tx=0x7f703000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.780+0000 7f702effd700 1 -- 172.21.15.111:0/601349158 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703000ab10 con 0x7f70380cfc90 2023-11-28T13:00:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.780+0000 7f702effd700 1 -- 172.21.15.111:0/601349158 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703000ac90 con 0x7f70380cfc90 2023-11-28T13:00:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.780+0000 7f702effd700 1 -- 172.21.15.111:0/601349158 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70300070d0 con 0x7f70380cfc90 2023-11-28T13:00:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.780+0000 7f7040752700 1 -- 172.21.15.111:0/601349158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70340094a0 con 0x7f70380cfc90 2023-11-28T13:00:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.780+0000 7f7040752700 1 -- 172.21.15.111:0/601349158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70380cd060 con 0x7f70380cfc90 2023-11-28T13:00:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.781+0000 7f7040752700 1 -- 172.21.15.111:0/601349158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70380ccd80 con 0x7f70380cfc90 2023-11-28T13:00:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.781+0000 7f702effd700 1 -- 172.21.15.111:0/601349158 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7030010070 con 0x7f70380cfc90 2023-11-28T13:00:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.782+0000 7f702effd700 1 --2- 172.21.15.111:0/601349158 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f701c0682f0 0x7f701c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.782+0000 7f702effd700 1 -- 172.21.15.111:0/601349158 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6282+0+0 (secure 0 0 0) 0x7f7030085570 con 0x7f70380cfc90 2023-11-28T13:00:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.782+0000 7f703e4ee700 1 --2- 172.21.15.111:0/601349158 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f701c0682f0 0x7f701c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.782+0000 7f703e4ee700 1 --2- 172.21.15.111:0/601349158 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f701c0682f0 0x7f701c06a7a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f70381597e0 tx=0x7f7028009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:11.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:11.786+0000 7f702effd700 1 -- 172.21.15.111:0/601349158 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7030051c70 con 0x7f70380cfc90 2023-11-28T13:00:12.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.010+0000 7f7040752700 1 -- 172.21.15.111:0/601349158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f70380cd340 con 0x7f70380cfc90 2023-11-28T13:00:12.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.015+0000 7f702effd700 1 -- 172.21.15.111:0/601349158 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7030055290 con 0x7f70380cfc90 2023-11-28T13:00:12.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.021+0000 7f7040752700 1 -- 172.21.15.111:0/601349158 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f701c0682f0 msgr2=0x7f701c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.021+0000 7f7040752700 1 --2- 172.21.15.111:0/601349158 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f701c0682f0 0x7f701c06a7a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f70381597e0 tx=0x7f7028009210 comp rx=0 tx=0).stop 2023-11-28T13:00:12.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.021+0000 7f7040752700 1 -- 172.21.15.111:0/601349158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70380cfc90 msgr2=0x7f70380c1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.021+0000 7f7040752700 1 --2- 172.21.15.111:0/601349158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70380cfc90 0x7f70380c1930 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f703000cca0 tx=0x7f703000a060 comp rx=0 tx=0).stop 2023-11-28T13:00:12.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.021+0000 7f7040752700 1 -- 172.21.15.111:0/601349158 shutdown_connections 2023-11-28T13:00:12.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.021+0000 7f7040752700 1 --2- 172.21.15.111:0/601349158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70380cfc90 0x7f70380c1930 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.021+0000 7f7040752700 1 --2- 172.21.15.111:0/601349158 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f701c0682f0 0x7f701c06a7a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.021+0000 7f7040752700 1 --2- 172.21.15.111:0/601349158 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70380cc5a0 0x7f70380c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.021+0000 7f7040752700 1 --2- 172.21.15.111:0/601349158 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70380cdd20 0x7f70380b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.021+0000 7f7040752700 1 -- 172.21.15.111:0/601349158 >> 172.21.15.111:0/601349158 conn(0x7f7038098ee0 msgr2=0x7f70380ae1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:12.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.021+0000 7f7040752700 1 -- 172.21.15.111:0/601349158 shutdown_connections 2023-11-28T13:00:12.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.023+0000 7f7040752700 1 -- 172.21.15.111:0/601349158 wait complete. 2023-11-28T13:00:12.029 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 6 and data pool 7 2023-11-28T13:00:12.046 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:00:12.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.165+0000 7f92bd456700 1 Processor -- start 2023-11-28T13:00:12.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.165+0000 7f92bd456700 1 -- start start 2023-11-28T13:00:12.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92bd456700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f92b815cbf0 0x7f92b80b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92bd456700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f92b80b40a0 0x7f92b806b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92bd456700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f92b80cffd0 0x7f92b806b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92bd456700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b80cfdb0 con 0x7f92b80b40a0 2023-11-28T13:00:12.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92bd456700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b80cc740 con 0x7f92b80cffd0 2023-11-28T13:00:12.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92bd456700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b80b35e0 con 0x7f92b815cbf0 2023-11-28T13:00:12.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92b67fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f92b80b40a0 0x7f92b806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92b67fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f92b80b40a0 0x7f92b806b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35164/0 (socket says 172.21.15.111:35164) 2023-11-28T13:00:12.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92b67fc700 1 -- 172.21.15.111:0/2502097351 learned_addr learned my addr 172.21.15.111:0/2502097351 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:12.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92b77fe700 1 --2- 172.21.15.111:0/2502097351 >> v2:172.21.15.149:3300/0 conn(0x7f92b80cffd0 0x7f92b806b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92b6ffd700 1 --2- 172.21.15.111:0/2502097351 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f92b815cbf0 0x7f92b80b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92b67fc700 1 -- 172.21.15.111:0/2502097351 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f92b815cbf0 msgr2=0x7f92b80b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:12.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92b67fc700 1 --2- 172.21.15.111:0/2502097351 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f92b815cbf0 0x7f92b80b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92b67fc700 1 -- 172.21.15.111:0/2502097351 >> v2:172.21.15.149:3300/0 conn(0x7f92b80cffd0 msgr2=0x7f92b806b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:12.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92b67fc700 1 --2- 172.21.15.111:0/2502097351 >> v2:172.21.15.149:3300/0 conn(0x7f92b80cffd0 0x7f92b806b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92b67fc700 1 -- 172.21.15.111:0/2502097351 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b80cbe10 con 0x7f92b80b40a0 2023-11-28T13:00:12.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92b77fe700 1 --2- 172.21.15.111:0/2502097351 >> v2:172.21.15.149:3300/0 conn(0x7f92b80cffd0 0x7f92b806b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:12.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92b6ffd700 1 --2- 172.21.15.111:0/2502097351 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f92b815cbf0 0x7f92b80b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:12.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.166+0000 7f92b67fc700 1 --2- 172.21.15.111:0/2502097351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92b80b40a0 0x7f92b806b380 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f92b809a6e0 tx=0x7f92a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9ef661d20b253b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:12.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.167+0000 7f92b5ffb700 1 -- 172.21.15.111:0/2502097351 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a800b870 con 0x7f92b80b40a0 2023-11-28T13:00:12.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.167+0000 7f92b5ffb700 1 -- 172.21.15.111:0/2502097351 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a800b9d0 con 0x7f92b80b40a0 2023-11-28T13:00:12.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.167+0000 7f92b5ffb700 1 -- 172.21.15.111:0/2502097351 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a800bd60 con 0x7f92b80b40a0 2023-11-28T13:00:12.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.167+0000 7f92bd456700 1 -- 172.21.15.111:0/2502097351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92b80b40a0 msgr2=0x7f92b806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.167+0000 7f92bd456700 1 --2- 172.21.15.111:0/2502097351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92b80b40a0 0x7f92b806b380 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f92b809a6e0 tx=0x7f92a8009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.167+0000 7f92bd456700 1 -- 172.21.15.111:0/2502097351 shutdown_connections 2023-11-28T13:00:12.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.167+0000 7f92bd456700 1 --2- 172.21.15.111:0/2502097351 >> v2:172.21.15.149:3300/0 conn(0x7f92b80cffd0 0x7f92b806b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.167+0000 7f92bd456700 1 --2- 172.21.15.111:0/2502097351 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f92b815cbf0 0x7f92b80b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.167+0000 7f92bd456700 1 --2- 172.21.15.111:0/2502097351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92b80b40a0 0x7f92b806b380 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.167+0000 7f92bd456700 1 -- 172.21.15.111:0/2502097351 >> 172.21.15.111:0/2502097351 conn(0x7f92b809a040 msgr2=0x7f92b80d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:12.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.167+0000 7f92bd456700 1 -- 172.21.15.111:0/2502097351 shutdown_connections 2023-11-28T13:00:12.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.167+0000 7f92bd456700 1 -- 172.21.15.111:0/2502097351 wait complete. 2023-11-28T13:00:12.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.168+0000 7f92bd456700 1 Processor -- start 2023-11-28T13:00:12.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.169+0000 7f92bd456700 1 -- start start 2023-11-28T13:00:12.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.169+0000 7f92bd456700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92b80b40a0 0x7f92b80c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.169+0000 7f92bd456700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f92b80cffd0 0x7f92b806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.169+0000 7f92bd456700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f92b815cbf0 0x7f92b80702a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.169+0000 7f92bd456700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b815ad90 con 0x7f92b80b40a0 2023-11-28T13:00:12.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.169+0000 7f92bd456700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b815db00 con 0x7f92b80cffd0 2023-11-28T13:00:12.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.169+0000 7f92bd456700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b815dde0 con 0x7f92b815cbf0 2023-11-28T13:00:12.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.169+0000 7f92b6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92b80b40a0 0x7f92b80c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.169+0000 7f92b6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92b80b40a0 0x7f92b80c63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35168/0 (socket says 172.21.15.111:35168) 2023-11-28T13:00:12.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.169+0000 7f92b6ffd700 1 -- 172.21.15.111:0/1715577834 learned_addr learned my addr 172.21.15.111:0/1715577834 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:12.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f92b77fe700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f92b815cbf0 0x7f92b80702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f92b67fc700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f92b80cffd0 0x7f92b806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f92b6ffd700 1 -- 172.21.15.111:0/1715577834 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f92b815cbf0 msgr2=0x7f92b80702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f92b6ffd700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f92b815cbf0 0x7f92b80702a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f92b6ffd700 1 -- 172.21.15.111:0/1715577834 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f92b80cffd0 msgr2=0x7f92b806fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f92b6ffd700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f92b80cffd0 0x7f92b806fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f92b6ffd700 1 -- 172.21.15.111:0/1715577834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a8009550 con 0x7f92b80b40a0 2023-11-28T13:00:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f92b77fe700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f92b815cbf0 0x7f92b80702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f92b6ffd700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92b80b40a0 0x7f92b80c63f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00e810 tx=0x7f92ac00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f929f7fe700 1 -- 172.21.15.111:0/1715577834 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac00c020 con 0x7f92b80b40a0 2023-11-28T13:00:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f929f7fe700 1 -- 172.21.15.111:0/1715577834 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac004250 con 0x7f92b80b40a0 2023-11-28T13:00:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f92bd456700 1 -- 172.21.15.111:0/1715577834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b80cc3d0 con 0x7f92b80b40a0 2023-11-28T13:00:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f929f7fe700 1 -- 172.21.15.111:0/1715577834 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac0036a0 con 0x7f92b80b40a0 2023-11-28T13:00:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.170+0000 7f92bd456700 1 -- 172.21.15.111:0/1715577834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b80cc0f0 con 0x7f92b80b40a0 2023-11-28T13:00:12.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.171+0000 7f92bd456700 1 -- 172.21.15.111:0/1715577834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b80d06f0 con 0x7f92b80b40a0 2023-11-28T13:00:12.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.171+0000 7f929f7fe700 1 -- 172.21.15.111:0/1715577834 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f92ac003800 con 0x7f92b80b40a0 2023-11-28T13:00:12.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.172+0000 7f929f7fe700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f92a40682f0 0x7f92a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.172+0000 7f929f7fe700 1 -- 172.21.15.111:0/1715577834 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f92ac085ab0 con 0x7f92b80b40a0 2023-11-28T13:00:12.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.172+0000 7f92b67fc700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f92a40682f0 0x7f92a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.172+0000 7f92b67fc700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f92a40682f0 0x7f92a406a7a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f92b8070900 tx=0x7f92a8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:12.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.176+0000 7f929f7fe700 1 -- 172.21.15.111:0/1715577834 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f92ac0521d0 con 0x7f92b80b40a0 2023-11-28T13:00:12.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.362+0000 7f92bd456700 1 -- 172.21.15.111:0/1715577834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92b815e9f0 con 0x7f92b80b40a0 2023-11-28T13:00:12.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.363+0000 7f929f7fe700 1 -- 172.21.15.111:0/1715577834 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+18189 (secure 0 0 0) 0x7f92ac00f040 con 0x7f92b80b40a0 2023-11-28T13:00:12.366 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:12.367 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":40,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:12.011647+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:09.793782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:10.979525+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:00:12.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.366+0000 7f92bd456700 1 -- 172.21.15.111:0/1715577834 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f92a40682f0 msgr2=0x7f92a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.366+0000 7f92bd456700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f92a40682f0 0x7f92a406a7a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f92b8070900 tx=0x7f92a8008030 comp rx=0 tx=0).stop 2023-11-28T13:00:12.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.366+0000 7f92bd456700 1 -- 172.21.15.111:0/1715577834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92b80b40a0 msgr2=0x7f92b80c63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.366+0000 7f92bd456700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92b80b40a0 0x7f92b80c63f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00e810 tx=0x7f92ac00ebd0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.366+0000 7f92bd456700 1 -- 172.21.15.111:0/1715577834 shutdown_connections 2023-11-28T13:00:12.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.366+0000 7f92bd456700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f92b815cbf0 0x7f92b80702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.366+0000 7f92bd456700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92b80b40a0 0x7f92b80c63f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.366+0000 7f92bd456700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f92a40682f0 0x7f92a406a7a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.366+0000 7f92bd456700 1 --2- 172.21.15.111:0/1715577834 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f92b80cffd0 0x7f92b806fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.366+0000 7f92bd456700 1 -- 172.21.15.111:0/1715577834 >> 172.21.15.111:0/1715577834 conn(0x7f92b809a040 msgr2=0x7f92b80b4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:12.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.366+0000 7f92bd456700 1 -- 172.21.15.111:0/1715577834 shutdown_connections 2023-11-28T13:00:12.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.366+0000 7f92bd456700 1 -- 172.21.15.111:0/1715577834 wait complete. 2023-11-28T13:00:12.379 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:00:12.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.498+0000 7f6c22d10700 1 Processor -- start 2023-11-28T13:00:12.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.498+0000 7f6c22d10700 1 -- start start 2023-11-28T13:00:12.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c22d10700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6c1c0af5a0 0x7f6c1c0b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c22d10700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6c1c0af9f0 0x7f6c1c0b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c22d10700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1c06ac80 con 0x7f6c1c15fed0 2023-11-28T13:00:12.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c22d10700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c1c0cee00 con 0x7f6c1c0af9f0 2023-11-28T13:00:12.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c22d10700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c0cf9a0 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c212ad700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6c1c0af9f0 0x7f6c1c0b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c20aac700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6c1c15fed0 0x7f6c1c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35954/0 (socket says 172.21.15.111:35954) 2023-11-28T13:00:12.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c20aac700 1 -- 172.21.15.111:0/67235399 learned_addr learned my addr 172.21.15.111:0/67235399 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:12.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c212ad700 1 -- 172.21.15.111:0/67235399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6c1c0af5a0 msgr2=0x7f6c1c0b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:12.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c212ad700 1 --2- 172.21.15.111:0/67235399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6c1c0af5a0 0x7f6c1c0b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c212ad700 1 -- 172.21.15.111:0/67235399 >> v1:172.21.15.111:6789/0 conn(0x7f6c1c15fed0 legacy=0x7f6c1c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c1b7fe700 1 -- 172.21.15.111:0/67235399 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001790052 0 0) 0x7f6c1c06ac80 con 0x7f6c1c15fed0 2023-11-28T13:00:12.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c212ad700 1 -- 172.21.15.111:0/67235399 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c06aed0 con 0x7f6c1c0af9f0 2023-11-28T13:00:12.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c212ad700 1 --2- 172.21.15.111:0/67235399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c1c0af9f0 0x7f6c1c0b09f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6c14009d90 tx=0x7f6c1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796a4d3e7e43145a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:12.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c1b7fe700 1 -- 172.21.15.111:0/67235399 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1400e070 con 0x7f6c1c0af9f0 2023-11-28T13:00:12.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c1b7fe700 1 -- 172.21.15.111:0/67235399 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1400bbd0 con 0x7f6c1c0af9f0 2023-11-28T13:00:12.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.499+0000 7f6c1b7fe700 1 -- 172.21.15.111:0/67235399 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c14012430 con 0x7f6c1c0af9f0 2023-11-28T13:00:12.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.500+0000 7f6c22d10700 1 -- 172.21.15.111:0/67235399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c1c0af9f0 msgr2=0x7f6c1c0b09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.500+0000 7f6c22d10700 1 --2- 172.21.15.111:0/67235399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c1c0af9f0 0x7f6c1c0b09f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6c14009d90 tx=0x7f6c1400b330 comp rx=0 tx=0).stop 2023-11-28T13:00:12.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.500+0000 7f6c22d10700 1 -- 172.21.15.111:0/67235399 shutdown_connections 2023-11-28T13:00:12.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.500+0000 7f6c22d10700 1 --2- 172.21.15.111:0/67235399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c1c0af9f0 0x7f6c1c0b09f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.500+0000 7f6c22d10700 1 --2- 172.21.15.111:0/67235399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6c1c0af5a0 0x7f6c1c0b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.500+0000 7f6c22d10700 1 -- 172.21.15.111:0/67235399 >> 172.21.15.111:0/67235399 conn(0x7f6c1c06bca0 msgr2=0x7f6c1c0cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:12.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.500+0000 7f6c22d10700 1 -- 172.21.15.111:0/67235399 shutdown_connections 2023-11-28T13:00:12.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.500+0000 7f6c22d10700 1 -- 172.21.15.111:0/67235399 wait complete. 2023-11-28T13:00:12.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.501+0000 7f6c22d10700 1 Processor -- start 2023-11-28T13:00:12.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.501+0000 7f6c22d10700 1 -- start start 2023-11-28T13:00:12.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c22d10700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c1c0af5a0 0x7f6c1c0c3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c22d10700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6c1c0af9f0 0x7f6c1c14dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c22d10700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6c1c15fed0 0x7f6c1c0b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c22d10700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c14e140 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c22d10700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c0c9df0 con 0x7f6c1c15fed0 2023-11-28T13:00:12.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c22d10700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c0b47d0 con 0x7f6c1c0af9f0 2023-11-28T13:00:12.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c20aac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c1c0af5a0 0x7f6c1c0c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c20aac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c1c0af5a0 0x7f6c1c0c3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35182/0 (socket says 172.21.15.111:35182) 2023-11-28T13:00:12.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c20aac700 1 -- 172.21.15.111:0/3272461679 learned_addr learned my addr 172.21.15.111:0/3272461679 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:12.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c1bfff700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6c1c0af9f0 0x7f6c1c14dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c212ad700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6c1c15fed0 0x7f6c1c0b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c20aac700 1 -- 172.21.15.111:0/3272461679 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6c1c0af9f0 msgr2=0x7f6c1c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c20aac700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6c1c0af9f0 0x7f6c1c14dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.502+0000 7f6c20aac700 1 -- 172.21.15.111:0/3272461679 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6c1c15fed0 msgr2=0x7f6c1c0b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.503+0000 7f6c20aac700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6c1c15fed0 0x7f6c1c0b4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.503+0000 7f6c20aac700 1 -- 172.21.15.111:0/3272461679 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c14009a70 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.503+0000 7f6c1bfff700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6c1c0af9f0 0x7f6c1c14dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:12.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.503+0000 7f6c20aac700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c1c0af5a0 0x7f6c1c0c3700 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000c550 tx=0x7f6c1000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:12.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.503+0000 7f6c197fa700 1 -- 172.21.15.111:0/3272461679 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10004c80 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.503+0000 7f6c197fa700 1 -- 172.21.15.111:0/3272461679 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c10004e00 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.503+0000 7f6c22d10700 1 -- 172.21.15.111:0/3272461679 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1c1602a0 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.503+0000 7f6c197fa700 1 -- 172.21.15.111:0/3272461679 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10007310 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.503+0000 7f6c22d10700 1 -- 172.21.15.111:0/3272461679 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c1c0cea90 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.504+0000 7f6c22d10700 1 -- 172.21.15.111:0/3272461679 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1c0ce6b0 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.504+0000 7f6c197fa700 1 -- 172.21.15.111:0/3272461679 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6c1001a020 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.505+0000 7f6c197fa700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6c000682f0 0x7f6c0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.505+0000 7f6c1bfff700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6c000682f0 0x7f6c0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.505+0000 7f6c197fa700 1 -- 172.21.15.111:0/3272461679 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6c10085a20 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.505+0000 7f6c1bfff700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6c000682f0 0x7f6c0006a7a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c15ed00 tx=0x7f6c08008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:12.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.509+0000 7f6c197fa700 1 -- 172.21.15.111:0/3272461679 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6c100521f0 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.695+0000 7f6c22d10700 1 -- 172.21.15.111:0/3272461679 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c1c0b4ab0 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.696+0000 7f6c197fa700 1 -- 172.21.15.111:0/3272461679 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+18189 (secure 0 0 0) 0x7f6c10055810 con 0x7f6c1c0af5a0 2023-11-28T13:00:12.698 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:12.698 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":40,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:12.011647+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:09.793782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:10.979525+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:00:12.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.698+0000 7f6c22d10700 1 -- 172.21.15.111:0/3272461679 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6c000682f0 msgr2=0x7f6c0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.698+0000 7f6c22d10700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6c000682f0 0x7f6c0006a7a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c15ed00 tx=0x7f6c08008040 comp rx=0 tx=0).stop 2023-11-28T13:00:12.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.698+0000 7f6c22d10700 1 -- 172.21.15.111:0/3272461679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c1c0af5a0 msgr2=0x7f6c1c0c3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.698+0000 7f6c22d10700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c1c0af5a0 0x7f6c1c0c3700 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000c550 tx=0x7f6c1000c910 comp rx=0 tx=0).stop 2023-11-28T13:00:12.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.698+0000 7f6c22d10700 1 -- 172.21.15.111:0/3272461679 shutdown_connections 2023-11-28T13:00:12.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.698+0000 7f6c22d10700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6c000682f0 0x7f6c0006a7a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.698+0000 7f6c22d10700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6c1c15fed0 0x7f6c1c0b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.698+0000 7f6c22d10700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c1c0af5a0 0x7f6c1c0c3700 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.698+0000 7f6c22d10700 1 --2- 172.21.15.111:0/3272461679 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6c1c0af9f0 0x7f6c1c14dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.698+0000 7f6c22d10700 1 -- 172.21.15.111:0/3272461679 >> 172.21.15.111:0/3272461679 conn(0x7f6c1c06bca0 msgr2=0x7f6c1c06ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:12.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.698+0000 7f6c22d10700 1 -- 172.21.15.111:0/3272461679 shutdown_connections 2023-11-28T13:00:12.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.698+0000 7f6c22d10700 1 -- 172.21.15.111:0/3272461679 wait complete. 2023-11-28T13:00:12.712 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T13:00:12.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.833+0000 7fe739253700 1 Processor -- start 2023-11-28T13:00:12.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.834+0000 7fe739253700 1 -- start start 2023-11-28T13:00:12.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.834+0000 7fe739253700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe7340b0d70 0x7fe7341604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.834+0000 7fe739253700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe7340b1140 0x7fe7340ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.834+0000 7fe739253700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73415cd90 con 0x7fe7340b15b0 2023-11-28T13:00:12.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.834+0000 7fe739253700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7341613e0 con 0x7fe7340b0d70 2023-11-28T13:00:12.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.834+0000 7fe739253700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe734160b40 con 0x7fe7340b1140 2023-11-28T13:00:12.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.834+0000 7fe73359e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe7340b15b0 0x7fe7340cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:35972/0 (socket says 172.21.15.111:35972) 2023-11-28T13:00:12.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.834+0000 7fe73359e700 1 -- 172.21.15.111:0/2279995212 learned_addr learned my addr 172.21.15.111:0/2279995212 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:12.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260109654 0 0) 0x7fe72c001490 con 0x7fe7340b15b0 2023-11-28T13:00:12.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c0035b0 con 0x7fe7340b15b0 2023-11-28T13:00:12.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe732d9d700 1 --2- 172.21.15.111:0/2279995212 >> v2:172.21.15.149:3300/0 conn(0x7fe7340b0d70 0x7fe7341604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe73259c700 1 --2- 172.21.15.111:0/2279995212 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe7340b1140 0x7fe7340ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3061402443 0 0) 0x7fe71c0035b0 con 0x7fe7340b15b0 2023-11-28T13:00:12.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe7340b1140 msgr2=0x7fe7340ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:12.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 --2- 172.21.15.111:0/2279995212 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe7340b1140 0x7fe7340ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 >> v2:172.21.15.149:3300/0 conn(0x7fe7340b0d70 msgr2=0x7fe7341604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:12.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 --2- 172.21.15.111:0/2279995212 >> v2:172.21.15.149:3300/0 conn(0x7fe7340b0d70 0x7fe7341604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe734160e20 con 0x7fe7340b15b0 2023-11-28T13:00:12.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fe72c001b80 con 0x7fe7340b15b0 2023-11-28T13:00:12.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 >> v1:172.21.15.111:6789/0 conn(0x7fe7340b15b0 legacy=0x7fe7340cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 --2- 172.21.15.111:0/2279995212 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe7340b1140 0x7fe71c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 --2- 172.21.15.111:0/2279995212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe71c003f80 0x7fe71c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 --2- 172.21.15.111:0/2279995212 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe71c004d90 0x7fe71c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7341613e0 con 0x7fe71c003f80 2023-11-28T13:00:12.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe734160b40 con 0x7fe71c004d90 2023-11-28T13:00:12.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe71c00ba10 con 0x7fe7340b1140 2023-11-28T13:00:12.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe73259c700 1 --2- 172.21.15.111:0/2279995212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe71c003f80 0x7fe71c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe732d9d700 1 --2- 172.21.15.111:0/2279995212 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe7340b1140 0x7fe71c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe73359e700 1 --2- 172.21.15.111:0/2279995212 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe71c004d90 0x7fe71c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe73259c700 1 -- 172.21.15.111:0/2279995212 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe7340b1140 msgr2=0x7fe71c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:12.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe73259c700 1 --2- 172.21.15.111:0/2279995212 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe7340b1140 0x7fe71c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe73259c700 1 -- 172.21.15.111:0/2279995212 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe71c004d90 msgr2=0x7fe71c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:12.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe73259c700 1 --2- 172.21.15.111:0/2279995212 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe71c004d90 0x7fe71c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe73259c700 1 -- 172.21.15.111:0/2279995212 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71c0035b0 con 0x7fe71c003f80 2023-11-28T13:00:12.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.835+0000 7fe73359e700 1 --2- 172.21.15.111:0/2279995212 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe71c004d90 0x7fe71c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:12.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.836+0000 7fe73259c700 1 --2- 172.21.15.111:0/2279995212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe71c003f80 0x7fe71c004850 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe724009b60 tx=0x7fe71c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695d2cfd22f62c74 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:12.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.836+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72400e6c0 con 0x7fe71c003f80 2023-11-28T13:00:12.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.836+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72400e820 con 0x7fe71c003f80 2023-11-28T13:00:12.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.836+0000 7fe731d9b700 1 -- 172.21.15.111:0/2279995212 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72400eb90 con 0x7fe71c003f80 2023-11-28T13:00:12.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.836+0000 7fe739253700 1 -- 172.21.15.111:0/2279995212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe71c003f80 msgr2=0x7fe71c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.836+0000 7fe739253700 1 --2- 172.21.15.111:0/2279995212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe71c003f80 0x7fe71c004850 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe724009b60 tx=0x7fe71c005880 comp rx=0 tx=0).stop 2023-11-28T13:00:12.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.836+0000 7fe73359e700 1 -- 172.21.15.111:0/2279995212 reap_dead start 2023-11-28T13:00:12.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.836+0000 7fe739253700 1 -- 172.21.15.111:0/2279995212 shutdown_connections 2023-11-28T13:00:12.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.836+0000 7fe739253700 1 -- 172.21.15.111:0/2279995212 >> 172.21.15.111:0/2279995212 conn(0x7fe734098ee0 msgr2=0x7fe73406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:12.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.836+0000 7fe739253700 1 -- 172.21.15.111:0/2279995212 shutdown_connections 2023-11-28T13:00:12.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.836+0000 7fe739253700 1 -- 172.21.15.111:0/2279995212 wait complete. 2023-11-28T13:00:12.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.838+0000 7fe739253700 1 Processor -- start 2023-11-28T13:00:12.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.838+0000 7fe739253700 1 -- start start 2023-11-28T13:00:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.839+0000 7fe739253700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe71c003f80 0x7fe7341507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.839+0000 7fe739253700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe734150f60 0x7fe73415a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.839+0000 7fe739253700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7340ad6c0 0x7fe7340adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.839+0000 7fe739253700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73415f600 con 0x7fe7340ad6c0 2023-11-28T13:00:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.839+0000 7fe739253700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7340ae070 con 0x7fe71c003f80 2023-11-28T13:00:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.839+0000 7fe739253700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7340c3780 con 0x7fe734150f60 2023-11-28T13:00:12.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.839+0000 7fe73359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7340ad6c0 0x7fe7340adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.839+0000 7fe73359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7340ad6c0 0x7fe7340adb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35206/0 (socket says 172.21.15.111:35206) 2023-11-28T13:00:12.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.839+0000 7fe73359e700 1 -- 172.21.15.111:0/2647334429 learned_addr learned my addr 172.21.15.111:0/2647334429 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:12.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.839+0000 7fe732d9d700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe71c003f80 0x7fe7341507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.839+0000 7fe73259c700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe734150f60 0x7fe73415a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.840+0000 7fe73359e700 1 -- 172.21.15.111:0/2647334429 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe734150f60 msgr2=0x7fe73415a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.840+0000 7fe73359e700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe734150f60 0x7fe73415a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.840+0000 7fe73359e700 1 -- 172.21.15.111:0/2647334429 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe71c003f80 msgr2=0x7fe7341507b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:12.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.840+0000 7fe73359e700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe71c003f80 0x7fe7341507b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:12.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.840+0000 7fe73359e700 1 -- 172.21.15.111:0/2647334429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7340af340 con 0x7fe7340ad6c0 2023-11-28T13:00:12.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.840+0000 7fe73359e700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7340ad6c0 0x7fe7340adb30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe72c0013d0 tx=0x7fe72c00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:12.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.840+0000 7fe71b7fe700 1 -- 172.21.15.111:0/2647334429 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c018070 con 0x7fe7340ad6c0 2023-11-28T13:00:12.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.840+0000 7fe71b7fe700 1 -- 172.21.15.111:0/2647334429 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72c00ebb0 con 0x7fe7340ad6c0 2023-11-28T13:00:12.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.840+0000 7fe739253700 1 -- 172.21.15.111:0/2647334429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe734161100 con 0x7fe7340ad6c0 2023-11-28T13:00:12.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.840+0000 7fe71b7fe700 1 -- 172.21.15.111:0/2647334429 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c00ed60 con 0x7fe7340ad6c0 2023-11-28T13:00:12.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.840+0000 7fe739253700 1 -- 172.21.15.111:0/2647334429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7340ccc60 con 0x7fe7340ad6c0 2023-11-28T13:00:12.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.841+0000 7fe739253700 1 -- 172.21.15.111:0/2647334429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7340c35e0 con 0x7fe7340ad6c0 2023-11-28T13:00:12.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.842+0000 7fe71b7fe700 1 -- 172.21.15.111:0/2647334429 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe72c027430 con 0x7fe7340ad6c0 2023-11-28T13:00:12.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.842+0000 7fe71b7fe700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe7200682f0 0x7fe72006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:12.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.842+0000 7fe71b7fe700 1 -- 172.21.15.111:0/2647334429 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe72c0864c0 con 0x7fe7340ad6c0 2023-11-28T13:00:12.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.842+0000 7fe732d9d700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe7200682f0 0x7fe72006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:12.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.842+0000 7fe732d9d700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe7200682f0 0x7fe72006a7a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe728009cc0 tx=0x7fe7280091c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:12.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:12.846+0000 7fe71b7fe700 1 -- 172.21.15.111:0/2647334429 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe72c052c20 con 0x7fe7340ad6c0 2023-11-28T13:00:13.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:13.073+0000 7fe739253700 1 -- 172.21.15.111:0/2647334429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe7340c3980 con 0x7fe7340ad6c0 2023-11-28T13:00:14.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.035+0000 7fe71b7fe700 1 -- 172.21.15.111:0/2647334429 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe7340c3980 con 0x7fe7340ad6c0 2023-11-28T13:00:14.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.038+0000 7fe739253700 1 -- 172.21.15.111:0/2647334429 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe7200682f0 msgr2=0x7fe72006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:14.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.038+0000 7fe739253700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe7200682f0 0x7fe72006a7a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe728009cc0 tx=0x7fe7280091c0 comp rx=0 tx=0).stop 2023-11-28T13:00:14.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.038+0000 7fe739253700 1 -- 172.21.15.111:0/2647334429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7340ad6c0 msgr2=0x7fe7340adb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:14.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.038+0000 7fe739253700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7340ad6c0 0x7fe7340adb30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe72c0013d0 tx=0x7fe72c00e440 comp rx=0 tx=0).stop 2023-11-28T13:00:14.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.038+0000 7fe739253700 1 -- 172.21.15.111:0/2647334429 shutdown_connections 2023-11-28T13:00:14.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.038+0000 7fe739253700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7340ad6c0 0x7fe7340adb30 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:14.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.038+0000 7fe739253700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe7200682f0 0x7fe72006a7a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:14.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.038+0000 7fe739253700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe71c003f80 0x7fe7341507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:14.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.038+0000 7fe739253700 1 --2- 172.21.15.111:0/2647334429 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe734150f60 0x7fe73415a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:14.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.038+0000 7fe739253700 1 -- 172.21.15.111:0/2647334429 >> 172.21.15.111:0/2647334429 conn(0x7fe734098ee0 msgr2=0x7fe7340af6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:14.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.038+0000 7fe739253700 1 -- 172.21.15.111:0/2647334429 shutdown_connections 2023-11-28T13:00:14.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.038+0000 7fe739253700 1 -- 172.21.15.111:0/2647334429 wait complete. 2023-11-28T13:00:14.054 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T13:00:14.055 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T13:00:14.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.182+0000 7feea3f8a700 1 Processor -- start 2023-11-28T13:00:14.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.182+0000 7feea3f8a700 1 -- start start 2023-11-28T13:00:14.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea3f8a700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fee9c159310 0x7fee9c0ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:14.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea3f8a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fee9c1596e0 0x7fee9c0ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:14.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea3f8a700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9c15de70 con 0x7fee9c15cfd0 2023-11-28T13:00:14.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea3f8a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9c0ced10 con 0x7fee9c1596e0 2023-11-28T13:00:14.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea3f8a700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9c0ceff0 con 0x7fee9c159310 2023-11-28T13:00:14.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea1525700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fee9c1596e0 0x7fee9c0ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:14.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea1525700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fee9c1596e0 0x7fee9c0ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35212/0 (socket says 172.21.15.111:35212) 2023-11-28T13:00:14.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea1525700 1 -- 172.21.15.111:0/2065050377 learned_addr learned my addr 172.21.15.111:0/2065050377 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:14.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea1d26700 1 --2- 172.21.15.111:0/2065050377 >> v2:172.21.15.149:3300/0 conn(0x7fee9c159310 0x7fee9c0ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:14.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea1525700 1 -- 172.21.15.111:0/2065050377 >> v2:172.21.15.149:3300/0 conn(0x7fee9c159310 msgr2=0x7fee9c0ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:14.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea1525700 1 --2- 172.21.15.111:0/2065050377 >> v2:172.21.15.149:3300/0 conn(0x7fee9c159310 0x7fee9c0ce1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:14.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea1525700 1 -- 172.21.15.111:0/2065050377 >> v1:172.21.15.149:6789/0 conn(0x7fee9c15cfd0 legacy=0x7fee9c0b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:14.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea1525700 1 -- 172.21.15.111:0/2065050377 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9c0cf2d0 con 0x7fee9c1596e0 2023-11-28T13:00:14.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea1525700 1 --2- 172.21.15.111:0/2065050377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee9c1596e0 0x7fee9c0ce6e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0ca1c0 tx=0x7fee8c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98a3892bceb1389 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:14.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.183+0000 7feea0d24700 1 -- 172.21.15.111:0/2065050377 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8c004230 con 0x7fee9c1596e0 2023-11-28T13:00:14.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.184+0000 7feea0d24700 1 -- 172.21.15.111:0/2065050377 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8c004390 con 0x7fee9c1596e0 2023-11-28T13:00:14.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.184+0000 7feea0d24700 1 -- 172.21.15.111:0/2065050377 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8c0036a0 con 0x7fee9c1596e0 2023-11-28T13:00:14.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.184+0000 7feea3f8a700 1 -- 172.21.15.111:0/2065050377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee9c1596e0 msgr2=0x7fee9c0ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:14.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.184+0000 7feea3f8a700 1 --2- 172.21.15.111:0/2065050377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee9c1596e0 0x7fee9c0ce6e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0ca1c0 tx=0x7fee8c009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:14.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.184+0000 7feea3f8a700 1 -- 172.21.15.111:0/2065050377 shutdown_connections 2023-11-28T13:00:14.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.184+0000 7feea3f8a700 1 --2- 172.21.15.111:0/2065050377 >> v2:172.21.15.149:3300/0 conn(0x7fee9c159310 0x7fee9c0ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:14.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.184+0000 7feea3f8a700 1 --2- 172.21.15.111:0/2065050377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee9c1596e0 0x7fee9c0ce6e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:14.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.184+0000 7feea3f8a700 1 -- 172.21.15.111:0/2065050377 >> 172.21.15.111:0/2065050377 conn(0x7fee9c06db10 msgr2=0x7fee9c06df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:14.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.184+0000 7feea3f8a700 1 -- 172.21.15.111:0/2065050377 shutdown_connections 2023-11-28T13:00:14.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.184+0000 7feea3f8a700 1 -- 172.21.15.111:0/2065050377 wait complete. 2023-11-28T13:00:14.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.185+0000 7feea3f8a700 1 Processor -- start 2023-11-28T13:00:14.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.185+0000 7feea3f8a700 1 -- start start 2023-11-28T13:00:14.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea3f8a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee9c159310 0x7fee9c153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:14.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea3f8a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee9c1596e0 0x7fee9c15b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:14.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea3f8a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee9c15cfd0 0x7fee9c0b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:14.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea3f8a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c15bab0 con 0x7fee9c15cfd0 2023-11-28T13:00:14.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea3f8a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c15bd90 con 0x7fee9c159310 2023-11-28T13:00:14.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea3f8a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c15c070 con 0x7fee9c1596e0 2023-11-28T13:00:14.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea2527700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee9c15cfd0 0x7fee9c0b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:14.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea2527700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee9c15cfd0 0x7fee9c0b3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35218/0 (socket says 172.21.15.111:35218) 2023-11-28T13:00:14.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea2527700 1 -- 172.21.15.111:0/4022122717 learned_addr learned my addr 172.21.15.111:0/4022122717 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:14.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea1d26700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee9c159310 0x7fee9c153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:14.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea1525700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee9c1596e0 0x7fee9c15b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:14.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea2527700 1 -- 172.21.15.111:0/4022122717 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee9c1596e0 msgr2=0x7fee9c15b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:14.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea2527700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee9c1596e0 0x7fee9c15b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:14.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea2527700 1 -- 172.21.15.111:0/4022122717 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee9c159310 msgr2=0x7fee9c153a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:14.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea2527700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee9c159310 0x7fee9c153a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:14.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea2527700 1 -- 172.21.15.111:0/4022122717 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c009550 con 0x7fee9c15cfd0 2023-11-28T13:00:14.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.186+0000 7feea1d26700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee9c159310 0x7fee9c153a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:14.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.187+0000 7feea2527700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee9c15cfd0 0x7fee9c0b3240 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fee9400b970 tx=0x7fee9400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:14.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.187+0000 7fee927fc700 1 -- 172.21.15.111:0/4022122717 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9400c9b0 con 0x7fee9c15cfd0 2023-11-28T13:00:14.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.187+0000 7fee927fc700 1 -- 172.21.15.111:0/4022122717 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9400cb10 con 0x7fee9c15cfd0 2023-11-28T13:00:14.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.187+0000 7feea3f8a700 1 -- 172.21.15.111:0/4022122717 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee9c0cb3d0 con 0x7fee9c15cfd0 2023-11-28T13:00:14.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.187+0000 7fee927fc700 1 -- 172.21.15.111:0/4022122717 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94014440 con 0x7fee9c15cfd0 2023-11-28T13:00:14.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.187+0000 7feea3f8a700 1 -- 172.21.15.111:0/4022122717 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee9c0cb0f0 con 0x7fee9c15cfd0 2023-11-28T13:00:14.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.188+0000 7feea3f8a700 1 -- 172.21.15.111:0/4022122717 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee9c0cc900 con 0x7fee9c15cfd0 2023-11-28T13:00:14.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.188+0000 7fee927fc700 1 -- 172.21.15.111:0/4022122717 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fee9401b030 con 0x7fee9c15cfd0 2023-11-28T13:00:14.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.188+0000 7fee927fc700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fee800682f0 0x7fee8006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:14.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.189+0000 7fee927fc700 1 -- 172.21.15.111:0/4022122717 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fee94085910 con 0x7fee9c15cfd0 2023-11-28T13:00:14.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.189+0000 7feea1d26700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fee800682f0 0x7fee8006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:14.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.189+0000 7feea1d26700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fee800682f0 0x7fee8006a7a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fee98006fd0 tx=0x7fee98008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:14.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.193+0000 7fee927fc700 1 -- 172.21.15.111:0/4022122717 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fee94052030 con 0x7fee9c15cfd0 2023-11-28T13:00:14.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:14.419+0000 7feea3f8a700 1 -- 172.21.15.111:0/4022122717 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fee9c067170 con 0x7fee9c15cfd0 2023-11-28T13:00:15.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.044+0000 7fee927fc700 1 -- 172.21.15.111:0/4022122717 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7fee94055650 con 0x7fee9c15cfd0 2023-11-28T13:00:15.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.047+0000 7feea3f8a700 1 -- 172.21.15.111:0/4022122717 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fee800682f0 msgr2=0x7fee8006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.047+0000 7feea3f8a700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fee800682f0 0x7fee8006a7a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fee98006fd0 tx=0x7fee98008040 comp rx=0 tx=0).stop 2023-11-28T13:00:15.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.047+0000 7feea3f8a700 1 -- 172.21.15.111:0/4022122717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee9c15cfd0 msgr2=0x7fee9c0b3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.047+0000 7feea3f8a700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee9c15cfd0 0x7fee9c0b3240 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fee9400b970 tx=0x7fee9400bd30 comp rx=0 tx=0).stop 2023-11-28T13:00:15.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.047+0000 7feea3f8a700 1 -- 172.21.15.111:0/4022122717 shutdown_connections 2023-11-28T13:00:15.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.047+0000 7feea3f8a700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee9c15cfd0 0x7fee9c0b3240 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.047+0000 7feea3f8a700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fee800682f0 0x7fee8006a7a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.047+0000 7feea3f8a700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee9c159310 0x7fee9c153a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.047+0000 7feea3f8a700 1 --2- 172.21.15.111:0/4022122717 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee9c1596e0 0x7fee9c15b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.047+0000 7feea3f8a700 1 -- 172.21.15.111:0/4022122717 >> 172.21.15.111:0/4022122717 conn(0x7fee9c06db10 msgr2=0x7fee9c1580f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:15.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.047+0000 7feea3f8a700 1 -- 172.21.15.111:0/4022122717 shutdown_connections 2023-11-28T13:00:15.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.047+0000 7feea3f8a700 1 -- 172.21.15.111:0/4022122717 wait complete. 2023-11-28T13:00:15.063 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T13:00:15.063 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:00:15.069 INFO:tasks.ceph.mds.d.smithi149.stderr:2023-11-28T13:00:15.067+0000 7f4308044700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T13:00:15.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.185+0000 7f63a993c700 1 Processor -- start 2023-11-28T13:00:15.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.185+0000 7f63a993c700 1 -- start start 2023-11-28T13:00:15.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.185+0000 7f63a993c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f63a4158cb0 0x7f63a40ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.185+0000 7f63a993c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63a4071020 0x7f63a406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.185+0000 7f63a993c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a406d870 con 0x7f63a4159080 2023-11-28T13:00:15.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.185+0000 7f63a993c700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a406cc60 con 0x7f63a4158cb0 2023-11-28T13:00:15.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.185+0000 7f63a993c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a406cf40 con 0x7f63a4071020 2023-11-28T13:00:15.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a27fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f63a4159080 0x7f63a4070910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36002/0 (socket says 172.21.15.111:36002) 2023-11-28T13:00:15.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a27fc700 1 -- 172.21.15.111:0/1218865287 learned_addr learned my addr 172.21.15.111:0/1218865287 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:15.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966863513 0 0) 0x7f63a406d870 con 0x7f63a4159080 2023-11-28T13:00:15.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a2ffd700 1 --2- 172.21.15.111:0/1218865287 >> v2:172.21.15.149:3300/0 conn(0x7f63a4158cb0 0x7f63a40ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f638c0046c0 con 0x7f63a4159080 2023-11-28T13:00:15.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a37fe700 1 --2- 172.21.15.111:0/1218865287 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63a4071020 0x7f63a406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3537277409 0 0) 0x7f638c0046c0 con 0x7f63a4159080 2023-11-28T13:00:15.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63a4071020 msgr2=0x7f63a406dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:15.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 --2- 172.21.15.111:0/1218865287 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63a4071020 0x7f63a406dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 >> v2:172.21.15.149:3300/0 conn(0x7f63a4158cb0 msgr2=0x7f63a40ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:15.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 --2- 172.21.15.111:0/1218865287 >> v2:172.21.15.149:3300/0 conn(0x7f63a4158cb0 0x7f63a40ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a406d220 con 0x7f63a4159080 2023-11-28T13:00:15.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f6398001670 con 0x7f63a4159080 2023-11-28T13:00:15.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 >> v1:172.21.15.111:6789/0 conn(0x7f63a4159080 legacy=0x7f63a4070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 --2- 172.21.15.111:0/1218865287 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f638c004b10 0x7f638c004f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 --2- 172.21.15.111:0/1218865287 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63a4071020 0x7f638c005960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 --2- 172.21.15.111:0/1218865287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f638c005ea0 0x7f638c006310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a406cc60 con 0x7f638c005ea0 2023-11-28T13:00:15.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a406cf40 con 0x7f638c004b10 2023-11-28T13:00:15.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638c00cb20 con 0x7f63a4071020 2023-11-28T13:00:15.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a37fe700 1 --2- 172.21.15.111:0/1218865287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f638c005ea0 0x7f638c006310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a2ffd700 1 --2- 172.21.15.111:0/1218865287 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f638c004b10 0x7f638c004f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.186+0000 7f63a27fc700 1 --2- 172.21.15.111:0/1218865287 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63a4071020 0x7f638c005960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.187+0000 7f63a37fe700 1 -- 172.21.15.111:0/1218865287 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63a4071020 msgr2=0x7f638c005960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:15.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.187+0000 7f63a37fe700 1 --2- 172.21.15.111:0/1218865287 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63a4071020 0x7f638c005960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.187+0000 7f63a37fe700 1 -- 172.21.15.111:0/1218865287 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f638c004b10 msgr2=0x7f638c004f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:15.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.187+0000 7f63a37fe700 1 --2- 172.21.15.111:0/1218865287 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f638c004b10 0x7f638c004f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.187+0000 7f63a37fe700 1 -- 172.21.15.111:0/1218865287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c0046c0 con 0x7f638c005ea0 2023-11-28T13:00:15.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.187+0000 7f63a37fe700 1 --2- 172.21.15.111:0/1218865287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f638c005ea0 0x7f638c006310 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f639c009fc0 tx=0x7f638c006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=106f0b2112cb6d0c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:15.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.187+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c00ebd0 con 0x7f638c005ea0 2023-11-28T13:00:15.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.187+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c00f040 con 0x7f638c005ea0 2023-11-28T13:00:15.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.187+0000 7f63a1ffb700 1 -- 172.21.15.111:0/1218865287 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c014480 con 0x7f638c005ea0 2023-11-28T13:00:15.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.187+0000 7f63a993c700 1 -- 172.21.15.111:0/1218865287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f638c005ea0 msgr2=0x7f638c006310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.187+0000 7f63a993c700 1 --2- 172.21.15.111:0/1218865287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f638c005ea0 0x7f638c006310 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f639c009fc0 tx=0x7f638c006990 comp rx=0 tx=0).stop 2023-11-28T13:00:15.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.187+0000 7f63a37fe700 1 -- 172.21.15.111:0/1218865287 reap_dead start 2023-11-28T13:00:15.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.188+0000 7f63a993c700 1 -- 172.21.15.111:0/1218865287 shutdown_connections 2023-11-28T13:00:15.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.188+0000 7f63a993c700 1 -- 172.21.15.111:0/1218865287 >> 172.21.15.111:0/1218865287 conn(0x7f63a40cb490 msgr2=0x7f63a40cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:15.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.188+0000 7f63a993c700 1 -- 172.21.15.111:0/1218865287 shutdown_connections 2023-11-28T13:00:15.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.188+0000 7f63a993c700 1 -- 172.21.15.111:0/1218865287 wait complete. 2023-11-28T13:00:15.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.189+0000 7f63a993c700 1 Processor -- start 2023-11-28T13:00:15.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.189+0000 7f63a993c700 1 -- start start 2023-11-28T13:00:15.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a993c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f638c005ea0 0x7f63a4156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a993c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a40acde0 0x7f63a40c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a993c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63a41dffe0 0x7f63a41e0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a993c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a41e3470 con 0x7f63a40acde0 2023-11-28T13:00:15.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a993c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a41e09c0 con 0x7f638c005ea0 2023-11-28T13:00:15.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a993c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40c8950 con 0x7f63a41dffe0 2023-11-28T13:00:15.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a40acde0 0x7f63a40c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a40acde0 0x7f63a40c8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35238/0 (socket says 172.21.15.111:35238) 2023-11-28T13:00:15.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a27fc700 1 -- 172.21.15.111:0/3498595920 learned_addr learned my addr 172.21.15.111:0/3498595920 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:15.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a27fc700 1 -- 172.21.15.111:0/3498595920 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63a41dffe0 msgr2=0x7f63a41e0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a27fc700 1 --2- 172.21.15.111:0/3498595920 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63a41dffe0 0x7f63a41e0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a27fc700 1 -- 172.21.15.111:0/3498595920 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f638c005ea0 msgr2=0x7f63a4156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a27fc700 1 --2- 172.21.15.111:0/3498595920 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f638c005ea0 0x7f63a4156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a27fc700 1 -- 172.21.15.111:0/3498595920 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a4071b20 con 0x7f63a40acde0 2023-11-28T13:00:15.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a27fc700 1 --2- 172.21.15.111:0/3498595920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a40acde0 0x7f63a40c8410 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6398007340 tx=0x7f6398007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:15.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f638bfff700 1 -- 172.21.15.111:0/3498595920 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6398007820 con 0x7f63a40acde0 2023-11-28T13:00:15.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f638bfff700 1 -- 172.21.15.111:0/3498595920 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639800f7f0 con 0x7f63a40acde0 2023-11-28T13:00:15.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a993c700 1 -- 172.21.15.111:0/3498595920 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a406d500 con 0x7f63a40acde0 2023-11-28T13:00:15.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f638bfff700 1 -- 172.21.15.111:0/3498595920 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639800fba0 con 0x7f63a40acde0 2023-11-28T13:00:15.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.190+0000 7f63a993c700 1 -- 172.21.15.111:0/3498595920 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a40c9300 con 0x7f63a40acde0 2023-11-28T13:00:15.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.191+0000 7f63a993c700 1 -- 172.21.15.111:0/3498595920 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a4067880 con 0x7f63a40acde0 2023-11-28T13:00:15.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.192+0000 7f638bfff700 1 -- 172.21.15.111:0/3498595920 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f639800fd00 con 0x7f63a40acde0 2023-11-28T13:00:15.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.192+0000 7f638bfff700 1 --2- 172.21.15.111:0/3498595920 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63900682f0 0x7f639006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.192+0000 7f638bfff700 1 -- 172.21.15.111:0/3498595920 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6398085f60 con 0x7f63a40acde0 2023-11-28T13:00:15.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.192+0000 7f63a2ffd700 1 --2- 172.21.15.111:0/3498595920 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63900682f0 0x7f639006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.192+0000 7f63a2ffd700 1 --2- 172.21.15.111:0/3498595920 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63900682f0 0x7f639006a7a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6394006f10 tx=0x7f6394006e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:15.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.196+0000 7f638bfff700 1 -- 172.21.15.111:0/3498595920 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6398052680 con 0x7f63a40acde0 2023-11-28T13:00:15.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.420+0000 7f63a993c700 1 -- 172.21.15.111:0/3498595920 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63a40c8b80 con 0x7f63a40acde0 2023-11-28T13:00:15.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.421+0000 7f638bfff700 1 -- 172.21.15.111:0/3498595920 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f6398055ca0 con 0x7f63a40acde0 2023-11-28T13:00:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:15.423 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3254653481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3254653481},{"type":"v1","addr":"172.21.15.149:6835","nonce":3254653481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1890904311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1890904311},{"type":"v1","addr":"172.21.15.111:6836","nonce":1890904311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1592804351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1592804351},{"type":"v1","addr":"172.21.15.111:6837","nonce":1592804351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:12.011963+0000","modified":"2023-11-28T13:00:15.034385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5352":{"gid":5352,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.149:6834/3579364312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":3579364312},{"type":"v1","addr":"172.21.15.149:6834","nonce":3579364312}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T13:00:15.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.423+0000 7f63a993c700 1 -- 172.21.15.111:0/3498595920 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63900682f0 msgr2=0x7f639006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.423+0000 7f63a993c700 1 --2- 172.21.15.111:0/3498595920 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63900682f0 0x7f639006a7a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6394006f10 tx=0x7f6394006e80 comp rx=0 tx=0).stop 2023-11-28T13:00:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.423+0000 7f63a993c700 1 -- 172.21.15.111:0/3498595920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a40acde0 msgr2=0x7f63a40c8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.423+0000 7f63a993c700 1 --2- 172.21.15.111:0/3498595920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a40acde0 0x7f63a40c8410 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6398007340 tx=0x7f6398007700 comp rx=0 tx=0).stop 2023-11-28T13:00:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.423+0000 7f63a993c700 1 -- 172.21.15.111:0/3498595920 shutdown_connections 2023-11-28T13:00:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.423+0000 7f63a993c700 1 --2- 172.21.15.111:0/3498595920 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63a41dffe0 0x7f63a41e0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.423+0000 7f63a993c700 1 --2- 172.21.15.111:0/3498595920 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63900682f0 0x7f639006a7a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.423+0000 7f63a993c700 1 --2- 172.21.15.111:0/3498595920 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f638c005ea0 0x7f63a4156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.423+0000 7f63a993c700 1 --2- 172.21.15.111:0/3498595920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63a40acde0 0x7f63a40c8410 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.423+0000 7f63a993c700 1 -- 172.21.15.111:0/3498595920 >> 172.21.15.111:0/3498595920 conn(0x7f63a40cb490 msgr2=0x7f63a40ccc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.423+0000 7f63a993c700 1 -- 172.21.15.111:0/3498595920 shutdown_connections 2023-11-28T13:00:15.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.423+0000 7f63a993c700 1 -- 172.21.15.111:0/3498595920 wait complete. 2023-11-28T13:00:15.426 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 17 2023-11-28T13:00:15.437 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:00:15.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.554+0000 7f4714d23700 1 Processor -- start 2023-11-28T13:00:15.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.554+0000 7f4714d23700 1 -- start start 2023-11-28T13:00:15.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.555+0000 7f4714d23700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f471015a5d0 0x7f47100ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.555+0000 7f4714d23700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f47100b0160 0x7f47100ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.555+0000 7f4714d23700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47100cb2d0 0x7f471015e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.555+0000 7f4714d23700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47100cc1b0 con 0x7f47100b0160 2023-11-28T13:00:15.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.555+0000 7f4714d23700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47100cc390 con 0x7f471015a5d0 2023-11-28T13:00:15.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.555+0000 7f4714d23700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47100cc670 con 0x7f47100cb2d0 2023-11-28T13:00:15.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.555+0000 7f470dd9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f47100b0160 0x7f47100ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470dd9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f47100b0160 0x7f47100ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35244/0 (socket says 172.21.15.111:35244) 2023-11-28T13:00:15.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470dd9b700 1 -- 172.21.15.111:0/1487941611 learned_addr learned my addr 172.21.15.111:0/1487941611 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:15.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470e59c700 1 --2- 172.21.15.111:0/1487941611 >> v2:172.21.15.149:3300/0 conn(0x7f471015a5d0 0x7f47100ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470dd9b700 1 -- 172.21.15.111:0/1487941611 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47100cb2d0 msgr2=0x7f471015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:15.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470ed9d700 1 --2- 172.21.15.111:0/1487941611 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47100cb2d0 0x7f471015e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470dd9b700 1 --2- 172.21.15.111:0/1487941611 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47100cb2d0 0x7f471015e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470dd9b700 1 -- 172.21.15.111:0/1487941611 >> v2:172.21.15.149:3300/0 conn(0x7f471015a5d0 msgr2=0x7f47100ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:15.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470dd9b700 1 --2- 172.21.15.111:0/1487941611 >> v2:172.21.15.149:3300/0 conn(0x7f471015a5d0 0x7f47100ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470dd9b700 1 -- 172.21.15.111:0/1487941611 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47100cc950 con 0x7f47100b0160 2023-11-28T13:00:15.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470e59c700 1 --2- 172.21.15.111:0/1487941611 >> v2:172.21.15.149:3300/0 conn(0x7f471015a5d0 0x7f47100ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:15.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470dd9b700 1 --2- 172.21.15.111:0/1487941611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47100b0160 0x7f47100ced90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4710067d30 tx=0x7f4700009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7004ffe1706416d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:15.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470d59a700 1 -- 172.21.15.111:0/1487941611 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470000b870 con 0x7f47100b0160 2023-11-28T13:00:15.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470d59a700 1 -- 172.21.15.111:0/1487941611 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470000b9d0 con 0x7f47100b0160 2023-11-28T13:00:15.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.556+0000 7f470d59a700 1 -- 172.21.15.111:0/1487941611 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470000bd60 con 0x7f47100b0160 2023-11-28T13:00:15.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.557+0000 7f4714d23700 1 -- 172.21.15.111:0/1487941611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47100b0160 msgr2=0x7f47100ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.557+0000 7f4714d23700 1 --2- 172.21.15.111:0/1487941611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47100b0160 0x7f47100ced90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4710067d30 tx=0x7f4700009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.557+0000 7f4714d23700 1 -- 172.21.15.111:0/1487941611 shutdown_connections 2023-11-28T13:00:15.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.557+0000 7f4714d23700 1 --2- 172.21.15.111:0/1487941611 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47100cb2d0 0x7f471015e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.557+0000 7f4714d23700 1 --2- 172.21.15.111:0/1487941611 >> v2:172.21.15.149:3300/0 conn(0x7f471015a5d0 0x7f47100ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.557+0000 7f4714d23700 1 --2- 172.21.15.111:0/1487941611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47100b0160 0x7f47100ced90 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.557+0000 7f4714d23700 1 -- 172.21.15.111:0/1487941611 >> 172.21.15.111:0/1487941611 conn(0x7f471009a040 msgr2=0x7f47100d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:15.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.557+0000 7f4714d23700 1 -- 172.21.15.111:0/1487941611 shutdown_connections 2023-11-28T13:00:15.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.557+0000 7f4714d23700 1 -- 172.21.15.111:0/1487941611 wait complete. 2023-11-28T13:00:15.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.558+0000 7f4714d23700 1 Processor -- start 2023-11-28T13:00:15.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.558+0000 7f4714d23700 1 -- start start 2023-11-28T13:00:15.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f4714d23700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47100b0160 0x7f47101542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f4714d23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47100cb2d0 0x7f47100c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f4714d23700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f471015a5d0 0x7f47101580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f4714d23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4710154820 con 0x7f47100cb2d0 2023-11-28T13:00:15.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f4714d23700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47100ca540 con 0x7f471015a5d0 2023-11-28T13:00:15.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f4714d23700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47100ca820 con 0x7f47100b0160 2023-11-28T13:00:15.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f470dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47100cb2d0 0x7f47100c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f470dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47100cb2d0 0x7f47100c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35250/0 (socket says 172.21.15.111:35250) 2023-11-28T13:00:15.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f470dd9b700 1 -- 172.21.15.111:0/3932039507 learned_addr learned my addr 172.21.15.111:0/3932039507 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:15.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f470e59c700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47100b0160 0x7f47101542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f470ed9d700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f471015a5d0 0x7f47101580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f470dd9b700 1 -- 172.21.15.111:0/3932039507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47100b0160 msgr2=0x7f47101542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f470dd9b700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47100b0160 0x7f47101542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f470dd9b700 1 -- 172.21.15.111:0/3932039507 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f471015a5d0 msgr2=0x7f47101580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f470dd9b700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f471015a5d0 0x7f47101580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f470dd9b700 1 -- 172.21.15.111:0/3932039507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4700009550 con 0x7f47100cb2d0 2023-11-28T13:00:15.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f470ed9d700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f471015a5d0 0x7f47101580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:15.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f470dd9b700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47100cb2d0 0x7f47100c9b80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f471006d8f0 tx=0x7f4700004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:15.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f46feffd700 1 -- 172.21.15.111:0/3932039507 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700004310 con 0x7f47100cb2d0 2023-11-28T13:00:15.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f46feffd700 1 -- 172.21.15.111:0/3932039507 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470001f070 con 0x7f47100cb2d0 2023-11-28T13:00:15.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f4714d23700 1 -- 172.21.15.111:0/3932039507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47100ccf10 con 0x7f47100cb2d0 2023-11-28T13:00:15.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.559+0000 7f46feffd700 1 -- 172.21.15.111:0/3932039507 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47000036a0 con 0x7f47100cb2d0 2023-11-28T13:00:15.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.560+0000 7f4714d23700 1 -- 172.21.15.111:0/3932039507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47100ccc30 con 0x7f47100cb2d0 2023-11-28T13:00:15.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.560+0000 7f4714d23700 1 -- 172.21.15.111:0/3932039507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471015a9a0 con 0x7f47100cb2d0 2023-11-28T13:00:15.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.561+0000 7f46feffd700 1 -- 172.21.15.111:0/3932039507 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f470002a020 con 0x7f47100cb2d0 2023-11-28T13:00:15.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.561+0000 7f46feffd700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f46f00682f0 0x7f46f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.561+0000 7f46feffd700 1 -- 172.21.15.111:0/3932039507 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4700087330 con 0x7f47100cb2d0 2023-11-28T13:00:15.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.561+0000 7f470e59c700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f46f00682f0 0x7f46f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.562+0000 7f470e59c700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f46f00682f0 0x7f46f006a7a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f47100cacd0 tx=0x7f4704008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:15.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.566+0000 7f46feffd700 1 -- 172.21.15.111:0/3932039507 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4700053a50 con 0x7f47100cb2d0 2023-11-28T13:00:15.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.749+0000 7f4714d23700 1 -- 172.21.15.111:0/3932039507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4710067170 con 0x7f47100cb2d0 2023-11-28T13:00:15.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.750+0000 7f46feffd700 1 -- 172.21.15.111:0/3932039507 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+18176 (secure 0 0 0) 0x7f4700057070 con 0x7f47100cb2d0 2023-11-28T13:00:15.752 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:15.752 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":41,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:13.013725+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:09.793782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:10.979525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:00:15.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.752+0000 7f4714d23700 1 -- 172.21.15.111:0/3932039507 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f46f00682f0 msgr2=0x7f46f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.752+0000 7f4714d23700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f46f00682f0 0x7f46f006a7a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f47100cacd0 tx=0x7f4704008040 comp rx=0 tx=0).stop 2023-11-28T13:00:15.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.752+0000 7f4714d23700 1 -- 172.21.15.111:0/3932039507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47100cb2d0 msgr2=0x7f47100c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.752+0000 7f4714d23700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47100cb2d0 0x7f47100c9b80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f471006d8f0 tx=0x7f4700004750 comp rx=0 tx=0).stop 2023-11-28T13:00:15.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.753+0000 7f4714d23700 1 -- 172.21.15.111:0/3932039507 shutdown_connections 2023-11-28T13:00:15.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.753+0000 7f4714d23700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f46f00682f0 0x7f46f006a7a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.753+0000 7f4714d23700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f471015a5d0 0x7f47101580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.753+0000 7f4714d23700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47100b0160 0x7f47101542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.753+0000 7f4714d23700 1 --2- 172.21.15.111:0/3932039507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47100cb2d0 0x7f47100c9b80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.753+0000 7f4714d23700 1 -- 172.21.15.111:0/3932039507 >> 172.21.15.111:0/3932039507 conn(0x7f471009a040 msgr2=0x7f4710154f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:15.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.753+0000 7f4714d23700 1 -- 172.21.15.111:0/3932039507 shutdown_connections 2023-11-28T13:00:15.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.753+0000 7f4714d23700 1 -- 172.21.15.111:0/3932039507 wait complete. 2023-11-28T13:00:15.766 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T13:00:15.767 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:00:15.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.885+0000 7f42c2e5d700 1 Processor -- start 2023-11-28T13:00:15.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.885+0000 7f42c2e5d700 1 -- start start 2023-11-28T13:00:15.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.885+0000 7f42c2e5d700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f42bc0d2df0 0x7f42bc0cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42c2e5d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f42bc0b4fc0 0x7f42bc0d31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42c2e5d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42bc0b1c60 0x7f42bc0cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42c2e5d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42bc0d3d30 con 0x7f42bc0b4fc0 2023-11-28T13:00:15.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42c2e5d700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42bc0d1950 con 0x7f42bc0d2df0 2023-11-28T13:00:15.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42c2e5d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc0b2500 con 0x7f42bc0b1c60 2023-11-28T13:00:15.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42bbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f42bc0b4fc0 0x7f42bc0d31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42bbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f42bc0b4fc0 0x7f42bc0d31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35256/0 (socket says 172.21.15.111:35256) 2023-11-28T13:00:15.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42bbfff700 1 -- 172.21.15.111:0/3005639755 learned_addr learned my addr 172.21.15.111:0/3005639755 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:15.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42c0bf9700 1 --2- 172.21.15.111:0/3005639755 >> v2:172.21.15.149:3300/0 conn(0x7f42bc0d2df0 0x7f42bc0cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42c13fa700 1 --2- 172.21.15.111:0/3005639755 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42bc0b1c60 0x7f42bc0cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42bbfff700 1 -- 172.21.15.111:0/3005639755 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42bc0b1c60 msgr2=0x7f42bc0cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:15.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42bbfff700 1 --2- 172.21.15.111:0/3005639755 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42bc0b1c60 0x7f42bc0cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42bbfff700 1 -- 172.21.15.111:0/3005639755 >> v2:172.21.15.149:3300/0 conn(0x7f42bc0d2df0 msgr2=0x7f42bc0cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:15.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42bbfff700 1 --2- 172.21.15.111:0/3005639755 >> v2:172.21.15.149:3300/0 conn(0x7f42bc0d2df0 0x7f42bc0cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42bbfff700 1 -- 172.21.15.111:0/3005639755 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc0d3a50 con 0x7f42bc0b4fc0 2023-11-28T13:00:15.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42c0bf9700 1 --2- 172.21.15.111:0/3005639755 >> v2:172.21.15.149:3300/0 conn(0x7f42bc0d2df0 0x7f42bc0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:15.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42bbfff700 1 --2- 172.21.15.111:0/3005639755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc0b4fc0 0x7f42bc0d31c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0d27f0 tx=0x7f42a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7143dbcaf718261b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:15.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42bb7fe700 1 -- 172.21.15.111:0/3005639755 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a800b870 con 0x7f42bc0b4fc0 2023-11-28T13:00:15.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42bb7fe700 1 -- 172.21.15.111:0/3005639755 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a800b9d0 con 0x7f42bc0b4fc0 2023-11-28T13:00:15.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.886+0000 7f42bb7fe700 1 -- 172.21.15.111:0/3005639755 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a800bd60 con 0x7f42bc0b4fc0 2023-11-28T13:00:15.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.887+0000 7f42c2e5d700 1 -- 172.21.15.111:0/3005639755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc0b4fc0 msgr2=0x7f42bc0d31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.887+0000 7f42c2e5d700 1 --2- 172.21.15.111:0/3005639755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc0b4fc0 0x7f42bc0d31c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0d27f0 tx=0x7f42a8009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.887+0000 7f42c2e5d700 1 -- 172.21.15.111:0/3005639755 shutdown_connections 2023-11-28T13:00:15.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.887+0000 7f42c2e5d700 1 --2- 172.21.15.111:0/3005639755 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42bc0b1c60 0x7f42bc0cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.887+0000 7f42c2e5d700 1 --2- 172.21.15.111:0/3005639755 >> v2:172.21.15.149:3300/0 conn(0x7f42bc0d2df0 0x7f42bc0cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.887+0000 7f42c2e5d700 1 --2- 172.21.15.111:0/3005639755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc0b4fc0 0x7f42bc0d31c0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.887+0000 7f42c2e5d700 1 -- 172.21.15.111:0/3005639755 >> 172.21.15.111:0/3005639755 conn(0x7f42bc0719a0 msgr2=0x7f42bc071da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:15.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.887+0000 7f42c2e5d700 1 -- 172.21.15.111:0/3005639755 shutdown_connections 2023-11-28T13:00:15.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.887+0000 7f42c2e5d700 1 -- 172.21.15.111:0/3005639755 wait complete. 2023-11-28T13:00:15.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.888+0000 7f42c2e5d700 1 Processor -- start 2023-11-28T13:00:15.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.888+0000 7f42c2e5d700 1 -- start start 2023-11-28T13:00:15.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c2e5d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc0b1c60 0x7f42bc152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c2e5d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42bc0b4fc0 0x7f42bc0a9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c2e5d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42bc0d2df0 0x7f42bc0b68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c2e5d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc1528a0 con 0x7f42bc0b1c60 2023-11-28T13:00:15.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c2e5d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc0aa280 con 0x7f42bc0b4fc0 2023-11-28T13:00:15.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c2e5d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc0aa560 con 0x7f42bc0d2df0 2023-11-28T13:00:15.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c0bf9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc0b1c60 0x7f42bc152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c0bf9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc0b1c60 0x7f42bc152360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35260/0 (socket says 172.21.15.111:35260) 2023-11-28T13:00:15.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c0bf9700 1 -- 172.21.15.111:0/866001421 learned_addr learned my addr 172.21.15.111:0/866001421 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:15.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42bbfff700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42bc0b4fc0 0x7f42bc0a9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c13fa700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42bc0d2df0 0x7f42bc0b68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c0bf9700 1 -- 172.21.15.111:0/866001421 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42bc0d2df0 msgr2=0x7f42bc0b68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c0bf9700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42bc0d2df0 0x7f42bc0b68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c0bf9700 1 -- 172.21.15.111:0/866001421 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42bc0b4fc0 msgr2=0x7f42bc0a9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:15.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c0bf9700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42bc0b4fc0 0x7f42bc0a9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:15.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c0bf9700 1 -- 172.21.15.111:0/866001421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a8009550 con 0x7f42bc0b1c60 2023-11-28T13:00:15.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.889+0000 7f42c13fa700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42bc0d2df0 0x7f42bc0b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:15.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.890+0000 7f42c0bf9700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc0b1c60 0x7f42bc152360 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f42b000e810 tx=0x7f42b000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:15.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.890+0000 7f42b97fa700 1 -- 172.21.15.111:0/866001421 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b0013610 con 0x7f42bc0b1c60 2023-11-28T13:00:15.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.890+0000 7f42b97fa700 1 -- 172.21.15.111:0/866001421 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b0013770 con 0x7f42bc0b1c60 2023-11-28T13:00:15.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.890+0000 7f42c2e5d700 1 -- 172.21.15.111:0/866001421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42bc0b88f0 con 0x7f42bc0b1c60 2023-11-28T13:00:15.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.890+0000 7f42b97fa700 1 -- 172.21.15.111:0/866001421 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b0013b20 con 0x7f42bc0b1c60 2023-11-28T13:00:15.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.890+0000 7f42c2e5d700 1 -- 172.21.15.111:0/866001421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42bc0d15e0 con 0x7f42bc0b1c60 2023-11-28T13:00:15.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.891+0000 7f42c2e5d700 1 -- 172.21.15.111:0/866001421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42bc0b21b0 con 0x7f42bc0b1c60 2023-11-28T13:00:15.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.891+0000 7f42b97fa700 1 -- 172.21.15.111:0/866001421 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f42b0023070 con 0x7f42bc0b1c60 2023-11-28T13:00:15.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.891+0000 7f42b97fa700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42a00682f0 0x7f42a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:15.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.892+0000 7f42b97fa700 1 -- 172.21.15.111:0/866001421 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f42b0085a20 con 0x7f42bc0b1c60 2023-11-28T13:00:15.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.892+0000 7f42bbfff700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42a00682f0 0x7f42a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:15.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.892+0000 7f42bbfff700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42a00682f0 0x7f42a006a7a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f42a8009840 tx=0x7f42a8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:15.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:15.896+0000 7f42b97fa700 1 -- 172.21.15.111:0/866001421 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f42b00520b0 con 0x7f42bc0b1c60 2023-11-28T13:00:16.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.077+0000 7f42c2e5d700 1 -- 172.21.15.111:0/866001421 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f42bc0aa6d0 con 0x7f42a00682f0 2023-11-28T13:00:16.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.086+0000 7f42b97fa700 1 -- 172.21.15.111:0/866001421 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+300126 (secure 0 0 0) 0x7f42bc0aa6d0 con 0x7f42a00682f0 2023-11-28T13:00:16.090 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:16.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.089+0000 7f42c2e5d700 1 -- 172.21.15.111:0/866001421 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42a00682f0 msgr2=0x7f42a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.089+0000 7f42c2e5d700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42a00682f0 0x7f42a006a7a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f42a8009840 tx=0x7f42a8008030 comp rx=0 tx=0).stop 2023-11-28T13:00:16.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.089+0000 7f42c2e5d700 1 -- 172.21.15.111:0/866001421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc0b1c60 msgr2=0x7f42bc152360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.089+0000 7f42c2e5d700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc0b1c60 0x7f42bc152360 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f42b000e810 tx=0x7f42b000eb20 comp rx=0 tx=0).stop 2023-11-28T13:00:16.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.089+0000 7f42c2e5d700 1 -- 172.21.15.111:0/866001421 shutdown_connections 2023-11-28T13:00:16.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.089+0000 7f42c2e5d700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f42bc0d2df0 0x7f42bc0b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.089+0000 7f42c2e5d700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42bc0b1c60 0x7f42bc152360 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.089+0000 7f42c2e5d700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42a00682f0 0x7f42a006a7a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.089+0000 7f42c2e5d700 1 --2- 172.21.15.111:0/866001421 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42bc0b4fc0 0x7f42bc0a9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.089+0000 7f42c2e5d700 1 -- 172.21.15.111:0/866001421 >> 172.21.15.111:0/866001421 conn(0x7f42bc0719a0 msgr2=0x7f42bc0b26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:16.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.089+0000 7f42c2e5d700 1 -- 172.21.15.111:0/866001421 shutdown_connections 2023-11-28T13:00:16.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.089+0000 7f42c2e5d700 1 -- 172.21.15.111:0/866001421 wait complete. 2023-11-28T13:00:16.094 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:00:16.105 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2023-11-28T13:00:14.351189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":25,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":70032,"kb_used_data":4560,"kb_used_omap":0,"kb_used_meta":65408,"kb_avail":754904688,"statfs":{"total":773094113280,"available":773022400512,"internally_reserved":0,"allocated":4669440,"data_stored":2706556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66977792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000696"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622561+0000","last_change":"2023-11-28T13:00:10.622561+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622561+0000","last_undegraded":"2023-11-28T13:00:10.622561+0000","last_fullsized":"2023-11-28T13:00:10.622561+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:47.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.618665+0000","last_change":"2023-11-28T13:00:10.618665+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.618665+0000","last_undegraded":"2023-11-28T13:00:10.618665+0000","last_fullsized":"2023-11-28T13:00:10.618665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:48:31.549366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621901+0000","last_change":"2023-11-28T13:00:10.621901+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621901+0000","last_undegraded":"2023-11-28T13:00:10.621901+0000","last_fullsized":"2023-11-28T13:00:10.621901+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:57:53.793229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619096+0000","last_change":"2023-11-28T13:00:10.619096+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619096+0000","last_undegraded":"2023-11-28T13:00:10.619096+0000","last_fullsized":"2023-11-28T13:00:10.619096+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.624870+0000","last_change":"2023-11-28T13:00:10.624870+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.624870+0000","last_undegraded":"2023-11-28T13:00:10.624870+0000","last_fullsized":"2023-11-28T13:00:10.624870+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:02:41.353263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619341+0000","last_change":"2023-11-28T13:00:10.619341+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619341+0000","last_undegraded":"2023-11-28T13:00:10.619341+0000","last_fullsized":"2023-11-28T13:00:10.619341+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:03:04.306890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621647+0000","last_change":"2023-11-28T13:00:10.621647+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621647+0000","last_undegraded":"2023-11-28T13:00:10.621647+0000","last_fullsized":"2023-11-28T13:00:10.621647+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.627237+0000","last_change":"2023-11-28T13:00:10.627237+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.627237+0000","last_undegraded":"2023-11-28T13:00:10.627237+0000","last_fullsized":"2023-11-28T13:00:10.627237+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619562+0000","last_change":"2023-11-28T13:00:10.619562+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619562+0000","last_undegraded":"2023-11-28T13:00:10.619562+0000","last_fullsized":"2023-11-28T13:00:10.619562+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:01:31.408692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.625891+0000","last_change":"2023-11-28T13:00:10.625891+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.625891+0000","last_undegraded":"2023-11-28T13:00:10.625891+0000","last_fullsized":"2023-11-28T13:00:10.625891+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:26.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.620076+0000","last_change":"2023-11-28T13:00:10.620076+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.620076+0000","last_undegraded":"2023-11-28T13:00:10.620076+0000","last_fullsized":"2023-11-28T13:00:10.620076+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621487+0000","last_change":"2023-11-28T13:00:10.621487+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621487+0000","last_undegraded":"2023-11-28T13:00:10.621487+0000","last_fullsized":"2023-11-28T13:00:10.621487+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:28:07.771314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.624473+0000","last_change":"2023-11-28T13:00:10.624473+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.624473+0000","last_undegraded":"2023-11-28T13:00:10.624473+0000","last_fullsized":"2023-11-28T13:00:10.624473+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:19:16.875338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619210+0000","last_change":"2023-11-28T13:00:10.619210+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619210+0000","last_undegraded":"2023-11-28T13:00:10.619210+0000","last_fullsized":"2023-11-28T13:00:10.619210+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.623596+0000","last_change":"2023-11-28T13:00:10.623596+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.623596+0000","last_undegraded":"2023-11-28T13:00:10.623596+0000","last_fullsized":"2023-11-28T13:00:10.623596+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:08:35.062283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.626302+0000","last_change":"2023-11-28T13:00:10.626302+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.626302+0000","last_undegraded":"2023-11-28T13:00:10.626302+0000","last_fullsized":"2023-11-28T13:00:10.626302+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621592+0000","last_change":"2023-11-28T13:00:10.621592+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621592+0000","last_undegraded":"2023-11-28T13:00:10.621592+0000","last_fullsized":"2023-11-28T13:00:10.621592+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:19.809502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619154+0000","last_change":"2023-11-28T13:00:10.619154+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619154+0000","last_undegraded":"2023-11-28T13:00:10.619154+0000","last_fullsized":"2023-11-28T13:00:10.619154+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:36:34.014681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622098+0000","last_change":"2023-11-28T13:00:10.622098+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622098+0000","last_undegraded":"2023-11-28T13:00:10.622098+0000","last_fullsized":"2023-11-28T13:00:10.622098+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622534+0000","last_change":"2023-11-28T13:00:10.622534+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622534+0000","last_undegraded":"2023-11-28T13:00:10.622534+0000","last_fullsized":"2023-11-28T13:00:10.622534+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622647+0000","last_change":"2023-11-28T13:00:10.622647+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622647+0000","last_undegraded":"2023-11-28T13:00:10.622647+0000","last_fullsized":"2023-11-28T13:00:10.622647+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:17:03.747332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622101+0000","last_change":"2023-11-28T13:00:10.622101+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622101+0000","last_undegraded":"2023-11-28T13:00:10.622101+0000","last_fullsized":"2023-11-28T13:00:10.622101+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.625935+0000","last_change":"2023-11-28T13:00:10.625935+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.625935+0000","last_undegraded":"2023-11-28T13:00:10.625935+0000","last_fullsized":"2023-11-28T13:00:10.625935+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:37:07.844503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619380+0000","last_change":"2023-11-28T13:00:10.619380+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619380+0000","last_undegraded":"2023-11-28T13:00:10.619380+0000","last_fullsized":"2023-11-28T13:00:10.619380+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:22:30.539161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.625846+0000","last_change":"2023-11-28T13:00:10.625846+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.625846+0000","last_undegraded":"2023-11-28T13:00:10.625846+0000","last_fullsized":"2023-11-28T13:00:10.625846+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806154+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T13:00:05.806154+0000","last_peered":"2023-11-28T13:00:05.806154+0000","last_clean":"2023-11-28T13:00:05.806154+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:00:05.806154+0000","last_undegraded":"2023-11-28T13:00:05.806154+0000","last_fullsized":"2023-11-28T13:00:05.806154+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622988+0000","last_change":"2023-11-28T13:00:10.622988+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622988+0000","last_undegraded":"2023-11-28T13:00:10.622988+0000","last_fullsized":"2023-11-28T13:00:10.622988+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806797+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T13:00:05.806797+0000","last_peered":"2023-11-28T13:00:05.806797+0000","last_clean":"2023-11-28T13:00:05.806797+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:00:05.806797+0000","last_undegraded":"2023-11-28T13:00:05.806797+0000","last_fullsized":"2023-11-28T13:00:05.806797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619075+0000","last_change":"2023-11-28T13:00:10.619075+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619075+0000","last_undegraded":"2023-11-28T13:00:10.619075+0000","last_fullsized":"2023-11-28T13:00:10.619075+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:26.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.255891+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T13:00:05.255891+0000","last_peered":"2023-11-28T13:00:05.255891+0000","last_clean":"2023-11-28T13:00:05.255891+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:00:05.255891+0000","last_undegraded":"2023-11-28T13:00:05.255891+0000","last_fullsized":"2023-11-28T13:00:05.255891+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621665+0000","last_change":"2023-11-28T13:00:10.621665+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621665+0000","last_undegraded":"2023-11-28T13:00:10.621665+0000","last_fullsized":"2023-11-28T13:00:10.621665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:22:30.539161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806796+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T13:00:05.806796+0000","last_peered":"2023-11-28T13:00:05.806796+0000","last_clean":"2023-11-28T13:00:05.806796+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:00:05.806796+0000","last_undegraded":"2023-11-28T13:00:05.806796+0000","last_fullsized":"2023-11-28T13:00:05.806796+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":179,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806772+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T13:00:05.806772+0000","last_peered":"2023-11-28T13:00:05.806772+0000","last_clean":"2023-11-28T13:00:05.806772+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:00:05.806772+0000","last_undegraded":"2023-11-28T13:00:05.806772+0000","last_fullsized":"2023-11-28T13:00:05.806772+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621578+0000","last_change":"2023-11-28T13:00:10.621578+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621578+0000","last_undegraded":"2023-11-28T13:00:10.621578+0000","last_fullsized":"2023-11-28T13:00:10.621578+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:02:30.352479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806793+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T13:00:05.806793+0000","last_peered":"2023-11-28T13:00:05.806793+0000","last_clean":"2023-11-28T13:00:05.806793+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:00:05.806793+0000","last_undegraded":"2023-11-28T13:00:05.806793+0000","last_fullsized":"2023-11-28T13:00:05.806793+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621478+0000","last_change":"2023-11-28T13:00:10.621478+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621478+0000","last_undegraded":"2023-11-28T13:00:10.621478+0000","last_fullsized":"2023-11-28T13:00:10.621478+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:40.321176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806781+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T13:00:05.806781+0000","last_peered":"2023-11-28T13:00:05.806781+0000","last_clean":"2023-11-28T13:00:05.806781+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:00:05.806781+0000","last_undegraded":"2023-11-28T13:00:05.806781+0000","last_fullsized":"2023-11-28T13:00:05.806781+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.618918+0000","last_change":"2023-11-28T13:00:10.618918+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.618918+0000","last_undegraded":"2023-11-28T13:00:10.618918+0000","last_fullsized":"2023-11-28T13:00:10.618918+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:50:51.288479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806162+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T13:00:05.806162+0000","last_peered":"2023-11-28T13:00:05.806162+0000","last_clean":"2023-11-28T13:00:05.806162+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:00:05.806162+0000","last_undegraded":"2023-11-28T13:00:05.806162+0000","last_fullsized":"2023-11-28T13:00:05.806162+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619023+0000","last_change":"2023-11-28T13:00:10.619023+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619023+0000","last_undegraded":"2023-11-28T13:00:10.619023+0000","last_fullsized":"2023-11-28T13:00:10.619023+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:47.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.251149+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T13:00:05.251149+0000","last_peered":"2023-11-28T13:00:05.251149+0000","last_clean":"2023-11-28T13:00:05.251149+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:00:05.251149+0000","last_undegraded":"2023-11-28T13:00:05.251149+0000","last_fullsized":"2023-11-28T13:00:05.251149+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9580,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362260,"statfs":{"total":96636764160,"available":96626954240,"internally_reserved":0,"allocated":1019904,"data_stored":776228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.3279999999999998,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.2919999999999998,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.47999999999999998}]},{"osd":1,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.39000000000000001}]},{"osd":4,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.441},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.59099999999999997}]},{"osd":5,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.52100000000000002}]},{"osd":6,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.56200000000000006}]}]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8436,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363404,"statfs":{"total":96636764160,"available":96628125696,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.376},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.38800000000000001}]},{"osd":1,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.433},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.53700000000000003}]},{"osd":2,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.46100000000000002}]},{"osd":3,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.47099999999999997}]},{"osd":4,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.48999999999999999}]},{"osd":5,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.505}]},{"osd":7,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.58299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8488,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363352,"statfs":{"total":96636764160,"available":96628072448,"internally_reserved":0,"allocated":425984,"data_stored":185860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40400000000000003}]},{"osd":2,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.38}]},{"osd":3,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.436}]},{"osd":4,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.58999999999999997}]},{"osd":5,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.874},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.74399999999999999}]},{"osd":6,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.71899999999999997}]},{"osd":7,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.65400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9592,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362248,"statfs":{"total":96636764160,"available":96626941952,"internally_reserved":0,"allocated":1032192,"data_stored":782718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.32100000000000001}]},{"osd":2,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.35299999999999998}]},{"osd":3,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.377}]},{"osd":4,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47999999999999998}]},{"osd":5,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.432},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51600000000000001}]},{"osd":6,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46400000000000002}]},{"osd":7,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.57499999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8372,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363468,"statfs":{"total":96636764160,"available":96628191232,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.36499999999999999}]},{"osd":1,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.316},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.41499999999999998}]},{"osd":3,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.39300000000000002}]},{"osd":4,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.71899999999999997}]},{"osd":5,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.628},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70099999999999996}]},{"osd":6,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.65700000000000003}]},{"osd":7,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.68700000000000006}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.318,"15min":0.318},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.39700000000000002}]},{"osd":1,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.41199999999999998}]},{"osd":2,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":0.437},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":0.48799999999999999}]},{"osd":4,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":8.2889999999999997,"5min":8.2889999999999997,"15min":8.2889999999999997},"last":0.57799999999999996}]},{"osd":5,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":0.627}]},{"osd":6,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":7.0899999999999999,"5min":7.0899999999999999,"15min":7.0899999999999999},"last":0.67600000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8628,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363212,"statfs":{"total":96636764160,"available":96627929088,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.495},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.47699999999999998}]},{"osd":1,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.53200000000000003}]},{"osd":2,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.54400000000000004}]},{"osd":5,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.436},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.55900000000000005}]},{"osd":6,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.61399999999999999}]},{"osd":7,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57299999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8436,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363404,"statfs":{"total":96636764160,"available":96628125696,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.69599999999999995}]},{"osd":1,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.627},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.50600000000000001}]},{"osd":2,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.4020000000000001,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.53800000000000003}]},{"osd":3,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":0.46600000000000003}]},{"osd":4,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.0739999999999998,"5min":3.0739999999999998,"15min":3.0739999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.65200000000000002}]},{"osd":6,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":3.7679999999999998,"5min":3.7679999999999998,"15min":3.7679999999999998},"last":0.49099999999999999}]},{"osd":7,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:00:16.110 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:00:16.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.261+0000 7fb9ddb11700 1 Processor -- start 2023-11-28T13:00:16.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.261+0000 7fb9ddb11700 1 -- start start 2023-11-28T13:00:16.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.262+0000 7fb9ddb11700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb9d815a5d0 0x7fb9d80ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:16.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.262+0000 7fb9ddb11700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9d80cb2d0 0x7fb9d80ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:16.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.262+0000 7fb9ddb11700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d80ae8b0 con 0x7fb9d80b0160 2023-11-28T13:00:16.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.262+0000 7fb9ddb11700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d80cc1b0 con 0x7fb9d815a5d0 2023-11-28T13:00:16.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.262+0000 7fb9ddb11700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d80cc390 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.262+0000 7fb9d7fff700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9d80cb2d0 0x7fb9d80ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:16.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.262+0000 7fb9d6ffd700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7fb9d80b0160 0x7fb9d815e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:44588/0 (socket says 172.21.15.111:44588) 2023-11-28T13:00:16.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.262+0000 7fb9d77fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb9d815a5d0 0x7fb9d80ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:16.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.262+0000 7fb9d6ffd700 1 -- 172.21.15.111:0/257896961 learned_addr learned my addr 172.21.15.111:0/257896961 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:16.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.262+0000 7fb9d67fc700 1 -- 172.21.15.111:0/257896961 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511543779 0 0) 0x7fb9d80ae8b0 con 0x7fb9d80b0160 2023-11-28T13:00:16.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.262+0000 7fb9d67fc700 1 -- 172.21.15.111:0/257896961 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c00035b0 con 0x7fb9d80b0160 2023-11-28T13:00:16.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.263+0000 7fb9d7fff700 1 -- 172.21.15.111:0/257896961 >> v2:172.21.15.149:3300/0 conn(0x7fb9d815a5d0 msgr2=0x7fb9d80ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:16.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.263+0000 7fb9d7fff700 1 --2- 172.21.15.111:0/257896961 >> v2:172.21.15.149:3300/0 conn(0x7fb9d815a5d0 0x7fb9d80ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.263+0000 7fb9d7fff700 1 -- 172.21.15.111:0/257896961 >> v1:172.21.15.149:6789/0 conn(0x7fb9d80b0160 legacy=0x7fb9d815e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.263+0000 7fb9d7fff700 1 -- 172.21.15.111:0/257896961 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d80cc670 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.263+0000 7fb9d77fe700 1 --2- 172.21.15.111:0/257896961 >> v2:172.21.15.149:3300/0 conn(0x7fb9d815a5d0 0x7fb9d80ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:16.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.263+0000 7fb9d7fff700 1 --2- 172.21.15.111:0/257896961 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9d80cb2d0 0x7fb9d80ced90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009d90 tx=0x7fb9d000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6e2095d8824564f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:16.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.263+0000 7fb9d67fc700 1 -- 172.21.15.111:0/257896961 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d000e070 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.263+0000 7fb9d67fc700 1 -- 172.21.15.111:0/257896961 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d000bb20 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.263+0000 7fb9d67fc700 1 -- 172.21.15.111:0/257896961 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0012430 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.263+0000 7fb9ddb11700 1 -- 172.21.15.111:0/257896961 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9d80cb2d0 msgr2=0x7fb9d80ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.263+0000 7fb9ddb11700 1 --2- 172.21.15.111:0/257896961 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9d80cb2d0 0x7fb9d80ced90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009d90 tx=0x7fb9d000b330 comp rx=0 tx=0).stop 2023-11-28T13:00:16.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.264+0000 7fb9ddb11700 1 -- 172.21.15.111:0/257896961 shutdown_connections 2023-11-28T13:00:16.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.264+0000 7fb9ddb11700 1 --2- 172.21.15.111:0/257896961 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9d80cb2d0 0x7fb9d80ced90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.264+0000 7fb9ddb11700 1 --2- 172.21.15.111:0/257896961 >> v2:172.21.15.149:3300/0 conn(0x7fb9d815a5d0 0x7fb9d80ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.264+0000 7fb9ddb11700 1 -- 172.21.15.111:0/257896961 >> 172.21.15.111:0/257896961 conn(0x7fb9d809a040 msgr2=0x7fb9d80d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:16.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.264+0000 7fb9ddb11700 1 -- 172.21.15.111:0/257896961 shutdown_connections 2023-11-28T13:00:16.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.264+0000 7fb9ddb11700 1 -- 172.21.15.111:0/257896961 wait complete. 2023-11-28T13:00:16.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.265+0000 7fb9ddb11700 1 Processor -- start 2023-11-28T13:00:16.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.265+0000 7fb9ddb11700 1 -- start start 2023-11-28T13:00:16.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9ddb11700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9d80b0160 0x7fb9d80c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:16.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9ddb11700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d80cb2d0 0x7fb9d8154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:16.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9ddb11700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9d815a5d0 0x7fb9d8154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:16.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9ddb11700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d80b3be0 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9ddb11700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d80b3e50 con 0x7fb9d80b0160 2023-11-28T13:00:16.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9ddb11700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d80b4130 con 0x7fb9d815a5d0 2023-11-28T13:00:16.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9d6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d80cb2d0 0x7fb9d8154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:16.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9d6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d80cb2d0 0x7fb9d8154880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35276/0 (socket says 172.21.15.111:35276) 2023-11-28T13:00:16.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9d6ffd700 1 -- 172.21.15.111:0/1045085231 learned_addr learned my addr 172.21.15.111:0/1045085231 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:16.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9d7fff700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9d815a5d0 0x7fb9d8154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:16.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9d77fe700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9d80b0160 0x7fb9d80c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:16.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9d6ffd700 1 -- 172.21.15.111:0/1045085231 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9d815a5d0 msgr2=0x7fb9d8154dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9d6ffd700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9d815a5d0 0x7fb9d8154dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9d6ffd700 1 -- 172.21.15.111:0/1045085231 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9d80b0160 msgr2=0x7fb9d80c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9d6ffd700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9d80b0160 0x7fb9d80c3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9d6ffd700 1 -- 172.21.15.111:0/1045085231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc009430 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9d6ffd700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d80cb2d0 0x7fb9d8154880 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8001eb0 tx=0x7fb9c800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:16.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9bffff700 1 -- 172.21.15.111:0/1045085231 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8005d70 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9bffff700 1 -- 172.21.15.111:0/1045085231 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c80070d0 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9ddb11700 1 -- 172.21.15.111:0/1045085231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d0009a70 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9bffff700 1 -- 172.21.15.111:0/1045085231 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8007460 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.266+0000 7fb9ddb11700 1 -- 172.21.15.111:0/1045085231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d80ccc30 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.268+0000 7fb9ddb11700 1 -- 172.21.15.111:0/1045085231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d80cc950 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.268+0000 7fb9bffff700 1 -- 172.21.15.111:0/1045085231 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb9c8019030 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.268+0000 7fb9bffff700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb9c40682f0 0x7fb9c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:16.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.268+0000 7fb9bffff700 1 -- 172.21.15.111:0/1045085231 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb9c8085060 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.268+0000 7fb9d77fe700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb9c40682f0 0x7fb9c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:16.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.269+0000 7fb9d77fe700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb9c40682f0 0x7fb9c406a7a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009d50 tx=0x7fb9cc0091c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:16.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.272+0000 7fb9bffff700 1 -- 172.21.15.111:0/1045085231 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb9c8051780 con 0x7fb9d80cb2d0 2023-11-28T13:00:16.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.453+0000 7fb9ddb11700 1 -- 172.21.15.111:0/1045085231 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb9d80b3fc0 con 0x7fb9c40682f0 2023-11-28T13:00:16.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.461+0000 7fb9bffff700 1 -- 172.21.15.111:0/1045085231 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+300126 (secure 0 0 0) 0x7fb9d80b3fc0 con 0x7fb9c40682f0 2023-11-28T13:00:16.465 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:16.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.464+0000 7fb9ddb11700 1 -- 172.21.15.111:0/1045085231 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb9c40682f0 msgr2=0x7fb9c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.464+0000 7fb9ddb11700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb9c40682f0 0x7fb9c406a7a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009d50 tx=0x7fb9cc0091c0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.464+0000 7fb9ddb11700 1 -- 172.21.15.111:0/1045085231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d80cb2d0 msgr2=0x7fb9d8154880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.464+0000 7fb9ddb11700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d80cb2d0 0x7fb9d8154880 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8001eb0 tx=0x7fb9c800c7b0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.464+0000 7fb9ddb11700 1 -- 172.21.15.111:0/1045085231 shutdown_connections 2023-11-28T13:00:16.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.464+0000 7fb9ddb11700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb9d815a5d0 0x7fb9d8154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.464+0000 7fb9ddb11700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb9c40682f0 0x7fb9c406a7a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.464+0000 7fb9ddb11700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb9d80b0160 0x7fb9d80c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.464+0000 7fb9ddb11700 1 --2- 172.21.15.111:0/1045085231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d80cb2d0 0x7fb9d8154880 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.464+0000 7fb9ddb11700 1 -- 172.21.15.111:0/1045085231 >> 172.21.15.111:0/1045085231 conn(0x7fb9d809a040 msgr2=0x7fb9d80cf750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:16.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.464+0000 7fb9ddb11700 1 -- 172.21.15.111:0/1045085231 shutdown_connections 2023-11-28T13:00:16.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.464+0000 7fb9ddb11700 1 -- 172.21.15.111:0/1045085231 wait complete. 2023-11-28T13:00:16.469 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:00:16.477 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2023-11-28T13:00:16.351493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":25,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":70032,"kb_used_data":4560,"kb_used_omap":0,"kb_used_meta":65408,"kb_avail":754904688,"statfs":{"total":773094113280,"available":773022400512,"internally_reserved":0,"allocated":4669440,"data_stored":2706556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66977792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.735324"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622561+0000","last_change":"2023-11-28T13:00:10.622561+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622561+0000","last_undegraded":"2023-11-28T13:00:10.622561+0000","last_fullsized":"2023-11-28T13:00:10.622561+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:47.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.618665+0000","last_change":"2023-11-28T13:00:10.618665+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.618665+0000","last_undegraded":"2023-11-28T13:00:10.618665+0000","last_fullsized":"2023-11-28T13:00:10.618665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:48:31.549366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621901+0000","last_change":"2023-11-28T13:00:10.621901+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621901+0000","last_undegraded":"2023-11-28T13:00:10.621901+0000","last_fullsized":"2023-11-28T13:00:10.621901+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:57:53.793229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619096+0000","last_change":"2023-11-28T13:00:10.619096+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619096+0000","last_undegraded":"2023-11-28T13:00:10.619096+0000","last_fullsized":"2023-11-28T13:00:10.619096+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.624870+0000","last_change":"2023-11-28T13:00:10.624870+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.624870+0000","last_undegraded":"2023-11-28T13:00:10.624870+0000","last_fullsized":"2023-11-28T13:00:10.624870+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:02:41.353263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619341+0000","last_change":"2023-11-28T13:00:10.619341+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619341+0000","last_undegraded":"2023-11-28T13:00:10.619341+0000","last_fullsized":"2023-11-28T13:00:10.619341+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:03:04.306890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621647+0000","last_change":"2023-11-28T13:00:10.621647+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621647+0000","last_undegraded":"2023-11-28T13:00:10.621647+0000","last_fullsized":"2023-11-28T13:00:10.621647+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.627237+0000","last_change":"2023-11-28T13:00:10.627237+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.627237+0000","last_undegraded":"2023-11-28T13:00:10.627237+0000","last_fullsized":"2023-11-28T13:00:10.627237+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619562+0000","last_change":"2023-11-28T13:00:10.619562+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619562+0000","last_undegraded":"2023-11-28T13:00:10.619562+0000","last_fullsized":"2023-11-28T13:00:10.619562+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:01:31.408692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.625891+0000","last_change":"2023-11-28T13:00:10.625891+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.625891+0000","last_undegraded":"2023-11-28T13:00:10.625891+0000","last_fullsized":"2023-11-28T13:00:10.625891+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:26.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.620076+0000","last_change":"2023-11-28T13:00:10.620076+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.620076+0000","last_undegraded":"2023-11-28T13:00:10.620076+0000","last_fullsized":"2023-11-28T13:00:10.620076+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621487+0000","last_change":"2023-11-28T13:00:10.621487+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621487+0000","last_undegraded":"2023-11-28T13:00:10.621487+0000","last_fullsized":"2023-11-28T13:00:10.621487+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:28:07.771314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.624473+0000","last_change":"2023-11-28T13:00:10.624473+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.624473+0000","last_undegraded":"2023-11-28T13:00:10.624473+0000","last_fullsized":"2023-11-28T13:00:10.624473+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:19:16.875338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619210+0000","last_change":"2023-11-28T13:00:10.619210+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619210+0000","last_undegraded":"2023-11-28T13:00:10.619210+0000","last_fullsized":"2023-11-28T13:00:10.619210+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.623596+0000","last_change":"2023-11-28T13:00:10.623596+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.623596+0000","last_undegraded":"2023-11-28T13:00:10.623596+0000","last_fullsized":"2023-11-28T13:00:10.623596+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:08:35.062283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.626302+0000","last_change":"2023-11-28T13:00:10.626302+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.626302+0000","last_undegraded":"2023-11-28T13:00:10.626302+0000","last_fullsized":"2023-11-28T13:00:10.626302+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621592+0000","last_change":"2023-11-28T13:00:10.621592+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621592+0000","last_undegraded":"2023-11-28T13:00:10.621592+0000","last_fullsized":"2023-11-28T13:00:10.621592+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:19.809502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619154+0000","last_change":"2023-11-28T13:00:10.619154+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619154+0000","last_undegraded":"2023-11-28T13:00:10.619154+0000","last_fullsized":"2023-11-28T13:00:10.619154+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:36:34.014681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622098+0000","last_change":"2023-11-28T13:00:10.622098+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622098+0000","last_undegraded":"2023-11-28T13:00:10.622098+0000","last_fullsized":"2023-11-28T13:00:10.622098+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622534+0000","last_change":"2023-11-28T13:00:10.622534+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622534+0000","last_undegraded":"2023-11-28T13:00:10.622534+0000","last_fullsized":"2023-11-28T13:00:10.622534+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622647+0000","last_change":"2023-11-28T13:00:10.622647+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622647+0000","last_undegraded":"2023-11-28T13:00:10.622647+0000","last_fullsized":"2023-11-28T13:00:10.622647+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:17:03.747332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622101+0000","last_change":"2023-11-28T13:00:10.622101+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622101+0000","last_undegraded":"2023-11-28T13:00:10.622101+0000","last_fullsized":"2023-11-28T13:00:10.622101+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.625935+0000","last_change":"2023-11-28T13:00:10.625935+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.625935+0000","last_undegraded":"2023-11-28T13:00:10.625935+0000","last_fullsized":"2023-11-28T13:00:10.625935+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:37:07.844503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619380+0000","last_change":"2023-11-28T13:00:10.619380+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619380+0000","last_undegraded":"2023-11-28T13:00:10.619380+0000","last_fullsized":"2023-11-28T13:00:10.619380+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:22:30.539161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.625846+0000","last_change":"2023-11-28T13:00:10.625846+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.625846+0000","last_undegraded":"2023-11-28T13:00:10.625846+0000","last_fullsized":"2023-11-28T13:00:10.625846+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806154+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T13:00:05.806154+0000","last_peered":"2023-11-28T13:00:05.806154+0000","last_clean":"2023-11-28T13:00:05.806154+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:00:05.806154+0000","last_undegraded":"2023-11-28T13:00:05.806154+0000","last_fullsized":"2023-11-28T13:00:05.806154+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622988+0000","last_change":"2023-11-28T13:00:10.622988+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622988+0000","last_undegraded":"2023-11-28T13:00:10.622988+0000","last_fullsized":"2023-11-28T13:00:10.622988+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806797+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T13:00:05.806797+0000","last_peered":"2023-11-28T13:00:05.806797+0000","last_clean":"2023-11-28T13:00:05.806797+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:00:05.806797+0000","last_undegraded":"2023-11-28T13:00:05.806797+0000","last_fullsized":"2023-11-28T13:00:05.806797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619075+0000","last_change":"2023-11-28T13:00:10.619075+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619075+0000","last_undegraded":"2023-11-28T13:00:10.619075+0000","last_fullsized":"2023-11-28T13:00:10.619075+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:26.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.255891+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T13:00:05.255891+0000","last_peered":"2023-11-28T13:00:05.255891+0000","last_clean":"2023-11-28T13:00:05.255891+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:00:05.255891+0000","last_undegraded":"2023-11-28T13:00:05.255891+0000","last_fullsized":"2023-11-28T13:00:05.255891+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621665+0000","last_change":"2023-11-28T13:00:10.621665+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621665+0000","last_undegraded":"2023-11-28T13:00:10.621665+0000","last_fullsized":"2023-11-28T13:00:10.621665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:22:30.539161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806796+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T13:00:05.806796+0000","last_peered":"2023-11-28T13:00:05.806796+0000","last_clean":"2023-11-28T13:00:05.806796+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:00:05.806796+0000","last_undegraded":"2023-11-28T13:00:05.806796+0000","last_fullsized":"2023-11-28T13:00:05.806796+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":179,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806772+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T13:00:05.806772+0000","last_peered":"2023-11-28T13:00:05.806772+0000","last_clean":"2023-11-28T13:00:05.806772+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:00:05.806772+0000","last_undegraded":"2023-11-28T13:00:05.806772+0000","last_fullsized":"2023-11-28T13:00:05.806772+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621578+0000","last_change":"2023-11-28T13:00:10.621578+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621578+0000","last_undegraded":"2023-11-28T13:00:10.621578+0000","last_fullsized":"2023-11-28T13:00:10.621578+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:02:30.352479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806793+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T13:00:05.806793+0000","last_peered":"2023-11-28T13:00:05.806793+0000","last_clean":"2023-11-28T13:00:05.806793+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:00:05.806793+0000","last_undegraded":"2023-11-28T13:00:05.806793+0000","last_fullsized":"2023-11-28T13:00:05.806793+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621478+0000","last_change":"2023-11-28T13:00:10.621478+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621478+0000","last_undegraded":"2023-11-28T13:00:10.621478+0000","last_fullsized":"2023-11-28T13:00:10.621478+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:40.321176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806781+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T13:00:05.806781+0000","last_peered":"2023-11-28T13:00:05.806781+0000","last_clean":"2023-11-28T13:00:05.806781+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:00:05.806781+0000","last_undegraded":"2023-11-28T13:00:05.806781+0000","last_fullsized":"2023-11-28T13:00:05.806781+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.618918+0000","last_change":"2023-11-28T13:00:10.618918+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.618918+0000","last_undegraded":"2023-11-28T13:00:10.618918+0000","last_fullsized":"2023-11-28T13:00:10.618918+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:50:51.288479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806162+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T13:00:05.806162+0000","last_peered":"2023-11-28T13:00:05.806162+0000","last_clean":"2023-11-28T13:00:05.806162+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:00:05.806162+0000","last_undegraded":"2023-11-28T13:00:05.806162+0000","last_fullsized":"2023-11-28T13:00:05.806162+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619023+0000","last_change":"2023-11-28T13:00:10.619023+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619023+0000","last_undegraded":"2023-11-28T13:00:10.619023+0000","last_fullsized":"2023-11-28T13:00:10.619023+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:47.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.251149+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T13:00:05.251149+0000","last_peered":"2023-11-28T13:00:05.251149+0000","last_clean":"2023-11-28T13:00:05.251149+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:00:05.251149+0000","last_undegraded":"2023-11-28T13:00:05.251149+0000","last_fullsized":"2023-11-28T13:00:05.251149+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9580,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362260,"statfs":{"total":96636764160,"available":96626954240,"internally_reserved":0,"allocated":1019904,"data_stored":776228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.3279999999999998,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.2919999999999998,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.47999999999999998}]},{"osd":1,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.39000000000000001}]},{"osd":4,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.441},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.59099999999999997}]},{"osd":5,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.52100000000000002}]},{"osd":6,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.56200000000000006}]}]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8436,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363404,"statfs":{"total":96636764160,"available":96628125696,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.376},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.38800000000000001}]},{"osd":1,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.433},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.53700000000000003}]},{"osd":2,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.46100000000000002}]},{"osd":3,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.47099999999999997}]},{"osd":4,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.48999999999999999}]},{"osd":5,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.505}]},{"osd":7,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.58299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8488,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363352,"statfs":{"total":96636764160,"available":96628072448,"internally_reserved":0,"allocated":425984,"data_stored":185860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40400000000000003}]},{"osd":2,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.38}]},{"osd":3,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.436}]},{"osd":4,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.58999999999999997}]},{"osd":5,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.874},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.74399999999999999}]},{"osd":6,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.71899999999999997}]},{"osd":7,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.65400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9592,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362248,"statfs":{"total":96636764160,"available":96626941952,"internally_reserved":0,"allocated":1032192,"data_stored":782718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.32100000000000001}]},{"osd":2,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.35299999999999998}]},{"osd":3,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.377}]},{"osd":4,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47999999999999998}]},{"osd":5,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.432},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51600000000000001}]},{"osd":6,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46400000000000002}]},{"osd":7,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.57499999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8372,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363468,"statfs":{"total":96636764160,"available":96628191232,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.36499999999999999}]},{"osd":1,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.316},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.41499999999999998}]},{"osd":3,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.39300000000000002}]},{"osd":4,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.71899999999999997}]},{"osd":5,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.628},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70099999999999996}]},{"osd":6,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.65700000000000003}]},{"osd":7,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.68700000000000006}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.318,"15min":0.318},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.39700000000000002}]},{"osd":1,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.41199999999999998}]},{"osd":2,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":0.437},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":0.48799999999999999}]},{"osd":4,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":8.2889999999999997,"5min":8.2889999999999997,"15min":8.2889999999999997},"last":0.57799999999999996}]},{"osd":5,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":0.627}]},{"osd":6,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":7.0899999999999999,"5min":7.0899999999999999,"15min":7.0899999999999999},"last":0.67600000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8628,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363212,"statfs":{"total":96636764160,"available":96627929088,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.495},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.47699999999999998}]},{"osd":1,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.53200000000000003}]},{"osd":2,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.54400000000000004}]},{"osd":5,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.436},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.55900000000000005}]},{"osd":6,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.61399999999999999}]},{"osd":7,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57299999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8436,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363404,"statfs":{"total":96636764160,"available":96628125696,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.69599999999999995}]},{"osd":1,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.627},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.50600000000000001}]},{"osd":2,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.4020000000000001,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.53800000000000003}]},{"osd":3,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":0.46600000000000003}]},{"osd":4,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.0739999999999998,"5min":3.0739999999999998,"15min":3.0739999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.65200000000000002}]},{"osd":6,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":3.7679999999999998,"5min":3.7679999999999998,"15min":3.7679999999999998},"last":0.49099999999999999}]},{"osd":7,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:00:16.482 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:00:16.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.637+0000 7f8f84ad0700 1 Processor -- start 2023-11-28T13:00:16.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.637+0000 7f8f84ad0700 1 -- start start 2023-11-28T13:00:16.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f84ad0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8f800b0440 0x7f8f800b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:16.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f84ad0700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f8f800ca2f0 0x7f8f800ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:16.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f84ad0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f800cc750 con 0x7f8f8015c6c0 2023-11-28T13:00:16.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f84ad0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f800b0260 con 0x7f8f800b0440 2023-11-28T13:00:16.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f84ad0700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f800b0e20 con 0x7f8f800ca2f0 2023-11-28T13:00:16.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f7dd9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8f800b0440 0x7f8f800b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:16.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f7e59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8f8015c6c0 0x7f8f800afb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36060/0 (socket says 172.21.15.111:36060) 2023-11-28T13:00:16.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f7e59c700 1 -- 172.21.15.111:0/4159788239 learned_addr learned my addr 172.21.15.111:0/4159788239 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:16.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f7ed9d700 1 --2- 172.21.15.111:0/4159788239 >> v2:172.21.15.149:3300/0 conn(0x7f8f800ca2f0 0x7f8f800ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:16.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f7dd9b700 1 -- 172.21.15.111:0/4159788239 >> v2:172.21.15.149:3300/0 conn(0x7f8f800ca2f0 msgr2=0x7f8f800ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:16.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f7dd9b700 1 --2- 172.21.15.111:0/4159788239 >> v2:172.21.15.149:3300/0 conn(0x7f8f800ca2f0 0x7f8f800ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f7dd9b700 1 -- 172.21.15.111:0/4159788239 >> v1:172.21.15.111:6789/0 conn(0x7f8f8015c6c0 legacy=0x7f8f800afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f7dd9b700 1 -- 172.21.15.111:0/4159788239 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f800cca30 con 0x7f8f800b0440 2023-11-28T13:00:16.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f7d59a700 1 -- 172.21.15.111:0/4159788239 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053708468 0 0) 0x7f8f800cc750 con 0x7f8f8015c6c0 2023-11-28T13:00:16.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f7dd9b700 1 --2- 172.21.15.111:0/4159788239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f800b0440 0x7f8f800b08e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8f80071ae0 tx=0x7f8f74009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1847721414079056 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:16.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.638+0000 7f8f7d59a700 1 -- 172.21.15.111:0/4159788239 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74011830 con 0x7f8f800b0440 2023-11-28T13:00:16.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.639+0000 7f8f7d59a700 1 -- 172.21.15.111:0/4159788239 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f74011990 con 0x7f8f800b0440 2023-11-28T13:00:16.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.639+0000 7f8f7d59a700 1 -- 172.21.15.111:0/4159788239 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74011d20 con 0x7f8f800b0440 2023-11-28T13:00:16.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.639+0000 7f8f84ad0700 1 -- 172.21.15.111:0/4159788239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f800b0440 msgr2=0x7f8f800b08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.639+0000 7f8f84ad0700 1 --2- 172.21.15.111:0/4159788239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f800b0440 0x7f8f800b08e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8f80071ae0 tx=0x7f8f74009b20 comp rx=0 tx=0).stop 2023-11-28T13:00:16.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.639+0000 7f8f84ad0700 1 -- 172.21.15.111:0/4159788239 shutdown_connections 2023-11-28T13:00:16.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.639+0000 7f8f84ad0700 1 --2- 172.21.15.111:0/4159788239 >> v2:172.21.15.149:3300/0 conn(0x7f8f800ca2f0 0x7f8f800ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.639+0000 7f8f84ad0700 1 --2- 172.21.15.111:0/4159788239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f800b0440 0x7f8f800b08e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.639+0000 7f8f84ad0700 1 -- 172.21.15.111:0/4159788239 >> 172.21.15.111:0/4159788239 conn(0x7f8f80098ee0 msgr2=0x7f8f8006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:16.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.639+0000 7f8f84ad0700 1 -- 172.21.15.111:0/4159788239 shutdown_connections 2023-11-28T13:00:16.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.639+0000 7f8f84ad0700 1 -- 172.21.15.111:0/4159788239 wait complete. 2023-11-28T13:00:16.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.640+0000 7f8f84ad0700 1 Processor -- start 2023-11-28T13:00:16.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.640+0000 7f8f84ad0700 1 -- start start 2023-11-28T13:00:16.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.641+0000 7f8f84ad0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f800b0440 0x7f8f801f2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:16.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.641+0000 7f8f84ad0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8f800ca2f0 0x7f8f800cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:16.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.641+0000 7f8f84ad0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8015c6c0 0x7f8f8015cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:16.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.641+0000 7f8f84ad0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8015a8c0 con 0x7f8f8015c6c0 2023-11-28T13:00:16.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.641+0000 7f8f84ad0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800d0850 con 0x7f8f800b0440 2023-11-28T13:00:16.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.641+0000 7f8f84ad0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f80159f20 con 0x7f8f800ca2f0 2023-11-28T13:00:16.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.641+0000 7f8f7ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8015c6c0 0x7f8f8015cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:16.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.641+0000 7f8f7ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8015c6c0 0x7f8f8015cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35292/0 (socket says 172.21.15.111:35292) 2023-11-28T13:00:16.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.641+0000 7f8f7ed9d700 1 -- 172.21.15.111:0/2847324767 learned_addr learned my addr 172.21.15.111:0/2847324767 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:16.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.641+0000 7f8f7e59c700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f800b0440 0x7f8f801f2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:16.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.641+0000 7f8f7dd9b700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8f800ca2f0 0x7f8f800cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:16.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.642+0000 7f8f7ed9d700 1 -- 172.21.15.111:0/2847324767 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8f800ca2f0 msgr2=0x7f8f800cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.642+0000 7f8f7ed9d700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8f800ca2f0 0x7f8f800cfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.642+0000 7f8f7ed9d700 1 -- 172.21.15.111:0/2847324767 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f800b0440 msgr2=0x7f8f801f2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.642+0000 7f8f7ed9d700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f800b0440 0x7f8f801f2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.642+0000 7f8f7ed9d700 1 -- 172.21.15.111:0/2847324767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f740094a0 con 0x7f8f8015c6c0 2023-11-28T13:00:16.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.642+0000 7f8f7e59c700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f800b0440 0x7f8f801f2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:16.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.642+0000 7f8f7ed9d700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8015c6c0 0x7f8f8015cbd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8f78007e60 tx=0x7f8f7800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:16.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.642+0000 7f8f6effd700 1 -- 172.21.15.111:0/2847324767 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7800ede0 con 0x7f8f8015c6c0 2023-11-28T13:00:16.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.642+0000 7f8f6effd700 1 -- 172.21.15.111:0/2847324767 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f78014420 con 0x7f8f8015c6c0 2023-11-28T13:00:16.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.642+0000 7f8f6effd700 1 -- 172.21.15.111:0/2847324767 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f780147d0 con 0x7f8f8015c6c0 2023-11-28T13:00:16.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.642+0000 7f8f84ad0700 1 -- 172.21.15.111:0/2847324767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f800ccff0 con 0x7f8f8015c6c0 2023-11-28T13:00:16.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.642+0000 7f8f84ad0700 1 -- 172.21.15.111:0/2847324767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f800ccd10 con 0x7f8f8015c6c0 2023-11-28T13:00:16.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.643+0000 7f8f84ad0700 1 -- 172.21.15.111:0/2847324767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f800cd2d0 con 0x7f8f8015c6c0 2023-11-28T13:00:16.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.643+0000 7f8f6effd700 1 -- 172.21.15.111:0/2847324767 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8f78012050 con 0x7f8f8015c6c0 2023-11-28T13:00:16.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.644+0000 7f8f6effd700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8f600682f0 0x7f8f6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:16.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.644+0000 7f8f6effd700 1 -- 172.21.15.111:0/2847324767 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8f78086400 con 0x7f8f8015c6c0 2023-11-28T13:00:16.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.644+0000 7f8f7e59c700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8f600682f0 0x7f8f6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:16.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.644+0000 7f8f7e59c700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8f600682f0 0x7f8f6006a7a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8f70000f60 tx=0x7f8f70007ca0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:16.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.648+0000 7f8f6effd700 1 -- 172.21.15.111:0/2847324767 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8f78052b20 con 0x7f8f8015c6c0 2023-11-28T13:00:16.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.828+0000 7f8f84ad0700 1 -- 172.21.15.111:0/2847324767 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8f8015a090 con 0x7f8f600682f0 2023-11-28T13:00:16.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.836+0000 7f8f6effd700 1 -- 172.21.15.111:0/2847324767 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+300126 (secure 0 0 0) 0x7f8f8015a090 con 0x7f8f600682f0 2023-11-28T13:00:16.841 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.839+0000 7f8f84ad0700 1 -- 172.21.15.111:0/2847324767 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8f600682f0 msgr2=0x7f8f6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.839+0000 7f8f84ad0700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8f600682f0 0x7f8f6006a7a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8f70000f60 tx=0x7f8f70007ca0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.839+0000 7f8f84ad0700 1 -- 172.21.15.111:0/2847324767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8015c6c0 msgr2=0x7f8f8015cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:16.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.839+0000 7f8f84ad0700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8015c6c0 0x7f8f8015cbd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8f78007e60 tx=0x7f8f7800e3c0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.839+0000 7f8f84ad0700 1 -- 172.21.15.111:0/2847324767 shutdown_connections 2023-11-28T13:00:16.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.839+0000 7f8f84ad0700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8015c6c0 0x7f8f8015cbd0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.839+0000 7f8f84ad0700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8f600682f0 0x7f8f6006a7a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.839+0000 7f8f84ad0700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8f800b0440 0x7f8f801f2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.839+0000 7f8f84ad0700 1 --2- 172.21.15.111:0/2847324767 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8f800ca2f0 0x7f8f800cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:16.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.839+0000 7f8f84ad0700 1 -- 172.21.15.111:0/2847324767 >> 172.21.15.111:0/2847324767 conn(0x7f8f80098ee0 msgr2=0x7f8f8006e400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:16.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.839+0000 7f8f84ad0700 1 -- 172.21.15.111:0/2847324767 shutdown_connections 2023-11-28T13:00:16.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:16.839+0000 7f8f84ad0700 1 -- 172.21.15.111:0/2847324767 wait complete. 2023-11-28T13:00:16.845 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:00:16.853 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2023-11-28T13:00:16.351493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":25,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":70032,"kb_used_data":4560,"kb_used_omap":0,"kb_used_meta":65408,"kb_avail":754904688,"statfs":{"total":773094113280,"available":773022400512,"internally_reserved":0,"allocated":4669440,"data_stored":2706556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66977792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.735324"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622561+0000","last_change":"2023-11-28T13:00:10.622561+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622561+0000","last_undegraded":"2023-11-28T13:00:10.622561+0000","last_fullsized":"2023-11-28T13:00:10.622561+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:47.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.618665+0000","last_change":"2023-11-28T13:00:10.618665+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.618665+0000","last_undegraded":"2023-11-28T13:00:10.618665+0000","last_fullsized":"2023-11-28T13:00:10.618665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:48:31.549366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621901+0000","last_change":"2023-11-28T13:00:10.621901+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621901+0000","last_undegraded":"2023-11-28T13:00:10.621901+0000","last_fullsized":"2023-11-28T13:00:10.621901+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:57:53.793229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619096+0000","last_change":"2023-11-28T13:00:10.619096+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619096+0000","last_undegraded":"2023-11-28T13:00:10.619096+0000","last_fullsized":"2023-11-28T13:00:10.619096+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.624870+0000","last_change":"2023-11-28T13:00:10.624870+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.624870+0000","last_undegraded":"2023-11-28T13:00:10.624870+0000","last_fullsized":"2023-11-28T13:00:10.624870+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:02:41.353263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619341+0000","last_change":"2023-11-28T13:00:10.619341+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619341+0000","last_undegraded":"2023-11-28T13:00:10.619341+0000","last_fullsized":"2023-11-28T13:00:10.619341+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:03:04.306890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621647+0000","last_change":"2023-11-28T13:00:10.621647+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621647+0000","last_undegraded":"2023-11-28T13:00:10.621647+0000","last_fullsized":"2023-11-28T13:00:10.621647+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.627237+0000","last_change":"2023-11-28T13:00:10.627237+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.627237+0000","last_undegraded":"2023-11-28T13:00:10.627237+0000","last_fullsized":"2023-11-28T13:00:10.627237+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619562+0000","last_change":"2023-11-28T13:00:10.619562+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619562+0000","last_undegraded":"2023-11-28T13:00:10.619562+0000","last_fullsized":"2023-11-28T13:00:10.619562+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:01:31.408692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.625891+0000","last_change":"2023-11-28T13:00:10.625891+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.625891+0000","last_undegraded":"2023-11-28T13:00:10.625891+0000","last_fullsized":"2023-11-28T13:00:10.625891+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:26.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.620076+0000","last_change":"2023-11-28T13:00:10.620076+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.620076+0000","last_undegraded":"2023-11-28T13:00:10.620076+0000","last_fullsized":"2023-11-28T13:00:10.620076+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621487+0000","last_change":"2023-11-28T13:00:10.621487+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621487+0000","last_undegraded":"2023-11-28T13:00:10.621487+0000","last_fullsized":"2023-11-28T13:00:10.621487+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:28:07.771314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.624473+0000","last_change":"2023-11-28T13:00:10.624473+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.624473+0000","last_undegraded":"2023-11-28T13:00:10.624473+0000","last_fullsized":"2023-11-28T13:00:10.624473+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:19:16.875338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619210+0000","last_change":"2023-11-28T13:00:10.619210+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619210+0000","last_undegraded":"2023-11-28T13:00:10.619210+0000","last_fullsized":"2023-11-28T13:00:10.619210+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.623596+0000","last_change":"2023-11-28T13:00:10.623596+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.623596+0000","last_undegraded":"2023-11-28T13:00:10.623596+0000","last_fullsized":"2023-11-28T13:00:10.623596+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:08:35.062283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.626302+0000","last_change":"2023-11-28T13:00:10.626302+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.626302+0000","last_undegraded":"2023-11-28T13:00:10.626302+0000","last_fullsized":"2023-11-28T13:00:10.626302+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621592+0000","last_change":"2023-11-28T13:00:10.621592+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621592+0000","last_undegraded":"2023-11-28T13:00:10.621592+0000","last_fullsized":"2023-11-28T13:00:10.621592+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:19.809502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619154+0000","last_change":"2023-11-28T13:00:10.619154+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619154+0000","last_undegraded":"2023-11-28T13:00:10.619154+0000","last_fullsized":"2023-11-28T13:00:10.619154+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:36:34.014681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622098+0000","last_change":"2023-11-28T13:00:10.622098+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622098+0000","last_undegraded":"2023-11-28T13:00:10.622098+0000","last_fullsized":"2023-11-28T13:00:10.622098+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622534+0000","last_change":"2023-11-28T13:00:10.622534+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622534+0000","last_undegraded":"2023-11-28T13:00:10.622534+0000","last_fullsized":"2023-11-28T13:00:10.622534+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622647+0000","last_change":"2023-11-28T13:00:10.622647+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622647+0000","last_undegraded":"2023-11-28T13:00:10.622647+0000","last_fullsized":"2023-11-28T13:00:10.622647+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:17:03.747332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622101+0000","last_change":"2023-11-28T13:00:10.622101+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622101+0000","last_undegraded":"2023-11-28T13:00:10.622101+0000","last_fullsized":"2023-11-28T13:00:10.622101+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.625935+0000","last_change":"2023-11-28T13:00:10.625935+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.625935+0000","last_undegraded":"2023-11-28T13:00:10.625935+0000","last_fullsized":"2023-11-28T13:00:10.625935+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:37:07.844503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619380+0000","last_change":"2023-11-28T13:00:10.619380+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619380+0000","last_undegraded":"2023-11-28T13:00:10.619380+0000","last_fullsized":"2023-11-28T13:00:10.619380+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:22:30.539161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.625846+0000","last_change":"2023-11-28T13:00:10.625846+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.625846+0000","last_undegraded":"2023-11-28T13:00:10.625846+0000","last_fullsized":"2023-11-28T13:00:10.625846+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806154+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T13:00:05.806154+0000","last_peered":"2023-11-28T13:00:05.806154+0000","last_clean":"2023-11-28T13:00:05.806154+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:00:05.806154+0000","last_undegraded":"2023-11-28T13:00:05.806154+0000","last_fullsized":"2023-11-28T13:00:05.806154+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622988+0000","last_change":"2023-11-28T13:00:10.622988+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622988+0000","last_undegraded":"2023-11-28T13:00:10.622988+0000","last_fullsized":"2023-11-28T13:00:10.622988+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806797+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T13:00:05.806797+0000","last_peered":"2023-11-28T13:00:05.806797+0000","last_clean":"2023-11-28T13:00:05.806797+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:00:05.806797+0000","last_undegraded":"2023-11-28T13:00:05.806797+0000","last_fullsized":"2023-11-28T13:00:05.806797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619075+0000","last_change":"2023-11-28T13:00:10.619075+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619075+0000","last_undegraded":"2023-11-28T13:00:10.619075+0000","last_fullsized":"2023-11-28T13:00:10.619075+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:26.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.255891+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T13:00:05.255891+0000","last_peered":"2023-11-28T13:00:05.255891+0000","last_clean":"2023-11-28T13:00:05.255891+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:00:05.255891+0000","last_undegraded":"2023-11-28T13:00:05.255891+0000","last_fullsized":"2023-11-28T13:00:05.255891+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621665+0000","last_change":"2023-11-28T13:00:10.621665+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621665+0000","last_undegraded":"2023-11-28T13:00:10.621665+0000","last_fullsized":"2023-11-28T13:00:10.621665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:22:30.539161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806796+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T13:00:05.806796+0000","last_peered":"2023-11-28T13:00:05.806796+0000","last_clean":"2023-11-28T13:00:05.806796+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:00:05.806796+0000","last_undegraded":"2023-11-28T13:00:05.806796+0000","last_fullsized":"2023-11-28T13:00:05.806796+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":179,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806772+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T13:00:05.806772+0000","last_peered":"2023-11-28T13:00:05.806772+0000","last_clean":"2023-11-28T13:00:05.806772+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:00:05.806772+0000","last_undegraded":"2023-11-28T13:00:05.806772+0000","last_fullsized":"2023-11-28T13:00:05.806772+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621578+0000","last_change":"2023-11-28T13:00:10.621578+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621578+0000","last_undegraded":"2023-11-28T13:00:10.621578+0000","last_fullsized":"2023-11-28T13:00:10.621578+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:02:30.352479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806793+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T13:00:05.806793+0000","last_peered":"2023-11-28T13:00:05.806793+0000","last_clean":"2023-11-28T13:00:05.806793+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:00:05.806793+0000","last_undegraded":"2023-11-28T13:00:05.806793+0000","last_fullsized":"2023-11-28T13:00:05.806793+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621478+0000","last_change":"2023-11-28T13:00:10.621478+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621478+0000","last_undegraded":"2023-11-28T13:00:10.621478+0000","last_fullsized":"2023-11-28T13:00:10.621478+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:40.321176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806781+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T13:00:05.806781+0000","last_peered":"2023-11-28T13:00:05.806781+0000","last_clean":"2023-11-28T13:00:05.806781+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:00:05.806781+0000","last_undegraded":"2023-11-28T13:00:05.806781+0000","last_fullsized":"2023-11-28T13:00:05.806781+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.618918+0000","last_change":"2023-11-28T13:00:10.618918+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.618918+0000","last_undegraded":"2023-11-28T13:00:10.618918+0000","last_fullsized":"2023-11-28T13:00:10.618918+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:50:51.288479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806162+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T13:00:05.806162+0000","last_peered":"2023-11-28T13:00:05.806162+0000","last_clean":"2023-11-28T13:00:05.806162+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:00:05.806162+0000","last_undegraded":"2023-11-28T13:00:05.806162+0000","last_fullsized":"2023-11-28T13:00:05.806162+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619023+0000","last_change":"2023-11-28T13:00:10.619023+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619023+0000","last_undegraded":"2023-11-28T13:00:10.619023+0000","last_fullsized":"2023-11-28T13:00:10.619023+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:47.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.251149+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T13:00:05.251149+0000","last_peered":"2023-11-28T13:00:05.251149+0000","last_clean":"2023-11-28T13:00:05.251149+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:00:05.251149+0000","last_undegraded":"2023-11-28T13:00:05.251149+0000","last_fullsized":"2023-11-28T13:00:05.251149+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9580,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362260,"statfs":{"total":96636764160,"available":96626954240,"internally_reserved":0,"allocated":1019904,"data_stored":776228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.3279999999999998,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.2919999999999998,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.47999999999999998}]},{"osd":1,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.39000000000000001}]},{"osd":4,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.441},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.59099999999999997}]},{"osd":5,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.52100000000000002}]},{"osd":6,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.56200000000000006}]}]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8436,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363404,"statfs":{"total":96636764160,"available":96628125696,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.376},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.38800000000000001}]},{"osd":1,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.433},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.53700000000000003}]},{"osd":2,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.46100000000000002}]},{"osd":3,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.47099999999999997}]},{"osd":4,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.48999999999999999}]},{"osd":5,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.505}]},{"osd":7,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.58299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8488,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363352,"statfs":{"total":96636764160,"available":96628072448,"internally_reserved":0,"allocated":425984,"data_stored":185860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40400000000000003}]},{"osd":2,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.38}]},{"osd":3,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.436}]},{"osd":4,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.58999999999999997}]},{"osd":5,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.874},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.74399999999999999}]},{"osd":6,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.71899999999999997}]},{"osd":7,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.65400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9592,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362248,"statfs":{"total":96636764160,"available":96626941952,"internally_reserved":0,"allocated":1032192,"data_stored":782718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.32100000000000001}]},{"osd":2,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.35299999999999998}]},{"osd":3,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.377}]},{"osd":4,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47999999999999998}]},{"osd":5,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.432},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51600000000000001}]},{"osd":6,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46400000000000002}]},{"osd":7,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.57499999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8372,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363468,"statfs":{"total":96636764160,"available":96628191232,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.36499999999999999}]},{"osd":1,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.316},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.41499999999999998}]},{"osd":3,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.39300000000000002}]},{"osd":4,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.71899999999999997}]},{"osd":5,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.628},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70099999999999996}]},{"osd":6,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.65700000000000003}]},{"osd":7,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.68700000000000006}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.318,"15min":0.318},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.39700000000000002}]},{"osd":1,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.41199999999999998}]},{"osd":2,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":0.437},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":0.48799999999999999}]},{"osd":4,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":8.2889999999999997,"5min":8.2889999999999997,"15min":8.2889999999999997},"last":0.57799999999999996}]},{"osd":5,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":0.627}]},{"osd":6,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":7.0899999999999999,"5min":7.0899999999999999,"15min":7.0899999999999999},"last":0.67600000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8628,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363212,"statfs":{"total":96636764160,"available":96627929088,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.495},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.47699999999999998}]},{"osd":1,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.53200000000000003}]},{"osd":2,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.54400000000000004}]},{"osd":5,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.436},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.55900000000000005}]},{"osd":6,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.61399999999999999}]},{"osd":7,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57299999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8436,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363404,"statfs":{"total":96636764160,"available":96628125696,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.69599999999999995}]},{"osd":1,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.627},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.50600000000000001}]},{"osd":2,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.4020000000000001,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.53800000000000003}]},{"osd":3,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":0.46600000000000003}]},{"osd":4,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.0739999999999998,"5min":3.0739999999999998,"15min":3.0739999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.65200000000000002}]},{"osd":6,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":3.7679999999999998,"5min":3.7679999999999998,"15min":3.7679999999999998},"last":0.49099999999999999}]},{"osd":7,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:00:16.858 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3f is not active+clean 2023-11-28T13:00:16.858 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.858 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3e is not active+clean 2023-11-28T13:00:16.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3d is not active+clean 2023-11-28T13:00:16.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3c is not active+clean 2023-11-28T13:00:16.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.859 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3b is not active+clean 2023-11-28T13:00:16.859 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3a is not active+clean 2023-11-28T13:00:16.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.39 is not active+clean 2023-11-28T13:00:16.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.38 is not active+clean 2023-11-28T13:00:16.860 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.860 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.37 is not active+clean 2023-11-28T13:00:16.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.36 is not active+clean 2023-11-28T13:00:16.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.35 is not active+clean 2023-11-28T13:00:16.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.34 is not active+clean 2023-11-28T13:00:16.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.861 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.33 is not active+clean 2023-11-28T13:00:16.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.32 is not active+clean 2023-11-28T13:00:16.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.31 is not active+clean 2023-11-28T13:00:16.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.862 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.30 is not active+clean 2023-11-28T13:00:16.862 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2f is not active+clean 2023-11-28T13:00:16.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2e is not active+clean 2023-11-28T13:00:16.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2d is not active+clean 2023-11-28T13:00:16.863 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.863 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2c is not active+clean 2023-11-28T13:00:16.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2b is not active+clean 2023-11-28T13:00:16.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2a is not active+clean 2023-11-28T13:00:16.864 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.864 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.29 is not active+clean 2023-11-28T13:00:16.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.28 is not active+clean 2023-11-28T13:00:16.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.27 is not active+clean 2023-11-28T13:00:16.865 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.865 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.26 is not active+clean 2023-11-28T13:00:16.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.25 is not active+clean 2023-11-28T13:00:16.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.24 is not active+clean 2023-11-28T13:00:16.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.23 is not active+clean 2023-11-28T13:00:16.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.866 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.22 is not active+clean 2023-11-28T13:00:16.866 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.21 is not active+clean 2023-11-28T13:00:16.867 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.20 is not active+clean 2023-11-28T13:00:16.867 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.867 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1e is not active+clean 2023-11-28T13:00:16.867 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2023-11-28T13:00:16.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.622561+0000', 'last_change': '2023-11-28T13:00:10.622561+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.622561+0000', 'last_undegraded': '2023-11-28T13:00:10.622561+0000', 'last_fullsized': '2023-11-28T13:00:10.622561+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T18:46:47.273034+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T13:00:16.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2023-11-28T13:00:16.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2023-11-28T13:00:16.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.618665+0000', 'last_change': '2023-11-28T13:00:10.618665+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.618665+0000', 'last_undegraded': '2023-11-28T13:00:10.618665+0000', 'last_fullsized': '2023-11-28T13:00:10.618665+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T17:48:31.549366+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-28T13:00:16.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2023-11-28T13:00:16.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2023-11-28T13:00:16.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.621901+0000', 'last_change': '2023-11-28T13:00:10.621901+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.621901+0000', 'last_undegraded': '2023-11-28T13:00:10.621901+0000', 'last_fullsized': '2023-11-28T13:00:10.621901+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T17:57:53.793229+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-28T13:00:16.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1d is not active+clean 2023-11-28T13:00:16.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2023-11-28T13:00:16.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.619096+0000', 'last_change': '2023-11-28T13:00:10.619096+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.619096+0000', 'last_undegraded': '2023-11-28T13:00:10.619096+0000', 'last_fullsized': '2023-11-28T13:00:10.619096+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T13:00:16.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1a is not active+clean 2023-11-28T13:00:16.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2023-11-28T13:00:16.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.624870+0000', 'last_change': '2023-11-28T13:00:10.624870+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.624870+0000', 'last_undegraded': '2023-11-28T13:00:10.624870+0000', 'last_fullsized': '2023-11-28T13:00:10.624870+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T20:02:41.353263+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T13:00:16.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2023-11-28T13:00:16.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2023-11-28T13:00:16.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.619341+0000', 'last_change': '2023-11-28T13:00:10.619341+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.619341+0000', 'last_undegraded': '2023-11-28T13:00:10.619341+0000', 'last_fullsized': '2023-11-28T13:00:10.619341+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T20:03:04.306890+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-28T13:00:16.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2023-11-28T13:00:16.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2023-11-28T13:00:16.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.621647+0000', 'last_change': '2023-11-28T13:00:10.621647+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.621647+0000', 'last_undegraded': '2023-11-28T13:00:10.621647+0000', 'last_fullsized': '2023-11-28T13:00:10.621647+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T13:00:16.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2023-11-28T13:00:16.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2023-11-28T13:00:16.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.627237+0000', 'last_change': '2023-11-28T13:00:10.627237+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.627237+0000', 'last_undegraded': '2023-11-28T13:00:10.627237+0000', 'last_fullsized': '2023-11-28T13:00:10.627237+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-28T13:00:16.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.16 is not active+clean 2023-11-28T13:00:16.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2023-11-28T13:00:16.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.619562+0000', 'last_change': '2023-11-28T13:00:10.619562+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.619562+0000', 'last_undegraded': '2023-11-28T13:00:10.619562+0000', 'last_fullsized': '2023-11-28T13:00:10.619562+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T14:01:31.408692+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-28T13:00:16.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2023-11-28T13:00:16.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2023-11-28T13:00:16.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.625891+0000', 'last_change': '2023-11-28T13:00:10.625891+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.625891+0000', 'last_undegraded': '2023-11-28T13:00:10.625891+0000', 'last_fullsized': '2023-11-28T13:00:10.625891+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T19:45:26.261331+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-28T13:00:16.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.14 is not active+clean 2023-11-28T13:00:16.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2023-11-28T13:00:16.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.620076+0000', 'last_change': '2023-11-28T13:00:10.620076+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.620076+0000', 'last_undegraded': '2023-11-28T13:00:10.620076+0000', 'last_fullsized': '2023-11-28T13:00:10.620076+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T13:00:16.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2023-11-28T13:00:16.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2023-11-28T13:00:16.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.621487+0000', 'last_change': '2023-11-28T13:00:10.621487+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.621487+0000', 'last_undegraded': '2023-11-28T13:00:10.621487+0000', 'last_fullsized': '2023-11-28T13:00:10.621487+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T22:28:07.771314+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-28T13:00:16.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2023-11-28T13:00:16.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2023-11-28T13:00:16.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.624473+0000', 'last_change': '2023-11-28T13:00:10.624473+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.624473+0000', 'last_undegraded': '2023-11-28T13:00:10.624473+0000', 'last_fullsized': '2023-11-28T13:00:10.624473+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T17:19:16.875338+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T13:00:16.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2023-11-28T13:00:16.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2023-11-28T13:00:16.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.619210+0000', 'last_change': '2023-11-28T13:00:10.619210+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.619210+0000', 'last_undegraded': '2023-11-28T13:00:10.619210+0000', 'last_fullsized': '2023-11-28T13:00:10.619210+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T13:00:16.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.10 is not active+clean 2023-11-28T13:00:16.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2023-11-28T13:00:16.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.623596+0000', 'last_change': '2023-11-28T13:00:10.623596+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.623596+0000', 'last_undegraded': '2023-11-28T13:00:10.623596+0000', 'last_fullsized': '2023-11-28T13:00:10.623596+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T17:08:35.062283+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T13:00:16.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2023-11-28T13:00:16.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2023-11-28T13:00:16.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.626302+0000', 'last_change': '2023-11-28T13:00:10.626302+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.626302+0000', 'last_undegraded': '2023-11-28T13:00:10.626302+0000', 'last_fullsized': '2023-11-28T13:00:10.626302+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-28T13:00:16.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.e is not active+clean 2023-11-28T13:00:16.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2023-11-28T13:00:16.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.621592+0000', 'last_change': '2023-11-28T13:00:10.621592+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.621592+0000', 'last_undegraded': '2023-11-28T13:00:10.621592+0000', 'last_fullsized': '2023-11-28T13:00:10.621592+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-30T00:01:19.809502+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-28T13:00:16.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2023-11-28T13:00:16.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2023-11-28T13:00:16.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.619154+0000', 'last_change': '2023-11-28T13:00:10.619154+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.619154+0000', 'last_undegraded': '2023-11-28T13:00:10.619154+0000', 'last_fullsized': '2023-11-28T13:00:10.619154+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T13:36:34.014681+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-28T13:00:16.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2023-11-28T13:00:16.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2023-11-28T13:00:16.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.622098+0000', 'last_change': '2023-11-28T13:00:10.622098+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.622098+0000', 'last_undegraded': '2023-11-28T13:00:10.622098+0000', 'last_fullsized': '2023-11-28T13:00:10.622098+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T13:00:16.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2023-11-28T13:00:16.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2023-11-28T13:00:16.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.622534+0000', 'last_change': '2023-11-28T13:00:10.622534+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.622534+0000', 'last_undegraded': '2023-11-28T13:00:10.622534+0000', 'last_fullsized': '2023-11-28T13:00:10.622534+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T13:00:16.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.a is not active+clean 2023-11-28T13:00:16.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2023-11-28T13:00:16.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.622647+0000', 'last_change': '2023-11-28T13:00:10.622647+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.622647+0000', 'last_undegraded': '2023-11-28T13:00:10.622647+0000', 'last_fullsized': '2023-11-28T13:00:10.622647+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-30T00:17:03.747332+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T13:00:16.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.b is not active+clean 2023-11-28T13:00:16.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2023-11-28T13:00:16.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.622101+0000', 'last_change': '2023-11-28T13:00:10.622101+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.622101+0000', 'last_undegraded': '2023-11-28T13:00:10.622101+0000', 'last_fullsized': '2023-11-28T13:00:10.622101+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T13:00:16.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.8 is not active+clean 2023-11-28T13:00:16.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2023-11-28T13:00:16.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.625935+0000', 'last_change': '2023-11-28T13:00:10.625935+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.625935+0000', 'last_undegraded': '2023-11-28T13:00:10.625935+0000', 'last_fullsized': '2023-11-28T13:00:10.625935+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-30T00:37:07.844503+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-28T13:00:16.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2023-11-28T13:00:16.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2023-11-28T13:00:16.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.619380+0000', 'last_change': '2023-11-28T13:00:10.619380+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.619380+0000', 'last_undegraded': '2023-11-28T13:00:10.619380+0000', 'last_fullsized': '2023-11-28T13:00:10.619380+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T19:22:30.539161+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T13:00:16.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1 is not active+clean 2023-11-28T13:00:16.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2023-11-28T13:00:16.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.625846+0000', 'last_change': '2023-11-28T13:00:10.625846+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.625846+0000', 'last_undegraded': '2023-11-28T13:00:10.625846+0000', 'last_fullsized': '2023-11-28T13:00:10.625846+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-11-28T13:00:16.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2023-11-28T13:00:16.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2023-11-28T13:00:16.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.622988+0000', 'last_change': '2023-11-28T13:00:10.622988+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.622988+0000', 'last_undegraded': '2023-11-28T13:00:10.622988+0000', 'last_fullsized': '2023-11-28T13:00:10.622988+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-11-28T13:00:16.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2023-11-28T13:00:16.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.882 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2023-11-28T13:00:16.882 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.619075+0000', 'last_change': '2023-11-28T13:00:10.619075+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.619075+0000', 'last_undegraded': '2023-11-28T13:00:10.619075+0000', 'last_fullsized': '2023-11-28T13:00:10.619075+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T19:45:26.261331+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T13:00:16.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2023-11-28T13:00:16.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2023-11-28T13:00:16.883 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.621665+0000', 'last_change': '2023-11-28T13:00:10.621665+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.621665+0000', 'last_undegraded': '2023-11-28T13:00:10.621665+0000', 'last_fullsized': '2023-11-28T13:00:10.621665+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T19:22:30.539161+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-11-28T13:00:16.883 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2023-11-28T13:00:16.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2023-11-28T13:00:16.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.621578+0000', 'last_change': '2023-11-28T13:00:10.621578+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.621578+0000', 'last_undegraded': '2023-11-28T13:00:10.621578+0000', 'last_fullsized': '2023-11-28T13:00:10.621578+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T13:02:30.352479+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-28T13:00:16.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2023-11-28T13:00:16.884 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.884 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2023-11-28T13:00:16.885 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.621478+0000', 'last_change': '2023-11-28T13:00:10.621478+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.621478+0000', 'last_undegraded': '2023-11-28T13:00:10.621478+0000', 'last_fullsized': '2023-11-28T13:00:10.621478+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T18:33:40.321176+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-28T13:00:16.885 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3 is not active+clean 2023-11-28T13:00:16.885 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.885 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2023-11-28T13:00:16.885 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.618918+0000', 'last_change': '2023-11-28T13:00:10.618918+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.618918+0000', 'last_undegraded': '2023-11-28T13:00:10.618918+0000', 'last_fullsized': '2023-11-28T13:00:10.618918+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T17:50:51.288479+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-11-28T13:00:16.886 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2 is not active+clean 2023-11-28T13:00:16.886 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-11-28T13:00:11.615718+0000', 'last_change': '2023-11-28T13:00:11.615718+0000', 'last_active': '2023-11-28T13:00:11.615718+0000', 'last_peered': '2023-11-28T13:00:11.615718+0000', 'last_clean': '2023-11-28T13:00:11.615718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:11.615718+0000', 'last_undegraded': '2023-11-28T13:00:11.615718+0000', 'last_fullsized': '2023-11-28T13:00:11.615718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:11.615718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-11-28T13:00:16.886 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2023-11-28T13:00:16.886 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 38, 'state': 'creating+peering', 'last_fresh': '2023-11-28T13:00:10.619023+0000', 'last_change': '2023-11-28T13:00:10.619023+0000', 'last_active': '2023-11-28T13:00:10.613366+0000', 'last_peered': '2023-11-28T13:00:10.613366+0000', 'last_clean': '2023-11-28T13:00:10.613366+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-28T13:00:10.619023+0000', 'last_undegraded': '2023-11-28T13:00:10.619023+0000', 'last_fullsized': '2023-11-28T13:00:10.619023+0000', 'mapping_epoch': 38, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 38, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'last_clean_scrub_stamp': '2023-11-28T13:00:10.613366+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-29T18:46:47.273034+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-11-28T13:00:16.887 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-11-28T13:00:17.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.011+0000 7fc12c982700 1 Processor -- start 2023-11-28T13:00:17.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.011+0000 7fc12c982700 1 -- start start 2023-11-28T13:00:17.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12c982700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc12806ba90 0x7fc1280cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:17.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12c982700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1280cb810 con 0x7fc1280d0490 2023-11-28T13:00:17.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12c982700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1280cc650 con 0x7fc12815c7b0 2023-11-28T13:00:17.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12c982700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1280cba40 con 0x7fc12806ba90 2023-11-28T13:00:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc125d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc1280d0490 0x7fc12806b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36074/0 (socket says 172.21.15.111:36074) 2023-11-28T13:00:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc125d9b700 1 -- 172.21.15.111:0/1443283918 learned_addr learned my addr 172.21.15.111:0/1443283918 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152457856 0 0) 0x7fc1280cb810 con 0x7fc1280d0490 2023-11-28T13:00:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc126d9d700 1 --2- 172.21.15.111:0/1443283918 >> v2:172.21.15.149:3300/0 conn(0x7fc12806ba90 0x7fc1280cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1100035b0 con 0x7fc1280d0490 2023-11-28T13:00:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2835218486 0 0) 0x7fc1100035b0 con 0x7fc1280d0490 2023-11-28T13:00:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 >> v2:172.21.15.149:3300/0 conn(0x7fc12806ba90 msgr2=0x7fc1280cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 --2- 172.21.15.111:0/1443283918 >> v2:172.21.15.149:3300/0 conn(0x7fc12806ba90 0x7fc1280cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 >> v1:172.21.15.149:6789/0 conn(0x7fc12815c7b0 legacy=0x7fc1280ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1280cbd20 con 0x7fc1280d0490 2023-11-28T13:00:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fc11c001670 con 0x7fc1280d0490 2023-11-28T13:00:17.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 >> v1:172.21.15.111:6789/0 conn(0x7fc1280d0490 legacy=0x7fc12806b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 --2- 172.21.15.111:0/1443283918 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc110003a00 0x7fc110003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 --2- 172.21.15.111:0/1443283918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc110004850 0x7fc110004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 --2- 172.21.15.111:0/1443283918 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc110005200 0x7fc11000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1280cba40 con 0x7fc110004850 2023-11-28T13:00:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11000c110 con 0x7fc110003a00 2023-11-28T13:00:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11000c2d0 con 0x7fc110005200 2023-11-28T13:00:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc125d9b700 1 --2- 172.21.15.111:0/1443283918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc110004850 0x7fc110004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.012+0000 7fc12659c700 1 --2- 172.21.15.111:0/1443283918 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc110003a00 0x7fc110003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc126d9d700 1 --2- 172.21.15.111:0/1443283918 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc110005200 0x7fc11000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc125d9b700 1 -- 172.21.15.111:0/1443283918 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc110005200 msgr2=0x7fc11000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:17.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc125d9b700 1 --2- 172.21.15.111:0/1443283918 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc110005200 0x7fc11000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc125d9b700 1 -- 172.21.15.111:0/1443283918 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc110003a00 msgr2=0x7fc110003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc125d9b700 1 --2- 172.21.15.111:0/1443283918 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc110003a00 0x7fc110003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc125d9b700 1 -- 172.21.15.111:0/1443283918 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1100035b0 con 0x7fc110004850 2023-11-28T13:00:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc126d9d700 1 -- 172.21.15.111:0/1443283918 reap_dead start 2023-11-28T13:00:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc12659c700 1 --2- 172.21.15.111:0/1443283918 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc110003a00 0x7fc110003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc125d9b700 1 --2- 172.21.15.111:0/1443283918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc110004850 0x7fc110004cc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00c2a0 tx=0x7fc110005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c745fc846a7628b2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c007c10 con 0x7fc110004850 2023-11-28T13:00:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c00cdc0 con 0x7fc110004850 2023-11-28T13:00:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc12559a700 1 -- 172.21.15.111:0/1443283918 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c005ba0 con 0x7fc110004850 2023-11-28T13:00:17.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc12c982700 1 -- 172.21.15.111:0/1443283918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc110004850 msgr2=0x7fc110004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc12c982700 1 --2- 172.21.15.111:0/1443283918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc110004850 0x7fc110004cc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00c2a0 tx=0x7fc110005880 comp rx=0 tx=0).stop 2023-11-28T13:00:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc12c982700 1 -- 172.21.15.111:0/1443283918 shutdown_connections 2023-11-28T13:00:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc12c982700 1 --2- 172.21.15.111:0/1443283918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc110004850 0x7fc110004cc0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.013+0000 7fc12c982700 1 -- 172.21.15.111:0/1443283918 >> 172.21.15.111:0/1443283918 conn(0x7fc12809a040 msgr2=0x7fc1280d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.014+0000 7fc12c982700 1 -- 172.21.15.111:0/1443283918 shutdown_connections 2023-11-28T13:00:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.014+0000 7fc12c982700 1 -- 172.21.15.111:0/1443283918 wait complete. 2023-11-28T13:00:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.015+0000 7fc12c982700 1 Processor -- start 2023-11-28T13:00:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.015+0000 7fc12c982700 1 -- start start 2023-11-28T13:00:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12c982700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc110004850 0x7fc12815f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12c982700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc12815dee0 0x7fc128148ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:17.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12c982700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc1280b6550 0x7fc1280b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:17.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12c982700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12815f0b0 con 0x7fc110004850 2023-11-28T13:00:17.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12c982700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12815e430 con 0x7fc1280b6550 2023-11-28T13:00:17.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12c982700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12815e710 con 0x7fc12815dee0 2023-11-28T13:00:17.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc110004850 0x7fc12815f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:17.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc110004850 0x7fc12815f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35308/0 (socket says 172.21.15.111:35308) 2023-11-28T13:00:17.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12659c700 1 -- 172.21.15.111:0/720546165 learned_addr learned my addr 172.21.15.111:0/720546165 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:17.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc125d9b700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc12815dee0 0x7fc128148ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:17.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc126d9d700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc1280b6550 0x7fc1280b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:17.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12659c700 1 -- 172.21.15.111:0/720546165 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc12815dee0 msgr2=0x7fc128148ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12659c700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc12815dee0 0x7fc128148ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12659c700 1 -- 172.21.15.111:0/720546165 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc1280b6550 msgr2=0x7fc1280b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12659c700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc1280b6550 0x7fc1280b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12659c700 1 -- 172.21.15.111:0/720546165 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1280cc2e0 con 0x7fc110004850 2023-11-28T13:00:17.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12659c700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc110004850 0x7fc12815f4d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc11800e810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:17.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc116ffd700 1 -- 172.21.15.111:0/720546165 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1180052e0 con 0x7fc110004850 2023-11-28T13:00:17.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc116ffd700 1 -- 172.21.15.111:0/720546165 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc118005440 con 0x7fc110004850 2023-11-28T13:00:17.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12c982700 1 -- 172.21.15.111:0/720546165 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1280cc000 con 0x7fc110004850 2023-11-28T13:00:17.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc116ffd700 1 -- 172.21.15.111:0/720546165 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc118003910 con 0x7fc110004850 2023-11-28T13:00:17.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.016+0000 7fc12c982700 1 -- 172.21.15.111:0/720546165 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12815cb80 con 0x7fc110004850 2023-11-28T13:00:17.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.017+0000 7fc12c982700 1 -- 172.21.15.111:0/720546165 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12815f320 con 0x7fc110004850 2023-11-28T13:00:17.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.018+0000 7fc116ffd700 1 -- 172.21.15.111:0/720546165 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc11801b030 con 0x7fc110004850 2023-11-28T13:00:17.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.018+0000 7fc116ffd700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc1080682f0 0x7fc10806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:17.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.018+0000 7fc116ffd700 1 -- 172.21.15.111:0/720546165 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc118085290 con 0x7fc110004850 2023-11-28T13:00:17.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.018+0000 7fc125d9b700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc1080682f0 0x7fc10806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:17.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.018+0000 7fc125d9b700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc1080682f0 0x7fc10806a7a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc12815fc20 tx=0x7fc11c007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:17.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.022+0000 7fc116ffd700 1 -- 172.21.15.111:0/720546165 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc11804d5e0 con 0x7fc110004850 2023-11-28T13:00:17.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.282+0000 7fc12c982700 1 -- 172.21.15.111:0/720546165 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fc128067170 con 0x7fc110004850 2023-11-28T13:00:17.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.283+0000 7fc116ffd700 1 -- 172.21.15.111:0/720546165 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1108 (secure 0 0 0) 0x7fc118019470 con 0x7fc110004850 2023-11-28T13:00:17.284 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:17.285 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":102,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":41,"num_osds":8,"num_up_osds":8,"osd_up_since":1701176326,"num_in_osds":8,"osd_in_since":1701176317,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":9}],"num_pgs":105,"num_pools":4,"num_objects":2,"data_bytes":590368,"bytes_used":71712768,"bytes_avail":773022400512,"bytes_total":773094113280,"unknown_pgs_ratio":0.60952383279800415,"inactive_pgs_ratio":0.30476191639900208},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"d","status":"up:active","gid":5352}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2023-11-28T13:00:00.349048+0000","services":{}},"progress_events":{}} 2023-11-28T13:00:17.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.285+0000 7fc12c982700 1 -- 172.21.15.111:0/720546165 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc1080682f0 msgr2=0x7fc10806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.285+0000 7fc12c982700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc1080682f0 0x7fc10806a7a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc12815fc20 tx=0x7fc11c007000 comp rx=0 tx=0).stop 2023-11-28T13:00:17.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.285+0000 7fc12c982700 1 -- 172.21.15.111:0/720546165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc110004850 msgr2=0x7fc12815f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.285+0000 7fc12c982700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc110004850 0x7fc12815f4d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc11800e810 comp rx=0 tx=0).stop 2023-11-28T13:00:17.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.285+0000 7fc12c982700 1 -- 172.21.15.111:0/720546165 shutdown_connections 2023-11-28T13:00:17.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.285+0000 7fc12c982700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc1080682f0 0x7fc10806a7a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.285+0000 7fc12c982700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc1280b6550 0x7fc1280b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.285+0000 7fc12c982700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc110004850 0x7fc12815f4d0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.285+0000 7fc12c982700 1 --2- 172.21.15.111:0/720546165 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc12815dee0 0x7fc128148ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.285+0000 7fc12c982700 1 -- 172.21.15.111:0/720546165 >> 172.21.15.111:0/720546165 conn(0x7fc12809a040 msgr2=0x7fc12806f440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:17.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.285+0000 7fc12c982700 1 -- 172.21.15.111:0/720546165 shutdown_connections 2023-11-28T13:00:17.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.285+0000 7fc12c982700 1 -- 172.21.15.111:0/720546165 wait complete. 2023-11-28T13:00:17.298 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-11-28T13:00:17.299 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:00:17.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.417+0000 7fe2e559f700 1 Processor -- start 2023-11-28T13:00:17.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.417+0000 7fe2e559f700 1 -- start start 2023-11-28T13:00:17.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.417+0000 7fe2e559f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2e00d1d70 0x7fe2e0167270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:17.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.417+0000 7fe2e559f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e006c790 con 0x7fe2e00b8510 2023-11-28T13:00:17.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.417+0000 7fe2e559f700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e00b88e0 con 0x7fe2e00cd460 2023-11-28T13:00:17.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.417+0000 7fe2e559f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2e00cdbd0 con 0x7fe2e00d1d70 2023-11-28T13:00:17.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.418+0000 7fe2df7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2e00d1d70 0x7fe2e0167270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:17.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.418+0000 7fe2deffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe2e00b8510 0x7fe2e006b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36094/0 (socket says 172.21.15.111:36094) 2023-11-28T13:00:17.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.418+0000 7fe2deffd700 1 -- 172.21.15.111:0/2692735218 learned_addr learned my addr 172.21.15.111:0/2692735218 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:17.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.418+0000 7fe2df7fe700 1 -- 172.21.15.111:0/2692735218 >> v1:172.21.15.149:6789/0 conn(0x7fe2e00cd460 legacy=0x7fe2e006c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.418+0000 7fe2df7fe700 1 -- 172.21.15.111:0/2692735218 >> v1:172.21.15.111:6789/0 conn(0x7fe2e00b8510 legacy=0x7fe2e006b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.418+0000 7fe2df7fe700 1 -- 172.21.15.111:0/2692735218 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e00cd8f0 con 0x7fe2e00d1d70 2023-11-28T13:00:17.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.418+0000 7fe2ddffb700 1 -- 172.21.15.111:0/2692735218 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825887182 0 0) 0x7fe2e006c790 con 0x7fe2e00b8510 2023-11-28T13:00:17.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.418+0000 7fe2df7fe700 1 --2- 172.21.15.111:0/2692735218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e00d1d70 0x7fe2e0167270 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8009d90 tx=0x7fe2d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7eb63353ddc6bd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:17.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.418+0000 7fe2ddffb700 1 -- 172.21.15.111:0/2692735218 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d800e070 con 0x7fe2e00d1d70 2023-11-28T13:00:17.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.418+0000 7fe2ddffb700 1 -- 172.21.15.111:0/2692735218 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d800bbd0 con 0x7fe2e00d1d70 2023-11-28T13:00:17.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.418+0000 7fe2ddffb700 1 -- 172.21.15.111:0/2692735218 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8012430 con 0x7fe2e00d1d70 2023-11-28T13:00:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.419+0000 7fe2e559f700 1 -- 172.21.15.111:0/2692735218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e00d1d70 msgr2=0x7fe2e0167270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.419+0000 7fe2e559f700 1 --2- 172.21.15.111:0/2692735218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e00d1d70 0x7fe2e0167270 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8009d90 tx=0x7fe2d800b330 comp rx=0 tx=0).stop 2023-11-28T13:00:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.419+0000 7fe2e559f700 1 -- 172.21.15.111:0/2692735218 shutdown_connections 2023-11-28T13:00:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.419+0000 7fe2e559f700 1 --2- 172.21.15.111:0/2692735218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e00d1d70 0x7fe2e0167270 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.419+0000 7fe2e559f700 1 -- 172.21.15.111:0/2692735218 >> 172.21.15.111:0/2692735218 conn(0x7fe2e00719a0 msgr2=0x7fe2e0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.419+0000 7fe2e559f700 1 -- 172.21.15.111:0/2692735218 shutdown_connections 2023-11-28T13:00:17.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.419+0000 7fe2e559f700 1 -- 172.21.15.111:0/2692735218 wait complete. 2023-11-28T13:00:17.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.420+0000 7fe2e559f700 1 Processor -- start 2023-11-28T13:00:17.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.420+0000 7fe2e559f700 1 -- start start 2023-11-28T13:00:17.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2e559f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe2e00b8510 0x7fe2e014e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:17.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2e559f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e00cd460 0x7fe2e01e4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:17.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2e559f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe2e00d1d70 0x7fe2e01e54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:17.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2e559f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e014e700 con 0x7fe2e00cd460 2023-11-28T13:00:17.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2e559f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e01e5b00 con 0x7fe2e00b8510 2023-11-28T13:00:17.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2e559f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e01e5de0 con 0x7fe2e00d1d70 2023-11-28T13:00:17.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2de7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e00cd460 0x7fe2e01e4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:17.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2de7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e00cd460 0x7fe2e01e4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35324/0 (socket says 172.21.15.111:35324) 2023-11-28T13:00:17.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2de7fc700 1 -- 172.21.15.111:0/1966105405 learned_addr learned my addr 172.21.15.111:0/1966105405 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:17.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2deffd700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe2e00b8510 0x7fe2e014e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:17.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2df7fe700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe2e00d1d70 0x7fe2e01e54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:17.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2de7fc700 1 -- 172.21.15.111:0/1966105405 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe2e00d1d70 msgr2=0x7fe2e01e54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2de7fc700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe2e00d1d70 0x7fe2e01e54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2de7fc700 1 -- 172.21.15.111:0/1966105405 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe2e00b8510 msgr2=0x7fe2e014e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2de7fc700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe2e00b8510 0x7fe2e014e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2de7fc700 1 -- 172.21.15.111:0/1966105405 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d8009a70 con 0x7fe2e00cd460 2023-11-28T13:00:17.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2de7fc700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e00cd460 0x7fe2e01e4f90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400c480 tx=0x7fe2d400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:17.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.421+0000 7fe2c77fe700 1 -- 172.21.15.111:0/1966105405 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d40052d0 con 0x7fe2e00cd460 2023-11-28T13:00:17.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.422+0000 7fe2c77fe700 1 -- 172.21.15.111:0/1966105405 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d4005430 con 0x7fe2e00cd460 2023-11-28T13:00:17.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.422+0000 7fe2c77fe700 1 -- 172.21.15.111:0/1966105405 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d40038c0 con 0x7fe2e00cd460 2023-11-28T13:00:17.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.422+0000 7fe2e559f700 1 -- 172.21.15.111:0/1966105405 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e00cfff0 con 0x7fe2e00cd460 2023-11-28T13:00:17.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.422+0000 7fe2e559f700 1 -- 172.21.15.111:0/1966105405 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e00cfd10 con 0x7fe2e00cd460 2023-11-28T13:00:17.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.423+0000 7fe2e559f700 1 -- 172.21.15.111:0/1966105405 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e0067880 con 0x7fe2e00cd460 2023-11-28T13:00:17.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.423+0000 7fe2c77fe700 1 -- 172.21.15.111:0/1966105405 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe2d4019030 con 0x7fe2e00cd460 2023-11-28T13:00:17.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.423+0000 7fe2c77fe700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2cc0682f0 0x7fe2cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:17.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.423+0000 7fe2c77fe700 1 -- 172.21.15.111:0/1966105405 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe2d4085060 con 0x7fe2e00cd460 2023-11-28T13:00:17.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.423+0000 7fe2deffd700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2cc0682f0 0x7fe2cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:17.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.424+0000 7fe2deffd700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2cc0682f0 0x7fe2cc06a7a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0007b10 tx=0x7fe2d0007a60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:17.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.427+0000 7fe2c77fe700 1 -- 172.21.15.111:0/1966105405 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2d4051840 con 0x7fe2e00cd460 2023-11-28T13:00:17.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.605+0000 7fe2e559f700 1 -- 172.21.15.111:0/1966105405 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2e00d2940 con 0x7fe2cc0682f0 2023-11-28T13:00:17.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.613+0000 7fe2c77fe700 1 -- 172.21.15.111:0/1966105405 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+300126 (secure 0 0 0) 0x7fe2e00d2940 con 0x7fe2cc0682f0 2023-11-28T13:00:17.617 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:17.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.616+0000 7fe2e559f700 1 -- 172.21.15.111:0/1966105405 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2cc0682f0 msgr2=0x7fe2cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.616+0000 7fe2e559f700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2cc0682f0 0x7fe2cc06a7a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0007b10 tx=0x7fe2d0007a60 comp rx=0 tx=0).stop 2023-11-28T13:00:17.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.616+0000 7fe2e559f700 1 -- 172.21.15.111:0/1966105405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e00cd460 msgr2=0x7fe2e01e4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:17.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.616+0000 7fe2e559f700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e00cd460 0x7fe2e01e4f90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400c480 tx=0x7fe2d400c790 comp rx=0 tx=0).stop 2023-11-28T13:00:17.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.616+0000 7fe2e559f700 1 -- 172.21.15.111:0/1966105405 shutdown_connections 2023-11-28T13:00:17.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.616+0000 7fe2e559f700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe2e00d1d70 0x7fe2e01e54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.616+0000 7fe2e559f700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2cc0682f0 0x7fe2cc06a7a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.616+0000 7fe2e559f700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe2e00b8510 0x7fe2e014e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.616+0000 7fe2e559f700 1 --2- 172.21.15.111:0/1966105405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2e00cd460 0x7fe2e01e4f90 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:17.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.616+0000 7fe2e559f700 1 -- 172.21.15.111:0/1966105405 >> 172.21.15.111:0/1966105405 conn(0x7fe2e00719a0 msgr2=0x7fe2e00b3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:17.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.616+0000 7fe2e559f700 1 -- 172.21.15.111:0/1966105405 shutdown_connections 2023-11-28T13:00:17.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:17.616+0000 7fe2e559f700 1 -- 172.21.15.111:0/1966105405 wait complete. 2023-11-28T13:00:17.621 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:00:17.629 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2023-11-28T13:00:16.351493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":25,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":70032,"kb_used_data":4560,"kb_used_omap":0,"kb_used_meta":65408,"kb_avail":754904688,"statfs":{"total":773094113280,"available":773022400512,"internally_reserved":0,"allocated":4669440,"data_stored":2706556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66977792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.735324"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622561+0000","last_change":"2023-11-28T13:00:10.622561+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622561+0000","last_undegraded":"2023-11-28T13:00:10.622561+0000","last_fullsized":"2023-11-28T13:00:10.622561+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:47.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.618665+0000","last_change":"2023-11-28T13:00:10.618665+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.618665+0000","last_undegraded":"2023-11-28T13:00:10.618665+0000","last_fullsized":"2023-11-28T13:00:10.618665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:48:31.549366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621901+0000","last_change":"2023-11-28T13:00:10.621901+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621901+0000","last_undegraded":"2023-11-28T13:00:10.621901+0000","last_fullsized":"2023-11-28T13:00:10.621901+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:57:53.793229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619096+0000","last_change":"2023-11-28T13:00:10.619096+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619096+0000","last_undegraded":"2023-11-28T13:00:10.619096+0000","last_fullsized":"2023-11-28T13:00:10.619096+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.624870+0000","last_change":"2023-11-28T13:00:10.624870+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.624870+0000","last_undegraded":"2023-11-28T13:00:10.624870+0000","last_fullsized":"2023-11-28T13:00:10.624870+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:02:41.353263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619341+0000","last_change":"2023-11-28T13:00:10.619341+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619341+0000","last_undegraded":"2023-11-28T13:00:10.619341+0000","last_fullsized":"2023-11-28T13:00:10.619341+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:03:04.306890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621647+0000","last_change":"2023-11-28T13:00:10.621647+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621647+0000","last_undegraded":"2023-11-28T13:00:10.621647+0000","last_fullsized":"2023-11-28T13:00:10.621647+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.627237+0000","last_change":"2023-11-28T13:00:10.627237+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.627237+0000","last_undegraded":"2023-11-28T13:00:10.627237+0000","last_fullsized":"2023-11-28T13:00:10.627237+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619562+0000","last_change":"2023-11-28T13:00:10.619562+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619562+0000","last_undegraded":"2023-11-28T13:00:10.619562+0000","last_fullsized":"2023-11-28T13:00:10.619562+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:01:31.408692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.625891+0000","last_change":"2023-11-28T13:00:10.625891+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.625891+0000","last_undegraded":"2023-11-28T13:00:10.625891+0000","last_fullsized":"2023-11-28T13:00:10.625891+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:26.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.620076+0000","last_change":"2023-11-28T13:00:10.620076+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.620076+0000","last_undegraded":"2023-11-28T13:00:10.620076+0000","last_fullsized":"2023-11-28T13:00:10.620076+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621487+0000","last_change":"2023-11-28T13:00:10.621487+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621487+0000","last_undegraded":"2023-11-28T13:00:10.621487+0000","last_fullsized":"2023-11-28T13:00:10.621487+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:28:07.771314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.624473+0000","last_change":"2023-11-28T13:00:10.624473+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.624473+0000","last_undegraded":"2023-11-28T13:00:10.624473+0000","last_fullsized":"2023-11-28T13:00:10.624473+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:19:16.875338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619210+0000","last_change":"2023-11-28T13:00:10.619210+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619210+0000","last_undegraded":"2023-11-28T13:00:10.619210+0000","last_fullsized":"2023-11-28T13:00:10.619210+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.623596+0000","last_change":"2023-11-28T13:00:10.623596+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.623596+0000","last_undegraded":"2023-11-28T13:00:10.623596+0000","last_fullsized":"2023-11-28T13:00:10.623596+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:08:35.062283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.626302+0000","last_change":"2023-11-28T13:00:10.626302+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.626302+0000","last_undegraded":"2023-11-28T13:00:10.626302+0000","last_fullsized":"2023-11-28T13:00:10.626302+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621592+0000","last_change":"2023-11-28T13:00:10.621592+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621592+0000","last_undegraded":"2023-11-28T13:00:10.621592+0000","last_fullsized":"2023-11-28T13:00:10.621592+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:19.809502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619154+0000","last_change":"2023-11-28T13:00:10.619154+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619154+0000","last_undegraded":"2023-11-28T13:00:10.619154+0000","last_fullsized":"2023-11-28T13:00:10.619154+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:36:34.014681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622098+0000","last_change":"2023-11-28T13:00:10.622098+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622098+0000","last_undegraded":"2023-11-28T13:00:10.622098+0000","last_fullsized":"2023-11-28T13:00:10.622098+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622534+0000","last_change":"2023-11-28T13:00:10.622534+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622534+0000","last_undegraded":"2023-11-28T13:00:10.622534+0000","last_fullsized":"2023-11-28T13:00:10.622534+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622647+0000","last_change":"2023-11-28T13:00:10.622647+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622647+0000","last_undegraded":"2023-11-28T13:00:10.622647+0000","last_fullsized":"2023-11-28T13:00:10.622647+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:17:03.747332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622101+0000","last_change":"2023-11-28T13:00:10.622101+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622101+0000","last_undegraded":"2023-11-28T13:00:10.622101+0000","last_fullsized":"2023-11-28T13:00:10.622101+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.625935+0000","last_change":"2023-11-28T13:00:10.625935+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.625935+0000","last_undegraded":"2023-11-28T13:00:10.625935+0000","last_fullsized":"2023-11-28T13:00:10.625935+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:37:07.844503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619380+0000","last_change":"2023-11-28T13:00:10.619380+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619380+0000","last_undegraded":"2023-11-28T13:00:10.619380+0000","last_fullsized":"2023-11-28T13:00:10.619380+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:22:30.539161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.625846+0000","last_change":"2023-11-28T13:00:10.625846+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.625846+0000","last_undegraded":"2023-11-28T13:00:10.625846+0000","last_fullsized":"2023-11-28T13:00:10.625846+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806154+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T13:00:05.806154+0000","last_peered":"2023-11-28T13:00:05.806154+0000","last_clean":"2023-11-28T13:00:05.806154+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:00:05.806154+0000","last_undegraded":"2023-11-28T13:00:05.806154+0000","last_fullsized":"2023-11-28T13:00:05.806154+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.622988+0000","last_change":"2023-11-28T13:00:10.622988+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.622988+0000","last_undegraded":"2023-11-28T13:00:10.622988+0000","last_fullsized":"2023-11-28T13:00:10.622988+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806797+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T13:00:05.806797+0000","last_peered":"2023-11-28T13:00:05.806797+0000","last_clean":"2023-11-28T13:00:05.806797+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:00:05.806797+0000","last_undegraded":"2023-11-28T13:00:05.806797+0000","last_fullsized":"2023-11-28T13:00:05.806797+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619075+0000","last_change":"2023-11-28T13:00:10.619075+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619075+0000","last_undegraded":"2023-11-28T13:00:10.619075+0000","last_fullsized":"2023-11-28T13:00:10.619075+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:26.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.255891+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T13:00:05.255891+0000","last_peered":"2023-11-28T13:00:05.255891+0000","last_clean":"2023-11-28T13:00:05.255891+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:00:05.255891+0000","last_undegraded":"2023-11-28T13:00:05.255891+0000","last_fullsized":"2023-11-28T13:00:05.255891+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621665+0000","last_change":"2023-11-28T13:00:10.621665+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621665+0000","last_undegraded":"2023-11-28T13:00:10.621665+0000","last_fullsized":"2023-11-28T13:00:10.621665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:22:30.539161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806796+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T13:00:05.806796+0000","last_peered":"2023-11-28T13:00:05.806796+0000","last_clean":"2023-11-28T13:00:05.806796+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:00:05.806796+0000","last_undegraded":"2023-11-28T13:00:05.806796+0000","last_fullsized":"2023-11-28T13:00:05.806796+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":179,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806772+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T13:00:05.806772+0000","last_peered":"2023-11-28T13:00:05.806772+0000","last_clean":"2023-11-28T13:00:05.806772+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:00:05.806772+0000","last_undegraded":"2023-11-28T13:00:05.806772+0000","last_fullsized":"2023-11-28T13:00:05.806772+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621578+0000","last_change":"2023-11-28T13:00:10.621578+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621578+0000","last_undegraded":"2023-11-28T13:00:10.621578+0000","last_fullsized":"2023-11-28T13:00:10.621578+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:02:30.352479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806793+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T13:00:05.806793+0000","last_peered":"2023-11-28T13:00:05.806793+0000","last_clean":"2023-11-28T13:00:05.806793+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:00:05.806793+0000","last_undegraded":"2023-11-28T13:00:05.806793+0000","last_fullsized":"2023-11-28T13:00:05.806793+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.621478+0000","last_change":"2023-11-28T13:00:10.621478+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.621478+0000","last_undegraded":"2023-11-28T13:00:10.621478+0000","last_fullsized":"2023-11-28T13:00:10.621478+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:40.321176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806781+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T13:00:05.806781+0000","last_peered":"2023-11-28T13:00:05.806781+0000","last_clean":"2023-11-28T13:00:05.806781+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:00:05.806781+0000","last_undegraded":"2023-11-28T13:00:05.806781+0000","last_fullsized":"2023-11-28T13:00:05.806781+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.618918+0000","last_change":"2023-11-28T13:00:10.618918+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.618918+0000","last_undegraded":"2023-11-28T13:00:10.618918+0000","last_fullsized":"2023-11-28T13:00:10.618918+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:50:51.288479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.806162+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T13:00:05.806162+0000","last_peered":"2023-11-28T13:00:05.806162+0000","last_clean":"2023-11-28T13:00:05.806162+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:00:05.806162+0000","last_undegraded":"2023-11-28T13:00:05.806162+0000","last_fullsized":"2023-11-28T13:00:05.806162+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:00:11.615718+0000","last_change":"2023-11-28T13:00:11.615718+0000","last_active":"2023-11-28T13:00:11.615718+0000","last_peered":"2023-11-28T13:00:11.615718+0000","last_clean":"2023-11-28T13:00:11.615718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:11.615718+0000","last_undegraded":"2023-11-28T13:00:11.615718+0000","last_fullsized":"2023-11-28T13:00:11.615718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":38,"state":"creating+peering","last_fresh":"2023-11-28T13:00:10.619023+0000","last_change":"2023-11-28T13:00:10.619023+0000","last_active":"2023-11-28T13:00:10.613366+0000","last_peered":"2023-11-28T13:00:10.613366+0000","last_clean":"2023-11-28T13:00:10.613366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:00:10.619023+0000","last_undegraded":"2023-11-28T13:00:10.619023+0000","last_fullsized":"2023-11-28T13:00:10.619023+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:47.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-28T13:00:05.251149+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T13:00:05.251149+0000","last_peered":"2023-11-28T13:00:05.251149+0000","last_clean":"2023-11-28T13:00:05.251149+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:00:05.251149+0000","last_undegraded":"2023-11-28T13:00:05.251149+0000","last_fullsized":"2023-11-28T13:00:05.251149+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9580,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362260,"statfs":{"total":96636764160,"available":96626954240,"internally_reserved":0,"allocated":1019904,"data_stored":776228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.3279999999999998,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.2919999999999998,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.47999999999999998}]},{"osd":1,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.53800000000000003}]},{"osd":3,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.39000000000000001}]},{"osd":4,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.441},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.59099999999999997}]},{"osd":5,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.52100000000000002}]},{"osd":6,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.56200000000000006}]}]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8436,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363404,"statfs":{"total":96636764160,"available":96628125696,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.376},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.38800000000000001}]},{"osd":1,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.433},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.53700000000000003}]},{"osd":2,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.46100000000000002}]},{"osd":3,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.47099999999999997}]},{"osd":4,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.48999999999999999}]},{"osd":5,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.505}]},{"osd":7,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.58299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8488,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363352,"statfs":{"total":96636764160,"available":96628072448,"internally_reserved":0,"allocated":425984,"data_stored":185860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40400000000000003}]},{"osd":2,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.38}]},{"osd":3,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.436}]},{"osd":4,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.58999999999999997}]},{"osd":5,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.874},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.74399999999999999}]},{"osd":6,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.71899999999999997}]},{"osd":7,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.65400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9592,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362248,"statfs":{"total":96636764160,"available":96626941952,"internally_reserved":0,"allocated":1032192,"data_stored":782718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.32100000000000001}]},{"osd":2,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.35299999999999998}]},{"osd":3,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.377}]},{"osd":4,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47999999999999998}]},{"osd":5,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.432},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51600000000000001}]},{"osd":6,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46400000000000002}]},{"osd":7,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.57499999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8372,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363468,"statfs":{"total":96636764160,"available":96628191232,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.36499999999999999}]},{"osd":1,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.316},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.41499999999999998}]},{"osd":3,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.39300000000000002}]},{"osd":4,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.71899999999999997}]},{"osd":5,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.628},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70099999999999996}]},{"osd":6,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.65700000000000003}]},{"osd":7,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.68700000000000006}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8500,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363340,"statfs":{"total":96636764160,"available":96628060160,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.318,"15min":0.318},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.39700000000000002}]},{"osd":1,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.41199999999999998}]},{"osd":2,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":0.437},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":0.48799999999999999}]},{"osd":4,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":8.2889999999999997,"5min":8.2889999999999997,"15min":8.2889999999999997},"last":0.57799999999999996}]},{"osd":5,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":0.627}]},{"osd":6,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":7.0899999999999999,"5min":7.0899999999999999,"15min":7.0899999999999999},"last":0.67600000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8628,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363212,"statfs":{"total":96636764160,"available":96627929088,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.495},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.47699999999999998}]},{"osd":1,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.53200000000000003}]},{"osd":2,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.54400000000000004}]},{"osd":5,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.436},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.55900000000000005}]},{"osd":6,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.61399999999999999}]},{"osd":7,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57299999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8436,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363404,"statfs":{"total":96636764160,"available":96628125696,"internally_reserved":0,"allocated":438272,"data_stored":192350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.69599999999999995}]},{"osd":1,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.627},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.50600000000000001}]},{"osd":2,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.4020000000000001,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.53800000000000003}]},{"osd":3,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":0.46600000000000003}]},{"osd":4,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.0739999999999998,"5min":3.0739999999999998,"15min":3.0739999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.65200000000000002}]},{"osd":6,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":3.7679999999999998,"5min":3.7679999999999998,"15min":3.7679999999999998},"last":0.49099999999999999}]},{"osd":7,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:00:20.635 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:00:20.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d99f8e700 1 Processor -- start 2023-11-28T13:00:20.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d99f8e700 1 -- start start 2023-11-28T13:00:20.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d99f8e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d9415c800 0x7f8d94160e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:20.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d99f8e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f8d940cd790 0x7f8d940cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:20.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d99f8e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9415cbd0 con 0x7f8d9406bfd0 2023-11-28T13:00:20.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d99f8e700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d940ce150 con 0x7f8d940cd790 2023-11-28T13:00:20.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d99f8e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9406af30 con 0x7f8d9415c800 2023-11-28T13:00:20.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d92ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8d9406bfd0 0x7f8d9406a6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36108/0 (socket says 172.21.15.111:36108) 2023-11-28T13:00:20.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d92ffd700 1 -- 172.21.15.111:0/3140483260 learned_addr learned my addr 172.21.15.111:0/3140483260 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:20.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d937fe700 1 --2- 172.21.15.111:0/3140483260 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d9415c800 0x7f8d94160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:20.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d93fff700 1 --2- 172.21.15.111:0/3140483260 >> v2:172.21.15.149:3300/0 conn(0x7f8d940cd790 0x7f8d940cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:20.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287245996 0 0) 0x7f8d9415cbd0 con 0x7f8d9406bfd0 2023-11-28T13:00:20.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d7c0035b0 con 0x7f8d9406bfd0 2023-11-28T13:00:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4265558429 0 0) 0x7f8d7c0035b0 con 0x7f8d9406bfd0 2023-11-28T13:00:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d9415c800 msgr2=0x7f8d94160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 --2- 172.21.15.111:0/3140483260 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d9415c800 0x7f8d94160e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 >> v2:172.21.15.149:3300/0 conn(0x7f8d940cd790 msgr2=0x7f8d940cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 --2- 172.21.15.111:0/3140483260 >> v2:172.21.15.149:3300/0 conn(0x7f8d940cd790 0x7f8d940cdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9406b210 con 0x7f8d9406bfd0 2023-11-28T13:00:20.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f8d88001670 con 0x7f8d9406bfd0 2023-11-28T13:00:20.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 >> v1:172.21.15.111:6789/0 conn(0x7f8d9406bfd0 legacy=0x7f8d9406a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:20.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 --2- 172.21.15.111:0/3140483260 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d7c003a00 0x7f8d7c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:20.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 --2- 172.21.15.111:0/3140483260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c004430 0x7f8d7c0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:20.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 --2- 172.21.15.111:0/3140483260 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d9415c800 0x7f8d7c004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:20.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d940ce150 con 0x7f8d7c004430 2023-11-28T13:00:20.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9406af30 con 0x7f8d7c003a00 2023-11-28T13:00:20.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c00ba10 con 0x7f8d9415c800 2023-11-28T13:00:20.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d93fff700 1 --2- 172.21.15.111:0/3140483260 >> v2:172.21.15.149:3300/0 conn(0x7f8d940cd790 0x7f8d940cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:20.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.768+0000 7f8d92ffd700 1 --2- 172.21.15.111:0/3140483260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c004430 0x7f8d7c0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d93fff700 1 --2- 172.21.15.111:0/3140483260 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d9415c800 0x7f8d7c004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d937fe700 1 --2- 172.21.15.111:0/3140483260 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d7c003a00 0x7f8d7c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d92ffd700 1 -- 172.21.15.111:0/3140483260 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d9415c800 msgr2=0x7f8d7c004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d92ffd700 1 --2- 172.21.15.111:0/3140483260 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d9415c800 0x7f8d7c004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d92ffd700 1 -- 172.21.15.111:0/3140483260 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d7c003a00 msgr2=0x7f8d7c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:20.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d92ffd700 1 --2- 172.21.15.111:0/3140483260 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d7c003a00 0x7f8d7c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:20.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d92ffd700 1 -- 172.21.15.111:0/3140483260 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c0035b0 con 0x7f8d7c004430 2023-11-28T13:00:20.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d93fff700 1 --2- 172.21.15.111:0/3140483260 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d9415c800 0x7f8d7c004fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:20.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d92ffd700 1 --2- 172.21.15.111:0/3140483260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c004430 0x7f8d7c0048f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800c320 tx=0x7f8d7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73aa78d72b1eb209 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:20.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8800ce60 con 0x7f8d7c004430 2023-11-28T13:00:20.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d88005960 con 0x7f8d7c004430 2023-11-28T13:00:20.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d927fc700 1 -- 172.21.15.111:0/3140483260 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88005cf0 con 0x7f8d7c004430 2023-11-28T13:00:20.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d99f8e700 1 -- 172.21.15.111:0/3140483260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c004430 msgr2=0x7f8d7c0048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:20.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d99f8e700 1 --2- 172.21.15.111:0/3140483260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d7c004430 0x7f8d7c0048f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800c320 tx=0x7f8d7c005880 comp rx=0 tx=0).stop 2023-11-28T13:00:20.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d93fff700 1 -- 172.21.15.111:0/3140483260 reap_dead start 2023-11-28T13:00:20.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d99f8e700 1 -- 172.21.15.111:0/3140483260 shutdown_connections 2023-11-28T13:00:20.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.769+0000 7f8d99f8e700 1 -- 172.21.15.111:0/3140483260 >> 172.21.15.111:0/3140483260 conn(0x7f8d9406db10 msgr2=0x7f8d9406df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:20.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.770+0000 7f8d99f8e700 1 -- 172.21.15.111:0/3140483260 shutdown_connections 2023-11-28T13:00:20.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.770+0000 7f8d99f8e700 1 -- 172.21.15.111:0/3140483260 wait complete. 2023-11-28T13:00:20.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.771+0000 7f8d99f8e700 1 Processor -- start 2023-11-28T13:00:20.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.771+0000 7f8d99f8e700 1 -- start start 2023-11-28T13:00:20.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.772+0000 7f8d99f8e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d7c004430 0x7f8d940ae530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:20.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.772+0000 7f8d99f8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d94154b10 0x7f8d94154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:20.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.772+0000 7f8d99f8e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d94156ab0 0x7f8d94156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:20.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.772+0000 7f8d99f8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94157460 con 0x7f8d94154b10 2023-11-28T13:00:20.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.772+0000 7f8d99f8e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d940b1810 con 0x7f8d7c004430 2023-11-28T13:00:20.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.772+0000 7f8d99f8e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d940b1af0 con 0x7f8d94156ab0 2023-11-28T13:00:20.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.772+0000 7f8d92ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d94154b10 0x7f8d94154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:20.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.772+0000 7f8d92ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d94154b10 0x7f8d94154580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35344/0 (socket says 172.21.15.111:35344) 2023-11-28T13:00:20.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.772+0000 7f8d92ffd700 1 -- 172.21.15.111:0/448286712 learned_addr learned my addr 172.21.15.111:0/448286712 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:20.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d937fe700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d7c004430 0x7f8d940ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:20.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d93fff700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d94156ab0 0x7f8d94156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:20.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d92ffd700 1 -- 172.21.15.111:0/448286712 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d94156ab0 msgr2=0x7f8d94156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:20.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d92ffd700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d94156ab0 0x7f8d94156f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:20.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d92ffd700 1 -- 172.21.15.111:0/448286712 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d7c004430 msgr2=0x7f8d940ae530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:20.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d92ffd700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d7c004430 0x7f8d940ae530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:20.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d92ffd700 1 -- 172.21.15.111:0/448286712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9406b7d0 con 0x7f8d94154b10 2023-11-28T13:00:20.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d93fff700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d94156ab0 0x7f8d94156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:20.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d92ffd700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d94154b10 0x7f8d94154580 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c005760 tx=0x7f8d88001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:20.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d7bfff700 1 -- 172.21.15.111:0/448286712 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88001680 con 0x7f8d94154b10 2023-11-28T13:00:20.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d7bfff700 1 -- 172.21.15.111:0/448286712 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d88004a80 con 0x7f8d94154b10 2023-11-28T13:00:20.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d7bfff700 1 -- 172.21.15.111:0/448286712 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88004e30 con 0x7f8d94154b10 2023-11-28T13:00:20.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d99f8e700 1 -- 172.21.15.111:0/448286712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9406b4f0 con 0x7f8d94154b10 2023-11-28T13:00:20.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.773+0000 7f8d99f8e700 1 -- 172.21.15.111:0/448286712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9406bab0 con 0x7f8d94154b10 2023-11-28T13:00:20.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.774+0000 7f8d99f8e700 1 -- 172.21.15.111:0/448286712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d940c3b00 con 0x7f8d94154b10 2023-11-28T13:00:20.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.775+0000 7f8d7bfff700 1 -- 172.21.15.111:0/448286712 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8d88024070 con 0x7f8d94154b10 2023-11-28T13:00:20.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.775+0000 7f8d7bfff700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8d800682f0 0x7f8d8006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:20.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.775+0000 7f8d7bfff700 1 -- 172.21.15.111:0/448286712 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8d88087000 con 0x7f8d94154b10 2023-11-28T13:00:20.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.775+0000 7f8d937fe700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8d800682f0 0x7f8d8006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:20.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.775+0000 7f8d937fe700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8d800682f0 0x7f8d8006a7a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8d940c7680 tx=0x7f8d84008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:20.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.780+0000 7f8d7bfff700 1 -- 172.21.15.111:0/448286712 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8d880537e0 con 0x7f8d94154b10 2023-11-28T13:00:20.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.958+0000 7f8d99f8e700 1 -- 172.21.15.111:0/448286712 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d94157460 con 0x7f8d800682f0 2023-11-28T13:00:20.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.966+0000 7f8d7bfff700 1 -- 172.21.15.111:0/448286712 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311138 (secure 0 0 0) 0x7f8d84004180 con 0x7f8d800682f0 2023-11-28T13:00:20.974 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:20.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.969+0000 7f8d99f8e700 1 -- 172.21.15.111:0/448286712 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8d800682f0 msgr2=0x7f8d8006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:20.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.969+0000 7f8d99f8e700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8d800682f0 0x7f8d8006a7a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8d940c7680 tx=0x7f8d84008040 comp rx=0 tx=0).stop 2023-11-28T13:00:20.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.969+0000 7f8d99f8e700 1 -- 172.21.15.111:0/448286712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d94154b10 msgr2=0x7f8d94154580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:20.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.969+0000 7f8d99f8e700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d94154b10 0x7f8d94154580 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c005760 tx=0x7f8d88001a00 comp rx=0 tx=0).stop 2023-11-28T13:00:20.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.970+0000 7f8d99f8e700 1 -- 172.21.15.111:0/448286712 shutdown_connections 2023-11-28T13:00:20.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.970+0000 7f8d99f8e700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d94156ab0 0x7f8d94156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:20.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.970+0000 7f8d99f8e700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8d800682f0 0x7f8d8006a7a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:20.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.970+0000 7f8d99f8e700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d7c004430 0x7f8d940ae530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:20.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.970+0000 7f8d99f8e700 1 --2- 172.21.15.111:0/448286712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d94154b10 0x7f8d94154580 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:20.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.970+0000 7f8d99f8e700 1 -- 172.21.15.111:0/448286712 >> 172.21.15.111:0/448286712 conn(0x7f8d9406db10 msgr2=0x7f8d940c81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:20.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.970+0000 7f8d99f8e700 1 -- 172.21.15.111:0/448286712 shutdown_connections 2023-11-28T13:00:20.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:20.970+0000 7f8d99f8e700 1 -- 172.21.15.111:0/448286712 wait complete. 2023-11-28T13:00:20.981 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:00:20.983 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2023-11-28T13:00:20.352437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":71816,"kb_used_data":4936,"kb_used_omap":0,"kb_used_meta":66816,"kb_avail":754902904,"statfs":{"total":773094113280,"available":773020573696,"internally_reserved":0,"allocated":5054464,"data_stored":2895084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":68419584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.338268"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.027495+0000","last_change":"2023-11-28T13:00:12.031931+0000","last_active":"2023-11-28T13:00:13.027495+0000","last_peered":"2023-11-28T13:00:13.027495+0000","last_clean":"2023-11-28T13:00:13.027495+0000","last_became_active":"2023-11-28T13:00:12.031682+0000","last_became_peered":"2023-11-28T13:00:12.031682+0000","last_unstale":"2023-11-28T13:00:13.027495+0000","last_undegraded":"2023-11-28T13:00:13.027495+0000","last_fullsized":"2023-11-28T13:00:13.027495+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:34:00.251384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741082+0000","last_change":"2023-11-28T13:00:12.036418+0000","last_active":"2023-11-28T13:00:13.741082+0000","last_peered":"2023-11-28T13:00:13.741082+0000","last_clean":"2023-11-28T13:00:13.741082+0000","last_became_active":"2023-11-28T13:00:12.036174+0000","last_became_peered":"2023-11-28T13:00:12.036174+0000","last_unstale":"2023-11-28T13:00:13.741082+0000","last_undegraded":"2023-11-28T13:00:13.741082+0000","last_fullsized":"2023-11-28T13:00:13.741082+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:18:11.225186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025476+0000","last_change":"2023-11-28T13:00:12.026871+0000","last_active":"2023-11-28T13:00:13.025476+0000","last_peered":"2023-11-28T13:00:13.025476+0000","last_clean":"2023-11-28T13:00:13.025476+0000","last_became_active":"2023-11-28T13:00:12.026590+0000","last_became_peered":"2023-11-28T13:00:12.026590+0000","last_unstale":"2023-11-28T13:00:13.025476+0000","last_undegraded":"2023-11-28T13:00:13.025476+0000","last_fullsized":"2023-11-28T13:00:13.025476+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:13:11.366568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741455+0000","last_change":"2023-11-28T13:00:12.033249+0000","last_active":"2023-11-28T13:00:13.741455+0000","last_peered":"2023-11-28T13:00:13.741455+0000","last_clean":"2023-11-28T13:00:13.741455+0000","last_became_active":"2023-11-28T13:00:12.032981+0000","last_became_peered":"2023-11-28T13:00:12.032981+0000","last_unstale":"2023-11-28T13:00:13.741455+0000","last_undegraded":"2023-11-28T13:00:13.741455+0000","last_fullsized":"2023-11-28T13:00:13.741455+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:48:17.540217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.740823+0000","last_change":"2023-11-28T13:00:12.033765+0000","last_active":"2023-11-28T13:00:13.740823+0000","last_peered":"2023-11-28T13:00:13.740823+0000","last_clean":"2023-11-28T13:00:13.740823+0000","last_became_active":"2023-11-28T13:00:12.033529+0000","last_became_peered":"2023-11-28T13:00:12.033529+0000","last_unstale":"2023-11-28T13:00:13.740823+0000","last_undegraded":"2023-11-28T13:00:13.740823+0000","last_fullsized":"2023-11-28T13:00:13.740823+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:35:09.447520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.317420+0000","last_change":"2023-11-28T13:00:12.030020+0000","last_active":"2023-11-28T13:00:13.317420+0000","last_peered":"2023-11-28T13:00:13.317420+0000","last_clean":"2023-11-28T13:00:13.317420+0000","last_became_active":"2023-11-28T13:00:12.029755+0000","last_became_peered":"2023-11-28T13:00:12.029755+0000","last_unstale":"2023-11-28T13:00:13.317420+0000","last_undegraded":"2023-11-28T13:00:13.317420+0000","last_fullsized":"2023-11-28T13:00:13.317420+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:36.759595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.027190+0000","last_change":"2023-11-28T13:00:12.040315+0000","last_active":"2023-11-28T13:00:13.027190+0000","last_peered":"2023-11-28T13:00:13.027190+0000","last_clean":"2023-11-28T13:00:13.027190+0000","last_became_active":"2023-11-28T13:00:12.040049+0000","last_became_peered":"2023-11-28T13:00:12.040049+0000","last_unstale":"2023-11-28T13:00:13.027190+0000","last_undegraded":"2023-11-28T13:00:13.027190+0000","last_fullsized":"2023-11-28T13:00:13.027190+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:13:26.169150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.740997+0000","last_change":"2023-11-28T13:00:12.036520+0000","last_active":"2023-11-28T13:00:13.740997+0000","last_peered":"2023-11-28T13:00:13.740997+0000","last_clean":"2023-11-28T13:00:13.740997+0000","last_became_active":"2023-11-28T13:00:12.036259+0000","last_became_peered":"2023-11-28T13:00:12.036259+0000","last_unstale":"2023-11-28T13:00:13.740997+0000","last_undegraded":"2023-11-28T13:00:13.740997+0000","last_fullsized":"2023-11-28T13:00:13.740997+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:45:48.228496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741612+0000","last_change":"2023-11-28T13:00:12.040217+0000","last_active":"2023-11-28T13:00:13.741612+0000","last_peered":"2023-11-28T13:00:13.741612+0000","last_clean":"2023-11-28T13:00:13.741612+0000","last_became_active":"2023-11-28T13:00:12.039950+0000","last_became_peered":"2023-11-28T13:00:12.039950+0000","last_unstale":"2023-11-28T13:00:13.741612+0000","last_undegraded":"2023-11-28T13:00:13.741612+0000","last_fullsized":"2023-11-28T13:00:13.741612+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:50:52.290831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741681+0000","last_change":"2023-11-28T13:00:12.039753+0000","last_active":"2023-11-28T13:00:13.741681+0000","last_peered":"2023-11-28T13:00:13.741681+0000","last_clean":"2023-11-28T13:00:13.741681+0000","last_became_active":"2023-11-28T13:00:12.039260+0000","last_became_peered":"2023-11-28T13:00:12.039260+0000","last_unstale":"2023-11-28T13:00:13.741681+0000","last_undegraded":"2023-11-28T13:00:13.741681+0000","last_fullsized":"2023-11-28T13:00:13.741681+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:19:13.419243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741470+0000","last_change":"2023-11-28T13:00:12.039652+0000","last_active":"2023-11-28T13:00:13.741470+0000","last_peered":"2023-11-28T13:00:13.741470+0000","last_clean":"2023-11-28T13:00:13.741470+0000","last_became_active":"2023-11-28T13:00:12.039440+0000","last_became_peered":"2023-11-28T13:00:12.039440+0000","last_unstale":"2023-11-28T13:00:13.741470+0000","last_undegraded":"2023-11-28T13:00:13.741470+0000","last_fullsized":"2023-11-28T13:00:13.741470+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:03:05.309242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741016+0000","last_change":"2023-11-28T13:00:12.028423+0000","last_active":"2023-11-28T13:00:13.741016+0000","last_peered":"2023-11-28T13:00:13.741016+0000","last_clean":"2023-11-28T13:00:13.741016+0000","last_became_active":"2023-11-28T13:00:12.028078+0000","last_became_peered":"2023-11-28T13:00:12.028078+0000","last_unstale":"2023-11-28T13:00:13.741016+0000","last_undegraded":"2023-11-28T13:00:13.741016+0000","last_fullsized":"2023-11-28T13:00:13.741016+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:34:00.251384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.740992+0000","last_change":"2023-11-28T13:00:12.038827+0000","last_active":"2023-11-28T13:00:13.740992+0000","last_peered":"2023-11-28T13:00:13.740992+0000","last_clean":"2023-11-28T13:00:13.740992+0000","last_became_active":"2023-11-28T13:00:12.037649+0000","last_became_peered":"2023-11-28T13:00:12.037649+0000","last_unstale":"2023-11-28T13:00:13.740992+0000","last_undegraded":"2023-11-28T13:00:13.740992+0000","last_fullsized":"2023-11-28T13:00:13.740992+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:57:54.795580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741638+0000","last_change":"2023-11-28T13:00:12.033832+0000","last_active":"2023-11-28T13:00:13.741638+0000","last_peered":"2023-11-28T13:00:13.741638+0000","last_clean":"2023-11-28T13:00:13.741638+0000","last_became_active":"2023-11-28T13:00:12.033351+0000","last_became_peered":"2023-11-28T13:00:12.033351+0000","last_unstale":"2023-11-28T13:00:13.741638+0000","last_undegraded":"2023-11-28T13:00:13.741638+0000","last_fullsized":"2023-11-28T13:00:13.741638+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:59:50.429502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.317445+0000","last_change":"2023-11-28T13:00:12.021159+0000","last_active":"2023-11-28T13:00:13.317445+0000","last_peered":"2023-11-28T13:00:13.317445+0000","last_clean":"2023-11-28T13:00:13.317445+0000","last_became_active":"2023-11-28T13:00:12.020912+0000","last_became_peered":"2023-11-28T13:00:12.020912+0000","last_unstale":"2023-11-28T13:00:13.317445+0000","last_undegraded":"2023-11-28T13:00:13.317445+0000","last_fullsized":"2023-11-28T13:00:13.317445+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:36:12.822043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.024727+0000","last_change":"2023-11-28T13:00:12.038809+0000","last_active":"2023-11-28T13:00:13.024727+0000","last_peered":"2023-11-28T13:00:13.024727+0000","last_clean":"2023-11-28T13:00:13.024727+0000","last_became_active":"2023-11-28T13:00:12.038450+0000","last_became_peered":"2023-11-28T13:00:12.038450+0000","last_unstale":"2023-11-28T13:00:13.024727+0000","last_undegraded":"2023-11-28T13:00:13.024727+0000","last_fullsized":"2023-11-28T13:00:13.024727+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:15.730073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025772+0000","last_change":"2023-11-28T13:00:12.034077+0000","last_active":"2023-11-28T13:00:13.025772+0000","last_peered":"2023-11-28T13:00:13.025772+0000","last_clean":"2023-11-28T13:00:13.025772+0000","last_became_active":"2023-11-28T13:00:12.033848+0000","last_became_peered":"2023-11-28T13:00:12.033848+0000","last_unstale":"2023-11-28T13:00:13.025772+0000","last_undegraded":"2023-11-28T13:00:13.025772+0000","last_fullsized":"2023-11-28T13:00:13.025772+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:34:06.384098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.024518+0000","last_change":"2023-11-28T13:00:12.035918+0000","last_active":"2023-11-28T13:00:13.024518+0000","last_peered":"2023-11-28T13:00:13.024518+0000","last_clean":"2023-11-28T13:00:13.024518+0000","last_became_active":"2023-11-28T13:00:12.035630+0000","last_became_peered":"2023-11-28T13:00:12.035630+0000","last_unstale":"2023-11-28T13:00:13.024518+0000","last_undegraded":"2023-11-28T13:00:13.024518+0000","last_fullsized":"2023-11-28T13:00:13.024518+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:13:08.205888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025054+0000","last_change":"2023-11-28T13:00:12.032977+0000","last_active":"2023-11-28T13:00:13.025054+0000","last_peered":"2023-11-28T13:00:13.025054+0000","last_clean":"2023-11-28T13:00:13.025054+0000","last_became_active":"2023-11-28T13:00:12.032767+0000","last_became_peered":"2023-11-28T13:00:12.032767+0000","last_unstale":"2023-11-28T13:00:13.025054+0000","last_undegraded":"2023-11-28T13:00:13.025054+0000","last_fullsized":"2023-11-28T13:00:13.025054+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:22:31.541512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741141+0000","last_change":"2023-11-28T13:00:12.038537+0000","last_active":"2023-11-28T13:00:13.741141+0000","last_peered":"2023-11-28T13:00:13.741141+0000","last_clean":"2023-11-28T13:00:13.741141+0000","last_became_active":"2023-11-28T13:00:12.038298+0000","last_became_peered":"2023-11-28T13:00:12.038298+0000","last_unstale":"2023-11-28T13:00:13.741141+0000","last_undegraded":"2023-11-28T13:00:13.741141+0000","last_fullsized":"2023-11-28T13:00:13.741141+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:37:08.846854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.317534+0000","last_change":"2023-11-28T13:00:12.026101+0000","last_active":"2023-11-28T13:00:13.317534+0000","last_peered":"2023-11-28T13:00:13.317534+0000","last_clean":"2023-11-28T13:00:13.317534+0000","last_became_active":"2023-11-28T13:00:12.025844+0000","last_became_peered":"2023-11-28T13:00:12.025844+0000","last_unstale":"2023-11-28T13:00:13.317534+0000","last_undegraded":"2023-11-28T13:00:13.317534+0000","last_fullsized":"2023-11-28T13:00:13.317534+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:48:32.551718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741439+0000","last_change":"2023-11-28T13:00:12.033340+0000","last_active":"2023-11-28T13:00:13.741439+0000","last_peered":"2023-11-28T13:00:13.741439+0000","last_clean":"2023-11-28T13:00:13.741439+0000","last_became_active":"2023-11-28T13:00:12.033093+0000","last_became_peered":"2023-11-28T13:00:12.033093+0000","last_unstale":"2023-11-28T13:00:13.741439+0000","last_undegraded":"2023-11-28T13:00:13.741439+0000","last_fullsized":"2023-11-28T13:00:13.741439+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:06.157368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025107+0000","last_change":"2023-11-28T13:00:12.029931+0000","last_active":"2023-11-28T13:00:13.025107+0000","last_peered":"2023-11-28T13:00:13.025107+0000","last_clean":"2023-11-28T13:00:13.025107+0000","last_became_active":"2023-11-28T13:00:12.029673+0000","last_became_peered":"2023-11-28T13:00:12.029673+0000","last_unstale":"2023-11-28T13:00:13.025107+0000","last_undegraded":"2023-11-28T13:00:13.025107+0000","last_fullsized":"2023-11-28T13:00:13.025107+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:16.333892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.024832+0000","last_change":"2023-11-28T13:00:12.029180+0000","last_active":"2023-11-28T13:00:13.024832+0000","last_peered":"2023-11-28T13:00:13.024832+0000","last_clean":"2023-11-28T13:00:13.024832+0000","last_became_active":"2023-11-28T13:00:12.028945+0000","last_became_peered":"2023-11-28T13:00:12.028945+0000","last_unstale":"2023-11-28T13:00:13.024832+0000","last_undegraded":"2023-11-28T13:00:13.024832+0000","last_fullsized":"2023-11-28T13:00:13.024832+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:17:04.749683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025559+0000","last_change":"2023-11-28T13:00:12.031717+0000","last_active":"2023-11-28T13:00:13.025559+0000","last_peered":"2023-11-28T13:00:13.025559+0000","last_clean":"2023-11-28T13:00:13.025559+0000","last_became_active":"2023-11-28T13:00:12.031323+0000","last_became_peered":"2023-11-28T13:00:12.031323+0000","last_unstale":"2023-11-28T13:00:13.025559+0000","last_undegraded":"2023-11-28T13:00:13.025559+0000","last_fullsized":"2023-11-28T13:00:13.025559+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:37:08.846854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741094+0000","last_change":"2023-11-28T13:00:12.041692+0000","last_active":"2023-11-28T13:00:13.741094+0000","last_peered":"2023-11-28T13:00:13.741094+0000","last_clean":"2023-11-28T13:00:13.741094+0000","last_became_active":"2023-11-28T13:00:12.041317+0000","last_became_peered":"2023-11-28T13:00:12.041317+0000","last_unstale":"2023-11-28T13:00:13.741094+0000","last_undegraded":"2023-11-28T13:00:13.741094+0000","last_fullsized":"2023-11-28T13:00:13.741094+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:09:51.905437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025563+0000","last_change":"2023-11-28T13:00:12.028094+0000","last_active":"2023-11-28T13:00:13.025563+0000","last_peered":"2023-11-28T13:00:13.025563+0000","last_clean":"2023-11-28T13:00:13.025563+0000","last_became_active":"2023-11-28T13:00:12.027834+0000","last_became_peered":"2023-11-28T13:00:12.027834+0000","last_unstale":"2023-11-28T13:00:13.025563+0000","last_undegraded":"2023-11-28T13:00:13.025563+0000","last_fullsized":"2023-11-28T13:00:13.025563+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:21:35.116481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.740842+0000","last_change":"2023-11-28T13:00:12.041250+0000","last_active":"2023-11-28T13:00:13.740842+0000","last_peered":"2023-11-28T13:00:13.740842+0000","last_clean":"2023-11-28T13:00:13.740842+0000","last_became_active":"2023-11-28T13:00:12.041066+0000","last_became_peered":"2023-11-28T13:00:12.041066+0000","last_unstale":"2023-11-28T13:00:13.740842+0000","last_undegraded":"2023-11-28T13:00:13.740842+0000","last_fullsized":"2023-11-28T13:00:13.740842+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:06.157368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025059+0000","last_change":"2023-11-28T13:00:12.033833+0000","last_active":"2023-11-28T13:00:13.025059+0000","last_peered":"2023-11-28T13:00:13.025059+0000","last_clean":"2023-11-28T13:00:13.025059+0000","last_became_active":"2023-11-28T13:00:12.033603+0000","last_became_peered":"2023-11-28T13:00:12.033603+0000","last_unstale":"2023-11-28T13:00:13.025059+0000","last_undegraded":"2023-11-28T13:00:13.025059+0000","last_fullsized":"2023-11-28T13:00:13.025059+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:13.358281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025228+0000","last_change":"2023-11-28T13:00:12.030408+0000","last_active":"2023-11-28T13:00:13.025228+0000","last_peered":"2023-11-28T13:00:13.025228+0000","last_clean":"2023-11-28T13:00:13.025228+0000","last_became_active":"2023-11-28T13:00:12.030051+0000","last_became_peered":"2023-11-28T13:00:12.030051+0000","last_unstale":"2023-11-28T13:00:13.025228+0000","last_undegraded":"2023-11-28T13:00:13.025228+0000","last_fullsized":"2023-11-28T13:00:13.025228+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:10:08.035683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.317314+0000","last_change":"2023-11-28T13:00:12.029241+0000","last_active":"2023-11-28T13:00:13.317314+0000","last_peered":"2023-11-28T13:00:13.317314+0000","last_clean":"2023-11-28T13:00:13.317314+0000","last_became_active":"2023-11-28T13:00:12.028979+0000","last_became_peered":"2023-11-28T13:00:12.028979+0000","last_unstale":"2023-11-28T13:00:13.317314+0000","last_undegraded":"2023-11-28T13:00:13.317314+0000","last_fullsized":"2023-11-28T13:00:13.317314+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:59:15.590926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.740809+0000","last_change":"2023-11-28T13:00:12.040717+0000","last_active":"2023-11-28T13:00:13.740809+0000","last_peered":"2023-11-28T13:00:13.740809+0000","last_clean":"2023-11-28T13:00:13.740809+0000","last_became_active":"2023-11-28T13:00:12.034068+0000","last_became_peered":"2023-11-28T13:00:12.034068+0000","last_unstale":"2023-11-28T13:00:13.740809+0000","last_undegraded":"2023-11-28T13:00:13.740809+0000","last_fullsized":"2023-11-28T13:00:13.740809+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:42:30.828919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.742512+0000","last_change":"2023-11-28T13:00:12.030601+0000","last_active":"2023-11-28T13:00:13.742512+0000","last_peered":"2023-11-28T13:00:13.742512+0000","last_clean":"2023-11-28T13:00:13.742512+0000","last_became_active":"2023-11-28T13:00:12.030319+0000","last_became_peered":"2023-11-28T13:00:12.030319+0000","last_unstale":"2023-11-28T13:00:13.742512+0000","last_undegraded":"2023-11-28T13:00:13.742512+0000","last_fullsized":"2023-11-28T13:00:13.742512+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:48.275386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"40'4","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.038797+0000","last_change":"2023-11-28T13:00:11.632883+0000","last_active":"2023-11-28T13:00:12.038797+0000","last_peered":"2023-11-28T13:00:12.038797+0000","last_clean":"2023-11-28T13:00:12.038797+0000","last_became_active":"2023-11-28T13:00:11.632050+0000","last_became_peered":"2023-11-28T13:00:11.632050+0000","last_unstale":"2023-11-28T13:00:12.038797+0000","last_undegraded":"2023-11-28T13:00:12.038797+0000","last_fullsized":"2023-11-28T13:00:12.038797+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:47.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025374+0000","last_change":"2023-11-28T13:00:12.023746+0000","last_active":"2023-11-28T13:00:13.025374+0000","last_peered":"2023-11-28T13:00:13.025374+0000","last_clean":"2023-11-28T13:00:13.025374+0000","last_became_active":"2023-11-28T13:00:12.023478+0000","last_became_peered":"2023-11-28T13:00:12.023478+0000","last_unstale":"2023-11-28T13:00:13.025374+0000","last_undegraded":"2023-11-28T13:00:13.025374+0000","last_fullsized":"2023-11-28T13:00:13.025374+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:09:51.905437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"40'1","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.038583+0000","last_change":"2023-11-28T13:00:11.638654+0000","last_active":"2023-11-28T13:00:12.038583+0000","last_peered":"2023-11-28T13:00:12.038583+0000","last_clean":"2023-11-28T13:00:12.038583+0000","last_became_active":"2023-11-28T13:00:11.638393+0000","last_became_peered":"2023-11-28T13:00:11.638393+0000","last_unstale":"2023-11-28T13:00:12.038583+0000","last_undegraded":"2023-11-28T13:00:12.038583+0000","last_fullsized":"2023-11-28T13:00:12.038583+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:48:31.549366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.740911+0000","last_change":"2023-11-28T13:00:12.033008+0000","last_active":"2023-11-28T13:00:13.740911+0000","last_peered":"2023-11-28T13:00:13.740911+0000","last_clean":"2023-11-28T13:00:13.740911+0000","last_became_active":"2023-11-28T13:00:12.032693+0000","last_became_peered":"2023-11-28T13:00:12.032693+0000","last_unstale":"2023-11-28T13:00:13.740911+0000","last_undegraded":"2023-11-28T13:00:13.740911+0000","last_fullsized":"2023-11-28T13:00:13.740911+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:13:26.169150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"40'2","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.045425+0000","last_change":"2023-11-28T13:00:11.636925+0000","last_active":"2023-11-28T13:00:12.045425+0000","last_peered":"2023-11-28T13:00:12.045425+0000","last_clean":"2023-11-28T13:00:12.045425+0000","last_became_active":"2023-11-28T13:00:11.636562+0000","last_became_peered":"2023-11-28T13:00:11.636562+0000","last_unstale":"2023-11-28T13:00:12.045425+0000","last_undegraded":"2023-11-28T13:00:12.045425+0000","last_fullsized":"2023-11-28T13:00:12.045425+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:57:53.793229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.742533+0000","last_change":"2023-11-28T13:00:12.039129+0000","last_active":"2023-11-28T13:00:13.742533+0000","last_peered":"2023-11-28T13:00:13.742533+0000","last_clean":"2023-11-28T13:00:13.742533+0000","last_became_active":"2023-11-28T13:00:12.038841+0000","last_became_peered":"2023-11-28T13:00:12.038841+0000","last_unstale":"2023-11-28T13:00:13.742533+0000","last_undegraded":"2023-11-28T13:00:13.742533+0000","last_fullsized":"2023-11-28T13:00:13.742533+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:16.333892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"40'2","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.044848+0000","last_change":"2023-11-28T13:00:11.642375+0000","last_active":"2023-11-28T13:00:12.044848+0000","last_peered":"2023-11-28T13:00:12.044848+0000","last_clean":"2023-11-28T13:00:12.044848+0000","last_became_active":"2023-11-28T13:00:11.642100+0000","last_became_peered":"2023-11-28T13:00:11.642100+0000","last_unstale":"2023-11-28T13:00:12.044848+0000","last_undegraded":"2023-11-28T13:00:12.044848+0000","last_fullsized":"2023-11-28T13:00:12.044848+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025287+0000","last_change":"2023-11-28T13:00:12.034725+0000","last_active":"2023-11-28T13:00:13.025287+0000","last_peered":"2023-11-28T13:00:13.025287+0000","last_clean":"2023-11-28T13:00:13.025287+0000","last_became_active":"2023-11-28T13:00:12.034465+0000","last_became_peered":"2023-11-28T13:00:12.034465+0000","last_unstale":"2023-11-28T13:00:13.025287+0000","last_undegraded":"2023-11-28T13:00:13.025287+0000","last_fullsized":"2023-11-28T13:00:13.025287+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:54:05.104616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"40'1","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.040178+0000","last_change":"2023-11-28T13:00:11.642628+0000","last_active":"2023-11-28T13:00:12.040178+0000","last_peered":"2023-11-28T13:00:12.040178+0000","last_clean":"2023-11-28T13:00:12.040178+0000","last_became_active":"2023-11-28T13:00:11.642186+0000","last_became_peered":"2023-11-28T13:00:11.642186+0000","last_unstale":"2023-11-28T13:00:12.040178+0000","last_undegraded":"2023-11-28T13:00:12.040178+0000","last_fullsized":"2023-11-28T13:00:12.040178+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:02:41.353263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741405+0000","last_change":"2023-11-28T13:00:12.032656+0000","last_active":"2023-11-28T13:00:13.741405+0000","last_peered":"2023-11-28T13:00:13.741405+0000","last_clean":"2023-11-28T13:00:13.741405+0000","last_became_active":"2023-11-28T13:00:12.032020+0000","last_became_peered":"2023-11-28T13:00:12.032020+0000","last_unstale":"2023-11-28T13:00:13.741405+0000","last_undegraded":"2023-11-28T13:00:13.741405+0000","last_fullsized":"2023-11-28T13:00:13.741405+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:00:32.580551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"40'1","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.035861+0000","last_change":"2023-11-28T13:00:11.636467+0000","last_active":"2023-11-28T13:00:12.035861+0000","last_peered":"2023-11-28T13:00:12.035861+0000","last_clean":"2023-11-28T13:00:12.035861+0000","last_became_active":"2023-11-28T13:00:11.636189+0000","last_became_peered":"2023-11-28T13:00:11.636189+0000","last_unstale":"2023-11-28T13:00:12.035861+0000","last_undegraded":"2023-11-28T13:00:12.035861+0000","last_fullsized":"2023-11-28T13:00:12.035861+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:03:04.306890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741007+0000","last_change":"2023-11-28T13:00:12.026045+0000","last_active":"2023-11-28T13:00:13.741007+0000","last_peered":"2023-11-28T13:00:13.741007+0000","last_clean":"2023-11-28T13:00:13.741007+0000","last_became_active":"2023-11-28T13:00:12.025804+0000","last_became_peered":"2023-11-28T13:00:12.025804+0000","last_unstale":"2023-11-28T13:00:13.741007+0000","last_undegraded":"2023-11-28T13:00:13.741007+0000","last_fullsized":"2023-11-28T13:00:13.741007+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:35:09.447520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.024935+0000","last_change":"2023-11-28T13:00:11.637574+0000","last_active":"2023-11-28T13:00:12.024935+0000","last_peered":"2023-11-28T13:00:12.024935+0000","last_clean":"2023-11-28T13:00:12.024935+0000","last_became_active":"2023-11-28T13:00:11.637250+0000","last_became_peered":"2023-11-28T13:00:11.637250+0000","last_unstale":"2023-11-28T13:00:12.024935+0000","last_undegraded":"2023-11-28T13:00:12.024935+0000","last_fullsized":"2023-11-28T13:00:12.024935+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025179+0000","last_change":"2023-11-28T13:00:12.027351+0000","last_active":"2023-11-28T13:00:13.025179+0000","last_peered":"2023-11-28T13:00:13.025179+0000","last_clean":"2023-11-28T13:00:13.025179+0000","last_became_active":"2023-11-28T13:00:12.027081+0000","last_became_peered":"2023-11-28T13:00:12.027081+0000","last_unstale":"2023-11-28T13:00:13.025179+0000","last_undegraded":"2023-11-28T13:00:13.025179+0000","last_fullsized":"2023-11-28T13:00:13.025179+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:48.275386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.018386+0000","last_change":"2023-11-28T13:00:11.636334+0000","last_active":"2023-11-28T13:00:12.018386+0000","last_peered":"2023-11-28T13:00:12.018386+0000","last_clean":"2023-11-28T13:00:12.018386+0000","last_became_active":"2023-11-28T13:00:11.636096+0000","last_became_peered":"2023-11-28T13:00:11.636096+0000","last_unstale":"2023-11-28T13:00:12.018386+0000","last_undegraded":"2023-11-28T13:00:12.018386+0000","last_fullsized":"2023-11-28T13:00:12.018386+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.024756+0000","last_change":"2023-11-28T13:00:12.035839+0000","last_active":"2023-11-28T13:00:13.024756+0000","last_peered":"2023-11-28T13:00:13.024756+0000","last_clean":"2023-11-28T13:00:13.024756+0000","last_became_active":"2023-11-28T13:00:12.035548+0000","last_became_peered":"2023-11-28T13:00:12.035548+0000","last_unstale":"2023-11-28T13:00:13.024756+0000","last_undegraded":"2023-11-28T13:00:13.024756+0000","last_fullsized":"2023-11-28T13:00:13.024756+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:13:10.667564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.018544+0000","last_change":"2023-11-28T13:00:11.633296+0000","last_active":"2023-11-28T13:00:12.018544+0000","last_peered":"2023-11-28T13:00:12.018544+0000","last_clean":"2023-11-28T13:00:12.018544+0000","last_became_active":"2023-11-28T13:00:11.633036+0000","last_became_peered":"2023-11-28T13:00:11.633036+0000","last_unstale":"2023-11-28T13:00:12.018544+0000","last_undegraded":"2023-11-28T13:00:12.018544+0000","last_fullsized":"2023-11-28T13:00:12.018544+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:01:31.408692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.027569+0000","last_change":"2023-11-28T13:00:12.037382+0000","last_active":"2023-11-28T13:00:13.027569+0000","last_peered":"2023-11-28T13:00:13.027569+0000","last_clean":"2023-11-28T13:00:13.027569+0000","last_became_active":"2023-11-28T13:00:12.037089+0000","last_became_peered":"2023-11-28T13:00:12.037089+0000","last_unstale":"2023-11-28T13:00:13.027569+0000","last_undegraded":"2023-11-28T13:00:13.027569+0000","last_fullsized":"2023-11-28T13:00:13.027569+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:18:11.225186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.025429+0000","last_change":"2023-11-28T13:00:11.642196+0000","last_active":"2023-11-28T13:00:12.025429+0000","last_peered":"2023-11-28T13:00:12.025429+0000","last_clean":"2023-11-28T13:00:12.025429+0000","last_became_active":"2023-11-28T13:00:11.641729+0000","last_became_peered":"2023-11-28T13:00:11.641729+0000","last_unstale":"2023-11-28T13:00:12.025429+0000","last_undegraded":"2023-11-28T13:00:12.025429+0000","last_fullsized":"2023-11-28T13:00:12.025429+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:26.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.317659+0000","last_change":"2023-11-28T13:00:12.026718+0000","last_active":"2023-11-28T13:00:13.317659+0000","last_peered":"2023-11-28T13:00:13.317659+0000","last_clean":"2023-11-28T13:00:13.317659+0000","last_became_active":"2023-11-28T13:00:12.026431+0000","last_became_peered":"2023-11-28T13:00:12.026431+0000","last_unstale":"2023-11-28T13:00:13.317659+0000","last_undegraded":"2023-11-28T13:00:13.317659+0000","last_fullsized":"2023-11-28T13:00:13.317659+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:00:32.580551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"40'3","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.042061+0000","last_change":"2023-11-28T13:00:11.644702+0000","last_active":"2023-11-28T13:00:12.042061+0000","last_peered":"2023-11-28T13:00:12.042061+0000","last_clean":"2023-11-28T13:00:12.042061+0000","last_became_active":"2023-11-28T13:00:11.644423+0000","last_became_peered":"2023-11-28T13:00:11.644423+0000","last_unstale":"2023-11-28T13:00:12.042061+0000","last_undegraded":"2023-11-28T13:00:12.042061+0000","last_fullsized":"2023-11-28T13:00:12.042061+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.027438+0000","last_change":"2023-11-28T13:00:12.040274+0000","last_active":"2023-11-28T13:00:13.027438+0000","last_peered":"2023-11-28T13:00:13.027438+0000","last_clean":"2023-11-28T13:00:13.027438+0000","last_became_active":"2023-11-28T13:00:12.040039+0000","last_became_peered":"2023-11-28T13:00:12.040039+0000","last_unstale":"2023-11-28T13:00:13.027438+0000","last_undegraded":"2023-11-28T13:00:13.027438+0000","last_fullsized":"2023-11-28T13:00:13.027438+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:35:09.447520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"40'6","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.048153+0000","last_change":"2023-11-28T13:00:11.635492+0000","last_active":"2023-11-28T13:00:12.048153+0000","last_peered":"2023-11-28T13:00:12.048153+0000","last_clean":"2023-11-28T13:00:12.048153+0000","last_became_active":"2023-11-28T13:00:11.635173+0000","last_became_peered":"2023-11-28T13:00:11.635173+0000","last_unstale":"2023-11-28T13:00:12.048153+0000","last_undegraded":"2023-11-28T13:00:12.048153+0000","last_fullsized":"2023-11-28T13:00:12.048153+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:28:07.771314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025262+0000","last_change":"2023-11-28T13:00:12.033559+0000","last_active":"2023-11-28T13:00:13.025262+0000","last_peered":"2023-11-28T13:00:13.025262+0000","last_clean":"2023-11-28T13:00:13.025262+0000","last_became_active":"2023-11-28T13:00:12.033306+0000","last_became_peered":"2023-11-28T13:00:12.033306+0000","last_unstale":"2023-11-28T13:00:13.025262+0000","last_undegraded":"2023-11-28T13:00:13.025262+0000","last_fullsized":"2023-11-28T13:00:13.025262+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:49.483771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"40'2","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.045950+0000","last_change":"2023-11-28T13:00:11.635851+0000","last_active":"2023-11-28T13:00:12.045950+0000","last_peered":"2023-11-28T13:00:12.045950+0000","last_clean":"2023-11-28T13:00:12.045950+0000","last_became_active":"2023-11-28T13:00:11.634751+0000","last_became_peered":"2023-11-28T13:00:11.634751+0000","last_unstale":"2023-11-28T13:00:12.045950+0000","last_undegraded":"2023-11-28T13:00:12.045950+0000","last_fullsized":"2023-11-28T13:00:12.045950+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:19:16.875338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741504+0000","last_change":"2023-11-28T13:00:12.032841+0000","last_active":"2023-11-28T13:00:13.741504+0000","last_peered":"2023-11-28T13:00:13.741504+0000","last_clean":"2023-11-28T13:00:13.741504+0000","last_became_active":"2023-11-28T13:00:12.032394+0000","last_became_peered":"2023-11-28T13:00:12.032394+0000","last_unstale":"2023-11-28T13:00:13.741504+0000","last_undegraded":"2023-11-28T13:00:13.741504+0000","last_fullsized":"2023-11-28T13:00:13.741504+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:36.759595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.024845+0000","last_change":"2023-11-28T13:00:11.640366+0000","last_active":"2023-11-28T13:00:12.024845+0000","last_peered":"2023-11-28T13:00:12.024845+0000","last_clean":"2023-11-28T13:00:12.024845+0000","last_became_active":"2023-11-28T13:00:11.640079+0000","last_became_peered":"2023-11-28T13:00:11.640079+0000","last_unstale":"2023-11-28T13:00:12.024845+0000","last_undegraded":"2023-11-28T13:00:12.024845+0000","last_fullsized":"2023-11-28T13:00:12.024845+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.317320+0000","last_change":"2023-11-28T13:00:12.026606+0000","last_active":"2023-11-28T13:00:13.317320+0000","last_peered":"2023-11-28T13:00:13.317320+0000","last_clean":"2023-11-28T13:00:13.317320+0000","last_became_active":"2023-11-28T13:00:12.026338+0000","last_became_peered":"2023-11-28T13:00:12.026338+0000","last_unstale":"2023-11-28T13:00:13.317320+0000","last_undegraded":"2023-11-28T13:00:13.317320+0000","last_fullsized":"2023-11-28T13:00:13.317320+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:01:32.411043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.015460+0000","last_change":"2023-11-28T13:00:11.631109+0000","last_active":"2023-11-28T13:00:12.015460+0000","last_peered":"2023-11-28T13:00:12.015460+0000","last_clean":"2023-11-28T13:00:12.015460+0000","last_became_active":"2023-11-28T13:00:11.630645+0000","last_became_peered":"2023-11-28T13:00:11.630645+0000","last_unstale":"2023-11-28T13:00:12.015460+0000","last_undegraded":"2023-11-28T13:00:12.015460+0000","last_fullsized":"2023-11-28T13:00:12.015460+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:08:35.062283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.027249+0000","last_change":"2023-11-28T13:00:12.037576+0000","last_active":"2023-11-28T13:00:13.027249+0000","last_peered":"2023-11-28T13:00:13.027249+0000","last_clean":"2023-11-28T13:00:13.027249+0000","last_became_active":"2023-11-28T13:00:12.037270+0000","last_became_peered":"2023-11-28T13:00:12.037270+0000","last_unstale":"2023-11-28T13:00:13.027249+0000","last_undegraded":"2023-11-28T13:00:13.027249+0000","last_fullsized":"2023-11-28T13:00:13.027249+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:35:08.999441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"40'8","reported_seq":19,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.045929+0000","last_change":"2023-11-28T13:00:11.637424+0000","last_active":"2023-11-28T13:00:12.045929+0000","last_peered":"2023-11-28T13:00:12.045929+0000","last_clean":"2023-11-28T13:00:12.045929+0000","last_became_active":"2023-11-28T13:00:11.637140+0000","last_became_peered":"2023-11-28T13:00:11.637140+0000","last_unstale":"2023-11-28T13:00:12.045929+0000","last_undegraded":"2023-11-28T13:00:12.045929+0000","last_fullsized":"2023-11-28T13:00:12.045929+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.740979+0000","last_change":"2023-11-28T13:00:12.041112+0000","last_active":"2023-11-28T13:00:13.740979+0000","last_peered":"2023-11-28T13:00:13.740979+0000","last_clean":"2023-11-28T13:00:13.740979+0000","last_became_active":"2023-11-28T13:00:12.036576+0000","last_became_peered":"2023-11-28T13:00:12.036576+0000","last_unstale":"2023-11-28T13:00:13.740979+0000","last_undegraded":"2023-11-28T13:00:13.740979+0000","last_fullsized":"2023-11-28T13:00:13.740979+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:56:13.358281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"40'1","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.042812+0000","last_change":"2023-11-28T13:00:11.638922+0000","last_active":"2023-11-28T13:00:12.042812+0000","last_peered":"2023-11-28T13:00:12.042812+0000","last_clean":"2023-11-28T13:00:12.042812+0000","last_became_active":"2023-11-28T13:00:11.638620+0000","last_became_peered":"2023-11-28T13:00:11.638620+0000","last_unstale":"2023-11-28T13:00:12.042812+0000","last_undegraded":"2023-11-28T13:00:12.042812+0000","last_fullsized":"2023-11-28T13:00:12.042812+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:01:19.809502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741646+0000","last_change":"2023-11-28T13:00:12.031749+0000","last_active":"2023-11-28T13:00:13.741646+0000","last_peered":"2023-11-28T13:00:13.741646+0000","last_clean":"2023-11-28T13:00:13.741646+0000","last_became_active":"2023-11-28T13:00:12.031531+0000","last_became_peered":"2023-11-28T13:00:12.031531+0000","last_unstale":"2023-11-28T13:00:13.741646+0000","last_undegraded":"2023-11-28T13:00:13.741646+0000","last_fullsized":"2023-11-28T13:00:13.741646+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:57:08.284812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"40'2","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.042829+0000","last_change":"2023-11-28T13:00:11.636924+0000","last_active":"2023-11-28T13:00:12.042829+0000","last_peered":"2023-11-28T13:00:12.042829+0000","last_clean":"2023-11-28T13:00:12.042829+0000","last_became_active":"2023-11-28T13:00:11.636649+0000","last_became_peered":"2023-11-28T13:00:11.636649+0000","last_unstale":"2023-11-28T13:00:12.042829+0000","last_undegraded":"2023-11-28T13:00:12.042829+0000","last_fullsized":"2023-11-28T13:00:12.042829+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:36:34.014681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.024625+0000","last_change":"2023-11-28T13:00:12.028899+0000","last_active":"2023-11-28T13:00:13.024625+0000","last_peered":"2023-11-28T13:00:13.024625+0000","last_clean":"2023-11-28T13:00:13.024625+0000","last_became_active":"2023-11-28T13:00:12.028620+0000","last_became_peered":"2023-11-28T13:00:12.028620+0000","last_unstale":"2023-11-28T13:00:13.024625+0000","last_undegraded":"2023-11-28T13:00:13.024625+0000","last_fullsized":"2023-11-28T13:00:13.024625+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:03:26.495438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"40'1","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.045700+0000","last_change":"2023-11-28T13:00:11.638068+0000","last_active":"2023-11-28T13:00:12.045700+0000","last_peered":"2023-11-28T13:00:12.045700+0000","last_clean":"2023-11-28T13:00:12.045700+0000","last_became_active":"2023-11-28T13:00:11.637563+0000","last_became_peered":"2023-11-28T13:00:11.637563+0000","last_unstale":"2023-11-28T13:00:12.045700+0000","last_undegraded":"2023-11-28T13:00:12.045700+0000","last_fullsized":"2023-11-28T13:00:12.045700+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025235+0000","last_change":"2023-11-28T13:00:12.038902+0000","last_active":"2023-11-28T13:00:13.025235+0000","last_peered":"2023-11-28T13:00:13.025235+0000","last_clean":"2023-11-28T13:00:13.025235+0000","last_became_active":"2023-11-28T13:00:12.038629+0000","last_became_peered":"2023-11-28T13:00:12.038629+0000","last_unstale":"2023-11-28T13:00:13.025235+0000","last_undegraded":"2023-11-28T13:00:13.025235+0000","last_fullsized":"2023-11-28T13:00:13.025235+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:27.263683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.015766+0000","last_change":"2023-11-28T13:00:11.632284+0000","last_active":"2023-11-28T13:00:12.015766+0000","last_peered":"2023-11-28T13:00:12.015766+0000","last_clean":"2023-11-28T13:00:12.015766+0000","last_became_active":"2023-11-28T13:00:11.632012+0000","last_became_peered":"2023-11-28T13:00:11.632012+0000","last_unstale":"2023-11-28T13:00:12.015766+0000","last_undegraded":"2023-11-28T13:00:12.015766+0000","last_fullsized":"2023-11-28T13:00:12.015766+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.317359+0000","last_change":"2023-11-28T13:00:12.027135+0000","last_active":"2023-11-28T13:00:13.317359+0000","last_peered":"2023-11-28T13:00:13.317359+0000","last_clean":"2023-11-28T13:00:13.317359+0000","last_became_active":"2023-11-28T13:00:12.026249+0000","last_became_peered":"2023-11-28T13:00:12.026249+0000","last_unstale":"2023-11-28T13:00:13.317359+0000","last_undegraded":"2023-11-28T13:00:13.317359+0000","last_fullsized":"2023-11-28T13:00:13.317359+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:50:52.290831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"40'2","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.040250+0000","last_change":"2023-11-28T13:00:11.642474+0000","last_active":"2023-11-28T13:00:12.040250+0000","last_peered":"2023-11-28T13:00:12.040250+0000","last_clean":"2023-11-28T13:00:12.040250+0000","last_became_active":"2023-11-28T13:00:11.641827+0000","last_became_peered":"2023-11-28T13:00:11.641827+0000","last_unstale":"2023-11-28T13:00:12.040250+0000","last_undegraded":"2023-11-28T13:00:12.040250+0000","last_fullsized":"2023-11-28T13:00:12.040250+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:17:03.747332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.742402+0000","last_change":"2023-11-28T13:00:12.037216+0000","last_active":"2023-11-28T13:00:13.742402+0000","last_peered":"2023-11-28T13:00:13.742402+0000","last_clean":"2023-11-28T13:00:13.742402+0000","last_became_active":"2023-11-28T13:00:12.036972+0000","last_became_peered":"2023-11-28T13:00:12.036972+0000","last_unstale":"2023-11-28T13:00:13.742402+0000","last_undegraded":"2023-11-28T13:00:13.742402+0000","last_fullsized":"2023-11-28T13:00:13.742402+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:49.483771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.024053+0000","last_change":"2023-11-28T13:00:11.639736+0000","last_active":"2023-11-28T13:00:12.024053+0000","last_peered":"2023-11-28T13:00:12.024053+0000","last_clean":"2023-11-28T13:00:12.024053+0000","last_became_active":"2023-11-28T13:00:11.639423+0000","last_became_peered":"2023-11-28T13:00:11.639423+0000","last_unstale":"2023-11-28T13:00:12.024053+0000","last_undegraded":"2023-11-28T13:00:12.024053+0000","last_fullsized":"2023-11-28T13:00:12.024053+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:05:48.481419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741075+0000","last_change":"2023-11-28T13:00:12.041038+0000","last_active":"2023-11-28T13:00:13.741075+0000","last_peered":"2023-11-28T13:00:13.741075+0000","last_clean":"2023-11-28T13:00:13.741075+0000","last_became_active":"2023-11-28T13:00:12.040786+0000","last_became_peered":"2023-11-28T13:00:12.040786+0000","last_unstale":"2023-11-28T13:00:13.741075+0000","last_undegraded":"2023-11-28T13:00:13.741075+0000","last_fullsized":"2023-11-28T13:00:13.741075+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:41.323528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.022541+0000","last_change":"2023-11-28T13:00:11.642051+0000","last_active":"2023-11-28T13:00:12.022541+0000","last_peered":"2023-11-28T13:00:12.022541+0000","last_clean":"2023-11-28T13:00:12.022541+0000","last_became_active":"2023-11-28T13:00:11.641772+0000","last_became_peered":"2023-11-28T13:00:11.641772+0000","last_unstale":"2023-11-28T13:00:12.022541+0000","last_undegraded":"2023-11-28T13:00:12.022541+0000","last_fullsized":"2023-11-28T13:00:12.022541+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:37:07.844503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741266+0000","last_change":"2023-11-28T13:00:12.026535+0000","last_active":"2023-11-28T13:00:13.741266+0000","last_peered":"2023-11-28T13:00:13.741266+0000","last_clean":"2023-11-28T13:00:13.741266+0000","last_became_active":"2023-11-28T13:00:12.026294+0000","last_became_peered":"2023-11-28T13:00:12.026294+0000","last_unstale":"2023-11-28T13:00:13.741266+0000","last_undegraded":"2023-11-28T13:00:13.741266+0000","last_fullsized":"2023-11-28T13:00:13.741266+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:36:35.017032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.027548+0000","last_change":"2023-11-28T13:00:11.640140+0000","last_active":"2023-11-28T13:00:12.027548+0000","last_peered":"2023-11-28T13:00:12.027548+0000","last_clean":"2023-11-28T13:00:12.027548+0000","last_became_active":"2023-11-28T13:00:11.639826+0000","last_became_peered":"2023-11-28T13:00:11.639826+0000","last_unstale":"2023-11-28T13:00:12.027548+0000","last_undegraded":"2023-11-28T13:00:12.027548+0000","last_fullsized":"2023-11-28T13:00:12.027548+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:22:30.539161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.024406+0000","last_change":"2023-11-28T13:00:12.026810+0000","last_active":"2023-11-28T13:00:13.024406+0000","last_peered":"2023-11-28T13:00:13.024406+0000","last_clean":"2023-11-28T13:00:13.024406+0000","last_became_active":"2023-11-28T13:00:12.026522+0000","last_became_peered":"2023-11-28T13:00:12.026522+0000","last_unstale":"2023-11-28T13:00:13.024406+0000","last_undegraded":"2023-11-28T13:00:13.024406+0000","last_fullsized":"2023-11-28T13:00:13.024406+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:10:37.465991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.018878+0000","last_change":"2023-11-28T13:00:11.642311+0000","last_active":"2023-11-28T13:00:12.018878+0000","last_peered":"2023-11-28T13:00:12.018878+0000","last_clean":"2023-11-28T13:00:12.018878+0000","last_became_active":"2023-11-28T13:00:11.642037+0000","last_became_peered":"2023-11-28T13:00:11.642037+0000","last_unstale":"2023-11-28T13:00:12.018878+0000","last_undegraded":"2023-11-28T13:00:12.018878+0000","last_fullsized":"2023-11-28T13:00:12.018878+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:12:15.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.022723+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T13:00:12.022723+0000","last_peered":"2023-11-28T13:00:12.022723+0000","last_clean":"2023-11-28T13:00:12.022723+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:00:12.022723+0000","last_undegraded":"2023-11-28T13:00:12.022723+0000","last_fullsized":"2023-11-28T13:00:12.022723+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.025471+0000","last_change":"2023-11-28T13:00:12.032869+0000","last_active":"2023-11-28T13:00:13.025471+0000","last_peered":"2023-11-28T13:00:13.025471+0000","last_clean":"2023-11-28T13:00:13.025471+0000","last_became_active":"2023-11-28T13:00:12.032611+0000","last_became_peered":"2023-11-28T13:00:12.032611+0000","last_unstale":"2023-11-28T13:00:13.025471+0000","last_undegraded":"2023-11-28T13:00:13.025471+0000","last_fullsized":"2023-11-28T13:00:13.025471+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:02:50.623785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"41'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.317629+0000","last_change":"2023-11-28T13:00:11.631447+0000","last_active":"2023-11-28T13:00:13.317629+0000","last_peered":"2023-11-28T13:00:13.317629+0000","last_clean":"2023-11-28T13:00:13.317629+0000","last_became_active":"2023-11-28T13:00:11.631159+0000","last_became_peered":"2023-11-28T13:00:11.631159+0000","last_unstale":"2023-11-28T13:00:13.317629+0000","last_undegraded":"2023-11-28T13:00:13.317629+0000","last_fullsized":"2023-11-28T13:00:13.317629+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:37:14.727722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.024215+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T13:00:12.024215+0000","last_peered":"2023-11-28T13:00:12.024215+0000","last_clean":"2023-11-28T13:00:12.024215+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:00:12.024215+0000","last_undegraded":"2023-11-28T13:00:12.024215+0000","last_fullsized":"2023-11-28T13:00:12.024215+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741558+0000","last_change":"2023-11-28T13:00:12.031186+0000","last_active":"2023-11-28T13:00:13.741558+0000","last_peered":"2023-11-28T13:00:13.741558+0000","last_clean":"2023-11-28T13:00:13.741558+0000","last_became_active":"2023-11-28T13:00:12.030888+0000","last_became_peered":"2023-11-28T13:00:12.030888+0000","last_unstale":"2023-11-28T13:00:13.741558+0000","last_undegraded":"2023-11-28T13:00:13.741558+0000","last_fullsized":"2023-11-28T13:00:13.741558+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:48:32.551718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.026182+0000","last_change":"2023-11-28T13:00:11.644735+0000","last_active":"2023-11-28T13:00:12.026182+0000","last_peered":"2023-11-28T13:00:12.026182+0000","last_clean":"2023-11-28T13:00:12.026182+0000","last_became_active":"2023-11-28T13:00:11.644491+0000","last_became_peered":"2023-11-28T13:00:11.644491+0000","last_unstale":"2023-11-28T13:00:12.026182+0000","last_undegraded":"2023-11-28T13:00:12.026182+0000","last_fullsized":"2023-11-28T13:00:12.026182+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:26.261331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.023192+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T13:00:12.023192+0000","last_peered":"2023-11-28T13:00:12.023192+0000","last_clean":"2023-11-28T13:00:12.023192+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:00:12.023192+0000","last_undegraded":"2023-11-28T13:00:12.023192+0000","last_fullsized":"2023-11-28T13:00:12.023192+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.741746+0000","last_change":"2023-11-28T13:00:12.034207+0000","last_active":"2023-11-28T13:00:13.741746+0000","last_peered":"2023-11-28T13:00:13.741746+0000","last_clean":"2023-11-28T13:00:13.741746+0000","last_became_active":"2023-11-28T13:00:12.033947+0000","last_became_peered":"2023-11-28T13:00:12.033947+0000","last_unstale":"2023-11-28T13:00:13.741746+0000","last_undegraded":"2023-11-28T13:00:13.741746+0000","last_fullsized":"2023-11-28T13:00:13.741746+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:36:12.822043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"40'2","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.040417+0000","last_change":"2023-11-28T13:00:11.638988+0000","last_active":"2023-11-28T13:00:12.040417+0000","last_peered":"2023-11-28T13:00:12.040417+0000","last_clean":"2023-11-28T13:00:12.040417+0000","last_became_active":"2023-11-28T13:00:11.638736+0000","last_became_peered":"2023-11-28T13:00:11.638736+0000","last_unstale":"2023-11-28T13:00:12.040417+0000","last_undegraded":"2023-11-28T13:00:12.040417+0000","last_fullsized":"2023-11-28T13:00:12.040417+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:22:30.539161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":29,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.024844+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T13:00:12.024844+0000","last_peered":"2023-11-28T13:00:12.024844+0000","last_clean":"2023-11-28T13:00:12.024844+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:00:12.024844+0000","last_undegraded":"2023-11-28T13:00:12.024844+0000","last_fullsized":"2023-11-28T13:00:12.024844+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":182,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.027645+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T13:00:12.027645+0000","last_peered":"2023-11-28T13:00:12.027645+0000","last_clean":"2023-11-28T13:00:12.027645+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:00:12.027645+0000","last_undegraded":"2023-11-28T13:00:12.027645+0000","last_fullsized":"2023-11-28T13:00:12.027645+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.027532+0000","last_change":"2023-11-28T13:00:12.041806+0000","last_active":"2023-11-28T13:00:13.027532+0000","last_peered":"2023-11-28T13:00:13.027532+0000","last_clean":"2023-11-28T13:00:13.027532+0000","last_became_active":"2023-11-28T13:00:12.041540+0000","last_became_peered":"2023-11-28T13:00:12.041540+0000","last_unstale":"2023-11-28T13:00:13.027532+0000","last_undegraded":"2023-11-28T13:00:13.027532+0000","last_fullsized":"2023-11-28T13:00:13.027532+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:06.157368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.022391+0000","last_change":"2023-11-28T13:00:11.637900+0000","last_active":"2023-11-28T13:00:12.022391+0000","last_peered":"2023-11-28T13:00:12.022391+0000","last_clean":"2023-11-28T13:00:12.022391+0000","last_became_active":"2023-11-28T13:00:11.637604+0000","last_became_peered":"2023-11-28T13:00:11.637604+0000","last_unstale":"2023-11-28T13:00:12.022391+0000","last_undegraded":"2023-11-28T13:00:12.022391+0000","last_fullsized":"2023-11-28T13:00:12.022391+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:02:30.352479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.027722+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T13:00:12.027722+0000","last_peered":"2023-11-28T13:00:12.027722+0000","last_clean":"2023-11-28T13:00:12.027722+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:00:12.027722+0000","last_undegraded":"2023-11-28T13:00:12.027722+0000","last_fullsized":"2023-11-28T13:00:12.027722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.740712+0000","last_change":"2023-11-28T13:00:12.037914+0000","last_active":"2023-11-28T13:00:13.740712+0000","last_peered":"2023-11-28T13:00:13.740712+0000","last_clean":"2023-11-28T13:00:13.740712+0000","last_became_active":"2023-11-28T13:00:12.037293+0000","last_became_peered":"2023-11-28T13:00:12.037293+0000","last_unstale":"2023-11-28T13:00:13.740712+0000","last_undegraded":"2023-11-28T13:00:13.740712+0000","last_fullsized":"2023-11-28T13:00:13.740712+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:35:08.999441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.021757+0000","last_change":"2023-11-28T13:00:11.637940+0000","last_active":"2023-11-28T13:00:12.021757+0000","last_peered":"2023-11-28T13:00:12.021757+0000","last_clean":"2023-11-28T13:00:12.021757+0000","last_became_active":"2023-11-28T13:00:11.637660+0000","last_became_peered":"2023-11-28T13:00:11.637660+0000","last_unstale":"2023-11-28T13:00:12.021757+0000","last_undegraded":"2023-11-28T13:00:12.021757+0000","last_fullsized":"2023-11-28T13:00:12.021757+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:40.321176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":30,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.026255+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T13:00:12.026255+0000","last_peered":"2023-11-28T13:00:12.026255+0000","last_clean":"2023-11-28T13:00:12.026255+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:00:12.026255+0000","last_undegraded":"2023-11-28T13:00:12.026255+0000","last_fullsized":"2023-11-28T13:00:12.026255+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.317634+0000","last_change":"2023-11-28T13:00:12.027079+0000","last_active":"2023-11-28T13:00:13.317634+0000","last_peered":"2023-11-28T13:00:13.317634+0000","last_clean":"2023-11-28T13:00:13.317634+0000","last_became_active":"2023-11-28T13:00:12.026829+0000","last_became_peered":"2023-11-28T13:00:12.026829+0000","last_unstale":"2023-11-28T13:00:13.317634+0000","last_undegraded":"2023-11-28T13:00:13.317634+0000","last_fullsized":"2023-11-28T13:00:13.317634+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:36:35.017032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.018962+0000","last_change":"2023-11-28T13:00:11.642157+0000","last_active":"2023-11-28T13:00:12.018962+0000","last_peered":"2023-11-28T13:00:12.018962+0000","last_clean":"2023-11-28T13:00:12.018962+0000","last_became_active":"2023-11-28T13:00:11.641916+0000","last_became_peered":"2023-11-28T13:00:11.641916+0000","last_unstale":"2023-11-28T13:00:12.018962+0000","last_undegraded":"2023-11-28T13:00:12.018962+0000","last_fullsized":"2023-11-28T13:00:12.018962+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:50:51.288479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.022816+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T13:00:12.022816+0000","last_peered":"2023-11-28T13:00:12.022816+0000","last_clean":"2023-11-28T13:00:12.022816+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:00:12.022816+0000","last_undegraded":"2023-11-28T13:00:12.022816+0000","last_fullsized":"2023-11-28T13:00:12.022816+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-28T13:00:13.317528+0000","last_change":"2023-11-28T13:00:12.031376+0000","last_active":"2023-11-28T13:00:13.317528+0000","last_peered":"2023-11-28T13:00:13.317528+0000","last_clean":"2023-11-28T13:00:13.317528+0000","last_became_active":"2023-11-28T13:00:12.030466+0000","last_became_peered":"2023-11-28T13:00:12.030466+0000","last_unstale":"2023-11-28T13:00:13.317528+0000","last_undegraded":"2023-11-28T13:00:13.317528+0000","last_fullsized":"2023-11-28T13:00:13.317528+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:11.615718+0000","last_clean_scrub_stamp":"2023-11-28T13:00:11.615718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:03:05.309242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.025184+0000","last_change":"2023-11-28T13:00:11.642375+0000","last_active":"2023-11-28T13:00:12.025184+0000","last_peered":"2023-11-28T13:00:12.025184+0000","last_clean":"2023-11-28T13:00:12.025184+0000","last_became_active":"2023-11-28T13:00:11.642148+0000","last_became_peered":"2023-11-28T13:00:11.642148+0000","last_unstale":"2023-11-28T13:00:12.025184+0000","last_undegraded":"2023-11-28T13:00:12.025184+0000","last_fullsized":"2023-11-28T13:00:12.025184+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:10.613366+0000","last_clean_scrub_stamp":"2023-11-28T13:00:10.613366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:47.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-11-28T13:00:12.024050+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T13:00:12.024050+0000","last_peered":"2023-11-28T13:00:12.024050+0000","last_clean":"2023-11-28T13:00:12.024050+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:00:12.024050+0000","last_undegraded":"2023-11-28T13:00:12.024050+0000","last_fullsized":"2023-11-28T13:00:12.024050+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9824,"kb_used_data":1048,"kb_used_omap":0,"kb_used_meta":8768,"kb_avail":94362016,"statfs":{"total":96636764160,"available":96626704384,"internally_reserved":0,"allocated":1073152,"data_stored":804169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8978432},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.3279999999999998,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.2919999999999998,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.59599999999999997}]},{"osd":1,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.54000000000000004}]},{"osd":2,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.61299999999999999}]},{"osd":3,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.66300000000000003}]},{"osd":4,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.55200000000000005}]},{"osd":5,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.41199999999999998}]},{"osd":6,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.58499999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8672,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363168,"statfs":{"total":96636764160,"available":96627884032,"internally_reserved":0,"allocated":483328,"data_stored":214367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.38200000000000001}]},{"osd":1,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.434},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.51500000000000001}]},{"osd":2,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.45900000000000002}]},{"osd":3,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.46800000000000003}]},{"osd":4,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.44600000000000001}]},{"osd":5,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.502},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.54100000000000004}]},{"osd":7,"last update":"Tue Nov 28 12:59:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.55400000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8732,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363108,"statfs":{"total":96636764160,"available":96627822592,"internally_reserved":0,"allocated":479232,"data_stored":213781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.24399999999999999}]},{"osd":2,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.31},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.34599999999999997}]},{"osd":3,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.29999999999999999}]},{"osd":4,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.46100000000000002}]},{"osd":5,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.40999999999999998}]},{"osd":6,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.45500000000000002}]},{"osd":7,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.42499999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9824,"kb_used_data":1048,"kb_used_omap":0,"kb_used_meta":8768,"kb_avail":94362016,"statfs":{"total":96636764160,"available":96626704384,"internally_reserved":0,"allocated":1073152,"data_stored":804113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8978432},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.29199999999999998}]},{"osd":2,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.34899999999999998}]},{"osd":3,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.308},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.32300000000000001}]},{"osd":4,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.437},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47699999999999998}]},{"osd":5,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.55500000000000005}]},{"osd":6,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.57199999999999995}]},{"osd":7,"last update":"Tue Nov 28 12:59:50 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.56399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8552,"kb_used_data":480,"kb_used_omap":0,"kb_used_meta":8064,"kb_avail":94363288,"statfs":{"total":96636764160,"available":96628006912,"internally_reserved":0,"allocated":491520,"data_stored":215281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8257536},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.245},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.29799999999999999}]},{"osd":1,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.35299999999999998}]},{"osd":3,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.40600000000000003}]},{"osd":4,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56599999999999995}]},{"osd":5,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.499},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.53300000000000003}]},{"osd":6,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.45800000000000002}]},{"osd":7,"last update":"Tue Nov 28 12:59:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.64200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8740,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363100,"statfs":{"total":96636764160,"available":96627814400,"internally_reserved":0,"allocated":487424,"data_stored":214333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.318,"15min":0.318},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.32700000000000001}]},{"osd":1,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.311},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.28899999999999998}]},{"osd":2,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.7160000000000002,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":0.377}]},{"osd":4,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":8.2889999999999997,"5min":8.2889999999999997,"15min":8.2889999999999997},"last":0.75900000000000001}]},{"osd":5,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":0.60199999999999998}]},{"osd":6,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.75}]},{"osd":7,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":7.0899999999999999,"5min":7.0899999999999999,"15min":7.0899999999999999},"last":0.81499999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8804,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363036,"statfs":{"total":96636764160,"available":96627748864,"internally_reserved":0,"allocated":487424,"data_stored":215259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56200000000000006}]},{"osd":1,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.495},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.48499999999999999}]},{"osd":2,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.51000000000000001}]},{"osd":3,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.52700000000000002}]},{"osd":5,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.378},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46400000000000002}]},{"osd":6,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.45400000000000001}]},{"osd":7,"last update":"Tue Nov 28 12:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57399999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8668,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94363172,"statfs":{"total":96636764160,"available":96627888128,"internally_reserved":0,"allocated":479232,"data_stored":213781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.59599999999999997}]},{"osd":1,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.48199999999999998}]},{"osd":2,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":2.4020000000000001,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.40600000000000003}]},{"osd":3,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":0.60799999999999998}]},{"osd":4,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.0739999999999998,"5min":3.0739999999999998,"15min":3.0739999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.66800000000000004}]},{"osd":6,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":3.7679999999999998,"5min":3.7679999999999998,"15min":3.7679999999999998},"last":0.58799999999999997}]},{"osd":7,"last update":"Tue Nov 28 12:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.67800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:00:20.992 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T13:00:20.993 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2023-11-28T13:00:21.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.149+0000 7fbac71bd700 1 Processor -- start 2023-11-28T13:00:21.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.149+0000 7fbac71bd700 1 -- start start 2023-11-28T13:00:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.149+0000 7fbac71bd700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fbac006bff0 0x7fbac015d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.149+0000 7fbac71bd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac00b1d50 con 0x7fbac0160ca0 2023-11-28T13:00:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.149+0000 7fbac71bd700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac015cd40 con 0x7fbac00cde60 2023-11-28T13:00:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.149+0000 7fbac71bd700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac0161140 con 0x7fbac006bff0 2023-11-28T13:00:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.149+0000 7fbabffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbac0160ca0 0x7fbac00cd750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36128/0 (socket says 172.21.15.111:36128) 2023-11-28T13:00:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.149+0000 7fbabffff700 1 -- 172.21.15.111:0/2254027261 learned_addr learned my addr 172.21.15.111:0/2254027261 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.149+0000 7fbac4f59700 1 --2- 172.21.15.111:0/2254027261 >> v2:172.21.15.149:3300/0 conn(0x7fbac006bff0 0x7fbac015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.149+0000 7fbabf7fe700 1 -- 172.21.15.111:0/2254027261 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671240377 0 0) 0x7fbac00b1d50 con 0x7fbac0160ca0 2023-11-28T13:00:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.149+0000 7fbabf7fe700 1 -- 172.21.15.111:0/2254027261 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab00035b0 con 0x7fbac0160ca0 2023-11-28T13:00:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.151+0000 7fbabf7fe700 1 -- 172.21.15.111:0/2254027261 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3546754618 0 0) 0x7fbab00035b0 con 0x7fbac0160ca0 2023-11-28T13:00:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.151+0000 7fbac4f59700 1 -- 172.21.15.111:0/2254027261 >> v1:172.21.15.149:6789/0 conn(0x7fbac00cde60 legacy=0x7fbac006a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.151+0000 7fbac4f59700 1 -- 172.21.15.111:0/2254027261 >> v1:172.21.15.111:6789/0 conn(0x7fbac0160ca0 legacy=0x7fbac00cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.151+0000 7fbac4f59700 1 -- 172.21.15.111:0/2254027261 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac015d020 con 0x7fbac006bff0 2023-11-28T13:00:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.151+0000 7fbac4f59700 1 --2- 172.21.15.111:0/2254027261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbac006bff0 0x7fbac015d820 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009840 tx=0x7fbab4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caea347880917d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.151+0000 7fbabf7fe700 1 -- 172.21.15.111:0/2254027261 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac001670 con 0x7fbac006bff0 2023-11-28T13:00:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.151+0000 7fbabf7fe700 1 -- 172.21.15.111:0/2254027261 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab4004030 con 0x7fbac006bff0 2023-11-28T13:00:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.151+0000 7fbabf7fe700 1 -- 172.21.15.111:0/2254027261 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab40043a0 con 0x7fbac006bff0 2023-11-28T13:00:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.151+0000 7fbac71bd700 1 -- 172.21.15.111:0/2254027261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbac006bff0 msgr2=0x7fbac015d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.151+0000 7fbac71bd700 1 --2- 172.21.15.111:0/2254027261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbac006bff0 0x7fbac015d820 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbab4009840 tx=0x7fbab4009b50 comp rx=0 tx=0).stop 2023-11-28T13:00:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.151+0000 7fbac71bd700 1 -- 172.21.15.111:0/2254027261 shutdown_connections 2023-11-28T13:00:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.151+0000 7fbac71bd700 1 --2- 172.21.15.111:0/2254027261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbac006bff0 0x7fbac015d820 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.151+0000 7fbac71bd700 1 -- 172.21.15.111:0/2254027261 >> 172.21.15.111:0/2254027261 conn(0x7fbac006e790 msgr2=0x7fbac00cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.152+0000 7fbac71bd700 1 -- 172.21.15.111:0/2254027261 shutdown_connections 2023-11-28T13:00:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.152+0000 7fbac71bd700 1 -- 172.21.15.111:0/2254027261 wait complete. 2023-11-28T13:00:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.153+0000 7fbac71bd700 1 Processor -- start 2023-11-28T13:00:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.153+0000 7fbac71bd700 1 -- start start 2023-11-28T13:00:21.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.153+0000 7fbac71bd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbac006bff0 0x7fbac00ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:21.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.153+0000 7fbac71bd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbac00cde60 0x7fbac00b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:21.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbac71bd700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbac0160ca0 0x7fbac014ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:21.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbac71bd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00b5320 con 0x7fbac00cde60 2023-11-28T13:00:21.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbac71bd700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00ac9e0 con 0x7fbac0160ca0 2023-11-28T13:00:21.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbac71bd700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00b2ae0 con 0x7fbac006bff0 2023-11-28T13:00:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbabffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbac00cde60 0x7fbac00b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbabffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbac00cde60 0x7fbac00b4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35358/0 (socket says 172.21.15.111:35358) 2023-11-28T13:00:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbabffff700 1 -- 172.21.15.111:0/490141743 learned_addr learned my addr 172.21.15.111:0/490141743 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbabffff700 1 -- 172.21.15.111:0/490141743 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbac006bff0 msgr2=0x7fbac00ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbabffff700 1 --2- 172.21.15.111:0/490141743 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbac006bff0 0x7fbac00ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbabffff700 1 -- 172.21.15.111:0/490141743 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbac0160ca0 msgr2=0x7fbac014ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbac575a700 1 --2- 172.21.15.111:0/490141743 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbac0160ca0 0x7fbac014ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbabffff700 1 --2- 172.21.15.111:0/490141743 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbac0160ca0 0x7fbac014ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbabffff700 1 -- 172.21.15.111:0/490141743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab40094a0 con 0x7fbac00cde60 2023-11-28T13:00:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbabffff700 1 --2- 172.21.15.111:0/490141743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbac00cde60 0x7fbac00b4de0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00c520 tx=0x7fbaac00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbabd7fa700 1 -- 172.21.15.111:0/490141743 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac004c80 con 0x7fbac00cde60 2023-11-28T13:00:21.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbabd7fa700 1 -- 172.21.15.111:0/490141743 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac004e20 con 0x7fbac00cde60 2023-11-28T13:00:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbabd7fa700 1 -- 172.21.15.111:0/490141743 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac007310 con 0x7fbac00cde60 2023-11-28T13:00:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbac71bd700 1 -- 172.21.15.111:0/490141743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac00b2030 con 0x7fbac00cde60 2023-11-28T13:00:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.154+0000 7fbac71bd700 1 -- 172.21.15.111:0/490141743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac015d300 con 0x7fbac00cde60 2023-11-28T13:00:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.155+0000 7fbac71bd700 1 -- 172.21.15.111:0/490141743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac00b1600 con 0x7fbac00cde60 2023-11-28T13:00:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.156+0000 7fbabd7fa700 1 -- 172.21.15.111:0/490141743 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbaac01a020 con 0x7fbac00cde60 2023-11-28T13:00:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.156+0000 7fbabd7fa700 1 --2- 172.21.15.111:0/490141743 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbaa40682f0 0x7fbaa406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.156+0000 7fbabd7fa700 1 -- 172.21.15.111:0/490141743 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fbaac085af0 con 0x7fbac00cde60 2023-11-28T13:00:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.156+0000 7fbac4f59700 1 --2- 172.21.15.111:0/490141743 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbaa40682f0 0x7fbaa406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.156+0000 7fbac4f59700 1 --2- 172.21.15.111:0/490141743 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbaa40682f0 0x7fbaa406a7a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbac00b04a0 tx=0x7fbab4010000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:21.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.160+0000 7fbabd7fa700 1 -- 172.21.15.111:0/490141743 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbaac052190 con 0x7fbac00cde60 2023-11-28T13:00:21.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.417+0000 7fbac71bd700 1 -- 172.21.15.111:0/490141743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7fbac0067170 con 0x7fbac00cde60 2023-11-28T13:00:21.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.421+0000 7fbabd7fa700 1 -- 172.21.15.111:0/490141743 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 184+0+0 (secure 0 0 0) 0x7fbaac0557b0 con 0x7fbac00cde60 2023-11-28T13:00:21.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.423+0000 7fbac71bd700 1 -- 172.21.15.111:0/490141743 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbaa40682f0 msgr2=0x7fbaa406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:21.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.423+0000 7fbac71bd700 1 --2- 172.21.15.111:0/490141743 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbaa40682f0 0x7fbaa406a7a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbac00b04a0 tx=0x7fbab4010000 comp rx=0 tx=0).stop 2023-11-28T13:00:21.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.423+0000 7fbac71bd700 1 -- 172.21.15.111:0/490141743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbac00cde60 msgr2=0x7fbac00b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:21.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.423+0000 7fbac71bd700 1 --2- 172.21.15.111:0/490141743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbac00cde60 0x7fbac00b4de0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00c520 tx=0x7fbaac00c8e0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.423+0000 7fbac71bd700 1 -- 172.21.15.111:0/490141743 shutdown_connections 2023-11-28T13:00:21.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.423+0000 7fbac71bd700 1 --2- 172.21.15.111:0/490141743 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbaa40682f0 0x7fbaa406a7a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.423+0000 7fbac71bd700 1 --2- 172.21.15.111:0/490141743 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbac0160ca0 0x7fbac014ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.423+0000 7fbac71bd700 1 --2- 172.21.15.111:0/490141743 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbac006bff0 0x7fbac00ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.423+0000 7fbac71bd700 1 --2- 172.21.15.111:0/490141743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbac00cde60 0x7fbac00b4de0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.423+0000 7fbac71bd700 1 -- 172.21.15.111:0/490141743 >> 172.21.15.111:0/490141743 conn(0x7fbac006e790 msgr2=0x7fbac015c120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:21.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.423+0000 7fbac71bd700 1 -- 172.21.15.111:0/490141743 shutdown_connections 2023-11-28T13:00:21.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.423+0000 7fbac71bd700 1 -- 172.21.15.111:0/490141743 wait complete. 2023-11-28T13:00:21.425 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-11-28T13:00:21.435 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:00:21.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.554+0000 7fd0f73db700 1 Processor -- start 2023-11-28T13:00:21.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.554+0000 7fd0f73db700 1 -- start start 2023-11-28T13:00:21.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.555+0000 7fd0f73db700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd0f0160e90 0x7fd0f006bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:21.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.555+0000 7fd0f73db700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0f015cbd0 con 0x7fd0f00cd790 2023-11-28T13:00:21.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.555+0000 7fd0f73db700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0f006af30 con 0x7fd0f015c800 2023-11-28T13:00:21.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.555+0000 7fd0f73db700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f006b210 con 0x7fd0f0160e90 2023-11-28T13:00:21.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.555+0000 7fd0f4976700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd0f0160e90 0x7fd0f006bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:21.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.555+0000 7fd0f5978700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd0f00cd790 0x7fd0f00b29f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36146/0 (socket says 172.21.15.111:36146) 2023-11-28T13:00:21.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.555+0000 7fd0f5978700 1 -- 172.21.15.111:0/3197999874 learned_addr learned my addr 172.21.15.111:0/3197999874 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:21.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.556+0000 7fd0f4976700 1 -- 172.21.15.111:0/3197999874 >> v1:172.21.15.149:6789/0 conn(0x7fd0f015c800 legacy=0x7fd0f006a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:21.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.556+0000 7fd0f4976700 1 -- 172.21.15.111:0/3197999874 >> v1:172.21.15.111:6789/0 conn(0x7fd0f00cd790 legacy=0x7fd0f00b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:21.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.556+0000 7fd0f4976700 1 -- 172.21.15.111:0/3197999874 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f006b4f0 con 0x7fd0f0160e90 2023-11-28T13:00:21.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.556+0000 7fd0f4976700 1 --2- 172.21.15.111:0/3197999874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0f0160e90 0x7fd0f006bfd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009840 tx=0x7fd0ec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c9647e72a68b9f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:21.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.556+0000 7fd0e7fff700 1 -- 172.21.15.111:0/3197999874 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec004230 con 0x7fd0f0160e90 2023-11-28T13:00:21.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.556+0000 7fd0e7fff700 1 -- 172.21.15.111:0/3197999874 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ec004390 con 0x7fd0f0160e90 2023-11-28T13:00:21.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.556+0000 7fd0e7fff700 1 -- 172.21.15.111:0/3197999874 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec0036a0 con 0x7fd0f0160e90 2023-11-28T13:00:21.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.556+0000 7fd0f73db700 1 -- 172.21.15.111:0/3197999874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0f0160e90 msgr2=0x7fd0f006bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:21.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.556+0000 7fd0f73db700 1 --2- 172.21.15.111:0/3197999874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0f0160e90 0x7fd0f006bfd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009840 tx=0x7fd0ec009b50 comp rx=0 tx=0).stop 2023-11-28T13:00:21.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.557+0000 7fd0f73db700 1 -- 172.21.15.111:0/3197999874 shutdown_connections 2023-11-28T13:00:21.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.557+0000 7fd0f73db700 1 --2- 172.21.15.111:0/3197999874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0f0160e90 0x7fd0f006bfd0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.557+0000 7fd0f73db700 1 -- 172.21.15.111:0/3197999874 >> 172.21.15.111:0/3197999874 conn(0x7fd0f006db10 msgr2=0x7fd0f006df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:21.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.557+0000 7fd0f73db700 1 -- 172.21.15.111:0/3197999874 shutdown_connections 2023-11-28T13:00:21.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.557+0000 7fd0f73db700 1 -- 172.21.15.111:0/3197999874 wait complete. 2023-11-28T13:00:21.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.558+0000 7fd0f73db700 1 Processor -- start 2023-11-28T13:00:21.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.558+0000 7fd0f73db700 1 -- start start 2023-11-28T13:00:21.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.558+0000 7fd0f73db700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd0f00cd790 0x7fd0f00c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:21.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.558+0000 7fd0f73db700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd0f015c800 0x7fd0f00c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:21.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f73db700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0f0160e90 0x7fd0f01f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:21.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f73db700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00c9a30 con 0x7fd0f0160e90 2023-11-28T13:00:21.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f73db700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00b4f70 con 0x7fd0f015c800 2023-11-28T13:00:21.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f73db700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f00b5250 con 0x7fd0f00cd790 2023-11-28T13:00:21.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f5978700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0f0160e90 0x7fd0f01f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:21.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f5978700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0f0160e90 0x7fd0f01f6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35374/0 (socket says 172.21.15.111:35374) 2023-11-28T13:00:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f5978700 1 -- 172.21.15.111:0/1373989531 learned_addr learned my addr 172.21.15.111:0/1373989531 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f4976700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd0f015c800 0x7fd0f00c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f5177700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd0f00cd790 0x7fd0f00c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f5978700 1 -- 172.21.15.111:0/1373989531 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd0f00cd790 msgr2=0x7fd0f00c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f5978700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd0f00cd790 0x7fd0f00c8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f5978700 1 -- 172.21.15.111:0/1373989531 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd0f015c800 msgr2=0x7fd0f00c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f5978700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd0f015c800 0x7fd0f00c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f5978700 1 -- 172.21.15.111:0/1373989531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec0094a0 con 0x7fd0f0160e90 2023-11-28T13:00:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f5978700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0f0160e90 0x7fd0f01f6240 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800cbc0 tx=0x7fd0e800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0e5ffb700 1 -- 172.21.15.111:0/1373989531 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e800ab50 con 0x7fd0f0160e90 2023-11-28T13:00:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0e5ffb700 1 -- 172.21.15.111:0/1373989531 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e800acb0 con 0x7fd0f0160e90 2023-11-28T13:00:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f73db700 1 -- 172.21.15.111:0/1373989531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f006bab0 con 0x7fd0f0160e90 2023-11-28T13:00:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0e5ffb700 1 -- 172.21.15.111:0/1373989531 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8012600 con 0x7fd0f0160e90 2023-11-28T13:00:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.559+0000 7fd0f73db700 1 -- 172.21.15.111:0/1373989531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f006b7d0 con 0x7fd0f0160e90 2023-11-28T13:00:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.560+0000 7fd0f73db700 1 -- 172.21.15.111:0/1373989531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f00cdc10 con 0x7fd0f0160e90 2023-11-28T13:00:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.561+0000 7fd0e5ffb700 1 -- 172.21.15.111:0/1373989531 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd0e8019030 con 0x7fd0f0160e90 2023-11-28T13:00:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.561+0000 7fd0e5ffb700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd0d40682f0 0x7fd0d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.561+0000 7fd0e5ffb700 1 -- 172.21.15.111:0/1373989531 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd0e8085900 con 0x7fd0f0160e90 2023-11-28T13:00:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.561+0000 7fd0f5177700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd0d40682f0 0x7fd0d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:21.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.561+0000 7fd0f5177700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd0d40682f0 0x7fd0d406a7a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0004f30 tx=0x7fd0e0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:21.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.565+0000 7fd0e5ffb700 1 -- 172.21.15.111:0/1373989531 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd0e804b5e0 con 0x7fd0f0160e90 2023-11-28T13:00:21.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.790+0000 7fd0f73db700 1 -- 172.21.15.111:0/1373989531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0f0066d50 con 0x7fd0f0160e90 2023-11-28T13:00:21.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.791+0000 7fd0e5ffb700 1 -- 172.21.15.111:0/1373989531 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4660 (secure 0 0 0) 0x7fd0e8052070 con 0x7fd0f0160e90 2023-11-28T13:00:21.792 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:21.792 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3254653481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3254653481},{"type":"v1","addr":"172.21.15.149:6835","nonce":3254653481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1890904311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1890904311},{"type":"v1","addr":"172.21.15.111:6836","nonce":1890904311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1592804351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1592804351},{"type":"v1","addr":"172.21.15.111:6837","nonce":1592804351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:12.011963+0000","modified":"2023-11-28T13:00:15.034385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5352":{"gid":5352,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.149:6834/3579364312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":3579364312},{"type":"v1","addr":"172.21.15.149:6834","nonce":3579364312}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T13:00:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.793+0000 7fd0f73db700 1 -- 172.21.15.111:0/1373989531 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd0d40682f0 msgr2=0x7fd0d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.793+0000 7fd0f73db700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd0d40682f0 0x7fd0d406a7a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0004f30 tx=0x7fd0e0009040 comp rx=0 tx=0).stop 2023-11-28T13:00:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.793+0000 7fd0f73db700 1 -- 172.21.15.111:0/1373989531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0f0160e90 msgr2=0x7fd0f01f6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.793+0000 7fd0f73db700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0f0160e90 0x7fd0f01f6240 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800cbc0 tx=0x7fd0e800cf80 comp rx=0 tx=0).stop 2023-11-28T13:00:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.793+0000 7fd0f73db700 1 -- 172.21.15.111:0/1373989531 shutdown_connections 2023-11-28T13:00:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.793+0000 7fd0f73db700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0f0160e90 0x7fd0f01f6240 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.793+0000 7fd0f73db700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd0f00cd790 0x7fd0f00c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.793+0000 7fd0f73db700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd0d40682f0 0x7fd0d406a7a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.793+0000 7fd0f73db700 1 --2- 172.21.15.111:0/1373989531 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd0f015c800 0x7fd0f00c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:21.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.793+0000 7fd0f73db700 1 -- 172.21.15.111:0/1373989531 >> 172.21.15.111:0/1373989531 conn(0x7fd0f006db10 msgr2=0x7fd0f0156870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:21.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.793+0000 7fd0f73db700 1 -- 172.21.15.111:0/1373989531 shutdown_connections 2023-11-28T13:00:21.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:21.793+0000 7fd0f73db700 1 -- 172.21.15.111:0/1373989531 wait complete. 2023-11-28T13:00:21.795 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 17 2023-11-28T13:00:22.208 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T13:00:12.011963+0000', 'modified': '2023-11-28T13:00:15.034385+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5352}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5352': {'gid': 5352, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.149:6834/3579364312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.149:6832', 'nonce': 3579364312}, {'type': 'v1', 'addr': '172.21.15.149:6834', 'nonce': 3579364312}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T13:00:22.208 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T13:00:22.208 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-11-28T13:00:22.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.331+0000 7f3d3f053700 1 Processor -- start 2023-11-28T13:00:22.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.331+0000 7f3d3f053700 1 -- start start 2023-11-28T13:00:22.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d3f053700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f3d380d0490 0x7f3d380cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d3f053700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3d3806b380 0x7f3d3806b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d3f053700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d380cb810 con 0x7f3d3815c7b0 2023-11-28T13:00:22.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d3f053700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d380cc650 con 0x7f3d3806b380 2023-11-28T13:00:22.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d3f053700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d380cba40 con 0x7f3d380d0490 2023-11-28T13:00:22.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d3d5f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3d3806b380 0x7f3d3806b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d3cdef700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3d3815c7b0 0x7f3d380ad8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36156/0 (socket says 172.21.15.111:36156) 2023-11-28T13:00:22.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d3cdef700 1 -- 172.21.15.111:0/170736894 learned_addr learned my addr 172.21.15.111:0/170736894 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:22.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d3d5f0700 1 -- 172.21.15.111:0/170736894 >> v2:172.21.15.149:3300/0 conn(0x7f3d380d0490 msgr2=0x7f3d380cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:22.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d37fff700 1 --2- 172.21.15.111:0/170736894 >> v2:172.21.15.149:3300/0 conn(0x7f3d380d0490 0x7f3d380cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d3d5f0700 1 --2- 172.21.15.111:0/170736894 >> v2:172.21.15.149:3300/0 conn(0x7f3d380d0490 0x7f3d380cb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d377fe700 1 -- 172.21.15.111:0/170736894 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997455042 0 0) 0x7f3d380cb810 con 0x7f3d3815c7b0 2023-11-28T13:00:22.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d3d5f0700 1 -- 172.21.15.111:0/170736894 >> v1:172.21.15.111:6789/0 conn(0x7f3d3815c7b0 legacy=0x7f3d380ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d3d5f0700 1 -- 172.21.15.111:0/170736894 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3806bd20 con 0x7f3d3806b380 2023-11-28T13:00:22.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d3d5f0700 1 --2- 172.21.15.111:0/170736894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d3806b380 0x7f3d3806b7e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3d380c4780 tx=0x7f3d3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97814f34f6eb6b45 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:22.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d377fe700 1 -- 172.21.15.111:0/170736894 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3000e070 con 0x7f3d3806b380 2023-11-28T13:00:22.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d377fe700 1 -- 172.21.15.111:0/170736894 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3000bbd0 con 0x7f3d3806b380 2023-11-28T13:00:22.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.332+0000 7f3d377fe700 1 -- 172.21.15.111:0/170736894 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d30012430 con 0x7f3d3806b380 2023-11-28T13:00:22.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.333+0000 7f3d3f053700 1 -- 172.21.15.111:0/170736894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d3806b380 msgr2=0x7f3d3806b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.333+0000 7f3d3f053700 1 --2- 172.21.15.111:0/170736894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d3806b380 0x7f3d3806b7e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3d380c4780 tx=0x7f3d3000b330 comp rx=0 tx=0).stop 2023-11-28T13:00:22.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.333+0000 7f3d3f053700 1 -- 172.21.15.111:0/170736894 shutdown_connections 2023-11-28T13:00:22.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.333+0000 7f3d3f053700 1 --2- 172.21.15.111:0/170736894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d3806b380 0x7f3d3806b7e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.333+0000 7f3d3f053700 1 --2- 172.21.15.111:0/170736894 >> v2:172.21.15.149:3300/0 conn(0x7f3d380d0490 0x7f3d380cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.333+0000 7f3d3f053700 1 -- 172.21.15.111:0/170736894 >> 172.21.15.111:0/170736894 conn(0x7f3d3809a040 msgr2=0x7f3d380d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:22.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.333+0000 7f3d3f053700 1 -- 172.21.15.111:0/170736894 shutdown_connections 2023-11-28T13:00:22.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.333+0000 7f3d3f053700 1 -- 172.21.15.111:0/170736894 wait complete. 2023-11-28T13:00:22.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.334+0000 7f3d3f053700 1 Processor -- start 2023-11-28T13:00:22.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.334+0000 7f3d3f053700 1 -- start start 2023-11-28T13:00:22.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.335+0000 7f3d3f053700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d3806b380 0x7f3d380b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.335+0000 7f3d3f053700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d380d0490 0x7f3d380b0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.335+0000 7f3d3f053700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d3815c7b0 0x7f3d380b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.335+0000 7f3d3f053700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3815ea00 con 0x7f3d3815c7b0 2023-11-28T13:00:22.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.335+0000 7f3d3f053700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3815ece0 con 0x7f3d3806b380 2023-11-28T13:00:22.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.335+0000 7f3d3f053700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3815efc0 con 0x7f3d380d0490 2023-11-28T13:00:22.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.335+0000 7f3d3d5f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d3815c7b0 0x7f3d380b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.335+0000 7f3d3d5f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d3815c7b0 0x7f3d380b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35388/0 (socket says 172.21.15.111:35388) 2023-11-28T13:00:22.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.335+0000 7f3d3d5f0700 1 -- 172.21.15.111:0/478834582 learned_addr learned my addr 172.21.15.111:0/478834582 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:22.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.335+0000 7f3d3cdef700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d3806b380 0x7f3d380b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.335+0000 7f3d37fff700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d380d0490 0x7f3d380b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.336+0000 7f3d3d5f0700 1 -- 172.21.15.111:0/478834582 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d380d0490 msgr2=0x7f3d380b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.336+0000 7f3d3d5f0700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d380d0490 0x7f3d380b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.336+0000 7f3d3d5f0700 1 -- 172.21.15.111:0/478834582 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d3806b380 msgr2=0x7f3d380b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.336+0000 7f3d3d5f0700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d3806b380 0x7f3d380b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.336+0000 7f3d3d5f0700 1 -- 172.21.15.111:0/478834582 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d30009a70 con 0x7f3d3815c7b0 2023-11-28T13:00:22.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.336+0000 7f3d37fff700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d380d0490 0x7f3d380b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:22.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.336+0000 7f3d3cdef700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d3806b380 0x7f3d380b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:22.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.336+0000 7f3d3d5f0700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d3815c7b0 0x7f3d380b0bb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3d3806d8f0 tx=0x7f3d30004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:22.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.336+0000 7f3d357fa700 1 -- 172.21.15.111:0/478834582 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d30012430 con 0x7f3d3815c7b0 2023-11-28T13:00:22.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.336+0000 7f3d357fa700 1 -- 172.21.15.111:0/478834582 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d30020070 con 0x7f3d3815c7b0 2023-11-28T13:00:22.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.336+0000 7f3d357fa700 1 -- 172.21.15.111:0/478834582 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d30003ea0 con 0x7f3d3815c7b0 2023-11-28T13:00:22.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.336+0000 7f3d3f053700 1 -- 172.21.15.111:0/478834582 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d380cc000 con 0x7f3d3815c7b0 2023-11-28T13:00:22.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.336+0000 7f3d3f053700 1 -- 172.21.15.111:0/478834582 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d380cbd20 con 0x7f3d3815c7b0 2023-11-28T13:00:22.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.337+0000 7f3d357fa700 1 -- 172.21.15.111:0/478834582 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d3002e070 con 0x7f3d3815c7b0 2023-11-28T13:00:22.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.338+0000 7f3d357fa700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d1c0682f0 0x7f3d1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.338+0000 7f3d357fa700 1 -- 172.21.15.111:0/478834582 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3d30022070 con 0x7f3d3815c7b0 2023-11-28T13:00:22.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.338+0000 7f3d3cdef700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d1c0682f0 0x7f3d1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.338+0000 7f3d3cdef700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d1c0682f0 0x7f3d1c06a7a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0010e0 tx=0x7f3d2c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:22.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.339+0000 7f3d4b1dcb80 1 Processor -- start 2023-11-28T13:00:22.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.340+0000 7f3d4b1dcb80 1 -- start start 2023-11-28T13:00:22.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.340+0000 7f3d4b1dcb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c3810 0x55f4cb999a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.340+0000 7f3d4b1dcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 0x55f4cb9c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.340+0000 7f3d4b1dcb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9df770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.340+0000 7f3d4b1dcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cb883bc0 con 0x55f4cb9c1130 2023-11-28T13:00:22.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.340+0000 7f3d4b1dcb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cb8f2350 con 0x55f4cb9c4120 2023-11-28T13:00:22.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.340+0000 7f3d4b1dcb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f4cb8eb880 con 0x55f4cb9c3810 2023-11-28T13:00:22.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.340+0000 7f3d37fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 0x55f4cb9c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.340+0000 7f3d37fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 0x55f4cb9c3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35394/0 (socket says 172.21.15.111:35394) 2023-11-28T13:00:22.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.340+0000 7f3d37fff700 1 -- 172.21.15.111:0/2684940529 learned_addr learned my addr 172.21.15.111:0/2684940529 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:22.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.341+0000 7f3d3cdef700 1 --2- 172.21.15.111:0/2684940529 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c3810 0x55f4cb999a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.341+0000 7f3d37fff700 1 -- 172.21.15.111:0/2684940529 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c3810 msgr2=0x55f4cb999a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:22.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.341+0000 7f3d3d5f0700 1 --2- 172.21.15.111:0/2684940529 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9df770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.341+0000 7f3d37fff700 1 --2- 172.21.15.111:0/2684940529 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c3810 0x55f4cb999a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.341+0000 7f3d37fff700 1 -- 172.21.15.111:0/2684940529 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 msgr2=0x55f4cb9df770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:22.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.341+0000 7f3d37fff700 1 --2- 172.21.15.111:0/2684940529 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9df770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.341+0000 7f3d37fff700 1 -- 172.21.15.111:0/2684940529 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4cb76d5e0 con 0x55f4cb9c1130 2023-11-28T13:00:22.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.341+0000 7f3d37fff700 1 --2- 172.21.15.111:0/2684940529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 0x55f4cb9c3be0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400a930 tx=0x7f3d2400ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f3712e0b49eb86 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:22.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.341+0000 7f3d227fc700 1 -- 172.21.15.111:0/2684940529 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2400f8d0 con 0x55f4cb9c1130 2023-11-28T13:00:22.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.341+0000 7f3d227fc700 1 -- 172.21.15.111:0/2684940529 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2400c030 con 0x55f4cb9c1130 2023-11-28T13:00:22.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.341+0000 7f3d227fc700 1 -- 172.21.15.111:0/2684940529 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2400fc30 con 0x55f4cb9c1130 2023-11-28T13:00:22.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.341+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/2684940529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 msgr2=0x55f4cb9c3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.341+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/2684940529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 0x55f4cb9c3be0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400a930 tx=0x7f3d2400ac40 comp rx=0 tx=0).stop 2023-11-28T13:00:22.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.342+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/2684940529 shutdown_connections 2023-11-28T13:00:22.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.342+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/2684940529 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9df770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.342+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/2684940529 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c3810 0x55f4cb999a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.342+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/2684940529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 0x55f4cb9c3be0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.342+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/2684940529 >> 172.21.15.111:0/2684940529 conn(0x55f4cb996d60 msgr2=0x55f4cb9976f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:22.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.342+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/2684940529 shutdown_connections 2023-11-28T13:00:22.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.342+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/2684940529 wait complete. 2023-11-28T13:00:22.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.342+0000 7f3d4b1dcb80 1 -- start start 2023-11-28T13:00:22.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.342+0000 7f3d4b1dcb80 1 Processor -- start 2023-11-28T13:00:22.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.343+0000 7f3d4b1dcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 0x55f4cb9b66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.343+0000 7f3d4b1dcb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c3810 0x55f4cb9b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.343+0000 7f3d4b1dcb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9c5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.343+0000 7f3d4b1dcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cb76f8d0 con 0x55f4cb9c1130 2023-11-28T13:00:22.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.343+0000 7f3d4b1dcb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cb9b2810 con 0x55f4cb9c4120 2023-11-28T13:00:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.343+0000 7f3d4b1dcb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f4cb9c25f0 con 0x55f4cb9c3810 2023-11-28T13:00:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.343+0000 7f3d4b1dcb80 10 client.0 _ll_register_callbacks cb 0x55f4cb791240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:00:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.343+0000 7f3d3cdef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 0x55f4cb9b66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.343+0000 7f3d3cdef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 0x55f4cb9b66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35398/0 (socket says 172.21.15.111:35398) 2023-11-28T13:00:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.343+0000 7f3d3cdef700 1 -- 172.21.15.111:0/382014600 learned_addr learned my addr 172.21.15.111:0/382014600 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.343+0000 7f3d37fff700 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c3810 0x55f4cb9b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.343+0000 7f3d3d5f0700 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9c5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d3cdef700 1 -- 172.21.15.111:0/382014600 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c3810 msgr2=0x55f4cb9b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d3cdef700 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c3810 0x55f4cb9b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d3cdef700 1 -- 172.21.15.111:0/382014600 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 msgr2=0x55f4cb9c5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d3cdef700 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9c5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d3cdef700 1 -- 172.21.15.111:0/382014600 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2400e040 con 0x55f4cb9c1130 2023-11-28T13:00:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d3d5f0700 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9c5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d3cdef700 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 0x55f4cb9b66d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0077b0 tx=0x7f3d2c001200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51198abf839805a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d177fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:00:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d177fe700 1 -- 172.21.15.111:0/382014600 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00e8b0 con 0x55f4cb9c1130 2023-11-28T13:00:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d177fe700 1 -- 172.21.15.111:0/382014600 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3d2c021d20 con 0x55f4cb9c1130 2023-11-28T13:00:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/382014600 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f4cb8b9140 con 0x55f4cb9c1130 2023-11-28T13:00:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d177fe700 1 -- 172.21.15.111:0/382014600 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d2c00ea10 con 0x55f4cb9c1130 2023-11-28T13:00:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d4b1dcb80 10 client.5553 fetch_fsmap learned FSMap version 17 2023-11-28T13:00:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/382014600 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55f4cb719d90 con 0x55f4cb9c1130 2023-11-28T13:00:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d177fe700 1 -- 172.21.15.111:0/382014600 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f4cb8b9140 con 0x55f4cb9c1130 2023-11-28T13:00:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d177fe700 1 -- 172.21.15.111:0/382014600 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 17) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f3d2c029020 con 0x55f4cb9c1130 2023-11-28T13:00:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d4b1dcb80 10 client.5553 fetch_fsmap finished waiting for FSMap version 17 2023-11-28T13:00:22.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d4b1dcb80 10 client.5553 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-28T13:00:22.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d4b1dcb80 20 client.5553 populate_metadata read hostname 'smithi111' 2023-11-28T13:00:22.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.149:6832/3579364312,v1:172.21.15.149:6834/3579364312] conn(0x55f4cb9c6230 0x55f4cb9c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d4b1dcb80 4 client.5553 mds_command: new command op to 5352 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-28T13:00:22.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.344+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/382014600 --> [v2:172.21.15.149:6832/3579364312,v1:172.21.15.149:6834/3579364312] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f4cb76ff10 con 0x55f4cb9c6230 2023-11-28T13:00:22.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.345+0000 7f3d37fff700 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.149:6832/3579364312,v1:172.21.15.149:6834/3579364312] conn(0x55f4cb9c6230 0x55f4cb9c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.345+0000 7f3d177fe700 10 client.5553 ms_handle_connect on v2:172.21.15.149:6832/3579364312 2023-11-28T13:00:22.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.345+0000 7f3d37fff700 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.149:6832/3579364312,v1:172.21.15.149:6834/3579364312] conn(0x55f4cb9c6230 0x55f4cb9c1c50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99536c2802782461 server_cookie=e81b155cf3c6841a in_seq=0 out_seq=0 2023-11-28T13:00:22.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d177fe700 1 -- 172.21.15.111:0/382014600 <== mds.0 v2:172.21.15.149:6832/3579364312 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f4cb76ff10 con 0x55f4cb9c6230 2023-11-28T13:00:22.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d177fe700 10 client.5553 handle_command_reply: tid=0 2023-11-28T13:00:22.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d4b1dcb80 1 client.5553 shutdown 2023-11-28T13:00:22.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/382014600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 msgr2=0x55f4cb9b66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 0x55f4cb9b66d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0077b0 tx=0x7f3d2c001200 comp rx=0 tx=0).stop 2023-11-28T13:00:22.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/382014600 shutdown_connections 2023-11-28T13:00:22.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9c5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c1130 0x55f4cb9b66d0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.149:6832/3579364312,v1:172.21.15.149:6834/3579364312] conn(0x55f4cb9c6230 0x55f4cb9c1c50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/382014600 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c3810 0x55f4cb9b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d177fe700 0 client.5553 ms_handle_reset on v2:172.21.15.149:6832/3579364312 2023-11-28T13:00:22.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/382014600 >> 172.21.15.111:0/382014600 conn(0x55f4cb996d60 msgr2=0x55f4cb998230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:22.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/382014600 shutdown_connections 2023-11-28T13:00:22.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/382014600 wait complete. 2023-11-28T13:00:22.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.346+0000 7f3d4b1dcb80 20 client.5553 trim_cache size 0 max 16384 2023-11-28T13:00:22.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.364+0000 7f3d4b1dcb80 1 Processor -- start 2023-11-28T13:00:22.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.364+0000 7f3d4b1dcb80 1 -- start start 2023-11-28T13:00:22.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.364+0000 7f3d4b1dcb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c1130 0x55f4cb994010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.364+0000 7f3d4b1dcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 0x55f4cb9c04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.364+0000 7f3d4b1dcb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9c0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d4b1dcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cb9c0f60 con 0x55f4cb9c3810 2023-11-28T13:00:22.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d4b1dcb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cb9b1b00 con 0x55f4cb9c4120 2023-11-28T13:00:22.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d4b1dcb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f4cb995490 con 0x55f4cb9c1130 2023-11-28T13:00:22.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d37fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 0x55f4cb9c04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d37fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 0x55f4cb9c04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35408/0 (socket says 172.21.15.111:35408) 2023-11-28T13:00:22.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d37fff700 1 -- 172.21.15.111:0/344888924 learned_addr learned my addr 172.21.15.111:0/344888924 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:22.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d3d5f0700 1 --2- 172.21.15.111:0/344888924 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9c0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d3cdef700 1 --2- 172.21.15.111:0/344888924 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c1130 0x55f4cb994010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d37fff700 1 -- 172.21.15.111:0/344888924 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c1130 msgr2=0x55f4cb994010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:22.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d37fff700 1 --2- 172.21.15.111:0/344888924 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c1130 0x55f4cb994010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d37fff700 1 -- 172.21.15.111:0/344888924 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 msgr2=0x55f4cb9c0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:22.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d37fff700 1 --2- 172.21.15.111:0/344888924 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9c0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d37fff700 1 -- 172.21.15.111:0/344888924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4cb8b2140 con 0x55f4cb9c3810 2023-11-28T13:00:22.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d37fff700 1 --2- 172.21.15.111:0/344888924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 0x55f4cb9c04e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400a820 tx=0x7f3d24014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5792c28be9fb40e5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:22.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d22ffd700 1 -- 172.21.15.111:0/344888924 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24022040 con 0x55f4cb9c3810 2023-11-28T13:00:22.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d22ffd700 1 -- 172.21.15.111:0/344888924 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d24024040 con 0x55f4cb9c3810 2023-11-28T13:00:22.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.365+0000 7f3d22ffd700 1 -- 172.21.15.111:0/344888924 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24029480 con 0x55f4cb9c3810 2023-11-28T13:00:22.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.366+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/344888924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 msgr2=0x55f4cb9c04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.366+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/344888924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 0x55f4cb9c04e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400a820 tx=0x7f3d24014000 comp rx=0 tx=0).stop 2023-11-28T13:00:22.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.366+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/344888924 shutdown_connections 2023-11-28T13:00:22.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.366+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/344888924 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9c0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.366+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/344888924 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c1130 0x55f4cb994010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.366+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/344888924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 0x55f4cb9c04e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.366+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/344888924 >> 172.21.15.111:0/344888924 conn(0x55f4cb996d60 msgr2=0x55f4cb9e23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:22.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.366+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/344888924 shutdown_connections 2023-11-28T13:00:22.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.366+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/344888924 wait complete. 2023-11-28T13:00:22.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.366+0000 7f3d4b1dcb80 1 -- start start 2023-11-28T13:00:22.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.367+0000 7f3d4b1dcb80 1 Processor -- start 2023-11-28T13:00:22.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.367+0000 7f3d4b1dcb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c1130 0x55f4cba46af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.367+0000 7f3d4b1dcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 0x55f4cb9c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.367+0000 7f3d4b1dcb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.367+0000 7f3d4b1dcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cb9bcdf0 con 0x55f4cb9c3810 2023-11-28T13:00:22.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.367+0000 7f3d4b1dcb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cba47550 con 0x55f4cb9c4120 2023-11-28T13:00:22.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.367+0000 7f3d4b1dcb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f4cb9c5230 con 0x55f4cb9c1130 2023-11-28T13:00:22.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.367+0000 7f3d4b1dcb80 10 client.0 _ll_register_callbacks cb 0x55f4cb791240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:00:22.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.367+0000 7f3d37fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 0x55f4cb9c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.367+0000 7f3d37fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 0x55f4cb9c4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35414/0 (socket says 172.21.15.111:35414) 2023-11-28T13:00:22.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.367+0000 7f3d37fff700 1 -- 172.21.15.111:0/3784922144 learned_addr learned my addr 172.21.15.111:0/3784922144 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:22.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.367+0000 7f3d3cdef700 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c1130 0x55f4cba46af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.367+0000 7f3d3d5f0700 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d37fff700 1 -- 172.21.15.111:0/3784922144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c1130 msgr2=0x55f4cba46af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:22.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d37fff700 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c1130 0x55f4cba46af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d37fff700 1 -- 172.21.15.111:0/3784922144 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 msgr2=0x55f4cb9b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:22.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d37fff700 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9b2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d37fff700 1 -- 172.21.15.111:0/3784922144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4cb76d5e0 con 0x55f4cb9c3810 2023-11-28T13:00:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d3cdef700 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c1130 0x55f4cba46af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d37fff700 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 0x55f4cb9c4cf0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3d24017040 tx=0x7f3d240047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ed54fbdf1a9daf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d177fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:00:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d177fe700 1 -- 172.21.15.111:0/3784922144 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24022040 con 0x55f4cb9c3810 2023-11-28T13:00:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d177fe700 1 -- 172.21.15.111:0/3784922144 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3d240299d0 con 0x55f4cb9c3810 2023-11-28T13:00:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/3784922144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f4cb8a9160 con 0x55f4cb9c3810 2023-11-28T13:00:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d177fe700 1 -- 172.21.15.111:0/3784922144 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d240045d0 con 0x55f4cb9c3810 2023-11-28T13:00:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d4b1dcb80 10 client.5559 fetch_fsmap learned FSMap version 17 2023-11-28T13:00:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/3784922144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55f4cb740200 con 0x55f4cb9c3810 2023-11-28T13:00:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d177fe700 1 -- 172.21.15.111:0/3784922144 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f4cb8a9160 con 0x55f4cb9c3810 2023-11-28T13:00:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d177fe700 1 -- 172.21.15.111:0/3784922144 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 17) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f3d24029bb0 con 0x55f4cb9c3810 2023-11-28T13:00:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d4b1dcb80 10 client.5559 fetch_fsmap finished waiting for FSMap version 17 2023-11-28T13:00:22.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d4b1dcb80 10 client.5559 resolve_mds: resolved name 'd' to daemon mds.d 2023-11-28T13:00:22.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d4b1dcb80 20 client.5559 populate_metadata read hostname 'smithi111' 2023-11-28T13:00:22.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.149:6832/3579364312,v1:172.21.15.149:6834/3579364312] conn(0x55f4cb992a10 0x55f4cb992de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d4b1dcb80 4 client.5559 mds_command: new command op to 5352 tid=0[{"prefix": "status"}] 2023-11-28T13:00:22.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.368+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/3784922144 --> [v2:172.21.15.149:6832/3579364312,v1:172.21.15.149:6834/3579364312] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f4cb893170 con 0x55f4cb992a10 2023-11-28T13:00:22.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d3cdef700 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.149:6832/3579364312,v1:172.21.15.149:6834/3579364312] conn(0x55f4cb992a10 0x55f4cb992de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d177fe700 10 client.5559 ms_handle_connect on v2:172.21.15.149:6832/3579364312 2023-11-28T13:00:22.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d3cdef700 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.149:6832/3579364312,v1:172.21.15.149:6834/3579364312] conn(0x55f4cb992a10 0x55f4cb992de0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79f055d13b64c862 server_cookie=1a1e2eaf7e9da9ac in_seq=0 out_seq=0 2023-11-28T13:00:22.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d177fe700 1 -- 172.21.15.111:0/3784922144 <== mds.0 v2:172.21.15.149:6832/3579364312 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+318 (crc 0 0 0) 0x55f4cb893170 con 0x55f4cb992a10 2023-11-28T13:00:22.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d177fe700 10 client.5559 handle_command_reply: tid=0 2023-11-28T13:00:22.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d4b1dcb80 1 client.5559 shutdown 2023-11-28T13:00:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/3784922144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 msgr2=0x55f4cb9c4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 0x55f4cb9c4cf0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3d24017040 tx=0x7f3d240047c0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/3784922144 shutdown_connections 2023-11-28T13:00:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f4cb9c4120 0x55f4cb9b2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.149:6832/3579364312,v1:172.21.15.149:6834/3579364312] conn(0x55f4cb992a10 0x55f4cb992de0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f4cb9c1130 0x55f4cba46af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d177fe700 0 client.5559 ms_handle_reset on v2:172.21.15.149:6832/3579364312 2023-11-28T13:00:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d4b1dcb80 1 --2- 172.21.15.111:0/3784922144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f4cb9c3810 0x55f4cb9c4cf0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/3784922144 >> 172.21.15.111:0/3784922144 conn(0x55f4cb996d60 msgr2=0x55f4cb9e23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/3784922144 shutdown_connections 2023-11-28T13:00:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.369+0000 7f3d4b1dcb80 1 -- 172.21.15.111:0/3784922144 wait complete. 2023-11-28T13:00:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.370+0000 7f3d4b1dcb80 20 client.5559 trim_cache size 0 max 16384 2023-11-28T13:00:22.378 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:00:22.378 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "a1b2d00c-aeec-4b5d-93e3-41820c4189ba", 2023-11-28T13:00:22.378 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-11-28T13:00:22.378 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 5352, 2023-11-28T13:00:22.378 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-11-28T13:00:22.378 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-11-28T13:00:22.378 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-11-28T13:00:22.378 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 10.355829581, 2023-11-28T13:00:22.379 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 17, 2023-11-28T13:00:22.379 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 41, 2023-11-28T13:00:22.379 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 41, 2023-11-28T13:00:22.379 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 18.15971892 2023-11-28T13:00:22.379 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:00:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.370+0000 7f3d3f053700 1 -- 172.21.15.111:0/478834582 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d1c0682f0 msgr2=0x7f3d1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.370+0000 7f3d3f053700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d1c0682f0 0x7f3d1c06a7a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0010e0 tx=0x7f3d2c009040 comp rx=0 tx=0).stop 2023-11-28T13:00:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.370+0000 7f3d3f053700 1 -- 172.21.15.111:0/478834582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d3815c7b0 msgr2=0x7f3d380b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.370+0000 7f3d3f053700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d3815c7b0 0x7f3d380b0bb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3d3806d8f0 tx=0x7f3d30004ad0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.370+0000 7f3d3f053700 1 -- 172.21.15.111:0/478834582 shutdown_connections 2023-11-28T13:00:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.370+0000 7f3d3f053700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d3815c7b0 0x7f3d380b0bb0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.370+0000 7f3d3f053700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d1c0682f0 0x7f3d1c06a7a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.370+0000 7f3d3f053700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d3806b380 0x7f3d380b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.370+0000 7f3d3f053700 1 --2- 172.21.15.111:0/478834582 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d380d0490 0x7f3d380b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.370+0000 7f3d3f053700 1 -- 172.21.15.111:0/478834582 >> 172.21.15.111:0/478834582 conn(0x7f3d3809a040 msgr2=0x7f3d380705c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.370+0000 7f3d3f053700 1 -- 172.21.15.111:0/478834582 shutdown_connections 2023-11-28T13:00:22.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.370+0000 7f3d3f053700 1 -- 172.21.15.111:0/478834582 wait complete. 2023-11-28T13:00:22.382 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T13:00:22.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.501+0000 7fee70a92700 1 Processor -- start 2023-11-28T13:00:22.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.501+0000 7fee70a92700 1 -- start start 2023-11-28T13:00:22.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.502+0000 7fee70a92700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fee6c15c6c0 0x7fee6c0afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.502+0000 7fee70a92700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fee6c0b0440 0x7fee6c0b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.502+0000 7fee70a92700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee6c1602a0 0x7fee6c160720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.502+0000 7fee70a92700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6c0b0e20 con 0x7fee6c0b0440 2023-11-28T13:00:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.502+0000 7fee70a92700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6c0cc750 con 0x7fee6c15c6c0 2023-11-28T13:00:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.502+0000 7fee70a92700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c0cca30 con 0x7fee6c1602a0 2023-11-28T13:00:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.502+0000 7fee69d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fee6c0b0440 0x7fee6c0b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.503+0000 7fee69d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fee6c0b0440 0x7fee6c0b08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35420/0 (socket says 172.21.15.111:35420) 2023-11-28T13:00:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.503+0000 7fee69d9b700 1 -- 172.21.15.111:0/4262396996 learned_addr learned my addr 172.21.15.111:0/4262396996 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.503+0000 7fee6ad9d700 1 --2- 172.21.15.111:0/4262396996 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee6c1602a0 0x7fee6c160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.503+0000 7fee6a59c700 1 --2- 172.21.15.111:0/4262396996 >> v2:172.21.15.149:3300/0 conn(0x7fee6c15c6c0 0x7fee6c0afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.503+0000 7fee69d9b700 1 -- 172.21.15.111:0/4262396996 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee6c1602a0 msgr2=0x7fee6c160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.503+0000 7fee69d9b700 1 --2- 172.21.15.111:0/4262396996 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee6c1602a0 0x7fee6c160720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.503+0000 7fee69d9b700 1 -- 172.21.15.111:0/4262396996 >> v2:172.21.15.149:3300/0 conn(0x7fee6c15c6c0 msgr2=0x7fee6c0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.503+0000 7fee69d9b700 1 --2- 172.21.15.111:0/4262396996 >> v2:172.21.15.149:3300/0 conn(0x7fee6c15c6c0 0x7fee6c0afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.503+0000 7fee69d9b700 1 -- 172.21.15.111:0/4262396996 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c0ccd10 con 0x7fee6c0b0440 2023-11-28T13:00:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.503+0000 7fee69d9b700 1 --2- 172.21.15.111:0/4262396996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee6c0b0440 0x7fee6c0b08e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fee6c071950 tx=0x7fee5c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34afd1a03318733d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.503+0000 7fee6959a700 1 -- 172.21.15.111:0/4262396996 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c00b870 con 0x7fee6c0b0440 2023-11-28T13:00:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.503+0000 7fee6959a700 1 -- 172.21.15.111:0/4262396996 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5c00b9d0 con 0x7fee6c0b0440 2023-11-28T13:00:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.503+0000 7fee6959a700 1 -- 172.21.15.111:0/4262396996 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5c00bd60 con 0x7fee6c0b0440 2023-11-28T13:00:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.504+0000 7fee70a92700 1 -- 172.21.15.111:0/4262396996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee6c0b0440 msgr2=0x7fee6c0b08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.504+0000 7fee70a92700 1 --2- 172.21.15.111:0/4262396996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee6c0b0440 0x7fee6c0b08e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fee6c071950 tx=0x7fee5c009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.504+0000 7fee70a92700 1 -- 172.21.15.111:0/4262396996 shutdown_connections 2023-11-28T13:00:22.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.504+0000 7fee70a92700 1 --2- 172.21.15.111:0/4262396996 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee6c1602a0 0x7fee6c160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.504+0000 7fee70a92700 1 --2- 172.21.15.111:0/4262396996 >> v2:172.21.15.149:3300/0 conn(0x7fee6c15c6c0 0x7fee6c0afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.504+0000 7fee70a92700 1 --2- 172.21.15.111:0/4262396996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee6c0b0440 0x7fee6c0b08e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.504+0000 7fee70a92700 1 -- 172.21.15.111:0/4262396996 >> 172.21.15.111:0/4262396996 conn(0x7fee6c098ee0 msgr2=0x7fee6c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:22.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.504+0000 7fee70a92700 1 -- 172.21.15.111:0/4262396996 shutdown_connections 2023-11-28T13:00:22.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.504+0000 7fee70a92700 1 -- 172.21.15.111:0/4262396996 wait complete. 2023-11-28T13:00:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.505+0000 7fee70a92700 1 Processor -- start 2023-11-28T13:00:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.505+0000 7fee70a92700 1 -- start start 2023-11-28T13:00:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee70a92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee6c0b0440 0x7fee6c0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee70a92700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee6c15c6c0 0x7fee6c0ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee70a92700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee6c1602a0 0x7fee6c1548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee70a92700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c0c3700 con 0x7fee6c0b0440 2023-11-28T13:00:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee70a92700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c0c3950 con 0x7fee6c15c6c0 2023-11-28T13:00:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee70a92700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c0c3c30 con 0x7fee6c1602a0 2023-11-28T13:00:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee6a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee6c0b0440 0x7fee6c0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee6a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee6c0b0440 0x7fee6c0ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35426/0 (socket says 172.21.15.111:35426) 2023-11-28T13:00:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee6a59c700 1 -- 172.21.15.111:0/3094264483 learned_addr learned my addr 172.21.15.111:0/3094264483 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee69d9b700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee6c15c6c0 0x7fee6c0ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee6ad9d700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee6c1602a0 0x7fee6c1548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee6a59c700 1 -- 172.21.15.111:0/3094264483 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee6c1602a0 msgr2=0x7fee6c1548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee6a59c700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee6c1602a0 0x7fee6c1548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee6a59c700 1 -- 172.21.15.111:0/3094264483 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee6c15c6c0 msgr2=0x7fee6c0ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee6a59c700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee6c15c6c0 0x7fee6c0ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee6a59c700 1 -- 172.21.15.111:0/3094264483 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c009550 con 0x7fee6c0b0440 2023-11-28T13:00:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee6ad9d700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee6c1602a0 0x7fee6c1548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee69d9b700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee6c15c6c0 0x7fee6c0ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.506+0000 7fee6a59c700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee6c0b0440 0x7fee6c0ce590 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fee6000d800 tx=0x7fee6000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:22.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.507+0000 7fee5affd700 1 -- 172.21.15.111:0/3094264483 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee600125f0 con 0x7fee6c0b0440 2023-11-28T13:00:22.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.507+0000 7fee5affd700 1 -- 172.21.15.111:0/3094264483 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee60012750 con 0x7fee6c0b0440 2023-11-28T13:00:22.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.507+0000 7fee70a92700 1 -- 172.21.15.111:0/3094264483 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6c0cd2d0 con 0x7fee6c0b0440 2023-11-28T13:00:22.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.507+0000 7fee5affd700 1 -- 172.21.15.111:0/3094264483 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60012b00 con 0x7fee6c0b0440 2023-11-28T13:00:22.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.507+0000 7fee70a92700 1 -- 172.21.15.111:0/3094264483 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6c0ccff0 con 0x7fee6c0b0440 2023-11-28T13:00:22.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.508+0000 7fee70a92700 1 -- 172.21.15.111:0/3094264483 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6c15ca90 con 0x7fee6c0b0440 2023-11-28T13:00:22.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.508+0000 7fee5affd700 1 -- 172.21.15.111:0/3094264483 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fee60012c60 con 0x7fee6c0b0440 2023-11-28T13:00:22.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.508+0000 7fee5affd700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fee4c0682f0 0x7fee4c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:22.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.508+0000 7fee5affd700 1 -- 172.21.15.111:0/3094264483 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fee600862f0 con 0x7fee6c0b0440 2023-11-28T13:00:22.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.508+0000 7fee69d9b700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fee4c0682f0 0x7fee4c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:22.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.509+0000 7fee69d9b700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fee4c0682f0 0x7fee4c06a7a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fee5c014040 tx=0x7fee5c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:22.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.512+0000 7fee5affd700 1 -- 172.21.15.111:0/3094264483 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fee60052a10 con 0x7fee6c0b0440 2023-11-28T13:00:22.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.740+0000 7fee70a92700 1 -- 172.21.15.111:0/3094264483 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fee6c0cdf80 con 0x7fee6c0b0440 2023-11-28T13:00:22.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.740+0000 7fee5affd700 1 -- 172.21.15.111:0/3094264483 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7fee60056030 con 0x7fee6c0b0440 2023-11-28T13:00:22.741 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T13:00:22.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.742+0000 7fee70a92700 1 -- 172.21.15.111:0/3094264483 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fee4c0682f0 msgr2=0x7fee4c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.742+0000 7fee70a92700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fee4c0682f0 0x7fee4c06a7a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fee5c014040 tx=0x7fee5c008030 comp rx=0 tx=0).stop 2023-11-28T13:00:22.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.742+0000 7fee70a92700 1 -- 172.21.15.111:0/3094264483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee6c0b0440 msgr2=0x7fee6c0ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:22.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.742+0000 7fee70a92700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee6c0b0440 0x7fee6c0ce590 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fee6000d800 tx=0x7fee6000dbc0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.742+0000 7fee70a92700 1 -- 172.21.15.111:0/3094264483 shutdown_connections 2023-11-28T13:00:22.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.742+0000 7fee70a92700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fee6c1602a0 0x7fee6c1548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.742+0000 7fee70a92700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fee6c0b0440 0x7fee6c0ce590 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.742+0000 7fee70a92700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fee4c0682f0 0x7fee4c06a7a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.742+0000 7fee70a92700 1 --2- 172.21.15.111:0/3094264483 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fee6c15c6c0 0x7fee6c0ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:22.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.742+0000 7fee70a92700 1 -- 172.21.15.111:0/3094264483 >> 172.21.15.111:0/3094264483 conn(0x7fee6c098ee0 msgr2=0x7fee6c0af700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:22.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.742+0000 7fee70a92700 1 -- 172.21.15.111:0/3094264483 shutdown_connections 2023-11-28T13:00:22.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:22.742+0000 7fee70a92700 1 -- 172.21.15.111:0/3094264483 wait complete. 2023-11-28T13:00:22.755 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T13:00:22.755 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T13:00:22.755 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-11-28T13:00:22.755 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T13:00:22.755 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T13:00:22.755 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T13:00:22.755 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T13:00:22.755 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T13:00:22.755 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T13:00:22.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:22.756 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-11-28T13:00:22.772 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T13:00:22.773 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T13:00:22.773 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T13:00:22.773 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:00:22.773 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-11-28T13:00:22.773 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:00:22.773 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T13:00:22.773 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-11-28T13:00:22.773 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T13:00:22.773 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-11-28T13:00:22.773 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T13:00:22.773 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-11-28T13:00:22.774 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T13:00:22.774 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-11-28T13:00:22.774 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T13:00:22.774 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 42303sec preferred_lft 42303sec 2023-11-28T13:00:22.774 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::9195:85a0:8e3c:2da4/64 scope link noprefixroute 2023-11-28T13:00:22.774 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:00:22.774 INFO:teuthology.orchestra.run.smithi111.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T13:00:22.774 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 96:9a:a2:c9:67:8f brd ff:ff:ff:ff:ff:ff 2023-11-28T13:00:22.774 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T13:00:22.774 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:00:22.774 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::f893:44ff:fe17:3abf/64 scope link 2023-11-28T13:00:22.775 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:00:22.775 INFO:teuthology.orchestra.run.smithi111.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T13:00:22.775 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 96:9a:a2:c9:67:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T13:00:22.775 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::949a:a2ff:fec9:678f/64 scope link 2023-11-28T13:00:22.775 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:00:22.775 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T13:00:22.775 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ee:e6:45:35:2f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T13:00:22.775 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ece6:45ff:fe35:2fc2/64 scope link 2023-11-28T13:00:22.775 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:00:22.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:22.776 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T13:00:22.838 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-11-28T13:00:22.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:22.840 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-11-28T13:00:22.899 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-11-28T13:00:22.899 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T13:00:22.899 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T13:00:22.900 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T13:00:22.900 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T13:00:22.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:22.900 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T13:00:22.900 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T13:00:22.900 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T13:00:22.900 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T13:00:22.900 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T13:00:22.900 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T13:00:23.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:23.003 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-11-28T13:00:23.020 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T13:00:23.083 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T13:00:23.083 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T13:00:23.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:23.084 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-11-28T13:00:23.111 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T13:00:23.111 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T13:00:23.168 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T13:00:23.168 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T13:00:23.168 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T13:00:23.224 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T13:00:23.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:23.225 DEBUG:teuthology.orchestra.run.smithi111:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T13:00:23.311 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T13:00:23.311 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T13:00:23.312 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T13:00:23.320 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T13:00:23.320 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T13:00:23.321 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T13:00:23.321 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T13:00:23.321 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T13:00:23.346 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T13:00:23.346 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T13:00:23.346 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T13:00:23.347 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T13:00:23.347 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T13:00:23.347 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790" 2023-11-28T13:00:23.347 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790:/ 2023-11-28T13:00:23.347 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=a1b2d00c-aeec-4b5d-93e3-41820c4189ba" will pass to kernel 2023-11-28T13:00:23.350 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T13:00:23.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:23.350 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T13:00:23.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:23.382 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T13:00:23.470 INFO:teuthology.orchestra.run.smithi111.stdout:client5568 2023-11-28T13:00:23.470 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:00:23.470 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:00:23.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:23.488 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir) 2023-11-28T13:00:23.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:23.551 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir/childdir) 2023-11-28T13:00:23.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:23.609 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch rfile) 2023-11-28T13:00:23.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:23.668 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/pfile) 2023-11-28T13:00:23.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:23.732 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/childdir/cfile) 2023-11-28T13:00:23.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:23.789 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2023-11-28T13:00:23.851 INFO:teuthology.orchestra.run.smithi111.stdout:./ 2023-11-28T13:00:23.851 INFO:teuthology.orchestra.run.smithi111.stdout:./rfile 2023-11-28T13:00:23.851 INFO:teuthology.orchestra.run.smithi111.stdout:./parentdir 2023-11-28T13:00:23.851 INFO:teuthology.orchestra.run.smithi111.stdout:./parentdir/pfile 2023-11-28T13:00:23.852 INFO:teuthology.orchestra.run.smithi111.stdout:./parentdir/childdir 2023-11-28T13:00:23.852 INFO:teuthology.orchestra.run.smithi111.stdout:./parentdir/childdir/cfile 2023-11-28T13:00:23.852 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-28T13:00:23.852 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-11-28T13:00:23.853 DEBUG:teuthology.orchestra.run.smithi111:> import stat 2023-11-28T13:00:23.853 DEBUG:teuthology.orchestra.run.smithi111:> 2023-11-28T13:00:23.853 DEBUG:teuthology.orchestra.run.smithi111:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2023-11-28T13:00:23.853 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-11-28T13:00:23.957 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-11-28T13:00:24.146 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-28T13:00:24.146 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-11-28T13:00:24.146 DEBUG:teuthology.orchestra.run.smithi111:> import stat 2023-11-28T13:00:24.146 DEBUG:teuthology.orchestra.run.smithi111:> 2023-11-28T13:00:24.147 DEBUG:teuthology.orchestra.run.smithi111:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./rfile").st_ino) 2023-11-28T13:00:24.147 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-11-28T13:00:24.254 INFO:teuthology.orchestra.run.smithi111.stdout:1099511627778 2023-11-28T13:00:24.442 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-28T13:00:24.443 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-11-28T13:00:24.443 DEBUG:teuthology.orchestra.run.smithi111:> import stat 2023-11-28T13:00:24.443 DEBUG:teuthology.orchestra.run.smithi111:> 2023-11-28T13:00:24.443 DEBUG:teuthology.orchestra.run.smithi111:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir").st_ino) 2023-11-28T13:00:24.443 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-11-28T13:00:24.548 INFO:teuthology.orchestra.run.smithi111.stdout:1099511627776 2023-11-28T13:00:24.738 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-28T13:00:24.738 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-11-28T13:00:24.738 DEBUG:teuthology.orchestra.run.smithi111:> import stat 2023-11-28T13:00:24.738 DEBUG:teuthology.orchestra.run.smithi111:> 2023-11-28T13:00:24.739 DEBUG:teuthology.orchestra.run.smithi111:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/pfile").st_ino) 2023-11-28T13:00:24.739 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-11-28T13:00:24.844 INFO:teuthology.orchestra.run.smithi111.stdout:1099511628279 2023-11-28T13:00:25.033 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-28T13:00:25.033 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-11-28T13:00:25.033 DEBUG:teuthology.orchestra.run.smithi111:> import stat 2023-11-28T13:00:25.033 DEBUG:teuthology.orchestra.run.smithi111:> 2023-11-28T13:00:25.033 DEBUG:teuthology.orchestra.run.smithi111:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir").st_ino) 2023-11-28T13:00:25.033 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-11-28T13:00:25.139 INFO:teuthology.orchestra.run.smithi111.stdout:1099511627777 2023-11-28T13:00:25.328 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-28T13:00:25.328 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-11-28T13:00:25.328 DEBUG:teuthology.orchestra.run.smithi111:> import stat 2023-11-28T13:00:25.328 DEBUG:teuthology.orchestra.run.smithi111:> 2023-11-28T13:00:25.328 DEBUG:teuthology.orchestra.run.smithi111:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir/cfile").st_ino) 2023-11-28T13:00:25.329 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-11-28T13:00:25.434 INFO:teuthology.orchestra.run.smithi111.stdout:1099511628280 2023-11-28T13:00:25.625 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:00:25.625 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:00:25.682 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:00:25.682 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:00:25.741 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:00:25.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.901+0000 7f1573ed8700 1 Processor -- start 2023-11-28T13:00:25.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.901+0000 7f1573ed8700 1 -- start start 2023-11-28T13:00:25.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.902+0000 7f1573ed8700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f156c158cb0 0x7f156c0ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:25.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.902+0000 7f1573ed8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f156c071020 0x7f156c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:25.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.902+0000 7f1573ed8700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156c06d870 con 0x7f156c159080 2023-11-28T13:00:25.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.902+0000 7f1573ed8700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156c06cc60 con 0x7f156c158cb0 2023-11-28T13:00:25.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.902+0000 7f1573ed8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156c06cf40 con 0x7f156c071020 2023-11-28T13:00:25.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.902+0000 7f1572475700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f156c071020 0x7f156c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.902+0000 7f1571c74700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f156c158cb0 0x7f156c0ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.902+0000 7f1571473700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f156c159080 0x7f156c070910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:44758/0 (socket says 172.21.15.111:44758) 2023-11-28T13:00:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.902+0000 7f1571473700 1 -- 172.21.15.111:0/2331112511 learned_addr learned my addr 172.21.15.111:0/2331112511 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.903+0000 7f1570c72700 1 -- 172.21.15.111:0/2331112511 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602585991 0 0) 0x7f156c06d870 con 0x7f156c159080 2023-11-28T13:00:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.903+0000 7f1570c72700 1 -- 172.21.15.111:0/2331112511 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15580035b0 con 0x7f156c159080 2023-11-28T13:00:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.903+0000 7f1572475700 1 -- 172.21.15.111:0/2331112511 >> v2:172.21.15.149:3300/0 conn(0x7f156c158cb0 msgr2=0x7f156c0ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.903+0000 7f1572475700 1 --2- 172.21.15.111:0/2331112511 >> v2:172.21.15.149:3300/0 conn(0x7f156c158cb0 0x7f156c0ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.903+0000 7f1572475700 1 -- 172.21.15.111:0/2331112511 >> v1:172.21.15.149:6789/0 conn(0x7f156c159080 legacy=0x7f156c070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.903+0000 7f1572475700 1 -- 172.21.15.111:0/2331112511 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c06d220 con 0x7f156c071020 2023-11-28T13:00:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.903+0000 7f1571c74700 1 --2- 172.21.15.111:0/2331112511 >> v2:172.21.15.149:3300/0 conn(0x7f156c158cb0 0x7f156c0ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:25.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.903+0000 7f1572475700 1 --2- 172.21.15.111:0/2331112511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f156c071020 0x7f156c06dd00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1564009d90 tx=0x7f156400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8d1e430d474da87 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.903+0000 7f1570c72700 1 -- 172.21.15.111:0/2331112511 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156400e070 con 0x7f156c071020 2023-11-28T13:00:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.903+0000 7f1570c72700 1 -- 172.21.15.111:0/2331112511 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156400bb20 con 0x7f156c071020 2023-11-28T13:00:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.903+0000 7f1570c72700 1 -- 172.21.15.111:0/2331112511 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156400be90 con 0x7f156c071020 2023-11-28T13:00:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.904+0000 7f1573ed8700 1 -- 172.21.15.111:0/2331112511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f156c071020 msgr2=0x7f156c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.904+0000 7f1573ed8700 1 --2- 172.21.15.111:0/2331112511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f156c071020 0x7f156c06dd00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1564009d90 tx=0x7f156400b330 comp rx=0 tx=0).stop 2023-11-28T13:00:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.904+0000 7f1573ed8700 1 -- 172.21.15.111:0/2331112511 shutdown_connections 2023-11-28T13:00:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.904+0000 7f1573ed8700 1 --2- 172.21.15.111:0/2331112511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f156c071020 0x7f156c06dd00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.904+0000 7f1573ed8700 1 --2- 172.21.15.111:0/2331112511 >> v2:172.21.15.149:3300/0 conn(0x7f156c158cb0 0x7f156c0ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.904+0000 7f1573ed8700 1 -- 172.21.15.111:0/2331112511 >> 172.21.15.111:0/2331112511 conn(0x7f156c0cb490 msgr2=0x7f156c0cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:25.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.904+0000 7f1573ed8700 1 -- 172.21.15.111:0/2331112511 shutdown_connections 2023-11-28T13:00:25.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.904+0000 7f1573ed8700 1 -- 172.21.15.111:0/2331112511 wait complete. 2023-11-28T13:00:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.905+0000 7f1573ed8700 1 Processor -- start 2023-11-28T13:00:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.905+0000 7f1573ed8700 1 -- start start 2023-11-28T13:00:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1573ed8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f156c071020 0x7f156c0c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1573ed8700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f156c158cb0 0x7f156c0c6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1573ed8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f156c1568b0 0x7f156c1e2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1573ed8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156c0c3c80 con 0x7f156c071020 2023-11-28T13:00:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1573ed8700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156c0c6ac0 con 0x7f156c158cb0 2023-11-28T13:00:25.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1573ed8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156c0c3930 con 0x7f156c1568b0 2023-11-28T13:00:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1571c74700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f156c071020 0x7f156c0c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1571c74700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f156c071020 0x7f156c0c5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35444/0 (socket says 172.21.15.111:35444) 2023-11-28T13:00:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1571c74700 1 -- 172.21.15.111:0/3979382274 learned_addr learned my addr 172.21.15.111:0/3979382274 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1572475700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f156c1568b0 0x7f156c1e2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1571473700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f156c158cb0 0x7f156c0c6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1571c74700 1 -- 172.21.15.111:0/3979382274 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f156c1568b0 msgr2=0x7f156c1e2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1571c74700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f156c1568b0 0x7f156c1e2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1571c74700 1 -- 172.21.15.111:0/3979382274 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f156c158cb0 msgr2=0x7f156c0c6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1571c74700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f156c158cb0 0x7f156c0c6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1571c74700 1 -- 172.21.15.111:0/3979382274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c00a640 con 0x7f156c071020 2023-11-28T13:00:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1571c74700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f156c071020 0x7f156c0c5a50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f155c00fc00 tx=0x7f155c00d260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f15627fc700 1 -- 172.21.15.111:0/3979382274 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c00dce0 con 0x7f156c071020 2023-11-28T13:00:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f15627fc700 1 -- 172.21.15.111:0/3979382274 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155c00de40 con 0x7f156c071020 2023-11-28T13:00:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f1573ed8700 1 -- 172.21.15.111:0/3979382274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1564009a70 con 0x7f156c071020 2023-11-28T13:00:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.906+0000 7f15627fc700 1 -- 172.21.15.111:0/3979382274 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c012680 con 0x7f156c071020 2023-11-28T13:00:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.907+0000 7f1573ed8700 1 -- 172.21.15.111:0/3979382274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f156c071b20 con 0x7f156c071020 2023-11-28T13:00:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.907+0000 7f1573ed8700 1 -- 172.21.15.111:0/3979382274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f156c06d500 con 0x7f156c071020 2023-11-28T13:00:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.908+0000 7f15627fc700 1 -- 172.21.15.111:0/3979382274 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f155c00e040 con 0x7f156c071020 2023-11-28T13:00:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.909+0000 7f15627fc700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f15500682f0 0x7f155006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.909+0000 7f15627fc700 1 -- 172.21.15.111:0/3979382274 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f155c087720 con 0x7f156c071020 2023-11-28T13:00:25.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.909+0000 7f1571473700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f15500682f0 0x7f155006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:25.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.909+0000 7f1571473700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f15500682f0 0x7f155006a7a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f156c1f13c0 tx=0x7f1568009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:25.912+0000 7f15627fc700 1 -- 172.21.15.111:0/3979382274 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f155c0535e0 con 0x7f156c071020 2023-11-28T13:00:26.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.095+0000 7f1573ed8700 1 -- 172.21.15.111:0/3979382274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f156c0c3f60 con 0x7f156c071020 2023-11-28T13:00:26.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.096+0000 7f15627fc700 1 -- 172.21.15.111:0/3979382274 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+18176 (secure 0 0 0) 0x7f155c053dd0 con 0x7f156c071020 2023-11-28T13:00:26.096 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:26.097 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":41,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:13.013725+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:09.793782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:10.979525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:00:26.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.098+0000 7f1573ed8700 1 -- 172.21.15.111:0/3979382274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f15500682f0 msgr2=0x7f155006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.098+0000 7f1573ed8700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f15500682f0 0x7f155006a7a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f156c1f13c0 tx=0x7f1568009040 comp rx=0 tx=0).stop 2023-11-28T13:00:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.098+0000 7f1573ed8700 1 -- 172.21.15.111:0/3979382274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f156c071020 msgr2=0x7f156c0c5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.098+0000 7f1573ed8700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f156c071020 0x7f156c0c5a50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f155c00fc00 tx=0x7f155c00d260 comp rx=0 tx=0).stop 2023-11-28T13:00:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.098+0000 7f1573ed8700 1 -- 172.21.15.111:0/3979382274 shutdown_connections 2023-11-28T13:00:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.098+0000 7f1573ed8700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f156c1568b0 0x7f156c1e2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.098+0000 7f1573ed8700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f156c071020 0x7f156c0c5a50 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.098+0000 7f1573ed8700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f15500682f0 0x7f155006a7a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.098+0000 7f1573ed8700 1 --2- 172.21.15.111:0/3979382274 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f156c158cb0 0x7f156c0c6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.098+0000 7f1573ed8700 1 -- 172.21.15.111:0/3979382274 >> 172.21.15.111:0/3979382274 conn(0x7f156c0cb490 msgr2=0x7f156c157e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.098+0000 7f1573ed8700 1 -- 172.21.15.111:0/3979382274 shutdown_connections 2023-11-28T13:00:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.098+0000 7f1573ed8700 1 -- 172.21.15.111:0/3979382274 wait complete. 2023-11-28T13:00:26.111 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T13:00:09.793782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T13:00:10.979525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T13:00:26.111 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T13:00:26.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.233+0000 7f149e3ef700 1 Processor -- start 2023-11-28T13:00:26.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.233+0000 7f149e3ef700 1 -- start start 2023-11-28T13:00:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.234+0000 7f149e3ef700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1498072460 0x7f149806b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.234+0000 7f149e3ef700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f149806b610 con 0x7f1498159440 2023-11-28T13:00:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.234+0000 7f149e3ef700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14980cb830 con 0x7f14980712e0 2023-11-28T13:00:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.234+0000 7f149e3ef700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149806a310 con 0x7f1498072460 2023-11-28T13:00:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.234+0000 7f1497fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1498159440 0x7f14980cafb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36230/0 (socket says 172.21.15.111:36230) 2023-11-28T13:00:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.234+0000 7f1497fff700 1 -- 172.21.15.111:0/3943281195 learned_addr learned my addr 172.21.15.111:0/3943281195 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.234+0000 7f14977fe700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1498072460 0x7f149806b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.234+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076484389 0 0) 0x7f1488001110 con 0x7f1498159440 2023-11-28T13:00:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14800035b0 con 0x7f1498159440 2023-11-28T13:00:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826385349 0 0) 0x7f14980cb830 con 0x7f14980712e0 2023-11-28T13:00:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1488001110 con 0x7f14980712e0 2023-11-28T13:00:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2385442245 0 0) 0x7f14800035b0 con 0x7f1498159440 2023-11-28T13:00:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1498072460 msgr2=0x7f149806b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1498072460 0x7f149806b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 >> v1:172.21.15.149:6789/0 conn(0x7f14980712e0 legacy=0x7f14980cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149806a5f0 con 0x7f1498159440 2023-11-28T13:00:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f1488001800 con 0x7f1498159440 2023-11-28T13:00:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 >> v1:172.21.15.111:6789/0 conn(0x7f1498159440 legacy=0x7f14980cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1498072460 0x7f1480003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1480004120 0x7f1480004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1480004d90 0x7f1480005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149806a310 con 0x7f1480004120 2023-11-28T13:00:26.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148000bac0 con 0x7f1480004d90 2023-11-28T13:00:26.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148000bc80 con 0x7f1498072460 2023-11-28T13:00:26.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f14977fe700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1480004120 0x7f1480004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1497fff700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1498072460 0x7f1480003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f149c98c700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1480004d90 0x7f1480005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f14977fe700 1 -- 172.21.15.111:0/3943281195 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1498072460 msgr2=0x7f1480003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:26.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f14977fe700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1498072460 0x7f1480003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f14977fe700 1 -- 172.21.15.111:0/3943281195 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1480004d90 msgr2=0x7f1480005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:26.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f14977fe700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1480004d90 0x7f1480005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f14977fe700 1 -- 172.21.15.111:0/3943281195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14800035b0 con 0x7f1480004120 2023-11-28T13:00:26.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f149c98c700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1480004d90 0x7f1480005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:26.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1497fff700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1498072460 0x7f1480003bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:26.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f14977fe700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1480004120 0x7f1480004850 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f148c009b30 tx=0x7f1480005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0991174a066a83 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:26.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c00e6c0 con 0x7f1480004120 2023-11-28T13:00:26.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148c00e820 con 0x7f1480004120 2023-11-28T13:00:26.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.235+0000 7f1496ffd700 1 -- 172.21.15.111:0/3943281195 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c00eb90 con 0x7f1480004120 2023-11-28T13:00:26.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.236+0000 7f149e3ef700 1 -- 172.21.15.111:0/3943281195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1480004120 msgr2=0x7f1480004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.236+0000 7f149e3ef700 1 --2- 172.21.15.111:0/3943281195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1480004120 0x7f1480004850 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f148c009b30 tx=0x7f1480005880 comp rx=0 tx=0).stop 2023-11-28T13:00:26.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.236+0000 7f149c98c700 1 -- 172.21.15.111:0/3943281195 reap_dead start 2023-11-28T13:00:26.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.236+0000 7f149e3ef700 1 -- 172.21.15.111:0/3943281195 shutdown_connections 2023-11-28T13:00:26.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.236+0000 7f149e3ef700 1 -- 172.21.15.111:0/3943281195 >> 172.21.15.111:0/3943281195 conn(0x7f14980b0db0 msgr2=0x7f14980b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:26.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.236+0000 7f149e3ef700 1 -- 172.21.15.111:0/3943281195 shutdown_connections 2023-11-28T13:00:26.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.236+0000 7f149e3ef700 1 -- 172.21.15.111:0/3943281195 wait complete. 2023-11-28T13:00:26.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.237+0000 7f149e3ef700 1 Processor -- start 2023-11-28T13:00:26.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.237+0000 7f149e3ef700 1 -- start start 2023-11-28T13:00:26.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.238+0000 7f149e3ef700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1480004120 0x7f14981487f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.238+0000 7f149e3ef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14980c51b0 0x7f14980c5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.238+0000 7f149e3ef700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f14980c2df0 0x7f14980c3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.238+0000 7f149e3ef700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14980c46f0 con 0x7f14980c51b0 2023-11-28T13:00:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.238+0000 7f149e3ef700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14980c5b60 con 0x7f14980c2df0 2023-11-28T13:00:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.238+0000 7f149e3ef700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14980c3ae0 con 0x7f1480004120 2023-11-28T13:00:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.238+0000 7f14977fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14980c51b0 0x7f14980c5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.238+0000 7f14977fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14980c51b0 0x7f14980c5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35466/0 (socket says 172.21.15.111:35466) 2023-11-28T13:00:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.238+0000 7f14977fe700 1 -- 172.21.15.111:0/3453151110 learned_addr learned my addr 172.21.15.111:0/3453151110 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.238+0000 7f1497fff700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1480004120 0x7f14981487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.238+0000 7f149c98c700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f14980c2df0 0x7f14980c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.239+0000 7f14977fe700 1 -- 172.21.15.111:0/3453151110 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1480004120 msgr2=0x7f14981487f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.239+0000 7f14977fe700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1480004120 0x7f14981487f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.239+0000 7f14977fe700 1 -- 172.21.15.111:0/3453151110 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f14980c2df0 msgr2=0x7f14980c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.239+0000 7f14977fe700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f14980c2df0 0x7f14980c3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.239+0000 7f14977fe700 1 -- 172.21.15.111:0/3453151110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149806abb0 con 0x7f14980c51b0 2023-11-28T13:00:26.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.239+0000 7f1497fff700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1480004120 0x7f14981487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:26.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.239+0000 7f149c98c700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f14980c2df0 0x7f14980c3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:26.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.239+0000 7f14977fe700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14980c51b0 0x7f14980c5620 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f14800047d0 tx=0x7f148c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:26.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.239+0000 7f1494ff9700 1 -- 172.21.15.111:0/3453151110 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c021070 con 0x7f14980c51b0 2023-11-28T13:00:26.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.239+0000 7f1494ff9700 1 -- 172.21.15.111:0/3453151110 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148c0045d0 con 0x7f14980c51b0 2023-11-28T13:00:26.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.239+0000 7f1494ff9700 1 -- 172.21.15.111:0/3453151110 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c00e6c0 con 0x7f14980c51b0 2023-11-28T13:00:26.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.239+0000 7f149e3ef700 1 -- 172.21.15.111:0/3453151110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149806a8d0 con 0x7f14980c51b0 2023-11-28T13:00:26.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.239+0000 7f149e3ef700 1 -- 172.21.15.111:0/3453151110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1498159810 con 0x7f14980c51b0 2023-11-28T13:00:26.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.240+0000 7f149e3ef700 1 -- 172.21.15.111:0/3453151110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14980c46f0 con 0x7f14980c51b0 2023-11-28T13:00:26.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.240+0000 7f1494ff9700 1 -- 172.21.15.111:0/3453151110 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f148c02e030 con 0x7f14980c51b0 2023-11-28T13:00:26.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.241+0000 7f1494ff9700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f14840682f0 0x7f148406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.241+0000 7f1494ff9700 1 -- 172.21.15.111:0/3453151110 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f148c087010 con 0x7f14980c51b0 2023-11-28T13:00:26.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.241+0000 7f1497fff700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f14840682f0 0x7f148406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.241+0000 7f1497fff700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f14840682f0 0x7f148406a7a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1488001010 tx=0x7f1488009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:26.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.245+0000 7f1494ff9700 1 -- 172.21.15.111:0/3453151110 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f148c0537e0 con 0x7f14980c51b0 2023-11-28T13:00:26.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.429+0000 7f149e3ef700 1 -- 172.21.15.111:0/3453151110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1498067170 con 0x7f14980c51b0 2023-11-28T13:00:26.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.429+0000 7f1494ff9700 1 -- 172.21.15.111:0/3453151110 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f148c02c020 con 0x7f14980c51b0 2023-11-28T13:00:26.430 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T13:00:26.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.432+0000 7f149e3ef700 1 -- 172.21.15.111:0/3453151110 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f14840682f0 msgr2=0x7f148406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.432+0000 7f149e3ef700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f14840682f0 0x7f148406a7a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1488001010 tx=0x7f1488009040 comp rx=0 tx=0).stop 2023-11-28T13:00:26.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.432+0000 7f149e3ef700 1 -- 172.21.15.111:0/3453151110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14980c51b0 msgr2=0x7f14980c5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.432+0000 7f149e3ef700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14980c51b0 0x7f14980c5620 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f14800047d0 tx=0x7f148c0047c0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.432+0000 7f149e3ef700 1 -- 172.21.15.111:0/3453151110 shutdown_connections 2023-11-28T13:00:26.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.432+0000 7f149e3ef700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f14840682f0 0x7f148406a7a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.432+0000 7f149e3ef700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f14980c2df0 0x7f14980c3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.432+0000 7f149e3ef700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1480004120 0x7f14981487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.432+0000 7f149e3ef700 1 --2- 172.21.15.111:0/3453151110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14980c51b0 0x7f14980c5620 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.432+0000 7f149e3ef700 1 -- 172.21.15.111:0/3453151110 >> 172.21.15.111:0/3453151110 conn(0x7f14980b0db0 msgr2=0x7f14980af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:26.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.432+0000 7f149e3ef700 1 -- 172.21.15.111:0/3453151110 shutdown_connections 2023-11-28T13:00:26.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.432+0000 7f149e3ef700 1 -- 172.21.15.111:0/3453151110 wait complete. 2023-11-28T13:00:26.444 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T13:00:26.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.566+0000 7ff340550700 1 Processor -- start 2023-11-28T13:00:26.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.566+0000 7ff340550700 1 -- start start 2023-11-28T13:00:26.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.567+0000 7ff340550700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff3381604b0 0x7ff3380ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.567+0000 7ff340550700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff3380cc550 0x7ff3380cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.567+0000 7ff340550700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff33815cd90 con 0x7ff3380b0d70 2023-11-28T13:00:26.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.567+0000 7ff340550700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff338160950 con 0x7ff3380cc550 2023-11-28T13:00:26.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.567+0000 7ff340550700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3380cced0 con 0x7ff3381604b0 2023-11-28T13:00:26.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.567+0000 7ff33eaed700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff3380cc550 0x7ff3380cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.567+0000 7ff33eaed700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff3380cc550 0x7ff3380cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35472/0 (socket says 172.21.15.111:35472) 2023-11-28T13:00:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.567+0000 7ff33eaed700 1 -- 172.21.15.111:0/387642170 learned_addr learned my addr 172.21.15.111:0/387642170 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.568+0000 7ff33daeb700 1 --2- 172.21.15.111:0/387642170 >> v2:172.21.15.149:3300/0 conn(0x7ff3381604b0 0x7ff3380ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.568+0000 7ff33eaed700 1 -- 172.21.15.111:0/387642170 >> v2:172.21.15.149:3300/0 conn(0x7ff3381604b0 msgr2=0x7ff3380ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.568+0000 7ff33eaed700 1 --2- 172.21.15.111:0/387642170 >> v2:172.21.15.149:3300/0 conn(0x7ff3381604b0 0x7ff3380ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.568+0000 7ff33eaed700 1 -- 172.21.15.111:0/387642170 >> v1:172.21.15.149:6789/0 conn(0x7ff3380b0d70 legacy=0x7ff3380b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.568+0000 7ff33eaed700 1 -- 172.21.15.111:0/387642170 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3381613e0 con 0x7ff3380cc550 2023-11-28T13:00:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.568+0000 7ff33eaed700 1 --2- 172.21.15.111:0/387642170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3380cc550 0x7ff3380cc990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ff33806b310 tx=0x7ff33000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5586b18aa81e46 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.568+0000 7ff33d2ea700 1 -- 172.21.15.111:0/387642170 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33000e070 con 0x7ff3380cc550 2023-11-28T13:00:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.568+0000 7ff33d2ea700 1 -- 172.21.15.111:0/387642170 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33000b9c0 con 0x7ff3380cc550 2023-11-28T13:00:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.568+0000 7ff33d2ea700 1 -- 172.21.15.111:0/387642170 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33000bd70 con 0x7ff3380cc550 2023-11-28T13:00:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.568+0000 7ff340550700 1 -- 172.21.15.111:0/387642170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3380cc550 msgr2=0x7ff3380cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.568+0000 7ff340550700 1 --2- 172.21.15.111:0/387642170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3380cc550 0x7ff3380cc990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ff33806b310 tx=0x7ff33000b330 comp rx=0 tx=0).stop 2023-11-28T13:00:26.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.569+0000 7ff340550700 1 -- 172.21.15.111:0/387642170 shutdown_connections 2023-11-28T13:00:26.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.569+0000 7ff340550700 1 --2- 172.21.15.111:0/387642170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3380cc550 0x7ff3380cc990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.569+0000 7ff340550700 1 --2- 172.21.15.111:0/387642170 >> v2:172.21.15.149:3300/0 conn(0x7ff3381604b0 0x7ff3380ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.569+0000 7ff340550700 1 -- 172.21.15.111:0/387642170 >> 172.21.15.111:0/387642170 conn(0x7ff338098ee0 msgr2=0x7ff33806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:26.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.569+0000 7ff340550700 1 -- 172.21.15.111:0/387642170 shutdown_connections 2023-11-28T13:00:26.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.569+0000 7ff340550700 1 -- 172.21.15.111:0/387642170 wait complete. 2023-11-28T13:00:26.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.570+0000 7ff340550700 1 Processor -- start 2023-11-28T13:00:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.570+0000 7ff340550700 1 -- start start 2023-11-28T13:00:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.570+0000 7ff340550700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff3380b0d70 0x7ff3380cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.570+0000 7ff340550700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff3380cc550 0x7ff338157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.570+0000 7ff340550700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3381604b0 0x7ff3380c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff340550700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff338159b70 con 0x7ff3381604b0 2023-11-28T13:00:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff340550700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3381f2440 con 0x7ff3380cc550 2023-11-28T13:00:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff340550700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3381585f0 con 0x7ff3380b0d70 2023-11-28T13:00:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff33eaed700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3381604b0 0x7ff3380c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff33eaed700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3381604b0 0x7ff3380c5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35482/0 (socket says 172.21.15.111:35482) 2023-11-28T13:00:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff33eaed700 1 -- 172.21.15.111:0/3905746711 learned_addr learned my addr 172.21.15.111:0/3905746711 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:26.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff33daeb700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff3380cc550 0x7ff338157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff33e2ec700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff3380b0d70 0x7ff3380cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff33eaed700 1 -- 172.21.15.111:0/3905746711 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff3380b0d70 msgr2=0x7ff3380cbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff33eaed700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff3380b0d70 0x7ff3380cbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff33eaed700 1 -- 172.21.15.111:0/3905746711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff3380cc550 msgr2=0x7ff338157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff33eaed700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff3380cc550 0x7ff338157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff33eaed700 1 -- 172.21.15.111:0/3905746711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff330009b30 con 0x7ff3381604b0 2023-11-28T13:00:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff33eaed700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3381604b0 0x7ff3380c5280 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff338067c90 tx=0x7ff330004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff32effd700 1 -- 172.21.15.111:0/3905746711 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3300099b0 con 0x7ff3381604b0 2023-11-28T13:00:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff32effd700 1 -- 172.21.15.111:0/3905746711 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3300092e0 con 0x7ff3381604b0 2023-11-28T13:00:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff340550700 1 -- 172.21.15.111:0/3905746711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff338160e20 con 0x7ff3381604b0 2023-11-28T13:00:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff32effd700 1 -- 172.21.15.111:0/3905746711 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33001c640 con 0x7ff3381604b0 2023-11-28T13:00:26.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.571+0000 7ff340550700 1 -- 172.21.15.111:0/3905746711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff338160b40 con 0x7ff3381604b0 2023-11-28T13:00:26.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.572+0000 7ff340550700 1 -- 172.21.15.111:0/3905746711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff338161100 con 0x7ff3381604b0 2023-11-28T13:00:26.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.573+0000 7ff32effd700 1 -- 172.21.15.111:0/3905746711 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff330022030 con 0x7ff3381604b0 2023-11-28T13:00:26.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.573+0000 7ff32effd700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff31c0682f0 0x7ff31c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.573+0000 7ff32effd700 1 -- 172.21.15.111:0/3905746711 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff3300876a0 con 0x7ff3381604b0 2023-11-28T13:00:26.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.573+0000 7ff33e2ec700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff31c0682f0 0x7ff31c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.573+0000 7ff33e2ec700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff31c0682f0 0x7ff31c06a7a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff328000fb0 tx=0x7ff328009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:26.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.577+0000 7ff32effd700 1 -- 172.21.15.111:0/3905746711 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff330053dc0 con 0x7ff3381604b0 2023-11-28T13:00:26.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.758+0000 7ff340550700 1 -- 172.21.15.111:0/3905746711 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff33806d9a0 con 0x7ff3381604b0 2023-11-28T13:00:26.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.758+0000 7ff32effd700 1 -- 172.21.15.111:0/3905746711 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v41) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff3300573e0 con 0x7ff3381604b0 2023-11-28T13:00:26.759 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T13:00:26.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.760+0000 7ff340550700 1 -- 172.21.15.111:0/3905746711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff31c0682f0 msgr2=0x7ff31c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.760+0000 7ff340550700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff31c0682f0 0x7ff31c06a7a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff328000fb0 tx=0x7ff328009040 comp rx=0 tx=0).stop 2023-11-28T13:00:26.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.760+0000 7ff340550700 1 -- 172.21.15.111:0/3905746711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3381604b0 msgr2=0x7ff3380c5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.760+0000 7ff340550700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3381604b0 0x7ff3380c5280 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff338067c90 tx=0x7ff330004ad0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.760+0000 7ff340550700 1 -- 172.21.15.111:0/3905746711 shutdown_connections 2023-11-28T13:00:26.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.760+0000 7ff340550700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3381604b0 0x7ff3380c5280 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.760+0000 7ff340550700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff3380b0d70 0x7ff3380cbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.760+0000 7ff340550700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff31c0682f0 0x7ff31c06a7a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.760+0000 7ff340550700 1 --2- 172.21.15.111:0/3905746711 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff3380cc550 0x7ff338157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.760+0000 7ff340550700 1 -- 172.21.15.111:0/3905746711 >> 172.21.15.111:0/3905746711 conn(0x7ff338098ee0 msgr2=0x7ff33815c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.760+0000 7ff340550700 1 -- 172.21.15.111:0/3905746711 shutdown_connections 2023-11-28T13:00:26.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.760+0000 7ff340550700 1 -- 172.21.15.111:0/3905746711 wait complete. 2023-11-28T13:00:26.773 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T13:00:26.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.894+0000 7fc970fe2700 1 Processor -- start 2023-11-28T13:00:26.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.894+0000 7fc970fe2700 1 -- start start 2023-11-28T13:00:26.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.895+0000 7fc970fe2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc96c0b0d70 0x7fc96c1604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.895+0000 7fc970fe2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc96c0b1140 0x7fc96c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.895+0000 7fc970fe2700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc96c0b15b0 0x7fc96c0cc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.895+0000 7fc970fe2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96c15cd90 con 0x7fc96c0b1140 2023-11-28T13:00:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.895+0000 7fc970fe2700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96c1613e0 con 0x7fc96c0b15b0 2023-11-28T13:00:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.895+0000 7fc970fe2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c160b40 con 0x7fc96c0b0d70 2023-11-28T13:00:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc969d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc96c0b1140 0x7fc96c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc969d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc96c0b1140 0x7fc96c0ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35486/0 (socket says 172.21.15.111:35486) 2023-11-28T13:00:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc969d9b700 1 -- 172.21.15.111:0/3737959908 learned_addr learned my addr 172.21.15.111:0/3737959908 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc96ad9d700 1 --2- 172.21.15.111:0/3737959908 >> v2:172.21.15.149:3300/0 conn(0x7fc96c0b15b0 0x7fc96c0cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc96a59c700 1 --2- 172.21.15.111:0/3737959908 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc96c0b0d70 0x7fc96c1604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc969d9b700 1 -- 172.21.15.111:0/3737959908 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc96c0b0d70 msgr2=0x7fc96c1604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc969d9b700 1 --2- 172.21.15.111:0/3737959908 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc96c0b0d70 0x7fc96c1604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc969d9b700 1 -- 172.21.15.111:0/3737959908 >> v2:172.21.15.149:3300/0 conn(0x7fc96c0b15b0 msgr2=0x7fc96c0cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc969d9b700 1 --2- 172.21.15.111:0/3737959908 >> v2:172.21.15.149:3300/0 conn(0x7fc96c0b15b0 0x7fc96c0cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc969d9b700 1 -- 172.21.15.111:0/3737959908 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96c160e20 con 0x7fc96c0b1140 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc969d9b700 1 --2- 172.21.15.111:0/3737959908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c0b1140 0x7fc96c0ce6c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0cfb10 tx=0x7fc95c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e3ac640307a5f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc96959a700 1 -- 172.21.15.111:0/3737959908 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c00b870 con 0x7fc96c0b1140 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc96959a700 1 -- 172.21.15.111:0/3737959908 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95c00b9d0 con 0x7fc96c0b1140 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.896+0000 7fc96959a700 1 -- 172.21.15.111:0/3737959908 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c00bd60 con 0x7fc96c0b1140 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.897+0000 7fc970fe2700 1 -- 172.21.15.111:0/3737959908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c0b1140 msgr2=0x7fc96c0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.897+0000 7fc970fe2700 1 --2- 172.21.15.111:0/3737959908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c0b1140 0x7fc96c0ce6c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0cfb10 tx=0x7fc95c009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.897+0000 7fc970fe2700 1 -- 172.21.15.111:0/3737959908 shutdown_connections 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.897+0000 7fc970fe2700 1 --2- 172.21.15.111:0/3737959908 >> v2:172.21.15.149:3300/0 conn(0x7fc96c0b15b0 0x7fc96c0cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.897+0000 7fc970fe2700 1 --2- 172.21.15.111:0/3737959908 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc96c0b0d70 0x7fc96c1604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.897+0000 7fc970fe2700 1 --2- 172.21.15.111:0/3737959908 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c0b1140 0x7fc96c0ce6c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.897+0000 7fc970fe2700 1 -- 172.21.15.111:0/3737959908 >> 172.21.15.111:0/3737959908 conn(0x7fc96c098ee0 msgr2=0x7fc96c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:26.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.897+0000 7fc970fe2700 1 -- 172.21.15.111:0/3737959908 shutdown_connections 2023-11-28T13:00:26.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.897+0000 7fc970fe2700 1 -- 172.21.15.111:0/3737959908 wait complete. 2023-11-28T13:00:26.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.898+0000 7fc970fe2700 1 Processor -- start 2023-11-28T13:00:26.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.898+0000 7fc970fe2700 1 -- start start 2023-11-28T13:00:26.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc970fe2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc96c0b0d70 0x7fc96c14a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc970fe2700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc96c0b1140 0x7fc96c1f5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc970fe2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c0b15b0 0x7fc96c1f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc970fe2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c0c48c0 con 0x7fc96c0b15b0 2023-11-28T13:00:26.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc970fe2700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c0c4ba0 con 0x7fc96c0b1140 2023-11-28T13:00:26.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc970fe2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c0c4e80 con 0x7fc96c0b0d70 2023-11-28T13:00:26.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc96ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c0b15b0 0x7fc96c1f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc96ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c0b15b0 0x7fc96c1f3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35496/0 (socket says 172.21.15.111:35496) 2023-11-28T13:00:26.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc96ad9d700 1 -- 172.21.15.111:0/2175906793 learned_addr learned my addr 172.21.15.111:0/2175906793 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:26.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc969d9b700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc96c0b1140 0x7fc96c1f5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc96a59c700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc96c0b0d70 0x7fc96c14a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc96ad9d700 1 -- 172.21.15.111:0/2175906793 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc96c0b0d70 msgr2=0x7fc96c14a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc96ad9d700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc96c0b0d70 0x7fc96c14a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc96ad9d700 1 -- 172.21.15.111:0/2175906793 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc96c0b1140 msgr2=0x7fc96c1f5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:26.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc96ad9d700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc96c0b1140 0x7fc96c1f5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.899+0000 7fc96ad9d700 1 -- 172.21.15.111:0/2175906793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c009550 con 0x7fc96c0b15b0 2023-11-28T13:00:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.900+0000 7fc96ad9d700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c0b15b0 0x7fc96c1f3200 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc964007de0 tx=0x7fc96400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.900+0000 7fc95affd700 1 -- 172.21.15.111:0/2175906793 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96400ed30 con 0x7fc96c0b15b0 2023-11-28T13:00:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.900+0000 7fc95affd700 1 -- 172.21.15.111:0/2175906793 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc964014420 con 0x7fc96c0b15b0 2023-11-28T13:00:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.900+0000 7fc95affd700 1 -- 172.21.15.111:0/2175906793 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9640147d0 con 0x7fc96c0b15b0 2023-11-28T13:00:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.900+0000 7fc970fe2700 1 -- 172.21.15.111:0/2175906793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc96c0af340 con 0x7fc96c0b15b0 2023-11-28T13:00:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.900+0000 7fc970fe2700 1 -- 172.21.15.111:0/2175906793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96c161100 con 0x7fc96c0b15b0 2023-11-28T13:00:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.901+0000 7fc970fe2700 1 -- 172.21.15.111:0/2175906793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96c0cec00 con 0x7fc96c0b15b0 2023-11-28T13:00:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.901+0000 7fc95affd700 1 -- 172.21.15.111:0/2175906793 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc964012050 con 0x7fc96c0b15b0 2023-11-28T13:00:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.901+0000 7fc95affd700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc94c0682f0 0x7fc94c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.902+0000 7fc95affd700 1 -- 172.21.15.111:0/2175906793 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc9640862f0 con 0x7fc96c0b15b0 2023-11-28T13:00:26.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.902+0000 7fc96a59c700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc94c0682f0 0x7fc94c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:26.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.902+0000 7fc96a59c700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc94c0682f0 0x7fc94c06a7a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0cb5f0 tx=0x7fc960008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:26.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:26.906+0000 7fc95affd700 1 -- 172.21.15.111:0/2175906793 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc964052a10 con 0x7fc96c0b15b0 2023-11-28T13:00:27.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.088+0000 7fc970fe2700 1 -- 172.21.15.111:0/2175906793 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc96c067170 con 0x7fc96c0b15b0 2023-11-28T13:00:27.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.089+0000 7fc95affd700 1 -- 172.21.15.111:0/2175906793 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc964056030 con 0x7fc96c0b15b0 2023-11-28T13:00:27.089 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T13:00:27.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.091+0000 7fc970fe2700 1 -- 172.21.15.111:0/2175906793 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc94c0682f0 msgr2=0x7fc94c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.091+0000 7fc970fe2700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc94c0682f0 0x7fc94c06a7a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0cb5f0 tx=0x7fc960008040 comp rx=0 tx=0).stop 2023-11-28T13:00:27.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.091+0000 7fc970fe2700 1 -- 172.21.15.111:0/2175906793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c0b15b0 msgr2=0x7fc96c1f3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.091+0000 7fc970fe2700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c0b15b0 0x7fc96c1f3200 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc964007de0 tx=0x7fc96400e3c0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.091+0000 7fc970fe2700 1 -- 172.21.15.111:0/2175906793 shutdown_connections 2023-11-28T13:00:27.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.091+0000 7fc970fe2700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc96c0b15b0 0x7fc96c1f3200 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.091+0000 7fc970fe2700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc96c0b0d70 0x7fc96c14a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.091+0000 7fc970fe2700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc94c0682f0 0x7fc94c06a7a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.091+0000 7fc970fe2700 1 --2- 172.21.15.111:0/2175906793 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc96c0b1140 0x7fc96c1f5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.091+0000 7fc970fe2700 1 -- 172.21.15.111:0/2175906793 >> 172.21.15.111:0/2175906793 conn(0x7fc96c098ee0 msgr2=0x7fc96c0cef00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:27.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.091+0000 7fc970fe2700 1 -- 172.21.15.111:0/2175906793 shutdown_connections 2023-11-28T13:00:27.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.091+0000 7fc970fe2700 1 -- 172.21.15.111:0/2175906793 wait complete. 2023-11-28T13:00:27.103 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T13:00:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.225+0000 7f58985dd700 1 Processor -- start 2023-11-28T13:00:27.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.225+0000 7f58985dd700 1 -- start start 2023-11-28T13:00:27.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.226+0000 7f58985dd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f589015c5e0 0x7f589006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.226+0000 7f58985dd700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f589006b720 0x7f589006a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.226+0000 7f58985dd700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589015c9b0 con 0x7f589006acf0 2023-11-28T13:00:27.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.226+0000 7f58985dd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58900cce90 con 0x7f589015c5e0 2023-11-28T13:00:27.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.226+0000 7f58985dd700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5890160e60 con 0x7f589006b720 2023-11-28T13:00:27.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.226+0000 7f5896379700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f589015c5e0 0x7f589006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.226+0000 7f5896379700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f589015c5e0 0x7f589006a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35500/0 (socket says 172.21.15.111:35500) 2023-11-28T13:00:27.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.226+0000 7f5896379700 1 -- 172.21.15.111:0/2091911994 learned_addr learned my addr 172.21.15.111:0/2091911994 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:27.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.227+0000 7f5895b78700 1 --2- 172.21.15.111:0/2091911994 >> v2:172.21.15.149:3300/0 conn(0x7f589006b720 0x7f589006a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.227+0000 7f5896379700 1 -- 172.21.15.111:0/2091911994 >> v2:172.21.15.149:3300/0 conn(0x7f589006b720 msgr2=0x7f589006a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:27.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.227+0000 7f5896379700 1 --2- 172.21.15.111:0/2091911994 >> v2:172.21.15.149:3300/0 conn(0x7f589006b720 0x7f589006a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.227+0000 7f5896379700 1 -- 172.21.15.111:0/2091911994 >> v1:172.21.15.149:6789/0 conn(0x7f589006acf0 legacy=0x7f58900712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:27.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.227+0000 7f5896379700 1 -- 172.21.15.111:0/2091911994 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58900cd0a0 con 0x7f589015c5e0 2023-11-28T13:00:27.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.227+0000 7f5895b78700 1 --2- 172.21.15.111:0/2091911994 >> v2:172.21.15.149:3300/0 conn(0x7f589006b720 0x7f589006a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:27.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.227+0000 7f5896379700 1 --2- 172.21.15.111:0/2091911994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f589015c5e0 0x7f589006a270 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f589006e340 tx=0x7f5880009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2ccad350bea341 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:27.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.227+0000 7f5895377700 1 -- 172.21.15.111:0/2091911994 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5880004230 con 0x7f589015c5e0 2023-11-28T13:00:27.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.227+0000 7f5895377700 1 -- 172.21.15.111:0/2091911994 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5880004390 con 0x7f589015c5e0 2023-11-28T13:00:27.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.227+0000 7f5895377700 1 -- 172.21.15.111:0/2091911994 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58800036a0 con 0x7f589015c5e0 2023-11-28T13:00:27.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.228+0000 7f58985dd700 1 -- 172.21.15.111:0/2091911994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f589015c5e0 msgr2=0x7f589006a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.228+0000 7f58985dd700 1 --2- 172.21.15.111:0/2091911994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f589015c5e0 0x7f589006a270 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f589006e340 tx=0x7f5880009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.228+0000 7f58985dd700 1 -- 172.21.15.111:0/2091911994 shutdown_connections 2023-11-28T13:00:27.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.228+0000 7f58985dd700 1 --2- 172.21.15.111:0/2091911994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f589015c5e0 0x7f589006a270 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.228+0000 7f58985dd700 1 --2- 172.21.15.111:0/2091911994 >> v2:172.21.15.149:3300/0 conn(0x7f589006b720 0x7f589006a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.228+0000 7f58985dd700 1 -- 172.21.15.111:0/2091911994 >> 172.21.15.111:0/2091911994 conn(0x7f58900b0db0 msgr2=0x7f58900b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:27.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.228+0000 7f58985dd700 1 -- 172.21.15.111:0/2091911994 shutdown_connections 2023-11-28T13:00:27.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.228+0000 7f58985dd700 1 -- 172.21.15.111:0/2091911994 wait complete. 2023-11-28T13:00:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.229+0000 7f58985dd700 1 Processor -- start 2023-11-28T13:00:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.229+0000 7f58985dd700 1 -- start start 2023-11-28T13:00:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.230+0000 7f58985dd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f589006acf0 0x7f5890151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.230+0000 7f58985dd700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f589006b720 0x7f589015ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.230+0000 7f58985dd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f589015c5e0 0x7f58900b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.230+0000 7f58985dd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5890151f70 con 0x7f589015c5e0 2023-11-28T13:00:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.230+0000 7f58985dd700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589015f160 con 0x7f589006b720 2023-11-28T13:00:27.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.230+0000 7f58985dd700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58901f25d0 con 0x7f589006acf0 2023-11-28T13:00:27.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.230+0000 7f5896b7a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f589015c5e0 0x7f58900b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.230+0000 7f5896b7a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f589015c5e0 0x7f58900b42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35504/0 (socket says 172.21.15.111:35504) 2023-11-28T13:00:27.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.230+0000 7f5896b7a700 1 -- 172.21.15.111:0/2068474402 learned_addr learned my addr 172.21.15.111:0/2068474402 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:27.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.230+0000 7f5895b78700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f589006b720 0x7f589015ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.230+0000 7f5896379700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f589006acf0 0x7f5890151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.231+0000 7f5896b7a700 1 -- 172.21.15.111:0/2068474402 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f589006acf0 msgr2=0x7f5890151a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.231+0000 7f5896b7a700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f589006acf0 0x7f5890151a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.231+0000 7f5896b7a700 1 -- 172.21.15.111:0/2068474402 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f589006b720 msgr2=0x7f589015ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.231+0000 7f5896b7a700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f589006b720 0x7f589015ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.231+0000 7f5896b7a700 1 -- 172.21.15.111:0/2068474402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5880009550 con 0x7f589015c5e0 2023-11-28T13:00:27.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.231+0000 7f5896379700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f589006acf0 0x7f5890151a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:27.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.231+0000 7f5896b7a700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f589015c5e0 0x7f58900b42c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f588800b970 tx=0x7f588800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:27.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.231+0000 7f5886ffd700 1 -- 172.21.15.111:0/2068474402 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588800c930 con 0x7f589015c5e0 2023-11-28T13:00:27.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.231+0000 7f5886ffd700 1 -- 172.21.15.111:0/2068474402 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588800cad0 con 0x7f589015c5e0 2023-11-28T13:00:27.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.231+0000 7f58985dd700 1 -- 172.21.15.111:0/2068474402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58900cd660 con 0x7f589015c5e0 2023-11-28T13:00:27.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.231+0000 7f5886ffd700 1 -- 172.21.15.111:0/2068474402 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588800ce80 con 0x7f589015c5e0 2023-11-28T13:00:27.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.231+0000 7f58985dd700 1 -- 172.21.15.111:0/2068474402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58900cd380 con 0x7f589015c5e0 2023-11-28T13:00:27.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.232+0000 7f58985dd700 1 -- 172.21.15.111:0/2068474402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5890067170 con 0x7f589015c5e0 2023-11-28T13:00:27.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.232+0000 7f5886ffd700 1 -- 172.21.15.111:0/2068474402 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f588801b030 con 0x7f589015c5e0 2023-11-28T13:00:27.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.233+0000 7f5886ffd700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f58740682f0 0x7f587406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.233+0000 7f5886ffd700 1 -- 172.21.15.111:0/2068474402 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5888085810 con 0x7f589015c5e0 2023-11-28T13:00:27.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.233+0000 7f5896379700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f58740682f0 0x7f587406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.233+0000 7f5896379700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f58740682f0 0x7f587406a7a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5880009130 tx=0x7f5880008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:27.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.237+0000 7f5886ffd700 1 -- 172.21.15.111:0/2068474402 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f588804d5e0 con 0x7f589015c5e0 2023-11-28T13:00:27.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.418+0000 7f58985dd700 1 -- 172.21.15.111:0/2068474402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5890160a40 con 0x7f589015c5e0 2023-11-28T13:00:27.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.418+0000 7f5886ffd700 1 -- 172.21.15.111:0/2068474402 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5888019020 con 0x7f589015c5e0 2023-11-28T13:00:27.419 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T13:00:27.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.420+0000 7f58985dd700 1 -- 172.21.15.111:0/2068474402 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f58740682f0 msgr2=0x7f587406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.420+0000 7f58985dd700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f58740682f0 0x7f587406a7a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5880009130 tx=0x7f5880008030 comp rx=0 tx=0).stop 2023-11-28T13:00:27.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.420+0000 7f58985dd700 1 -- 172.21.15.111:0/2068474402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f589015c5e0 msgr2=0x7f58900b42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.420+0000 7f58985dd700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f589015c5e0 0x7f58900b42c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f588800b970 tx=0x7f588800bc80 comp rx=0 tx=0).stop 2023-11-28T13:00:27.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.420+0000 7f58985dd700 1 -- 172.21.15.111:0/2068474402 shutdown_connections 2023-11-28T13:00:27.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.420+0000 7f58985dd700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f589015c5e0 0x7f58900b42c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.420+0000 7f58985dd700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f589006acf0 0x7f5890151a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.420+0000 7f58985dd700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f58740682f0 0x7f587406a7a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.420+0000 7f58985dd700 1 --2- 172.21.15.111:0/2068474402 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f589006b720 0x7f589015ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.420+0000 7f58985dd700 1 -- 172.21.15.111:0/2068474402 >> 172.21.15.111:0/2068474402 conn(0x7f58900b0db0 msgr2=0x7f58900af200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:27.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.420+0000 7f58985dd700 1 -- 172.21.15.111:0/2068474402 shutdown_connections 2023-11-28T13:00:27.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.420+0000 7f58985dd700 1 -- 172.21.15.111:0/2068474402 wait complete. 2023-11-28T13:00:27.433 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T13:00:27.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.552+0000 7f90bbd24700 1 Processor -- start 2023-11-28T13:00:27.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.553+0000 7f90bbd24700 1 -- start start 2023-11-28T13:00:27.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.553+0000 7f90bbd24700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f90b40cdd90 0x7f90b40b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.553+0000 7f90bbd24700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b40ad420 con 0x7f90b40aecc0 2023-11-28T13:00:27.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.553+0000 7f90bbd24700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b40ac810 con 0x7f90b4159960 2023-11-28T13:00:27.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.553+0000 7f90bbd24700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b4159d30 con 0x7f90b40cdd90 2023-11-28T13:00:27.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90ba2c1700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f90b40aecc0 0x7f90b40ad8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36292/0 (socket says 172.21.15.111:36292) 2023-11-28T13:00:27.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90ba2c1700 1 -- 172.21.15.111:0/1950668261 learned_addr learned my addr 172.21.15.111:0/1950668261 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:27.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b92bf700 1 --2- 172.21.15.111:0/1950668261 >> v2:172.21.15.149:3300/0 conn(0x7f90b40cdd90 0x7f90b40b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269875527 0 0) 0x7f90b40ad420 con 0x7f90b40aecc0 2023-11-28T13:00:27.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a00035b0 con 0x7f90b40aecc0 2023-11-28T13:00:27.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2295902703 0 0) 0x7f90a00035b0 con 0x7f90b40aecc0 2023-11-28T13:00:27.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 >> v2:172.21.15.149:3300/0 conn(0x7f90b40cdd90 msgr2=0x7f90b40b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:27.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 --2- 172.21.15.111:0/1950668261 >> v2:172.21.15.149:3300/0 conn(0x7f90b40cdd90 0x7f90b40b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 >> v1:172.21.15.149:6789/0 conn(0x7f90b4159960 legacy=0x7f90b40cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b40acaf0 con 0x7f90b40aecc0 2023-11-28T13:00:27.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f90ac0019f0 con 0x7f90b40aecc0 2023-11-28T13:00:27.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 >> v1:172.21.15.111:6789/0 conn(0x7f90b40aecc0 legacy=0x7f90b40ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 --2- 172.21.15.111:0/1950668261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90a0003a00 0x7f90a0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 --2- 172.21.15.111:0/1950668261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a0004850 0x7f90a0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 --2- 172.21.15.111:0/1950668261 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90a0005200 0x7f90a000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b4159d30 con 0x7f90a0004850 2023-11-28T13:00:27.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a000c080 con 0x7f90a0003a00 2023-11-28T13:00:27.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a000c240 con 0x7f90a0005200 2023-11-28T13:00:27.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b92bf700 1 --2- 172.21.15.111:0/1950668261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a0004850 0x7f90a0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b9ac0700 1 --2- 172.21.15.111:0/1950668261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90a0003a00 0x7f90a0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90ba2c1700 1 --2- 172.21.15.111:0/1950668261 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90a0005200 0x7f90a000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b92bf700 1 -- 172.21.15.111:0/1950668261 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90a0005200 msgr2=0x7f90a000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:27.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b92bf700 1 --2- 172.21.15.111:0/1950668261 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90a0005200 0x7f90a000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b92bf700 1 -- 172.21.15.111:0/1950668261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90a0003a00 msgr2=0x7f90a0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:27.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b92bf700 1 --2- 172.21.15.111:0/1950668261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90a0003a00 0x7f90a0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b92bf700 1 -- 172.21.15.111:0/1950668261 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a00035b0 con 0x7f90a0004850 2023-11-28T13:00:27.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90ba2c1700 1 -- 172.21.15.111:0/1950668261 reap_dead start 2023-11-28T13:00:27.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.554+0000 7f90b9ac0700 1 --2- 172.21.15.111:0/1950668261 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90a0003a00 0x7f90a0003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:27.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.555+0000 7f90b92bf700 1 --2- 172.21.15.111:0/1950668261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a0004850 0x7f90a0004cc0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f90b00096f0 tx=0x7f90a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf54b15ba54651a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:27.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.555+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b000e6c0 con 0x7f90a0004850 2023-11-28T13:00:27.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.555+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b000e820 con 0x7f90a0004850 2023-11-28T13:00:27.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.555+0000 7f90b8abe700 1 -- 172.21.15.111:0/1950668261 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b000ebb0 con 0x7f90a0004850 2023-11-28T13:00:27.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.555+0000 7f90bbd24700 1 -- 172.21.15.111:0/1950668261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a0004850 msgr2=0x7f90a0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.555+0000 7f90bbd24700 1 --2- 172.21.15.111:0/1950668261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a0004850 0x7f90a0004cc0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f90b00096f0 tx=0x7f90a0005880 comp rx=0 tx=0).stop 2023-11-28T13:00:27.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.555+0000 7f90bbd24700 1 -- 172.21.15.111:0/1950668261 shutdown_connections 2023-11-28T13:00:27.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.555+0000 7f90bbd24700 1 --2- 172.21.15.111:0/1950668261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a0004850 0x7f90a0004cc0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.555+0000 7f90bbd24700 1 -- 172.21.15.111:0/1950668261 >> 172.21.15.111:0/1950668261 conn(0x7f90b4098ee0 msgr2=0x7f90b406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:27.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.555+0000 7f90bbd24700 1 -- 172.21.15.111:0/1950668261 shutdown_connections 2023-11-28T13:00:27.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.555+0000 7f90bbd24700 1 -- 172.21.15.111:0/1950668261 wait complete. 2023-11-28T13:00:27.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.557+0000 7f90bbd24700 1 Processor -- start 2023-11-28T13:00:27.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.557+0000 7f90bbd24700 1 -- start start 2023-11-28T13:00:27.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.557+0000 7f90bbd24700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90a0004850 0x7f90b414a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.557+0000 7f90bbd24700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90b40cc590 0x7f90b406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.557+0000 7f90bbd24700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90b40cc9e0 0x7f90b406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.557+0000 7f90bbd24700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b4153910 con 0x7f90b40cc590 2023-11-28T13:00:27.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.557+0000 7f90bbd24700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b40cb300 con 0x7f90b40cc9e0 2023-11-28T13:00:27.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.557+0000 7f90bbd24700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b40cb5e0 con 0x7f90a0004850 2023-11-28T13:00:27.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.557+0000 7f90b92bf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90b40cc590 0x7f90b406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.557+0000 7f90b92bf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90b40cc590 0x7f90b406ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35524/0 (socket says 172.21.15.111:35524) 2023-11-28T13:00:27.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90b92bf700 1 -- 172.21.15.111:0/2727423761 learned_addr learned my addr 172.21.15.111:0/2727423761 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:27.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90ba2c1700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90b40cc9e0 0x7f90b406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90b9ac0700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90a0004850 0x7f90b414a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90b92bf700 1 -- 172.21.15.111:0/2727423761 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90a0004850 msgr2=0x7f90b414a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90b92bf700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90a0004850 0x7f90b414a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90b92bf700 1 -- 172.21.15.111:0/2727423761 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90b40cc9e0 msgr2=0x7f90b406f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90b92bf700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90b40cc9e0 0x7f90b406f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90b92bf700 1 -- 172.21.15.111:0/2727423761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b40ad0b0 con 0x7f90b40cc590 2023-11-28T13:00:27.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90b9ac0700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90a0004850 0x7f90b414a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:27.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90ba2c1700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90b40cc9e0 0x7f90b406f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:27.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90b92bf700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90b40cc590 0x7f90b406ed40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f90b000a010 tx=0x7f90b00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:27.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90aa7fc700 1 -- 172.21.15.111:0/2727423761 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b000e6c0 con 0x7f90b40cc590 2023-11-28T13:00:27.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90aa7fc700 1 -- 172.21.15.111:0/2727423761 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b0003ad0 con 0x7f90b40cc590 2023-11-28T13:00:27.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90bbd24700 1 -- 172.21.15.111:0/2727423761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b40acdd0 con 0x7f90b40cc590 2023-11-28T13:00:27.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90aa7fc700 1 -- 172.21.15.111:0/2727423761 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b0006e60 con 0x7f90b40cc590 2023-11-28T13:00:27.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.558+0000 7f90bbd24700 1 -- 172.21.15.111:0/2727423761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b40b1fd0 con 0x7f90b40cc590 2023-11-28T13:00:27.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.559+0000 7f90aa7fc700 1 -- 172.21.15.111:0/2727423761 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f90b002b030 con 0x7f90b40cc590 2023-11-28T13:00:27.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.559+0000 7f90bbd24700 1 -- 172.21.15.111:0/2727423761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b415af00 con 0x7f90b40cc590 2023-11-28T13:00:27.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.560+0000 7f90aa7fc700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90980682f0 0x7f909806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:27.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.560+0000 7f90aa7fc700 1 -- 172.21.15.111:0/2727423761 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f90b0086a20 con 0x7f90b40cc590 2023-11-28T13:00:27.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.560+0000 7f90b9ac0700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90980682f0 0x7f909806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:27.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.560+0000 7f90b9ac0700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90980682f0 0x7f909806a7a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f90a4000c00 tx=0x7f90a4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:27.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.564+0000 7f90aa7fc700 1 -- 172.21.15.111:0/2727423761 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f90b415af00 con 0x7f90b40cc590 2023-11-28T13:00:27.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.763+0000 7f90bbd24700 1 -- 172.21.15.111:0/2727423761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f90b415af00 con 0x7f90b40cc590 2023-11-28T13:00:27.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.763+0000 7f90aa7fc700 1 -- 172.21.15.111:0/2727423761 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v41) v1 ==== 80+0+0 (secure 0 0 0) 0x7f90b415af00 con 0x7f90b40cc590 2023-11-28T13:00:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.765+0000 7f90bbd24700 1 -- 172.21.15.111:0/2727423761 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90980682f0 msgr2=0x7f909806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.765+0000 7f90bbd24700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90980682f0 0x7f909806a7a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f90a4000c00 tx=0x7f90a4009040 comp rx=0 tx=0).stop 2023-11-28T13:00:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.765+0000 7f90bbd24700 1 -- 172.21.15.111:0/2727423761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90b40cc590 msgr2=0x7f90b406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.765+0000 7f90bbd24700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90b40cc590 0x7f90b406ed40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f90b000a010 tx=0x7f90b00047c0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.765+0000 7f90bbd24700 1 -- 172.21.15.111:0/2727423761 shutdown_connections 2023-11-28T13:00:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.765+0000 7f90bbd24700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f90980682f0 0x7f909806a7a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.765+0000 7f90bbd24700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f90b40cc9e0 0x7f90b406f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.765+0000 7f90bbd24700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90a0004850 0x7f90b414a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.765+0000 7f90bbd24700 1 --2- 172.21.15.111:0/2727423761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90b40cc590 0x7f90b406ed40 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:27.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.765+0000 7f90bbd24700 1 -- 172.21.15.111:0/2727423761 >> 172.21.15.111:0/2727423761 conn(0x7f90b4098ee0 msgr2=0x7f90b406e060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:27.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.765+0000 7f90bbd24700 1 -- 172.21.15.111:0/2727423761 shutdown_connections 2023-11-28T13:00:27.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:27.765+0000 7f90bbd24700 1 -- 172.21.15.111:0/2727423761 wait complete. 2023-11-28T13:00:27.767 INFO:teuthology.orchestra.run.smithi111.stderr:listed 0 entries 2023-11-28T13:00:27.777 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T13:00:27.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:27.778 DEBUG:teuthology.orchestra.run.smithi111:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T13:00:27.837 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T13:00:27.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:00:27.837 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T13:00:27.871 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:00:28.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.031+0000 7f41de6f1700 1 Processor -- start 2023-11-28T13:00:28.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.031+0000 7f41de6f1700 1 -- start start 2023-11-28T13:00:28.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41de6f1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f41d8159520 0x7f41d80cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:28.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41de6f1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d806c0a0 0x7f41d80cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:28.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41de6f1700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d80b1510 con 0x7f41d80aeb10 2023-11-28T13:00:28.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41de6f1700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d81598f0 con 0x7f41d8159520 2023-11-28T13:00:28.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41de6f1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80b0f50 con 0x7f41d806c0a0 2023-11-28T13:00:28.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41d7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f41d8159520 0x7f41d80cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:28.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41dcc8e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f41d80aeb10 0x7f41d80af530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36312/0 (socket says 172.21.15.111:36312) 2023-11-28T13:00:28.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41dcc8e700 1 -- 172.21.15.111:0/3347770833 learned_addr learned my addr 172.21.15.111:0/3347770833 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:28.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41d77fe700 1 --2- 172.21.15.111:0/3347770833 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d806c0a0 0x7f41d80cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:28.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41d6ffd700 1 -- 172.21.15.111:0/3347770833 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979729834 0 0) 0x7f41d80b1510 con 0x7f41d80aeb10 2023-11-28T13:00:28.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41d6ffd700 1 -- 172.21.15.111:0/3347770833 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c00035b0 con 0x7f41d80aeb10 2023-11-28T13:00:28.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41d7fff700 1 -- 172.21.15.111:0/3347770833 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d806c0a0 msgr2=0x7f41d80cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:28.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41d7fff700 1 --2- 172.21.15.111:0/3347770833 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d806c0a0 0x7f41d80cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.032+0000 7f41d7fff700 1 -- 172.21.15.111:0/3347770833 >> v1:172.21.15.111:6789/0 conn(0x7f41d80aeb10 legacy=0x7f41d80af530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:28.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.033+0000 7f41d7fff700 1 -- 172.21.15.111:0/3347770833 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d80b1230 con 0x7f41d8159520 2023-11-28T13:00:28.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.033+0000 7f41d7fff700 1 --2- 172.21.15.111:0/3347770833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d8159520 0x7f41d80cd7a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009840 tx=0x7f41c8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ad9267de18381d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:28.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.033+0000 7f41d6ffd700 1 -- 172.21.15.111:0/3347770833 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c8011830 con 0x7f41d8159520 2023-11-28T13:00:28.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.033+0000 7f41d6ffd700 1 -- 172.21.15.111:0/3347770833 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c8011990 con 0x7f41d8159520 2023-11-28T13:00:28.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.033+0000 7f41d6ffd700 1 -- 172.21.15.111:0/3347770833 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c8011d20 con 0x7f41d8159520 2023-11-28T13:00:28.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.034+0000 7f41de6f1700 1 -- 172.21.15.111:0/3347770833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d8159520 msgr2=0x7f41d80cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:28.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.034+0000 7f41de6f1700 1 --2- 172.21.15.111:0/3347770833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d8159520 0x7f41d80cd7a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009840 tx=0x7f41c8009b50 comp rx=0 tx=0).stop 2023-11-28T13:00:28.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.034+0000 7f41de6f1700 1 -- 172.21.15.111:0/3347770833 shutdown_connections 2023-11-28T13:00:28.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.034+0000 7f41de6f1700 1 --2- 172.21.15.111:0/3347770833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d8159520 0x7f41d80cd7a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.034+0000 7f41de6f1700 1 --2- 172.21.15.111:0/3347770833 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d806c0a0 0x7f41d80cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.034+0000 7f41de6f1700 1 -- 172.21.15.111:0/3347770833 >> 172.21.15.111:0/3347770833 conn(0x7f41d806bca0 msgr2=0x7f41d80cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:28.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.034+0000 7f41de6f1700 1 -- 172.21.15.111:0/3347770833 shutdown_connections 2023-11-28T13:00:28.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.034+0000 7f41de6f1700 1 -- 172.21.15.111:0/3347770833 wait complete. 2023-11-28T13:00:28.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.035+0000 7f41de6f1700 1 Processor -- start 2023-11-28T13:00:28.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.035+0000 7f41de6f1700 1 -- start start 2023-11-28T13:00:28.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41de6f1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d806c0a0 0x7f41d8151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:28.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41de6f1700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41d80aeb10 0x7f41d80b5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:28.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41de6f1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d8159520 0x7f41d8155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:28.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41de6f1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80c5230 con 0x7f41d8159520 2023-11-28T13:00:28.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41de6f1700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80c45b0 con 0x7f41d80aeb10 2023-11-28T13:00:28.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41de6f1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80c4890 con 0x7f41d806c0a0 2023-11-28T13:00:28.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41dcc8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d8159520 0x7f41d8155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:28.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41dcc8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d8159520 0x7f41d8155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35540/0 (socket says 172.21.15.111:35540) 2023-11-28T13:00:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41dcc8e700 1 -- 172.21.15.111:0/3944575085 learned_addr learned my addr 172.21.15.111:0/3944575085 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41d7fff700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d806c0a0 0x7f41d8151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41d77fe700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41d80aeb10 0x7f41d80b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41dcc8e700 1 -- 172.21.15.111:0/3944575085 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d806c0a0 msgr2=0x7f41d8151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41dcc8e700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d806c0a0 0x7f41d8151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41dcc8e700 1 -- 172.21.15.111:0/3944575085 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41d80aeb10 msgr2=0x7f41d80b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41dcc8e700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41d80aeb10 0x7f41d80b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.036+0000 7f41dcc8e700 1 -- 172.21.15.111:0/3944575085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c80094a0 con 0x7f41d8159520 2023-11-28T13:00:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.037+0000 7f41dcc8e700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d8159520 0x7f41d8155330 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f41d000caa0 tx=0x7f41d000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.037+0000 7f41d4ff9700 1 -- 172.21.15.111:0/3944575085 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d000a9e0 con 0x7f41d8159520 2023-11-28T13:00:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.037+0000 7f41de6f1700 1 -- 172.21.15.111:0/3944575085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d80acc90 con 0x7f41d8159520 2023-11-28T13:00:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.037+0000 7f41d4ff9700 1 -- 172.21.15.111:0/3944575085 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d000ab40 con 0x7f41d8159520 2023-11-28T13:00:28.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.037+0000 7f41de6f1700 1 -- 172.21.15.111:0/3944575085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d80ac9b0 con 0x7f41d8159520 2023-11-28T13:00:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.037+0000 7f41d4ff9700 1 -- 172.21.15.111:0/3944575085 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d0012400 con 0x7f41d8159520 2023-11-28T13:00:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.038+0000 7f41de6f1700 1 -- 172.21.15.111:0/3944575085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d80acf70 con 0x7f41d8159520 2023-11-28T13:00:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.038+0000 7f41d4ff9700 1 -- 172.21.15.111:0/3944575085 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f41d0019030 con 0x7f41d8159520 2023-11-28T13:00:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.039+0000 7f41d4ff9700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41c40682f0 0x7f41c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.039+0000 7f41d4ff9700 1 -- 172.21.15.111:0/3944575085 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f41d0085710 con 0x7f41d8159520 2023-11-28T13:00:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.039+0000 7f41d7fff700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41c40682f0 0x7f41c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:28.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.039+0000 7f41d7fff700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41c40682f0 0x7f41c406a7a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f41c8000c00 tx=0x7f41c8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:28.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.043+0000 7f41d4ff9700 1 -- 172.21.15.111:0/3944575085 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f41d004b5e0 con 0x7f41d8159520 2023-11-28T13:00:28.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.267+0000 7f41de6f1700 1 -- 172.21.15.111:0/3944575085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41d8071080 con 0x7f41d8159520 2023-11-28T13:00:28.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.268+0000 7f41d4ff9700 1 -- 172.21.15.111:0/3944575085 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f41d0051fa0 con 0x7f41d8159520 2023-11-28T13:00:28.269 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:28.269 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3254653481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3254653481},{"type":"v1","addr":"172.21.15.149:6835","nonce":3254653481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1890904311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1890904311},{"type":"v1","addr":"172.21.15.111:6836","nonce":1890904311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1592804351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1592804351},{"type":"v1","addr":"172.21.15.111:6837","nonce":1592804351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:12.011963+0000","modified":"2023-11-28T13:00:15.034385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5352":{"gid":5352,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.149:6834/3579364312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":3579364312},{"type":"v1","addr":"172.21.15.149:6834","nonce":3579364312}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T13:00:28.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.270+0000 7f41de6f1700 1 -- 172.21.15.111:0/3944575085 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41c40682f0 msgr2=0x7f41c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:28.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.270+0000 7f41de6f1700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41c40682f0 0x7f41c406a7a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f41c8000c00 tx=0x7f41c8008030 comp rx=0 tx=0).stop 2023-11-28T13:00:28.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.270+0000 7f41de6f1700 1 -- 172.21.15.111:0/3944575085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d8159520 msgr2=0x7f41d8155330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:28.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.270+0000 7f41de6f1700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d8159520 0x7f41d8155330 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f41d000caa0 tx=0x7f41d000cdb0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.270+0000 7f41de6f1700 1 -- 172.21.15.111:0/3944575085 shutdown_connections 2023-11-28T13:00:28.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.270+0000 7f41de6f1700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41d8159520 0x7f41d8155330 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.270+0000 7f41de6f1700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41d806c0a0 0x7f41d8151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.270+0000 7f41de6f1700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41c40682f0 0x7f41c406a7a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.270+0000 7f41de6f1700 1 --2- 172.21.15.111:0/3944575085 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41d80aeb10 0x7f41d80b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.270+0000 7f41de6f1700 1 -- 172.21.15.111:0/3944575085 >> 172.21.15.111:0/3944575085 conn(0x7f41d806bca0 msgr2=0x7f41d80adec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:28.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.270+0000 7f41de6f1700 1 -- 172.21.15.111:0/3944575085 shutdown_connections 2023-11-28T13:00:28.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.270+0000 7f41de6f1700 1 -- 172.21.15.111:0/3944575085 wait complete. 2023-11-28T13:00:28.272 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 17 2023-11-28T13:00:28.282 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2023-11-28T13:00:28.442 INFO:teuthology.orchestra.run.smithi149.stdout:{"message":"","return_code":0} 2023-11-28T13:00:28.442 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-11-28T13:00:28.443 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:00:28.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.562+0000 7fcb11391700 1 Processor -- start 2023-11-28T13:00:28.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.562+0000 7fcb11391700 1 -- start start 2023-11-28T13:00:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb11391700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb0c158cb0 0x7fcb0c0ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb11391700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcb0c159080 0x7fcb0c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb11391700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c06d870 con 0x7fcb0c0c9300 2023-11-28T13:00:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb11391700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb0c06cc60 con 0x7fcb0c159080 2023-11-28T13:00:28.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb11391700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c06cf40 con 0x7fcb0c158cb0 2023-11-28T13:00:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb0a7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcb0c159080 0x7fcb0c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb0b7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcb0c0c9300 0x7fcb0c070910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36326/0 (socket says 172.21.15.111:36326) 2023-11-28T13:00:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb0b7fe700 1 -- 172.21.15.111:0/995913361 learned_addr learned my addr 172.21.15.111:0/995913361 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb0affd700 1 --2- 172.21.15.111:0/995913361 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb0c158cb0 0x7fcb0c0ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb0a7fc700 1 -- 172.21.15.111:0/995913361 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb0c158cb0 msgr2=0x7fcb0c0ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb0a7fc700 1 --2- 172.21.15.111:0/995913361 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb0c158cb0 0x7fcb0c0ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb0a7fc700 1 -- 172.21.15.111:0/995913361 >> v1:172.21.15.111:6789/0 conn(0x7fcb0c0c9300 legacy=0x7fcb0c070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:28.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb09ffb700 1 -- 172.21.15.111:0/995913361 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036591428 0 0) 0x7fcb0c06d870 con 0x7fcb0c0c9300 2023-11-28T13:00:28.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb0a7fc700 1 -- 172.21.15.111:0/995913361 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c06d220 con 0x7fcb0c159080 2023-11-28T13:00:28.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb0a7fc700 1 --2- 172.21.15.111:0/995913361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb0c159080 0x7fcb0c06dd00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009840 tx=0x7fcafc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0280b66a996752 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:28.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb09ffb700 1 -- 172.21.15.111:0/995913361 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc011830 con 0x7fcb0c159080 2023-11-28T13:00:28.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb09ffb700 1 -- 172.21.15.111:0/995913361 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcafc011990 con 0x7fcb0c159080 2023-11-28T13:00:28.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.563+0000 7fcb09ffb700 1 -- 172.21.15.111:0/995913361 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc011d20 con 0x7fcb0c159080 2023-11-28T13:00:28.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.564+0000 7fcb11391700 1 -- 172.21.15.111:0/995913361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb0c159080 msgr2=0x7fcb0c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:28.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.564+0000 7fcb11391700 1 --2- 172.21.15.111:0/995913361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb0c159080 0x7fcb0c06dd00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009840 tx=0x7fcafc009b50 comp rx=0 tx=0).stop 2023-11-28T13:00:28.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.564+0000 7fcb11391700 1 -- 172.21.15.111:0/995913361 shutdown_connections 2023-11-28T13:00:28.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.564+0000 7fcb11391700 1 --2- 172.21.15.111:0/995913361 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb0c158cb0 0x7fcb0c0ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.564+0000 7fcb11391700 1 --2- 172.21.15.111:0/995913361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb0c159080 0x7fcb0c06dd00 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.564+0000 7fcb11391700 1 -- 172.21.15.111:0/995913361 >> 172.21.15.111:0/995913361 conn(0x7fcb0c0cb490 msgr2=0x7fcb0c0cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:28.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.564+0000 7fcb11391700 1 -- 172.21.15.111:0/995913361 shutdown_connections 2023-11-28T13:00:28.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.564+0000 7fcb11391700 1 -- 172.21.15.111:0/995913361 wait complete. 2023-11-28T13:00:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.565+0000 7fcb11391700 1 Processor -- start 2023-11-28T13:00:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.565+0000 7fcb11391700 1 -- start start 2023-11-28T13:00:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb11391700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb0c0c9300 0x7fcb0c0c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb11391700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb0c158cb0 0x7fcb0c15bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:28.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb11391700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb0c159bc0 0x7fcb0c15a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb11391700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c0af210 con 0x7fcb0c0c9300 2023-11-28T13:00:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb11391700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c1498a0 con 0x7fcb0c159bc0 2023-11-28T13:00:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb11391700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c0af420 con 0x7fcb0c158cb0 2023-11-28T13:00:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb0c0c9300 0x7fcb0c0c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb0c0c9300 0x7fcb0c0c21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35550/0 (socket says 172.21.15.111:35550) 2023-11-28T13:00:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb0affd700 1 -- 172.21.15.111:0/1610456992 learned_addr learned my addr 172.21.15.111:0/1610456992 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb0a7fc700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb0c158cb0 0x7fcb0c15bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb0b7fe700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb0c159bc0 0x7fcb0c15a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb0affd700 1 -- 172.21.15.111:0/1610456992 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb0c158cb0 msgr2=0x7fcb0c15bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:28.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb0affd700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb0c158cb0 0x7fcb0c15bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb0affd700 1 -- 172.21.15.111:0/1610456992 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb0c159bc0 msgr2=0x7fcb0c15a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb0affd700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb0c159bc0 0x7fcb0c15a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb0affd700 1 -- 172.21.15.111:0/1610456992 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc0094a0 con 0x7fcb0c0c9300 2023-11-28T13:00:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.566+0000 7fcb0b7fe700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb0c159bc0 0x7fcb0c15a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.567+0000 7fcb0affd700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb0c0c9300 0x7fcb0c0c21b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000e810 tx=0x7fcb0000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.567+0000 7fcaf37fe700 1 -- 172.21.15.111:0/1610456992 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00013610 con 0x7fcb0c0c9300 2023-11-28T13:00:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.567+0000 7fcaf37fe700 1 -- 172.21.15.111:0/1610456992 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb00013770 con 0x7fcb0c0c9300 2023-11-28T13:00:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.567+0000 7fcaf37fe700 1 -- 172.21.15.111:0/1610456992 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00013b20 con 0x7fcb0c0c9300 2023-11-28T13:00:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.567+0000 7fcb11391700 1 -- 172.21.15.111:0/1610456992 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb0c071b20 con 0x7fcb0c0c9300 2023-11-28T13:00:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.567+0000 7fcb11391700 1 -- 172.21.15.111:0/1610456992 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb0c06d500 con 0x7fcb0c0c9300 2023-11-28T13:00:28.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.568+0000 7fcb11391700 1 -- 172.21.15.111:0/1610456992 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb0c071020 con 0x7fcb0c0c9300 2023-11-28T13:00:28.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.568+0000 7fcaf37fe700 1 -- 172.21.15.111:0/1610456992 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcb00023070 con 0x7fcb0c0c9300 2023-11-28T13:00:28.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.568+0000 7fcaf37fe700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcaf80682f0 0x7fcaf806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:28.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.569+0000 7fcaf37fe700 1 -- 172.21.15.111:0/1610456992 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fcb00085b20 con 0x7fcb0c0c9300 2023-11-28T13:00:28.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.569+0000 7fcb0a7fc700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcaf80682f0 0x7fcaf806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:28.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.569+0000 7fcb0a7fc700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcaf80682f0 0x7fcaf806a7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcafc0090b0 tx=0x7fcafc01c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:28.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.573+0000 7fcaf37fe700 1 -- 172.21.15.111:0/1610456992 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcb000520d0 con 0x7fcb0c0c9300 2023-11-28T13:00:28.661 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:00:28.664 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:00:28.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.804+0000 7fcb11391700 1 -- 172.21.15.111:0/1610456992 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb0c0b0310 con 0x7fcb0c0c9300 2023-11-28T13:00:28.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.804+0000 7fcaf37fe700 1 -- 172.21.15.111:0/1610456992 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4660 (secure 0 0 0) 0x7fcb000556f0 con 0x7fcb0c0c9300 2023-11-28T13:00:28.805 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:28.805 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3254653481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3254653481},{"type":"v1","addr":"172.21.15.149:6835","nonce":3254653481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1890904311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1890904311},{"type":"v1","addr":"172.21.15.111:6836","nonce":1890904311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1592804351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1592804351},{"type":"v1","addr":"172.21.15.111:6837","nonce":1592804351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:12.011963+0000","modified":"2023-11-28T13:00:15.034385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5352":{"gid":5352,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.149:6834/3579364312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":3579364312},{"type":"v1","addr":"172.21.15.149:6834","nonce":3579364312}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T13:00:28.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.806+0000 7fcb11391700 1 -- 172.21.15.111:0/1610456992 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcaf80682f0 msgr2=0x7fcaf806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:28.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.806+0000 7fcb11391700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcaf80682f0 0x7fcaf806a7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcafc0090b0 tx=0x7fcafc01c040 comp rx=0 tx=0).stop 2023-11-28T13:00:28.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.806+0000 7fcb11391700 1 -- 172.21.15.111:0/1610456992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb0c0c9300 msgr2=0x7fcb0c0c21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:28.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.806+0000 7fcb11391700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb0c0c9300 0x7fcb0c0c21b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000e810 tx=0x7fcb0000eb20 comp rx=0 tx=0).stop 2023-11-28T13:00:28.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.806+0000 7fcb11391700 1 -- 172.21.15.111:0/1610456992 shutdown_connections 2023-11-28T13:00:28.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.806+0000 7fcb11391700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcaf80682f0 0x7fcaf806a7a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.806+0000 7fcb11391700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb0c159bc0 0x7fcb0c15a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.806+0000 7fcb11391700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb0c0c9300 0x7fcb0c0c21b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.806+0000 7fcb11391700 1 --2- 172.21.15.111:0/1610456992 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb0c158cb0 0x7fcb0c15bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:28.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.806+0000 7fcb11391700 1 -- 172.21.15.111:0/1610456992 >> 172.21.15.111:0/1610456992 conn(0x7fcb0c0cb490 msgr2=0x7fcb0c0cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:28.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.806+0000 7fcb11391700 1 -- 172.21.15.111:0/1610456992 shutdown_connections 2023-11-28T13:00:28.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:28.806+0000 7fcb11391700 1 -- 172.21.15.111:0/1610456992 wait complete. 2023-11-28T13:00:28.809 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 17 2023-11-28T13:00:28.819 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json tag path /parentdir mytag 2023-11-28T13:00:28.961 INFO:teuthology.orchestra.run.smithi149.stdout:{"return_code":0,"scrub_tag":"mytag","mode":"asynchronous"} 2023-11-28T13:00:28.961 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag" } 2023-11-28T13:00:38.962 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000000.00000000 scrub_tag 2023-11-28T13:00:39.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.012+0000 7f21bc928080 1 Processor -- start 2023-11-28T13:00:39.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.012+0000 7f21bc928080 1 -- start start 2023-11-28T13:00:39.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21bc928080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bac2e198d0 0x55bac2e594f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21bc928080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bac2e1a280 0x55bac2e1cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21bc928080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bac2e33760 0x55bac2e65050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21bc928080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bac2d55fe0 con 0x55bac2e1a280 2023-11-28T13:00:39.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21bc928080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bac2eeff50 con 0x55bac2e33760 2023-11-28T13:00:39.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21bc928080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bac2e447b0 con 0x55bac2e198d0 2023-11-28T13:00:39.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21b222a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bac2e1a280 0x55bac2e1cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21b222a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bac2e1a280 0x55bac2e1cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35566/0 (socket says 172.21.15.111:35566) 2023-11-28T13:00:39.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21b222a700 1 -- 172.21.15.111:0/2357530912 learned_addr learned my addr 172.21.15.111:0/2357530912 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:39.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21b2a2b700 1 --2- 172.21.15.111:0/2357530912 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bac2e198d0 0x55bac2e594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21b322c700 1 --2- 172.21.15.111:0/2357530912 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bac2e33760 0x55bac2e65050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21b222a700 1 -- 172.21.15.111:0/2357530912 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bac2e198d0 msgr2=0x55bac2e594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:39.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21b222a700 1 --2- 172.21.15.111:0/2357530912 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bac2e198d0 0x55bac2e594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21b222a700 1 -- 172.21.15.111:0/2357530912 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bac2e33760 msgr2=0x55bac2e65050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:39.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21b222a700 1 --2- 172.21.15.111:0/2357530912 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bac2e33760 0x55bac2e65050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21b222a700 1 -- 172.21.15.111:0/2357530912 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a8005d20 con 0x55bac2e1a280 2023-11-28T13:00:39.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21b2a2b700 1 --2- 172.21.15.111:0/2357530912 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bac2e198d0 0x55bac2e594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:39.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21b322c700 1 --2- 172.21.15.111:0/2357530912 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bac2e33760 0x55bac2e65050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:39.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.013+0000 7f21b222a700 1 --2- 172.21.15.111:0/2357530912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bac2e1a280 0x55bac2e1cf70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x55bac2e08a50 tx=0x7f21a800d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4209a84107e53020 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.014+0000 7f21b1228700 1 -- 172.21.15.111:0/2357530912 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8008b60 con 0x55bac2e1a280 2023-11-28T13:00:39.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.014+0000 7f21b1228700 1 -- 172.21.15.111:0/2357530912 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a8008cc0 con 0x55bac2e1a280 2023-11-28T13:00:39.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.014+0000 7f21b1228700 1 -- 172.21.15.111:0/2357530912 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a801e690 con 0x55bac2e1a280 2023-11-28T13:00:39.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.014+0000 7f21bc928080 1 -- 172.21.15.111:0/2357530912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bac2e1a280 msgr2=0x55bac2e1cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.014+0000 7f21bc928080 1 --2- 172.21.15.111:0/2357530912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bac2e1a280 0x55bac2e1cf70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x55bac2e08a50 tx=0x7f21a800d000 comp rx=0 tx=0).stop 2023-11-28T13:00:39.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.014+0000 7f21bc928080 1 -- 172.21.15.111:0/2357530912 shutdown_connections 2023-11-28T13:00:39.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.014+0000 7f21bc928080 1 --2- 172.21.15.111:0/2357530912 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bac2e33760 0x55bac2e65050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.014+0000 7f21bc928080 1 --2- 172.21.15.111:0/2357530912 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bac2e198d0 0x55bac2e594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.014+0000 7f21bc928080 1 --2- 172.21.15.111:0/2357530912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bac2e1a280 0x55bac2e1cf70 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.014+0000 7f21bc928080 1 -- 172.21.15.111:0/2357530912 >> 172.21.15.111:0/2357530912 conn(0x55bac2e635a0 msgr2=0x55bac2eef5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:39.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.014+0000 7f21bc928080 1 -- 172.21.15.111:0/2357530912 shutdown_connections 2023-11-28T13:00:39.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.014+0000 7f21bc928080 1 -- 172.21.15.111:0/2357530912 wait complete. 2023-11-28T13:00:39.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.015+0000 7f21bc928080 1 Processor -- start 2023-11-28T13:00:39.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.015+0000 7f21bc928080 1 -- start start 2023-11-28T13:00:39.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21bc928080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bac2e198d0 0x55bac2eed1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21bc928080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bac2e1a280 0x55bac2ee48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21bc928080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bac2e33760 0x55bac2eed810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21bc928080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bac2ee9900 con 0x55bac2e1a280 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21bc928080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bac2ee4df0 con 0x55bac2e33760 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21bc928080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bac2e475d0 con 0x55bac2e198d0 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21b222a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bac2e1a280 0x55bac2ee48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21b222a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bac2e1a280 0x55bac2ee48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35574/0 (socket says 172.21.15.111:35574) 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21b222a700 1 -- 172.21.15.111:0/2270477065 learned_addr learned my addr 172.21.15.111:0/2270477065 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21b2a2b700 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bac2e198d0 0x55bac2eed1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21b322c700 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bac2e33760 0x55bac2eed810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21b222a700 1 -- 172.21.15.111:0/2270477065 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bac2e198d0 msgr2=0x55bac2eed1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21b222a700 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bac2e198d0 0x55bac2eed1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21b222a700 1 -- 172.21.15.111:0/2270477065 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bac2e33760 msgr2=0x55bac2eed810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21b222a700 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bac2e33760 0x55bac2eed810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21b222a700 1 -- 172.21.15.111:0/2270477065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bac2d884f0 con 0x55bac2e1a280 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21b322c700 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bac2e33760 0x55bac2eed810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:39.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21b2a2b700 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bac2e198d0 0x55bac2eed1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:39.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.016+0000 7f21b222a700 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bac2e1a280 0x55bac2ee48b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f21a8021040 tx=0x7f21a8003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.017+0000 7f219bfff700 1 -- 172.21.15.111:0/2270477065 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a801ea20 con 0x55bac2e1a280 2023-11-28T13:00:39.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.017+0000 7f219bfff700 1 -- 172.21.15.111:0/2270477065 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a801eb80 con 0x55bac2e1a280 2023-11-28T13:00:39.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.017+0000 7f21bc928080 1 -- 172.21.15.111:0/2270477065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bac2d6e120 con 0x55bac2e1a280 2023-11-28T13:00:39.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.017+0000 7f219bfff700 1 -- 172.21.15.111:0/2270477065 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8029450 con 0x55bac2e1a280 2023-11-28T13:00:39.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.017+0000 7f21bc928080 1 -- 172.21.15.111:0/2270477065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bac2d61830 con 0x55bac2e1a280 2023-11-28T13:00:39.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.018+0000 7f219bfff700 1 -- 172.21.15.111:0/2270477065 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f21a80295b0 con 0x55bac2e1a280 2023-11-28T13:00:39.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.019+0000 7f219bfff700 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f21a0066c10 0x7f21a00690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.019+0000 7f219bfff700 1 -- 172.21.15.111:0/2270477065 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f21a8087c40 con 0x55bac2e1a280 2023-11-28T13:00:39.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.019+0000 7f21b2a2b700 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f21a0066c10 0x7f21a00690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.019+0000 7f21bc928080 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x55bac2e66060 0x55bac2eeaa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.019+0000 7f21bc928080 1 -- 172.21.15.111:0/2270477065 --> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] -- osd_op(unknown.0.0:1 6.10 6:0cf6ad0f:::10000000000.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) v8 -- 0x55bac2eeafc0 con 0x55bac2e66060 2023-11-28T13:00:39.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.019+0000 7f21b322c700 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x55bac2e66060 0x55bac2eeaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.019+0000 7f21b2a2b700 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f21a0066c10 0x7f21a00690c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f21a4005fd0 tx=0x7f21a4005e40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.019+0000 7f21b322c700 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x55bac2e66060 0x55bac2eeaa80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.020+0000 7f21b322c700 1 -- 172.21.15.111:0/2270477065 <== osd.0 v2:172.21.15.111:6800/103527 1 ==== osd_op_reply(1 10000000000.00000000 [getxattr (9) out=9b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f21ac003450 con 0x55bac2e66060 2023-11-28T13:00:39.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.020+0000 7f21bc928080 1 -- 172.21.15.111:0/2270477065 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x55bac2e66060 msgr2=0x55bac2eeaa80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.020+0000 7f21bc928080 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x55bac2e66060 0x55bac2eeaa80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.020+0000 7f21bc928080 1 -- 172.21.15.111:0/2270477065 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f21a0066c10 msgr2=0x7f21a00690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.020+0000 7f21bc928080 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f21a0066c10 0x7f21a00690c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f21a4005fd0 tx=0x7f21a4005e40 comp rx=0 tx=0).stop 2023-11-28T13:00:39.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.020+0000 7f21bc928080 1 -- 172.21.15.111:0/2270477065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bac2e1a280 msgr2=0x55bac2ee48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.020+0000 7f21bc928080 1 --2- 172.21.15.111:0/2270477065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bac2e1a280 0x55bac2ee48b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f21a8021040 tx=0x7f21a8003020 comp rx=0 tx=0).stop 2023-11-28T13:00:39.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.020+0000 7f21b322c700 1 -- 172.21.15.111:0/2270477065 reap_dead start 2023-11-28T13:00:39.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.020+0000 7f21bc928080 1 -- 172.21.15.111:0/2270477065 shutdown_connections 2023-11-28T13:00:39.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.020+0000 7f21bc928080 1 -- 172.21.15.111:0/2270477065 >> 172.21.15.111:0/2270477065 conn(0x55bac2e635a0 msgr2=0x55bac2eef5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:39.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.020+0000 7f21bc928080 1 -- 172.21.15.111:0/2270477065 shutdown_connections 2023-11-28T13:00:39.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.020+0000 7f21bc928080 1 -- 172.21.15.111:0/2270477065 wait complete. 2023-11-28T13:00:39.026 INFO:teuthology.orchestra.run.smithi111.stdout:mytag 2023-11-28T13:00:39.027 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000001.00000000 scrub_tag 2023-11-28T13:00:39.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.071+0000 7f9b1aba8080 1 Processor -- start 2023-11-28T13:00:39.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.071+0000 7f9b1aba8080 1 -- start start 2023-11-28T13:00:39.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b1aba8080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55c2b658ebc0 0x55c2b658d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b1aba8080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2b658ef90 0x55c2b66f9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b1aba8080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c2b65a7400 0x55c2b66f9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b1aba8080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b6537000 con 0x55c2b658ef90 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b1aba8080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b64c9fe0 con 0x55c2b65a7400 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b1aba8080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2b65d0dd0 con 0x55c2b658ebc0 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b0bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2b658ef90 0x55c2b66f9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b0bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2b658ef90 0x55c2b66f9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35584/0 (socket says 172.21.15.111:35584) 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b0bfff700 1 -- 172.21.15.111:0/943748784 learned_addr learned my addr 172.21.15.111:0/943748784 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b114ac700 1 --2- 172.21.15.111:0/943748784 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c2b65a7400 0x55c2b66f9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b10cab700 1 --2- 172.21.15.111:0/943748784 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55c2b658ebc0 0x55c2b658d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b0bfff700 1 -- 172.21.15.111:0/943748784 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55c2b658ebc0 msgr2=0x55c2b658d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b0bfff700 1 --2- 172.21.15.111:0/943748784 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55c2b658ebc0 0x55c2b658d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b0bfff700 1 -- 172.21.15.111:0/943748784 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c2b65a7400 msgr2=0x55c2b66f9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b0bfff700 1 --2- 172.21.15.111:0/943748784 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c2b65a7400 0x55c2b66f9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.072+0000 7f9b0bfff700 1 -- 172.21.15.111:0/943748784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc005d20 con 0x55c2b658ef90 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.073+0000 7f9b0bfff700 1 --2- 172.21.15.111:0/943748784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2b658ef90 0x55c2b66f9110 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9b04001db0 tx=0x7f9b0400b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255b0f11e508f92c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.073+0000 7f9b0affd700 1 -- 172.21.15.111:0/943748784 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b04014070 con 0x55c2b658ef90 2023-11-28T13:00:39.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.073+0000 7f9b0affd700 1 -- 172.21.15.111:0/943748784 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b04008030 con 0x55c2b658ef90 2023-11-28T13:00:39.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.073+0000 7f9b0affd700 1 -- 172.21.15.111:0/943748784 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0400bdf0 con 0x55c2b658ef90 2023-11-28T13:00:39.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.073+0000 7f9b1aba8080 1 -- 172.21.15.111:0/943748784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2b658ef90 msgr2=0x55c2b66f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.073+0000 7f9b1aba8080 1 --2- 172.21.15.111:0/943748784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2b658ef90 0x55c2b66f9110 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9b04001db0 tx=0x7f9b0400b4c0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.073+0000 7f9b1aba8080 1 -- 172.21.15.111:0/943748784 shutdown_connections 2023-11-28T13:00:39.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.073+0000 7f9b1aba8080 1 --2- 172.21.15.111:0/943748784 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c2b65a7400 0x55c2b66f9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.073+0000 7f9b1aba8080 1 --2- 172.21.15.111:0/943748784 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55c2b658ebc0 0x55c2b658d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.073+0000 7f9b1aba8080 1 --2- 172.21.15.111:0/943748784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2b658ef90 0x55c2b66f9110 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.073+0000 7f9b1aba8080 1 -- 172.21.15.111:0/943748784 >> 172.21.15.111:0/943748784 conn(0x55c2b65d75a0 msgr2=0x55c2b65cdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:39.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.073+0000 7f9b1aba8080 1 -- 172.21.15.111:0/943748784 shutdown_connections 2023-11-28T13:00:39.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.073+0000 7f9b1aba8080 1 -- 172.21.15.111:0/943748784 wait complete. 2023-11-28T13:00:39.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.074+0000 7f9b1aba8080 1 Processor -- start 2023-11-28T13:00:39.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.074+0000 7f9b1aba8080 1 -- start start 2023-11-28T13:00:39.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b1aba8080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c2b658ebc0 0x55c2b6669060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b1aba8080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55c2b658ef90 0x55c2b658d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b1aba8080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2b65a7400 0x55c2b6664630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b1aba8080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b66f2ac0 con 0x55c2b65a7400 2023-11-28T13:00:39.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b1aba8080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2b66695a0 con 0x55c2b658ebc0 2023-11-28T13:00:39.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b1aba8080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2b6668bd0 con 0x55c2b658ef90 2023-11-28T13:00:39.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b114ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2b65a7400 0x55c2b6664630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b114ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2b65a7400 0x55c2b6664630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35592/0 (socket says 172.21.15.111:35592) 2023-11-28T13:00:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b114ac700 1 -- 172.21.15.111:0/1061458602 learned_addr learned my addr 172.21.15.111:0/1061458602 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b0bfff700 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55c2b658ef90 0x55c2b658d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b10cab700 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c2b658ebc0 0x55c2b6669060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b114ac700 1 -- 172.21.15.111:0/1061458602 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55c2b658ef90 msgr2=0x55c2b658d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b114ac700 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55c2b658ef90 0x55c2b658d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b114ac700 1 -- 172.21.15.111:0/1061458602 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c2b658ebc0 msgr2=0x55c2b6669060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b114ac700 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55c2b658ebc0 0x55c2b6669060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b114ac700 1 -- 172.21.15.111:0/1061458602 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0400a040 con 0x55c2b65a7400 2023-11-28T13:00:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b114ac700 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2b65a7400 0x55c2b6664630 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c006fc0 tx=0x7f9b0c00d440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b09ffb700 1 -- 172.21.15.111:0/1061458602 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c0062e0 con 0x55c2b65a7400 2023-11-28T13:00:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b09ffb700 1 -- 172.21.15.111:0/1061458602 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0c00da60 con 0x55c2b65a7400 2023-11-28T13:00:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b09ffb700 1 -- 172.21.15.111:0/1061458602 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c00de10 con 0x55c2b65a7400 2023-11-28T13:00:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b1aba8080 1 -- 172.21.15.111:0/1061458602 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c2b64d5830 con 0x55c2b65a7400 2023-11-28T13:00:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.075+0000 7f9b1aba8080 1 -- 172.21.15.111:0/1061458602 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2b64e2360 con 0x55c2b65a7400 2023-11-28T13:00:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.077+0000 7f9b09ffb700 1 -- 172.21.15.111:0/1061458602 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9b0c01a030 con 0x55c2b65a7400 2023-11-28T13:00:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.077+0000 7f9b09ffb700 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9af4066c10 0x7f9af40690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.077+0000 7f9b09ffb700 1 -- 172.21.15.111:0/1061458602 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9b0c083ed0 con 0x55c2b65a7400 2023-11-28T13:00:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.077+0000 7f9b10cab700 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9af4066c10 0x7f9af40690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.077+0000 7f9b1aba8080 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55c2b6665570 0x55c2b65b94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.077+0000 7f9b1aba8080 1 -- 172.21.15.111:0/1061458602 --> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] -- osd_op(unknown.0.0:1 6.7 6:e2e07930:::10000000001.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) v8 -- 0x55c2b658d980 con 0x55c2b6665570 2023-11-28T13:00:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.078+0000 7f9b10cab700 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9af4066c10 0x7f9af40690c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x55c2b6593540 tx=0x7f9afc00f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.078+0000 7f9b0bfff700 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55c2b6665570 0x55c2b65b94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.078+0000 7f9b0bfff700 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55c2b6665570 0x55c2b65b94b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.078+0000 7f9b0bfff700 1 -- 172.21.15.111:0/1061458602 <== osd.5 v2:172.21.15.149:6808/103795 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (9) out=9b] v0'0 uv5 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f9b04016070 con 0x55c2b6665570 2023-11-28T13:00:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.078+0000 7f9b1aba8080 1 -- 172.21.15.111:0/1061458602 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55c2b6665570 msgr2=0x55c2b65b94b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.078+0000 7f9b1aba8080 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55c2b6665570 0x55c2b65b94b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.079+0000 7f9b1aba8080 1 -- 172.21.15.111:0/1061458602 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9af4066c10 msgr2=0x7f9af40690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.079+0000 7f9b1aba8080 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9af4066c10 0x7f9af40690c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x55c2b6593540 tx=0x7f9afc00f000 comp rx=0 tx=0).stop 2023-11-28T13:00:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.079+0000 7f9b1aba8080 1 -- 172.21.15.111:0/1061458602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2b65a7400 msgr2=0x55c2b6664630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.079+0000 7f9b1aba8080 1 --2- 172.21.15.111:0/1061458602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c2b65a7400 0x55c2b6664630 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c006fc0 tx=0x7f9b0c00d440 comp rx=0 tx=0).stop 2023-11-28T13:00:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.079+0000 7f9b114ac700 1 -- 172.21.15.111:0/1061458602 reap_dead start 2023-11-28T13:00:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.079+0000 7f9b1aba8080 1 -- 172.21.15.111:0/1061458602 shutdown_connections 2023-11-28T13:00:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.079+0000 7f9b1aba8080 1 -- 172.21.15.111:0/1061458602 >> 172.21.15.111:0/1061458602 conn(0x55c2b65d75a0 msgr2=0x55c2b65cd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.079+0000 7f9b1aba8080 1 -- 172.21.15.111:0/1061458602 shutdown_connections 2023-11-28T13:00:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.079+0000 7f9b1aba8080 1 -- 172.21.15.111:0/1061458602 wait complete. 2023-11-28T13:00:39.085 INFO:teuthology.orchestra.run.smithi111.stdout:mytag 2023-11-28T13:00:39.085 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2023-11-28T13:00:39.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.128+0000 7fda5d2bb080 1 Processor -- start 2023-11-28T13:00:39.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.128+0000 7fda5d2bb080 1 -- start start 2023-11-28T13:00:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.128+0000 7fda5d2bb080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5648574f4bc0 0x5648575347d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.128+0000 7fda5d2bb080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5648574f4f90 0x56485753ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.128+0000 7fda5d2bb080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56485750d400 0x56485751deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.128+0000 7fda5d2bb080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56485742ffe0 con 0x5648574f4bc0 2023-11-28T13:00:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.128+0000 7fda5d2bb080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648575ca420 con 0x5648574f4f90 2023-11-28T13:00:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.128+0000 7fda5d2bb080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5648574e2f20 con 0x56485750d400 2023-11-28T13:00:39.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.128+0000 7fda533be700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5648574f4bc0 0x5648575347d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.128+0000 7fda533be700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5648574f4bc0 0x5648575347d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35604/0 (socket says 172.21.15.111:35604) 2023-11-28T13:00:39.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.128+0000 7fda533be700 1 -- 172.21.15.111:0/3124585558 learned_addr learned my addr 172.21.15.111:0/3124585558 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:39.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.129+0000 7fda52bbd700 1 --2- 172.21.15.111:0/3124585558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5648574f4f90 0x56485753ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.129+0000 7fda533be700 1 -- 172.21.15.111:0/3124585558 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56485750d400 msgr2=0x56485751deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:39.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.129+0000 7fda53bbf700 1 --2- 172.21.15.111:0/3124585558 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56485750d400 0x56485751deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.129+0000 7fda533be700 1 --2- 172.21.15.111:0/3124585558 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56485750d400 0x56485751deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.129+0000 7fda533be700 1 -- 172.21.15.111:0/3124585558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5648574f4f90 msgr2=0x56485753ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:39.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.129+0000 7fda533be700 1 --2- 172.21.15.111:0/3124585558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5648574f4f90 0x56485753ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.129+0000 7fda533be700 1 -- 172.21.15.111:0/3124585558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda48005d20 con 0x5648574f4bc0 2023-11-28T13:00:39.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.129+0000 7fda533be700 1 --2- 172.21.15.111:0/3124585558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5648574f4bc0 0x5648575347d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00b1c0 tx=0x7fda4c00b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16ab255821ca41a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.129+0000 7fda51bbb700 1 -- 172.21.15.111:0/3124585558 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c00f5a0 con 0x5648574f4bc0 2023-11-28T13:00:39.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.129+0000 7fda51bbb700 1 -- 172.21.15.111:0/3124585558 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c00a040 con 0x5648574f4bc0 2023-11-28T13:00:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.129+0000 7fda51bbb700 1 -- 172.21.15.111:0/3124585558 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c00f960 con 0x5648574f4bc0 2023-11-28T13:00:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.129+0000 7fda5d2bb080 1 -- 172.21.15.111:0/3124585558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5648574f4bc0 msgr2=0x5648575347d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.129+0000 7fda5d2bb080 1 --2- 172.21.15.111:0/3124585558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5648574f4bc0 0x5648575347d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00b1c0 tx=0x7fda4c00b4d0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.130+0000 7fda5d2bb080 1 -- 172.21.15.111:0/3124585558 shutdown_connections 2023-11-28T13:00:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.130+0000 7fda5d2bb080 1 --2- 172.21.15.111:0/3124585558 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56485750d400 0x56485751deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.130+0000 7fda5d2bb080 1 --2- 172.21.15.111:0/3124585558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5648574f4bc0 0x5648575347d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.130+0000 7fda5d2bb080 1 --2- 172.21.15.111:0/3124585558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5648574f4f90 0x56485753ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.130+0000 7fda5d2bb080 1 -- 172.21.15.111:0/3124585558 >> 172.21.15.111:0/3124585558 conn(0x56485753d5a0 msgr2=0x56485753f650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.130+0000 7fda5d2bb080 1 -- 172.21.15.111:0/3124585558 shutdown_connections 2023-11-28T13:00:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.130+0000 7fda5d2bb080 1 -- 172.21.15.111:0/3124585558 wait complete. 2023-11-28T13:00:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.130+0000 7fda5d2bb080 1 Processor -- start 2023-11-28T13:00:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.130+0000 7fda5d2bb080 1 -- start start 2023-11-28T13:00:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda5d2bb080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5648574f4bc0 0x56485751e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda5d2bb080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5648574f4f90 0x5648575c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda5d2bb080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56485750d400 0x5648576631f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda5d2bb080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648575ca100 con 0x56485750d400 2023-11-28T13:00:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda5d2bb080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648575c7400 con 0x5648574f4bc0 2023-11-28T13:00:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda5d2bb080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5648575c9760 con 0x5648574f4f90 2023-11-28T13:00:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda53bbf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56485750d400 0x5648576631f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda533be700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5648574f4bc0 0x56485751e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda53bbf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56485750d400 0x5648576631f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35614/0 (socket says 172.21.15.111:35614) 2023-11-28T13:00:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda53bbf700 1 -- 172.21.15.111:0/1647822274 learned_addr learned my addr 172.21.15.111:0/1647822274 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda52bbd700 1 --2- 172.21.15.111:0/1647822274 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5648574f4f90 0x5648575c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda53bbf700 1 -- 172.21.15.111:0/1647822274 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5648574f4f90 msgr2=0x5648575c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda53bbf700 1 --2- 172.21.15.111:0/1647822274 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5648574f4f90 0x5648575c6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda53bbf700 1 -- 172.21.15.111:0/1647822274 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5648574f4bc0 msgr2=0x56485751e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda53bbf700 1 --2- 172.21.15.111:0/1647822274 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5648574f4bc0 0x56485751e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda53bbf700 1 -- 172.21.15.111:0/1647822274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c001b40 con 0x56485750d400 2023-11-28T13:00:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.131+0000 7fda53bbf700 1 --2- 172.21.15.111:0/1647822274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56485750d400 0x5648576631f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fda4400afd0 tx=0x7fda4400e4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.132+0000 7fda50bb9700 1 -- 172.21.15.111:0/1647822274 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4400eed0 con 0x56485750d400 2023-11-28T13:00:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.132+0000 7fda50bb9700 1 -- 172.21.15.111:0/1647822274 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda44013460 con 0x56485750d400 2023-11-28T13:00:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.132+0000 7fda5d2bb080 1 -- 172.21.15.111:0/1647822274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564857448120 con 0x56485750d400 2023-11-28T13:00:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.132+0000 7fda50bb9700 1 -- 172.21.15.111:0/1647822274 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda440137f0 con 0x56485750d400 2023-11-28T13:00:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.132+0000 7fda5d2bb080 1 -- 172.21.15.111:0/1647822274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56485743b830 con 0x56485750d400 2023-11-28T13:00:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.133+0000 7fda50bb9700 1 -- 172.21.15.111:0/1647822274 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda44013950 con 0x56485750d400 2023-11-28T13:00:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.133+0000 7fda50bb9700 1 --2- 172.21.15.111:0/1647822274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fda40066c10 0x7fda400690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.133+0000 7fda533be700 1 --2- 172.21.15.111:0/1647822274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fda40066c10 0x7fda400690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.133+0000 7fda50bb9700 1 -- 172.21.15.111:0/1647822274 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fda44084d00 con 0x56485750d400 2023-11-28T13:00:39.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda5d2bb080 1 --2- 172.21.15.111:0/1647822274 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x5648575c7b20 0x56485753da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda5d2bb080 1 -- 172.21.15.111:0/1647822274 --> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] -- osd_op(unknown.0.0:1 7.36 7:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) v8 -- 0x56485753dfc0 con 0x5648575c7b20 2023-11-28T13:00:39.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda52bbd700 1 --2- 172.21.15.111:0/1647822274 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x5648575c7b20 0x56485753da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda533be700 1 --2- 172.21.15.111:0/1647822274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fda40066c10 0x7fda400690c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00b1c0 tx=0x7fda4c002000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda52bbd700 1 --2- 172.21.15.111:0/1647822274 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x5648575c7b20 0x56485753da80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda52bbd700 1 -- 172.21.15.111:0/1647822274 <== osd.0 v2:172.21.15.111:6800/103527 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr (9) out=9b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7fda48019040 con 0x5648575c7b20 2023-11-28T13:00:39.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda5d2bb080 1 -- 172.21.15.111:0/1647822274 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x5648575c7b20 msgr2=0x56485753da80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda5d2bb080 1 --2- 172.21.15.111:0/1647822274 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x5648575c7b20 0x56485753da80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda5d2bb080 1 -- 172.21.15.111:0/1647822274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fda40066c10 msgr2=0x7fda400690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda5d2bb080 1 --2- 172.21.15.111:0/1647822274 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fda40066c10 0x7fda400690c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00b1c0 tx=0x7fda4c002000 comp rx=0 tx=0).stop 2023-11-28T13:00:39.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda5d2bb080 1 -- 172.21.15.111:0/1647822274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56485750d400 msgr2=0x5648576631f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda5d2bb080 1 --2- 172.21.15.111:0/1647822274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56485750d400 0x5648576631f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fda4400afd0 tx=0x7fda4400e4c0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda53bbf700 1 -- 172.21.15.111:0/1647822274 reap_dead start 2023-11-28T13:00:39.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda5d2bb080 1 -- 172.21.15.111:0/1647822274 shutdown_connections 2023-11-28T13:00:39.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda5d2bb080 1 -- 172.21.15.111:0/1647822274 >> 172.21.15.111:0/1647822274 conn(0x56485753d5a0 msgr2=0x564857533e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:39.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda5d2bb080 1 -- 172.21.15.111:0/1647822274 shutdown_connections 2023-11-28T13:00:39.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.134+0000 7fda5d2bb080 1 -- 172.21.15.111:0/1647822274 wait complete. 2023-11-28T13:00:39.146 INFO:teuthology.orchestra.run.smithi111.stdout:mytag 2023-11-28T13:00:39.146 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2023-11-28T13:00:39.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.188+0000 7f06acf16080 1 Processor -- start 2023-11-28T13:00:39.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.188+0000 7f06acf16080 1 -- start start 2023-11-28T13:00:39.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06acf16080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55b4077ce140 0x55b4077ed110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06acf16080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4077ce510 0x55b4077cc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06acf16080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55b4077d2d20 0x55b4077cc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06acf16080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b40774c000 con 0x55b4077ce510 2023-11-28T13:00:39.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06acf16080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4076defe0 con 0x55b4077d2d20 2023-11-28T13:00:39.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06acf16080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b4077cdd90 con 0x55b4077ce140 2023-11-28T13:00:39.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a2818700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4077ce510 0x55b4077cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a2818700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4077ce510 0x55b4077cc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35626/0 (socket says 172.21.15.111:35626) 2023-11-28T13:00:39.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a2818700 1 -- 172.21.15.111:0/1485658519 learned_addr learned my addr 172.21.15.111:0/1485658519 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:39.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a3019700 1 --2- 172.21.15.111:0/1485658519 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55b4077ce140 0x55b4077ed110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a381a700 1 --2- 172.21.15.111:0/1485658519 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55b4077d2d20 0x55b4077cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a2818700 1 -- 172.21.15.111:0/1485658519 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55b4077ce140 msgr2=0x55b4077ed110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:39.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a2818700 1 --2- 172.21.15.111:0/1485658519 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55b4077ce140 0x55b4077ed110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a2818700 1 -- 172.21.15.111:0/1485658519 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55b4077d2d20 msgr2=0x55b4077cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:39.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a2818700 1 --2- 172.21.15.111:0/1485658519 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55b4077d2d20 0x55b4077cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a2818700 1 -- 172.21.15.111:0/1485658519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0698005d20 con 0x55b4077ce510 2023-11-28T13:00:39.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a2818700 1 --2- 172.21.15.111:0/1485658519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4077ce510 0x55b4077cc000 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f069400b320 tx=0x7f069400b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20d135f18b14d145 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a1816700 1 -- 172.21.15.111:0/1485658519 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694014070 con 0x55b4077ce510 2023-11-28T13:00:39.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a1816700 1 -- 172.21.15.111:0/1485658519 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069400f3f0 con 0x55b4077ce510 2023-11-28T13:00:39.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.189+0000 7f06a1816700 1 -- 172.21.15.111:0/1485658519 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069400f7a0 con 0x55b4077ce510 2023-11-28T13:00:39.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.190+0000 7f06acf16080 1 -- 172.21.15.111:0/1485658519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4077ce510 msgr2=0x55b4077cc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.190+0000 7f06acf16080 1 --2- 172.21.15.111:0/1485658519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4077ce510 0x55b4077cc000 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f069400b320 tx=0x7f069400b630 comp rx=0 tx=0).stop 2023-11-28T13:00:39.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.190+0000 7f06acf16080 1 -- 172.21.15.111:0/1485658519 shutdown_connections 2023-11-28T13:00:39.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.190+0000 7f06acf16080 1 --2- 172.21.15.111:0/1485658519 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55b4077d2d20 0x55b4077cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.190+0000 7f06acf16080 1 --2- 172.21.15.111:0/1485658519 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55b4077ce140 0x55b4077ed110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.190+0000 7f06acf16080 1 --2- 172.21.15.111:0/1485658519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4077ce510 0x55b4077cc000 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.190+0000 7f06acf16080 1 -- 172.21.15.111:0/1485658519 >> 172.21.15.111:0/1485658519 conn(0x55b4077ec5a0 msgr2=0x55b4077d1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.190+0000 7f06acf16080 1 -- 172.21.15.111:0/1485658519 shutdown_connections 2023-11-28T13:00:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.190+0000 7f06acf16080 1 -- 172.21.15.111:0/1485658519 wait complete. 2023-11-28T13:00:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.190+0000 7f06acf16080 1 Processor -- start 2023-11-28T13:00:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.190+0000 7f06acf16080 1 -- start start 2023-11-28T13:00:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.191+0000 7f06acf16080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55b4077ce140 0x55b4077e1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.191+0000 7f06acf16080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4077ce510 0x55b4077d0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.191+0000 7f06acf16080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55b4077d2d20 0x55b4077d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.191+0000 7f06acf16080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b40787b140 con 0x55b4077ce510 2023-11-28T13:00:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.191+0000 7f06acf16080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4077cfbd0 con 0x55b4077ce140 2023-11-28T13:00:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.191+0000 7f06acf16080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b4077d1eb0 con 0x55b4077d2d20 2023-11-28T13:00:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.191+0000 7f06a2818700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4077ce510 0x55b4077d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.191+0000 7f06a2818700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4077ce510 0x55b4077d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35632/0 (socket says 172.21.15.111:35632) 2023-11-28T13:00:39.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.191+0000 7f06a2818700 1 -- 172.21.15.111:0/2502216963 learned_addr learned my addr 172.21.15.111:0/2502216963 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:39.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.191+0000 7f06a3019700 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55b4077ce140 0x55b4077e1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.191+0000 7f06a381a700 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55b4077d2d20 0x55b4077d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.192+0000 7f06a2818700 1 -- 172.21.15.111:0/2502216963 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55b4077d2d20 msgr2=0x55b4077d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.192+0000 7f06a2818700 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55b4077d2d20 0x55b4077d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.192+0000 7f06a2818700 1 -- 172.21.15.111:0/2502216963 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55b4077ce140 msgr2=0x55b4077e1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.192+0000 7f06a2818700 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55b4077ce140 0x55b4077e1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.192+0000 7f06a2818700 1 -- 172.21.15.111:0/2502216963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0694001ba0 con 0x55b4077ce510 2023-11-28T13:00:39.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.192+0000 7f06a2818700 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4077ce510 0x55b4077d0160 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x55b4077e2cf0 tx=0x7f0694000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.192+0000 7f06a0814700 1 -- 172.21.15.111:0/2502216963 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694014070 con 0x55b4077ce510 2023-11-28T13:00:39.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.192+0000 7f06a0814700 1 -- 172.21.15.111:0/2502216963 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069401c870 con 0x55b4077ce510 2023-11-28T13:00:39.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.192+0000 7f06acf16080 1 -- 172.21.15.111:0/2502216963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b4076ea830 con 0x55b4077ce510 2023-11-28T13:00:39.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.192+0000 7f06a0814700 1 -- 172.21.15.111:0/2502216963 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069401cc20 con 0x55b4077ce510 2023-11-28T13:00:39.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.192+0000 7f06acf16080 1 -- 172.21.15.111:0/2502216963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4076f7360 con 0x55b4077ce510 2023-11-28T13:00:39.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.193+0000 7f06a0814700 1 -- 172.21.15.111:0/2502216963 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f069402c070 con 0x55b4077ce510 2023-11-28T13:00:39.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.193+0000 7f06a0814700 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0690066c10 0x7f06900690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.193+0000 7f06a0814700 1 -- 172.21.15.111:0/2502216963 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f069400a040 con 0x55b4077ce510 2023-11-28T13:00:39.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.194+0000 7f06a3019700 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0690066c10 0x7f06900690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.194+0000 7f06acf16080 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x55b4077d0be0 0x55b4077e72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.194+0000 7f06acf16080 1 -- 172.21.15.111:0/2502216963 --> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] -- osd_op(unknown.0.0:1 7.19 7:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) v8 -- 0x55b4077eb270 con 0x55b4077d0be0 2023-11-28T13:00:39.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.194+0000 7f06a381a700 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x55b4077d0be0 0x55b4077e72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.194+0000 7f06a3019700 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0690066c10 0x7f06900690c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f069800c010 tx=0x7f069800f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.194+0000 7f06a381a700 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x55b4077d0be0 0x55b4077e72e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.195+0000 7f06a381a700 1 -- 172.21.15.111:0/2502216963 <== osd.6 v2:172.21.15.149:6816/103796 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (9) out=9b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f069c00d690 con 0x55b4077d0be0 2023-11-28T13:00:39.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.195+0000 7f06acf16080 1 -- 172.21.15.111:0/2502216963 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x55b4077d0be0 msgr2=0x55b4077e72e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.195+0000 7f06acf16080 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x55b4077d0be0 0x55b4077e72e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.195+0000 7f06acf16080 1 -- 172.21.15.111:0/2502216963 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0690066c10 msgr2=0x7f06900690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.195+0000 7f06acf16080 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0690066c10 0x7f06900690c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f069800c010 tx=0x7f069800f000 comp rx=0 tx=0).stop 2023-11-28T13:00:39.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.195+0000 7f06acf16080 1 -- 172.21.15.111:0/2502216963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4077ce510 msgr2=0x55b4077d0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.195+0000 7f06acf16080 1 --2- 172.21.15.111:0/2502216963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b4077ce510 0x55b4077d0160 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x55b4077e2cf0 tx=0x7f0694000ed0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.195+0000 7f06a381a700 1 -- 172.21.15.111:0/2502216963 reap_dead start 2023-11-28T13:00:39.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.195+0000 7f06acf16080 1 -- 172.21.15.111:0/2502216963 shutdown_connections 2023-11-28T13:00:39.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.195+0000 7f06acf16080 1 -- 172.21.15.111:0/2502216963 >> 172.21.15.111:0/2502216963 conn(0x55b4077ec5a0 msgr2=0x55b4077cc000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:39.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.195+0000 7f06acf16080 1 -- 172.21.15.111:0/2502216963 shutdown_connections 2023-11-28T13:00:39.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.195+0000 7f06acf16080 1 -- 172.21.15.111:0/2502216963 wait complete. 2023-11-28T13:00:39.206 INFO:teuthology.orchestra.run.smithi111.stdout:mytag 2023-11-28T13:00:39.206 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2023-11-28T13:00:39.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.288+0000 7f3ef7e01080 1 Processor -- start 2023-11-28T13:00:39.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.288+0000 7f3ef7e01080 1 -- start start 2023-11-28T13:00:39.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3ef7e01080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ddf7adabc0 0x55ddf7ad9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3ef7e01080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55ddf7adaf90 0x55ddf7ad9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3ef7e01080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55ddf7b0a710 0x55ddf7b1bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3ef7e01080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddf7bad3d0 con 0x55ddf7adabc0 2023-11-28T13:00:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3ef7e01080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddf7ac9350 con 0x55ddf7adaf90 2023-11-28T13:00:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3ef7e01080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ddf7ac90b0 con 0x55ddf7b0a710 2023-11-28T13:00:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3eedf04700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ddf7adabc0 0x55ddf7ad9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3eedf04700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ddf7adabc0 0x55ddf7ad9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35646/0 (socket says 172.21.15.111:35646) 2023-11-28T13:00:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3eedf04700 1 -- 172.21.15.111:0/502689467 learned_addr learned my addr 172.21.15.111:0/502689467 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3eedf04700 1 -- 172.21.15.111:0/502689467 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55ddf7b0a710 msgr2=0x55ddf7b1bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3eedf04700 1 --2- 172.21.15.111:0/502689467 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55ddf7b0a710 0x55ddf7b1bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3eedf04700 1 -- 172.21.15.111:0/502689467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55ddf7adaf90 msgr2=0x55ddf7ad9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:39.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3eedf04700 1 --2- 172.21.15.111:0/502689467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55ddf7adaf90 0x55ddf7ad9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3eedf04700 1 -- 172.21.15.111:0/502689467 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee4005de0 con 0x55ddf7adabc0 2023-11-28T13:00:39.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3eedf04700 1 --2- 172.21.15.111:0/502689467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ddf7adabc0 0x55ddf7ad9040 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4015040 tx=0x7f3ee400d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92ed88fe0600af3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3edffff700 1 -- 172.21.15.111:0/502689467 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4008670 con 0x55ddf7adabc0 2023-11-28T13:00:39.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3edffff700 1 -- 172.21.15.111:0/502689467 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee4008b40 con 0x55ddf7adabc0 2023-11-28T13:00:39.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.289+0000 7f3edffff700 1 -- 172.21.15.111:0/502689467 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee401c410 con 0x55ddf7adabc0 2023-11-28T13:00:39.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.290+0000 7f3ef7e01080 1 -- 172.21.15.111:0/502689467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ddf7adabc0 msgr2=0x55ddf7ad9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.290+0000 7f3ef7e01080 1 --2- 172.21.15.111:0/502689467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ddf7adabc0 0x55ddf7ad9040 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4015040 tx=0x7f3ee400d000 comp rx=0 tx=0).stop 2023-11-28T13:00:39.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.290+0000 7f3ef7e01080 1 -- 172.21.15.111:0/502689467 shutdown_connections 2023-11-28T13:00:39.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.290+0000 7f3ef7e01080 1 --2- 172.21.15.111:0/502689467 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55ddf7b0a710 0x55ddf7b1bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.290+0000 7f3ef7e01080 1 --2- 172.21.15.111:0/502689467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ddf7adabc0 0x55ddf7ad9040 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.290+0000 7f3ef7e01080 1 --2- 172.21.15.111:0/502689467 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55ddf7adaf90 0x55ddf7ad9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.290+0000 7f3ef7e01080 1 -- 172.21.15.111:0/502689467 >> 172.21.15.111:0/502689467 conn(0x55ddf7b235a0 msgr2=0x55ddf7b06fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:39.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.290+0000 7f3ef7e01080 1 -- 172.21.15.111:0/502689467 shutdown_connections 2023-11-28T13:00:39.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.290+0000 7f3ef7e01080 1 -- 172.21.15.111:0/502689467 wait complete. 2023-11-28T13:00:39.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.290+0000 7f3ef7e01080 1 Processor -- start 2023-11-28T13:00:39.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.291+0000 7f3ef7e01080 1 -- start start 2023-11-28T13:00:39.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.291+0000 7f3ef7e01080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ddf7adabc0 0x55ddf7b1c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.291+0000 7f3ef7e01080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55ddf7adaf90 0x55ddf7b06290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.291+0000 7f3ef7e01080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55ddf7b0a710 0x55ddf7bab390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.291+0000 7f3ef7e01080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddf7b05e70 con 0x55ddf7adabc0 2023-11-28T13:00:39.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.291+0000 7f3ef7e01080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddf7b055c0 con 0x55ddf7adaf90 2023-11-28T13:00:39.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.291+0000 7f3ef7e01080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ddf7ada470 con 0x55ddf7b0a710 2023-11-28T13:00:39.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.291+0000 7f3eedf04700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ddf7adabc0 0x55ddf7b1c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.291+0000 7f3eedf04700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ddf7adabc0 0x55ddf7b1c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35652/0 (socket says 172.21.15.111:35652) 2023-11-28T13:00:39.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.291+0000 7f3eedf04700 1 -- 172.21.15.111:0/2502621357 learned_addr learned my addr 172.21.15.111:0/2502621357 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:39.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.291+0000 7f3eed703700 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55ddf7adaf90 0x55ddf7b06290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.291+0000 7f3eee705700 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55ddf7b0a710 0x55ddf7bab390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.292+0000 7f3eedf04700 1 -- 172.21.15.111:0/2502621357 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55ddf7b0a710 msgr2=0x55ddf7bab390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.292+0000 7f3eedf04700 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55ddf7b0a710 0x55ddf7bab390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.292+0000 7f3eedf04700 1 -- 172.21.15.111:0/2502621357 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55ddf7adaf90 msgr2=0x55ddf7b06290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.292+0000 7f3eedf04700 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55ddf7adaf90 0x55ddf7b06290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.292+0000 7f3eedf04700 1 -- 172.21.15.111:0/2502621357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ddf7a214a0 con 0x55ddf7adabc0 2023-11-28T13:00:39.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.292+0000 7f3eedf04700 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ddf7adabc0 0x55ddf7b1c660 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400a8c0 tx=0x7f3ee4003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.292+0000 7f3edeffd700 1 -- 172.21.15.111:0/2502621357 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee401c930 con 0x55ddf7adabc0 2023-11-28T13:00:39.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.292+0000 7f3edeffd700 1 -- 172.21.15.111:0/2502621357 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee402a070 con 0x55ddf7adabc0 2023-11-28T13:00:39.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.292+0000 7f3ef7e01080 1 -- 172.21.15.111:0/2502621357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ddf7a484f0 con 0x55ddf7adabc0 2023-11-28T13:00:39.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.292+0000 7f3edeffd700 1 -- 172.21.15.111:0/2502621357 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee401ccb0 con 0x55ddf7adabc0 2023-11-28T13:00:39.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.292+0000 7f3ef7e01080 1 -- 172.21.15.111:0/2502621357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ddf7a2e120 con 0x55ddf7adabc0 2023-11-28T13:00:39.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.293+0000 7f3edeffd700 1 -- 172.21.15.111:0/2502621357 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3ee4035020 con 0x55ddf7adabc0 2023-11-28T13:00:39.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.293+0000 7f3edeffd700 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3ed0066c10 0x7f3ed00690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.294+0000 7f3edeffd700 1 -- 172.21.15.111:0/2502621357 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3ee4087210 con 0x55ddf7adabc0 2023-11-28T13:00:39.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.294+0000 7f3eed703700 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3ed0066c10 0x7f3ed00690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T13:00:39.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.294+0000 7f3ef7e01080 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x55ddf7c49270 0x55ddf7c496b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.294+0000 7f3ef7e01080 1 -- 172.21.15.111:0/2502621357 --> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] -- osd_op(unknown.0.0:1 7.12 7:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) v8 -- 0x55ddf7c49bf0 con 0x55ddf7c49270 2023-11-28T13:00:39.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.294+0000 7f3eed703700 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3ed0066c10 0x7f3ed00690c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x55ddf7bacaf0 tx=0x7f3ee000c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.294+0000 7f3eee705700 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x55ddf7c49270 0x55ddf7c496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.294+0000 7f3eee705700 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x55ddf7c49270 0x55ddf7c496b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.295+0000 7f3eee705700 1 -- 172.21.15.111:0/2502621357 <== osd.6 v2:172.21.15.149:6816/103796 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f3ee800c420 con 0x55ddf7c49270 2023-11-28T13:00:39.306 INFO:teuthology.orchestra.run.smithi111.stderr:error getting xattr cephfs_data/10000000002.00000000/scrub_tag: (61) No data available 2023-11-28T13:00:39.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.295+0000 7f3ef7e01080 1 -- 172.21.15.111:0/2502621357 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x55ddf7c49270 msgr2=0x55ddf7c496b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.295+0000 7f3ef7e01080 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x55ddf7c49270 0x55ddf7c496b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.295+0000 7f3ef7e01080 1 -- 172.21.15.111:0/2502621357 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3ed0066c10 msgr2=0x7f3ed00690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.295+0000 7f3ef7e01080 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3ed0066c10 0x7f3ed00690c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x55ddf7bacaf0 tx=0x7f3ee000c040 comp rx=0 tx=0).stop 2023-11-28T13:00:39.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.295+0000 7f3ef7e01080 1 -- 172.21.15.111:0/2502621357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ddf7adabc0 msgr2=0x55ddf7b1c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.295+0000 7f3ef7e01080 1 --2- 172.21.15.111:0/2502621357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ddf7adabc0 0x55ddf7b1c660 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400a8c0 tx=0x7f3ee4003020 comp rx=0 tx=0).stop 2023-11-28T13:00:39.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.295+0000 7f3eee705700 1 -- 172.21.15.111:0/2502621357 reap_dead start 2023-11-28T13:00:39.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.295+0000 7f3ef7e01080 1 -- 172.21.15.111:0/2502621357 shutdown_connections 2023-11-28T13:00:39.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.295+0000 7f3ef7e01080 1 -- 172.21.15.111:0/2502621357 >> 172.21.15.111:0/2502621357 conn(0x55ddf7b235a0 msgr2=0x55ddf7b1bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:39.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.295+0000 7f3ef7e01080 1 -- 172.21.15.111:0/2502621357 shutdown_connections 2023-11-28T13:00:39.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.295+0000 7f3ef7e01080 1 -- 172.21.15.111:0/2502621357 wait complete. 2023-11-28T13:00:39.307 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T13:00:39.307 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T13:00:39.390 DEBUG:teuthology.orchestra.run.smithi149:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T13:00:39.442 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T13:00:39.443 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:00:39.443 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:00:39.460 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284268k,nr_inodes=4071067,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18839 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-28T13:00:39.460 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T13:00:39.460 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:00:39.479 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8026740k,nr_inodes=2006685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16040 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=1609048k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-28T13:00:39.479 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:00:39.479 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:00:39.517 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284268k,nr_inodes=4071067,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18839 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-28T13:00:39.518 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T13:00:39.518 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:00:39.538 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8026740k,nr_inodes=2006685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16040 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=1609048k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-28T13:00:39.538 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:00:39.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.678+0000 7fbe3836c700 1 Processor -- start 2023-11-28T13:00:39.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.678+0000 7fbe3836c700 1 -- start start 2023-11-28T13:00:39.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.679+0000 7fbe3836c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fbe3015fed0 0x7fbe300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.679+0000 7fbe3836c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbe300cf7d0 0x7fbe300b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.679+0000 7fbe3836c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3006ac80 con 0x7fbe300af5a0 2023-11-28T13:00:39.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.679+0000 7fbe3836c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe300cee00 con 0x7fbe300cf7d0 2023-11-28T13:00:39.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.679+0000 7fbe3836c700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe3006aed0 con 0x7fbe3015fed0 2023-11-28T13:00:39.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.680+0000 7fbe35907700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbe300cf7d0 0x7fbe300b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.680+0000 7fbe35907700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbe300cf7d0 0x7fbe300b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35664/0 (socket says 172.21.15.111:35664) 2023-11-28T13:00:39.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.680+0000 7fbe35907700 1 -- 172.21.15.111:0/1805517541 learned_addr learned my addr 172.21.15.111:0/1805517541 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:39.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.680+0000 7fbe35907700 1 -- 172.21.15.111:0/1805517541 >> v2:172.21.15.149:3300/0 conn(0x7fbe3015fed0 msgr2=0x7fbe300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:39.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.680+0000 7fbe35907700 1 --2- 172.21.15.111:0/1805517541 >> v2:172.21.15.149:3300/0 conn(0x7fbe3015fed0 0x7fbe300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.680+0000 7fbe35907700 1 -- 172.21.15.111:0/1805517541 >> v1:172.21.15.149:6789/0 conn(0x7fbe300af5a0 legacy=0x7fbe300ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:39.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.680+0000 7fbe35907700 1 -- 172.21.15.111:0/1805517541 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe300cea90 con 0x7fbe300cf7d0 2023-11-28T13:00:39.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.680+0000 7fbe35907700 1 --2- 172.21.15.111:0/1805517541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe300cf7d0 0x7fbe300b04b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbe3006ced0 tx=0x7fbe2c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dece65a205aefd82 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.680+0000 7fbe35106700 1 -- 172.21.15.111:0/1805517541 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c004230 con 0x7fbe300cf7d0 2023-11-28T13:00:39.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.680+0000 7fbe35106700 1 -- 172.21.15.111:0/1805517541 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2c004390 con 0x7fbe300cf7d0 2023-11-28T13:00:39.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.680+0000 7fbe35106700 1 -- 172.21.15.111:0/1805517541 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c0036a0 con 0x7fbe300cf7d0 2023-11-28T13:00:39.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.681+0000 7fbe3836c700 1 -- 172.21.15.111:0/1805517541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe300cf7d0 msgr2=0x7fbe300b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.681+0000 7fbe3836c700 1 --2- 172.21.15.111:0/1805517541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe300cf7d0 0x7fbe300b04b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbe3006ced0 tx=0x7fbe2c009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.681+0000 7fbe3836c700 1 -- 172.21.15.111:0/1805517541 shutdown_connections 2023-11-28T13:00:39.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.681+0000 7fbe3836c700 1 --2- 172.21.15.111:0/1805517541 >> v2:172.21.15.149:3300/0 conn(0x7fbe3015fed0 0x7fbe300cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.681+0000 7fbe3836c700 1 --2- 172.21.15.111:0/1805517541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe300cf7d0 0x7fbe300b04b0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.681+0000 7fbe3836c700 1 -- 172.21.15.111:0/1805517541 >> 172.21.15.111:0/1805517541 conn(0x7fbe3006bca0 msgr2=0x7fbe300cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:39.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.681+0000 7fbe3836c700 1 -- 172.21.15.111:0/1805517541 shutdown_connections 2023-11-28T13:00:39.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.681+0000 7fbe3836c700 1 -- 172.21.15.111:0/1805517541 wait complete. 2023-11-28T13:00:39.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.682+0000 7fbe3836c700 1 Processor -- start 2023-11-28T13:00:39.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.682+0000 7fbe3836c700 1 -- start start 2023-11-28T13:00:39.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe3836c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe300af5a0 0x7fbe301f19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe3836c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbe300cf7d0 0x7fbe3015d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe3836c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbe3015fed0 0x7fbe3015d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe3836c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe30155970 con 0x7fbe300af5a0 2023-11-28T13:00:39.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe3836c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300b3990 con 0x7fbe300cf7d0 2023-11-28T13:00:39.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe3836c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe301f1f10 con 0x7fbe3015fed0 2023-11-28T13:00:39.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe36108700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe300af5a0 0x7fbe301f19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe36108700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe300af5a0 0x7fbe301f19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35668/0 (socket says 172.21.15.111:35668) 2023-11-28T13:00:39.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe36108700 1 -- 172.21.15.111:0/1589214991 learned_addr learned my addr 172.21.15.111:0/1589214991 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:39.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe35907700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbe300cf7d0 0x7fbe3015d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe36909700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbe3015fed0 0x7fbe3015d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe36108700 1 -- 172.21.15.111:0/1589214991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbe3015fed0 msgr2=0x7fbe3015d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe36108700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbe3015fed0 0x7fbe3015d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe36108700 1 -- 172.21.15.111:0/1589214991 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbe300cf7d0 msgr2=0x7fbe3015d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe36108700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbe300cf7d0 0x7fbe3015d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe36108700 1 -- 172.21.15.111:0/1589214991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2c009550 con 0x7fbe300af5a0 2023-11-28T13:00:39.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe36909700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbe3015fed0 0x7fbe3015d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:39.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe36108700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe300af5a0 0x7fbe301f19d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000b4d0 tx=0x7fbe2000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe26ffd700 1 -- 172.21.15.111:0/1589214991 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20011830 con 0x7fbe300af5a0 2023-11-28T13:00:39.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe26ffd700 1 -- 172.21.15.111:0/1589214991 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe20011990 con 0x7fbe300af5a0 2023-11-28T13:00:39.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe26ffd700 1 -- 172.21.15.111:0/1589214991 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20011d40 con 0x7fbe300af5a0 2023-11-28T13:00:39.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe3836c700 1 -- 172.21.15.111:0/1589214991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe300ce6b0 con 0x7fbe300af5a0 2023-11-28T13:00:39.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.683+0000 7fbe3836c700 1 -- 172.21.15.111:0/1589214991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe301602a0 con 0x7fbe300af5a0 2023-11-28T13:00:39.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.684+0000 7fbe3836c700 1 -- 172.21.15.111:0/1589214991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe300ce840 con 0x7fbe300af5a0 2023-11-28T13:00:39.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.685+0000 7fbe26ffd700 1 -- 172.21.15.111:0/1589214991 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbe20021070 con 0x7fbe300af5a0 2023-11-28T13:00:39.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.685+0000 7fbe26ffd700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbe140682f0 0x7fbe1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:39.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.685+0000 7fbe26ffd700 1 -- 172.21.15.111:0/1589214991 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fbe20085d30 con 0x7fbe300af5a0 2023-11-28T13:00:39.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.685+0000 7fbe35907700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbe140682f0 0x7fbe1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:39.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.685+0000 7fbe35907700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbe140682f0 0x7fbe1406a7a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbe3006ced0 tx=0x7fbe2c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:39.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.689+0000 7fbe26ffd700 1 -- 172.21.15.111:0/1589214991 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbe20052370 con 0x7fbe300af5a0 2023-11-28T13:00:39.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.911+0000 7fbe3836c700 1 -- 172.21.15.111:0/1589214991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe300c6d10 con 0x7fbe300af5a0 2023-11-28T13:00:39.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.912+0000 7fbe26ffd700 1 -- 172.21.15.111:0/1589214991 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4660 (secure 0 0 0) 0x7fbe20015020 con 0x7fbe300af5a0 2023-11-28T13:00:39.914 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:39.914 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3254653481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3254653481},{"type":"v1","addr":"172.21.15.149:6835","nonce":3254653481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1890904311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1890904311},{"type":"v1","addr":"172.21.15.111:6836","nonce":1890904311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1592804351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1592804351},{"type":"v1","addr":"172.21.15.111:6837","nonce":1592804351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:12.011963+0000","modified":"2023-11-28T13:00:15.034385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5352":{"gid":5352,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.149:6834/3579364312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":3579364312},{"type":"v1","addr":"172.21.15.149:6834","nonce":3579364312}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T13:00:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.913+0000 7fbe3836c700 1 -- 172.21.15.111:0/1589214991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbe140682f0 msgr2=0x7fbe1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.913+0000 7fbe3836c700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbe140682f0 0x7fbe1406a7a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbe3006ced0 tx=0x7fbe2c008030 comp rx=0 tx=0).stop 2023-11-28T13:00:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.913+0000 7fbe3836c700 1 -- 172.21.15.111:0/1589214991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe300af5a0 msgr2=0x7fbe301f19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.913+0000 7fbe3836c700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe300af5a0 0x7fbe301f19d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000b4d0 tx=0x7fbe2000b890 comp rx=0 tx=0).stop 2023-11-28T13:00:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.914+0000 7fbe3836c700 1 -- 172.21.15.111:0/1589214991 shutdown_connections 2023-11-28T13:00:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.914+0000 7fbe3836c700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbe3015fed0 0x7fbe3015d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.914+0000 7fbe3836c700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe300af5a0 0x7fbe301f19d0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.914+0000 7fbe3836c700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbe140682f0 0x7fbe1406a7a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.914+0000 7fbe3836c700 1 --2- 172.21.15.111:0/1589214991 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbe300cf7d0 0x7fbe3015d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.914+0000 7fbe3836c700 1 -- 172.21.15.111:0/1589214991 >> 172.21.15.111:0/1589214991 conn(0x7fbe3006bca0 msgr2=0x7fbe300aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.914+0000 7fbe3836c700 1 -- 172.21.15.111:0/1589214991 shutdown_connections 2023-11-28T13:00:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:39.914+0000 7fbe3836c700 1 -- 172.21.15.111:0/1589214991 wait complete. 2023-11-28T13:00:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 17 2023-11-28T13:00:39.927 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:00:40.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.045+0000 7f9fc76c1700 1 Processor -- start 2023-11-28T13:00:40.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.045+0000 7f9fc76c1700 1 -- start start 2023-11-28T13:00:40.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc76c1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fc015cbf0 0x7f9fc00b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc76c1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9fc00b40a0 0x7f9fc006b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc76c1700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9fc00cffd0 0x7f9fc006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc76c1700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fc00cfdb0 con 0x7f9fc00b40a0 2023-11-28T13:00:40.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc76c1700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fc00cc740 con 0x7f9fc00cffd0 2023-11-28T13:00:40.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc76c1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00b35e0 con 0x7f9fc015cbf0 2023-11-28T13:00:40.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc4c5c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9fc00b40a0 0x7f9fc006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc4c5c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9fc00b40a0 0x7f9fc006b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35678/0 (socket says 172.21.15.111:35678) 2023-11-28T13:00:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc4c5c700 1 -- 172.21.15.111:0/577583799 learned_addr learned my addr 172.21.15.111:0/577583799 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc545d700 1 --2- 172.21.15.111:0/577583799 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fc015cbf0 0x7f9fc00b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc5c5e700 1 --2- 172.21.15.111:0/577583799 >> v2:172.21.15.149:3300/0 conn(0x7f9fc00cffd0 0x7f9fc006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc4c5c700 1 -- 172.21.15.111:0/577583799 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fc015cbf0 msgr2=0x7f9fc00b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc4c5c700 1 --2- 172.21.15.111:0/577583799 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fc015cbf0 0x7f9fc00b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc4c5c700 1 -- 172.21.15.111:0/577583799 >> v2:172.21.15.149:3300/0 conn(0x7f9fc00cffd0 msgr2=0x7f9fc006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc4c5c700 1 --2- 172.21.15.111:0/577583799 >> v2:172.21.15.149:3300/0 conn(0x7f9fc00cffd0 0x7f9fc006b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.046+0000 7f9fc4c5c700 1 -- 172.21.15.111:0/577583799 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc00cbe10 con 0x7f9fc00b40a0 2023-11-28T13:00:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.047+0000 7f9fc4c5c700 1 --2- 172.21.15.111:0/577583799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc00b40a0 0x7f9fc006b380 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9fc009a6e0 tx=0x7f9fbc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaea4de64d9af995 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.047+0000 7f9fb7fff700 1 -- 172.21.15.111:0/577583799 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc011830 con 0x7f9fc00b40a0 2023-11-28T13:00:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.047+0000 7f9fb7fff700 1 -- 172.21.15.111:0/577583799 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fbc011990 con 0x7f9fc00b40a0 2023-11-28T13:00:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.047+0000 7f9fb7fff700 1 -- 172.21.15.111:0/577583799 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc011d20 con 0x7f9fc00b40a0 2023-11-28T13:00:40.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.047+0000 7f9fc76c1700 1 -- 172.21.15.111:0/577583799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc00b40a0 msgr2=0x7f9fc006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.047+0000 7f9fc76c1700 1 --2- 172.21.15.111:0/577583799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc00b40a0 0x7f9fc006b380 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9fc009a6e0 tx=0x7f9fbc009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.047+0000 7f9fc76c1700 1 -- 172.21.15.111:0/577583799 shutdown_connections 2023-11-28T13:00:40.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.047+0000 7f9fc76c1700 1 --2- 172.21.15.111:0/577583799 >> v2:172.21.15.149:3300/0 conn(0x7f9fc00cffd0 0x7f9fc006b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.047+0000 7f9fc76c1700 1 --2- 172.21.15.111:0/577583799 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fc015cbf0 0x7f9fc00b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.047+0000 7f9fc76c1700 1 --2- 172.21.15.111:0/577583799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc00b40a0 0x7f9fc006b380 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.047+0000 7f9fc76c1700 1 -- 172.21.15.111:0/577583799 >> 172.21.15.111:0/577583799 conn(0x7f9fc009a040 msgr2=0x7f9fc00d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:40.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.047+0000 7f9fc76c1700 1 -- 172.21.15.111:0/577583799 shutdown_connections 2023-11-28T13:00:40.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.047+0000 7f9fc76c1700 1 -- 172.21.15.111:0/577583799 wait complete. 2023-11-28T13:00:40.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.049+0000 7f9fc76c1700 1 Processor -- start 2023-11-28T13:00:40.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.049+0000 7f9fc76c1700 1 -- start start 2023-11-28T13:00:40.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.049+0000 7f9fc76c1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fc00b40a0 0x7f9fc00c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.049+0000 7f9fc76c1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc00cffd0 0x7f9fc01f17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.049+0000 7f9fc76c1700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9fc015cbf0 0x7f9fc00cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.049+0000 7f9fc76c1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00c8290 con 0x7f9fc00cffd0 2023-11-28T13:00:40.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.049+0000 7f9fc76c1700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00c4790 con 0x7f9fc015cbf0 2023-11-28T13:00:40.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.049+0000 7f9fc76c1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00ca7b0 con 0x7f9fc00b40a0 2023-11-28T13:00:40.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.049+0000 7f9fc4c5c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc00cffd0 0x7f9fc01f17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.049+0000 7f9fc4c5c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc00cffd0 0x7f9fc01f17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35684/0 (socket says 172.21.15.111:35684) 2023-11-28T13:00:40.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.049+0000 7f9fc4c5c700 1 -- 172.21.15.111:0/57586213 learned_addr learned my addr 172.21.15.111:0/57586213 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:40.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.050+0000 7f9fc4c5c700 1 -- 172.21.15.111:0/57586213 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fc00b40a0 msgr2=0x7f9fc00c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.050+0000 7f9fc4c5c700 1 --2- 172.21.15.111:0/57586213 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fc00b40a0 0x7f9fc00c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.050+0000 7f9fc4c5c700 1 -- 172.21.15.111:0/57586213 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9fc015cbf0 msgr2=0x7f9fc00cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.050+0000 7f9fc4c5c700 1 --2- 172.21.15.111:0/57586213 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9fc015cbf0 0x7f9fc00cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.050+0000 7f9fc4c5c700 1 -- 172.21.15.111:0/57586213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc009550 con 0x7f9fc00cffd0 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.050+0000 7f9fc4c5c700 1 --2- 172.21.15.111:0/57586213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc00cffd0 0x7f9fc01f17c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc000c00 tx=0x7f9fbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.050+0000 7f9fb5ffb700 1 -- 172.21.15.111:0/57586213 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc004300 con 0x7f9fc00cffd0 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.050+0000 7f9fb5ffb700 1 -- 172.21.15.111:0/57586213 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fbc01f070 con 0x7f9fc00cffd0 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.050+0000 7f9fc76c1700 1 -- 172.21.15.111:0/57586213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc00cc3d0 con 0x7f9fc00cffd0 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.050+0000 7f9fb5ffb700 1 -- 172.21.15.111:0/57586213 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc0036a0 con 0x7f9fc00cffd0 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.050+0000 7f9fc76c1700 1 -- 172.21.15.111:0/57586213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc00cc0f0 con 0x7f9fc00cffd0 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.051+0000 7f9fc76c1700 1 -- 172.21.15.111:0/57586213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc00d06f0 con 0x7f9fc00cffd0 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.052+0000 7f9fb5ffb700 1 -- 172.21.15.111:0/57586213 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9fbc02a020 con 0x7f9fc00cffd0 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.052+0000 7f9fb5ffb700 1 --2- 172.21.15.111:0/57586213 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9fa40682f0 0x7f9fa406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.052+0000 7f9fb5ffb700 1 -- 172.21.15.111:0/57586213 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9fbc087100 con 0x7f9fc00cffd0 2023-11-28T13:00:40.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.052+0000 7f9fc545d700 1 --2- 172.21.15.111:0/57586213 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9fa40682f0 0x7f9fa406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.052+0000 7f9fc545d700 1 --2- 172.21.15.111:0/57586213 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9fa40682f0 0x7f9fa406a7a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9fb0006b20 tx=0x7f9fb0006ab0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:40.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.056+0000 7f9fb5ffb700 1 -- 172.21.15.111:0/57586213 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9fbc0537c0 con 0x7f9fc00cffd0 2023-11-28T13:00:40.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.240+0000 7f9fc76c1700 1 -- 172.21.15.111:0/57586213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fc00cae90 con 0x7f9fc00cffd0 2023-11-28T13:00:40.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.241+0000 7f9fb5ffb700 1 -- 172.21.15.111:0/57586213 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+18176 (secure 0 0 0) 0x7f9fbc056de0 con 0x7f9fc00cffd0 2023-11-28T13:00:40.243 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:40.243 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":41,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:13.013725+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:09.793782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:10.979525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:00:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.243+0000 7f9fc76c1700 1 -- 172.21.15.111:0/57586213 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9fa40682f0 msgr2=0x7f9fa406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.243+0000 7f9fc76c1700 1 --2- 172.21.15.111:0/57586213 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9fa40682f0 0x7f9fa406a7a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9fb0006b20 tx=0x7f9fb0006ab0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.243+0000 7f9fc76c1700 1 -- 172.21.15.111:0/57586213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc00cffd0 msgr2=0x7f9fc01f17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.243+0000 7f9fc76c1700 1 --2- 172.21.15.111:0/57586213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc00cffd0 0x7f9fc01f17c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc000c00 tx=0x7f9fbc004750 comp rx=0 tx=0).stop 2023-11-28T13:00:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.243+0000 7f9fc76c1700 1 -- 172.21.15.111:0/57586213 shutdown_connections 2023-11-28T13:00:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.243+0000 7f9fc76c1700 1 --2- 172.21.15.111:0/57586213 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9fa40682f0 0x7f9fa406a7a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.243+0000 7f9fc76c1700 1 --2- 172.21.15.111:0/57586213 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9fc015cbf0 0x7f9fc00cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.243+0000 7f9fc76c1700 1 --2- 172.21.15.111:0/57586213 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fc00b40a0 0x7f9fc00c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.243+0000 7f9fc76c1700 1 --2- 172.21.15.111:0/57586213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fc00cffd0 0x7f9fc01f17c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.243+0000 7f9fc76c1700 1 -- 172.21.15.111:0/57586213 >> 172.21.15.111:0/57586213 conn(0x7f9fc009a040 msgr2=0x7f9fc00cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.244+0000 7f9fc76c1700 1 -- 172.21.15.111:0/57586213 shutdown_connections 2023-11-28T13:00:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.244+0000 7f9fc76c1700 1 -- 172.21.15.111:0/57586213 wait complete. 2023-11-28T13:00:40.257 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T13:00:09.793782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T13:00:10.979525+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T13:00:40.257 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T13:00:40.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.382+0000 7fdf77ccc700 1 Processor -- start 2023-11-28T13:00:40.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.382+0000 7fdf77ccc700 1 -- start start 2023-11-28T13:00:40.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf77ccc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdf700b0d70 0x7fdf701604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf77ccc700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fdf700b1140 0x7fdf700ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf77ccc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7015cd90 con 0x7fdf700cc550 2023-11-28T13:00:40.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf77ccc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf701613e0 con 0x7fdf700b0d70 2023-11-28T13:00:40.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf77ccc700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf70160b40 con 0x7fdf700b1140 2023-11-28T13:00:40.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf75a68700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdf700b0d70 0x7fdf701604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf76269700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdf700cc550 0x7fdf700af860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36472/0 (socket says 172.21.15.111:36472) 2023-11-28T13:00:40.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf76269700 1 -- 172.21.15.111:0/2703353974 learned_addr learned my addr 172.21.15.111:0/2703353974 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:40.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf75267700 1 --2- 172.21.15.111:0/2703353974 >> v2:172.21.15.149:3300/0 conn(0x7fdf700b1140 0x7fdf700ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf75a68700 1 -- 172.21.15.111:0/2703353974 >> v2:172.21.15.149:3300/0 conn(0x7fdf700b1140 msgr2=0x7fdf700ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:40.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf75a68700 1 --2- 172.21.15.111:0/2703353974 >> v2:172.21.15.149:3300/0 conn(0x7fdf700b1140 0x7fdf700ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf75a68700 1 -- 172.21.15.111:0/2703353974 >> v1:172.21.15.111:6789/0 conn(0x7fdf700cc550 legacy=0x7fdf700af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf75a68700 1 -- 172.21.15.111:0/2703353974 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf70160e20 con 0x7fdf700b0d70 2023-11-28T13:00:40.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf74a66700 1 -- 172.21.15.111:0/2703353974 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189263449 0 0) 0x7fdf68001490 con 0x7fdf700cc550 2023-11-28T13:00:40.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf75a68700 1 --2- 172.21.15.111:0/2703353974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700b0d70 0x7fdf701604b0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdf7006b310 tx=0x7fdf6c009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e89efc06994512 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:40.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf74a66700 1 -- 172.21.15.111:0/2703353974 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c005e30 con 0x7fdf700b0d70 2023-11-28T13:00:40.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.383+0000 7fdf74a66700 1 -- 172.21.15.111:0/2703353974 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c006030 con 0x7fdf700b0d70 2023-11-28T13:00:40.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.384+0000 7fdf74a66700 1 -- 172.21.15.111:0/2703353974 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c0063c0 con 0x7fdf700b0d70 2023-11-28T13:00:40.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.384+0000 7fdf77ccc700 1 -- 172.21.15.111:0/2703353974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700b0d70 msgr2=0x7fdf701604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.384+0000 7fdf77ccc700 1 --2- 172.21.15.111:0/2703353974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700b0d70 0x7fdf701604b0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdf7006b310 tx=0x7fdf6c009b00 comp rx=0 tx=0).stop 2023-11-28T13:00:40.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.384+0000 7fdf77ccc700 1 -- 172.21.15.111:0/2703353974 shutdown_connections 2023-11-28T13:00:40.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.384+0000 7fdf77ccc700 1 --2- 172.21.15.111:0/2703353974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700b0d70 0x7fdf701604b0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.384+0000 7fdf77ccc700 1 --2- 172.21.15.111:0/2703353974 >> v2:172.21.15.149:3300/0 conn(0x7fdf700b1140 0x7fdf700ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.384+0000 7fdf77ccc700 1 -- 172.21.15.111:0/2703353974 >> 172.21.15.111:0/2703353974 conn(0x7fdf70098ee0 msgr2=0x7fdf7006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:40.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.384+0000 7fdf77ccc700 1 -- 172.21.15.111:0/2703353974 shutdown_connections 2023-11-28T13:00:40.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.384+0000 7fdf77ccc700 1 -- 172.21.15.111:0/2703353974 wait complete. 2023-11-28T13:00:40.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.385+0000 7fdf77ccc700 1 Processor -- start 2023-11-28T13:00:40.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.385+0000 7fdf77ccc700 1 -- start start 2023-11-28T13:00:40.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf77ccc700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf700b0d70 0x7fdf70153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf77ccc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf700b1140 0x7fdf7015eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf77ccc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700cc550 0x7fdf7015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf77ccc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7015fa80 con 0x7fdf700cc550 2023-11-28T13:00:40.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf77ccc700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7015cf30 con 0x7fdf700b0d70 2023-11-28T13:00:40.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf77ccc700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7015d180 con 0x7fdf700b1140 2023-11-28T13:00:40.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf76269700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700cc550 0x7fdf7015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf76269700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700cc550 0x7fdf7015b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35700/0 (socket says 172.21.15.111:35700) 2023-11-28T13:00:40.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf76269700 1 -- 172.21.15.111:0/568390628 learned_addr learned my addr 172.21.15.111:0/568390628 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:40.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf75a68700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf700b0d70 0x7fdf70153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf75267700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf700b1140 0x7fdf7015eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf76269700 1 -- 172.21.15.111:0/568390628 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf700b1140 msgr2=0x7fdf7015eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf76269700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf700b1140 0x7fdf7015eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf76269700 1 -- 172.21.15.111:0/568390628 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf700b0d70 msgr2=0x7fdf70153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf76269700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf700b0d70 0x7fdf70153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.386+0000 7fdf76269700 1 -- 172.21.15.111:0/568390628 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c009430 con 0x7fdf700cc550 2023-11-28T13:00:40.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.387+0000 7fdf76269700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700cc550 0x7fdf7015b5d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800cb60 tx=0x7fdf6800ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:40.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.387+0000 7fdf667fc700 1 -- 172.21.15.111:0/568390628 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6800a9e0 con 0x7fdf700cc550 2023-11-28T13:00:40.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.387+0000 7fdf667fc700 1 -- 172.21.15.111:0/568390628 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6800ab40 con 0x7fdf700cc550 2023-11-28T13:00:40.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.387+0000 7fdf77ccc700 1 -- 172.21.15.111:0/568390628 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf700af340 con 0x7fdf700cc550 2023-11-28T13:00:40.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.387+0000 7fdf667fc700 1 -- 172.21.15.111:0/568390628 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf680070d0 con 0x7fdf700cc550 2023-11-28T13:00:40.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.387+0000 7fdf77ccc700 1 -- 172.21.15.111:0/568390628 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf70161100 con 0x7fdf700cc550 2023-11-28T13:00:40.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.388+0000 7fdf77ccc700 1 -- 172.21.15.111:0/568390628 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf700cec00 con 0x7fdf700cc550 2023-11-28T13:00:40.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.388+0000 7fdf667fc700 1 -- 172.21.15.111:0/568390628 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdf68010070 con 0x7fdf700cc550 2023-11-28T13:00:40.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.388+0000 7fdf667fc700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdf540682f0 0x7fdf5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.389+0000 7fdf667fc700 1 -- 172.21.15.111:0/568390628 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fdf68085530 con 0x7fdf700cc550 2023-11-28T13:00:40.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.389+0000 7fdf75a68700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdf540682f0 0x7fdf5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.389+0000 7fdf75a68700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdf540682f0 0x7fdf5406a7a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c000c00 tx=0x7fdf6c008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:40.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.393+0000 7fdf667fc700 1 -- 172.21.15.111:0/568390628 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdf68051c50 con 0x7fdf700cc550 2023-11-28T13:00:40.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.575+0000 7fdf77ccc700 1 -- 172.21.15.111:0/568390628 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdf7015cd90 con 0x7fdf700cc550 2023-11-28T13:00:40.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.575+0000 7fdf667fc700 1 -- 172.21.15.111:0/568390628 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdf7015cd90 con 0x7fdf700cc550 2023-11-28T13:00:40.577 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T13:00:40.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.577+0000 7fdf77ccc700 1 -- 172.21.15.111:0/568390628 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdf540682f0 msgr2=0x7fdf5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.577+0000 7fdf77ccc700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdf540682f0 0x7fdf5406a7a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c000c00 tx=0x7fdf6c008070 comp rx=0 tx=0).stop 2023-11-28T13:00:40.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.577+0000 7fdf77ccc700 1 -- 172.21.15.111:0/568390628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700cc550 msgr2=0x7fdf7015b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.577+0000 7fdf77ccc700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700cc550 0x7fdf7015b5d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800cb60 tx=0x7fdf6800ce70 comp rx=0 tx=0).stop 2023-11-28T13:00:40.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.577+0000 7fdf77ccc700 1 -- 172.21.15.111:0/568390628 shutdown_connections 2023-11-28T13:00:40.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.577+0000 7fdf77ccc700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700cc550 0x7fdf7015b5d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.577+0000 7fdf77ccc700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fdf540682f0 0x7fdf5406a7a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.577+0000 7fdf77ccc700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fdf700b0d70 0x7fdf70153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.577+0000 7fdf77ccc700 1 --2- 172.21.15.111:0/568390628 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fdf700b1140 0x7fdf7015eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.577+0000 7fdf77ccc700 1 -- 172.21.15.111:0/568390628 >> 172.21.15.111:0/568390628 conn(0x7fdf70098ee0 msgr2=0x7fdf700af700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:40.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.577+0000 7fdf77ccc700 1 -- 172.21.15.111:0/568390628 shutdown_connections 2023-11-28T13:00:40.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.577+0000 7fdf77ccc700 1 -- 172.21.15.111:0/568390628 wait complete. 2023-11-28T13:00:40.590 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T13:00:40.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.714+0000 7ff157bf4700 1 Processor -- start 2023-11-28T13:00:40.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.714+0000 7ff157bf4700 1 -- start start 2023-11-28T13:00:40.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff157bf4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff15015a3c0 0x7ff15006f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff157bf4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff1500cb790 0x7ff15006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff157bf4700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff15006a7b0 0x7ff15015df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff157bf4700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15006ac10 con 0x7ff15015a3c0 2023-11-28T13:00:40.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff157bf4700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15006b490 con 0x7ff15006a7b0 2023-11-28T13:00:40.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff157bf4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15006f680 con 0x7ff1500cb790 2023-11-28T13:00:40.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff155990700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff15015a3c0 0x7ff15006f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff155990700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff15015a3c0 0x7ff15006f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35704/0 (socket says 172.21.15.111:35704) 2023-11-28T13:00:40.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff155990700 1 -- 172.21.15.111:0/2167664046 learned_addr learned my addr 172.21.15.111:0/2167664046 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:40.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff155990700 1 -- 172.21.15.111:0/2167664046 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff1500cb790 msgr2=0x7ff15006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:40.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff155990700 1 --2- 172.21.15.111:0/2167664046 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff1500cb790 0x7ff15006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff156191700 1 --2- 172.21.15.111:0/2167664046 >> v2:172.21.15.149:3300/0 conn(0x7ff15006a7b0 0x7ff15015df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff155990700 1 -- 172.21.15.111:0/2167664046 >> v2:172.21.15.149:3300/0 conn(0x7ff15006a7b0 msgr2=0x7ff15015df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:40.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff155990700 1 --2- 172.21.15.111:0/2167664046 >> v2:172.21.15.149:3300/0 conn(0x7ff15006a7b0 0x7ff15015df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff155990700 1 -- 172.21.15.111:0/2167664046 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15015a790 con 0x7ff15015a3c0 2023-11-28T13:00:40.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff156191700 1 --2- 172.21.15.111:0/2167664046 >> v2:172.21.15.149:3300/0 conn(0x7ff15006a7b0 0x7ff15015df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:40.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff155990700 1 --2- 172.21.15.111:0/2167664046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff15015a3c0 0x7ff15006f140 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff15006e2b0 tx=0x7ff14c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b6b8e6b1701123 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:40.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff15498e700 1 -- 172.21.15.111:0/2167664046 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c00b870 con 0x7ff15015a3c0 2023-11-28T13:00:40.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff15498e700 1 -- 172.21.15.111:0/2167664046 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c00b9d0 con 0x7ff15015a3c0 2023-11-28T13:00:40.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.715+0000 7ff15498e700 1 -- 172.21.15.111:0/2167664046 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c00bd60 con 0x7ff15015a3c0 2023-11-28T13:00:40.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.716+0000 7ff157bf4700 1 -- 172.21.15.111:0/2167664046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff15015a3c0 msgr2=0x7ff15006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.716+0000 7ff157bf4700 1 --2- 172.21.15.111:0/2167664046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff15015a3c0 0x7ff15006f140 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff15006e2b0 tx=0x7ff14c009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.716+0000 7ff157bf4700 1 -- 172.21.15.111:0/2167664046 shutdown_connections 2023-11-28T13:00:40.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.716+0000 7ff157bf4700 1 --2- 172.21.15.111:0/2167664046 >> v2:172.21.15.149:3300/0 conn(0x7ff15006a7b0 0x7ff15015df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.716+0000 7ff157bf4700 1 --2- 172.21.15.111:0/2167664046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff15015a3c0 0x7ff15006f140 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.716+0000 7ff157bf4700 1 --2- 172.21.15.111:0/2167664046 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff1500cb790 0x7ff15006a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.716+0000 7ff157bf4700 1 -- 172.21.15.111:0/2167664046 >> 172.21.15.111:0/2167664046 conn(0x7ff1500b0db0 msgr2=0x7ff1500b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:40.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.716+0000 7ff157bf4700 1 -- 172.21.15.111:0/2167664046 shutdown_connections 2023-11-28T13:00:40.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.716+0000 7ff157bf4700 1 -- 172.21.15.111:0/2167664046 wait complete. 2023-11-28T13:00:40.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.717+0000 7ff157bf4700 1 Processor -- start 2023-11-28T13:00:40.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.717+0000 7ff157bf4700 1 -- start start 2023-11-28T13:00:40.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff157bf4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff15006a7b0 0x7ff1501456c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff157bf4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff1500cb790 0x7ff150147f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff157bf4700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff15015a3c0 0x7ff150148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff157bf4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff150145c00 con 0x7ff15006a7b0 2023-11-28T13:00:40.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff157bf4700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1500c7df0 con 0x7ff15015a3c0 2023-11-28T13:00:40.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff157bf4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff150145ee0 con 0x7ff1500cb790 2023-11-28T13:00:40.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff155990700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff15006a7b0 0x7ff1501456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff155990700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff15006a7b0 0x7ff1501456c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35710/0 (socket says 172.21.15.111:35710) 2023-11-28T13:00:40.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff155990700 1 -- 172.21.15.111:0/3482366708 learned_addr learned my addr 172.21.15.111:0/3482366708 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:40.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff156191700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff15015a3c0 0x7ff150148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff15518f700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff1500cb790 0x7ff150147f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff155990700 1 -- 172.21.15.111:0/3482366708 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff1500cb790 msgr2=0x7ff150147f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff155990700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff1500cb790 0x7ff150147f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff155990700 1 -- 172.21.15.111:0/3482366708 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff15015a3c0 msgr2=0x7ff150148460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff155990700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff15015a3c0 0x7ff150148460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff155990700 1 -- 172.21.15.111:0/3482366708 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c009550 con 0x7ff15006a7b0 2023-11-28T13:00:40.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff156191700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff15015a3c0 0x7ff150148460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:40.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.718+0000 7ff155990700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff15006a7b0 0x7ff1501456c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff1500b1930 tx=0x7ff14c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:40.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.719+0000 7ff1467fc700 1 -- 172.21.15.111:0/3482366708 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c004310 con 0x7ff15006a7b0 2023-11-28T13:00:40.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.719+0000 7ff1467fc700 1 -- 172.21.15.111:0/3482366708 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c01f070 con 0x7ff15006a7b0 2023-11-28T13:00:40.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.719+0000 7ff157bf4700 1 -- 172.21.15.111:0/3482366708 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15006b9c0 con 0x7ff15006a7b0 2023-11-28T13:00:40.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.719+0000 7ff1467fc700 1 -- 172.21.15.111:0/3482366708 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c0036a0 con 0x7ff15006a7b0 2023-11-28T13:00:40.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.719+0000 7ff157bf4700 1 -- 172.21.15.111:0/3482366708 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15006b6e0 con 0x7ff15006a7b0 2023-11-28T13:00:40.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.720+0000 7ff157bf4700 1 -- 172.21.15.111:0/3482366708 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1500c9ab0 con 0x7ff15006a7b0 2023-11-28T13:00:40.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.720+0000 7ff1467fc700 1 -- 172.21.15.111:0/3482366708 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff14c02a020 con 0x7ff15006a7b0 2023-11-28T13:00:40.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.720+0000 7ff1467fc700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff1340682f0 0x7ff13406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:40.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.721+0000 7ff1467fc700 1 -- 172.21.15.111:0/3482366708 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff14c0873f0 con 0x7ff15006a7b0 2023-11-28T13:00:40.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.721+0000 7ff15518f700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff1340682f0 0x7ff13406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:40.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.721+0000 7ff15518f700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff1340682f0 0x7ff13406a7a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff15015d9b0 tx=0x7ff140008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:40.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.725+0000 7ff1467fc700 1 -- 172.21.15.111:0/3482366708 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff14c053b10 con 0x7ff15006a7b0 2023-11-28T13:00:40.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.910+0000 7ff157bf4700 1 -- 172.21.15.111:0/3482366708 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff150067170 con 0x7ff15006a7b0 2023-11-28T13:00:40.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.911+0000 7ff1467fc700 1 -- 172.21.15.111:0/3482366708 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v41) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff14c057130 con 0x7ff15006a7b0 2023-11-28T13:00:40.912 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T13:00:40.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.913+0000 7ff157bf4700 1 -- 172.21.15.111:0/3482366708 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff1340682f0 msgr2=0x7ff13406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.913+0000 7ff157bf4700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff1340682f0 0x7ff13406a7a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff15015d9b0 tx=0x7ff140008040 comp rx=0 tx=0).stop 2023-11-28T13:00:40.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.913+0000 7ff157bf4700 1 -- 172.21.15.111:0/3482366708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff15006a7b0 msgr2=0x7ff1501456c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:40.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.913+0000 7ff157bf4700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff15006a7b0 0x7ff1501456c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff1500b1930 tx=0x7ff14c004750 comp rx=0 tx=0).stop 2023-11-28T13:00:40.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.913+0000 7ff157bf4700 1 -- 172.21.15.111:0/3482366708 shutdown_connections 2023-11-28T13:00:40.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.913+0000 7ff157bf4700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff1340682f0 0x7ff13406a7a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.913+0000 7ff157bf4700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff15015a3c0 0x7ff150148460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.913+0000 7ff157bf4700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff15006a7b0 0x7ff1501456c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.913+0000 7ff157bf4700 1 --2- 172.21.15.111:0/3482366708 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff1500cb790 0x7ff150147f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:40.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.913+0000 7ff157bf4700 1 -- 172.21.15.111:0/3482366708 >> 172.21.15.111:0/3482366708 conn(0x7ff1500b0db0 msgr2=0x7ff1500711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:40.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.913+0000 7ff157bf4700 1 -- 172.21.15.111:0/3482366708 shutdown_connections 2023-11-28T13:00:40.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:40.913+0000 7ff157bf4700 1 -- 172.21.15.111:0/3482366708 wait complete. 2023-11-28T13:00:40.926 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T13:00:41.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.047+0000 7ffa05db4700 1 Processor -- start 2023-11-28T13:00:41.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.047+0000 7ffa05db4700 1 -- start start 2023-11-28T13:00:41.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ffa05db4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffa001604b0 0x7ffa000ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ffa05db4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa000cc550 0x7ffa000cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ffa05db4700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0015cd90 con 0x7ffa000b0d70 2023-11-28T13:00:41.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ffa05db4700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa00160950 con 0x7ffa001604b0 2023-11-28T13:00:41.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ffa05db4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa000ccf10 con 0x7ffa000cc550 2023-11-28T13:00:41.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ff9feffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffa001604b0 0x7ffa000ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ff9feffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffa001604b0 0x7ffa000ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35722/0 (socket says 172.21.15.111:35722) 2023-11-28T13:00:41.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ff9feffd700 1 -- 172.21.15.111:0/2776985262 learned_addr learned my addr 172.21.15.111:0/2776985262 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:41.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ff9fffff700 1 --2- 172.21.15.111:0/2776985262 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa000cc550 0x7ffa000cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ff9feffd700 1 -- 172.21.15.111:0/2776985262 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa000cc550 msgr2=0x7ffa000cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:41.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ff9feffd700 1 --2- 172.21.15.111:0/2776985262 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa000cc550 0x7ffa000cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ff9feffd700 1 -- 172.21.15.111:0/2776985262 >> v1:172.21.15.149:6789/0 conn(0x7ffa000b0d70 legacy=0x7ffa000b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ff9feffd700 1 -- 172.21.15.111:0/2776985262 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa001613e0 con 0x7ffa001604b0 2023-11-28T13:00:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ff9feffd700 1 --2- 172.21.15.111:0/2776985262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa001604b0 0x7ffa000ce6c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40098f0 tx=0x7ff9f4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c2ccfa6536d99d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ff9fe7fc700 1 -- 172.21.15.111:0/2776985262 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f400b8a0 con 0x7ffa001604b0 2023-11-28T13:00:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ff9fe7fc700 1 -- 172.21.15.111:0/2776985262 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f400ba00 con 0x7ffa001604b0 2023-11-28T13:00:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.048+0000 7ff9fe7fc700 1 -- 172.21.15.111:0/2776985262 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f400bd90 con 0x7ffa001604b0 2023-11-28T13:00:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.049+0000 7ffa05db4700 1 -- 172.21.15.111:0/2776985262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa001604b0 msgr2=0x7ffa000ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.049+0000 7ffa05db4700 1 --2- 172.21.15.111:0/2776985262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa001604b0 0x7ffa000ce6c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40098f0 tx=0x7ff9f4009c00 comp rx=0 tx=0).stop 2023-11-28T13:00:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.049+0000 7ffa05db4700 1 -- 172.21.15.111:0/2776985262 shutdown_connections 2023-11-28T13:00:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.049+0000 7ffa05db4700 1 --2- 172.21.15.111:0/2776985262 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa000cc550 0x7ffa000cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.049+0000 7ffa05db4700 1 --2- 172.21.15.111:0/2776985262 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa001604b0 0x7ffa000ce6c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.049+0000 7ffa05db4700 1 -- 172.21.15.111:0/2776985262 >> 172.21.15.111:0/2776985262 conn(0x7ffa00098ee0 msgr2=0x7ffa0006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.049+0000 7ffa05db4700 1 -- 172.21.15.111:0/2776985262 shutdown_connections 2023-11-28T13:00:41.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.049+0000 7ffa05db4700 1 -- 172.21.15.111:0/2776985262 wait complete. 2023-11-28T13:00:41.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.050+0000 7ffa05db4700 1 Processor -- start 2023-11-28T13:00:41.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.050+0000 7ffa05db4700 1 -- start start 2023-11-28T13:00:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ffa05db4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa000b0d70 0x7ffa001f1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ffa05db4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa000cc550 0x7ffa000ce680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ffa05db4700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa001604b0 0x7ffa000cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ffa05db4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa001f2460 con 0x7ffa000b0d70 2023-11-28T13:00:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ffa05db4700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa001f2740 con 0x7ffa001604b0 2023-11-28T13:00:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ffa05db4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa001f2a20 con 0x7ffa000cc550 2023-11-28T13:00:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ff9ff7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa000b0d70 0x7ffa001f1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ff9ff7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa000b0d70 0x7ffa001f1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35726/0 (socket says 172.21.15.111:35726) 2023-11-28T13:00:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ff9ff7fe700 1 -- 172.21.15.111:0/767838786 learned_addr learned my addr 172.21.15.111:0/767838786 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ff9feffd700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa000cc550 0x7ffa000ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ff9fffff700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa001604b0 0x7ffa000cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ff9ff7fe700 1 -- 172.21.15.111:0/767838786 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa000cc550 msgr2=0x7ffa000ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ff9ff7fe700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa000cc550 0x7ffa000ce680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ff9ff7fe700 1 -- 172.21.15.111:0/767838786 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa001604b0 msgr2=0x7ffa000cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ff9ff7fe700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa001604b0 0x7ffa000cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ff9ff7fe700 1 -- 172.21.15.111:0/767838786 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f4009550 con 0x7ffa000b0d70 2023-11-28T13:00:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ff9feffd700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa000cc550 0x7ffa000ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.051+0000 7ff9ff7fe700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa000b0d70 0x7ffa001f1b10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000c4d0 tx=0x7ff9f000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.052+0000 7ff9e7fff700 1 -- 172.21.15.111:0/767838786 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00052d0 con 0x7ffa000b0d70 2023-11-28T13:00:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.052+0000 7ff9e7fff700 1 -- 172.21.15.111:0/767838786 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f0005430 con 0x7ffa000b0d70 2023-11-28T13:00:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.052+0000 7ffa05db4700 1 -- 172.21.15.111:0/767838786 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa00160e20 con 0x7ffa000b0d70 2023-11-28T13:00:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.052+0000 7ff9e7fff700 1 -- 172.21.15.111:0/767838786 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00038f0 con 0x7ffa000b0d70 2023-11-28T13:00:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.052+0000 7ffa05db4700 1 -- 172.21.15.111:0/767838786 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa00160b40 con 0x7ffa000b0d70 2023-11-28T13:00:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.052+0000 7ffa05db4700 1 -- 172.21.15.111:0/767838786 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa00161100 con 0x7ffa000b0d70 2023-11-28T13:00:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.053+0000 7ff9e7fff700 1 -- 172.21.15.111:0/767838786 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff9f0019030 con 0x7ffa000b0d70 2023-11-28T13:00:41.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.053+0000 7ff9e7fff700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9ec0682f0 0x7ff9ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.053+0000 7ff9e7fff700 1 -- 172.21.15.111:0/767838786 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff9f0084fd0 con 0x7ffa000b0d70 2023-11-28T13:00:41.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.053+0000 7ff9feffd700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9ec0682f0 0x7ff9ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.054+0000 7ff9feffd700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9ec0682f0 0x7ff9ec06a7a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4014040 tx=0x7ff9f4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:41.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.057+0000 7ff9e7fff700 1 -- 172.21.15.111:0/767838786 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff9f0051690 con 0x7ffa000b0d70 2023-11-28T13:00:41.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.240+0000 7ffa05db4700 1 -- 172.21.15.111:0/767838786 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffa0015cd90 con 0x7ffa000b0d70 2023-11-28T13:00:41.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.240+0000 7ff9e7fff700 1 -- 172.21.15.111:0/767838786 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff9f0017020 con 0x7ffa000b0d70 2023-11-28T13:00:41.242 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T13:00:41.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.242+0000 7ffa05db4700 1 -- 172.21.15.111:0/767838786 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9ec0682f0 msgr2=0x7ff9ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.242+0000 7ffa05db4700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9ec0682f0 0x7ff9ec06a7a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4014040 tx=0x7ff9f4008030 comp rx=0 tx=0).stop 2023-11-28T13:00:41.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.242+0000 7ffa05db4700 1 -- 172.21.15.111:0/767838786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa000b0d70 msgr2=0x7ffa001f1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.242+0000 7ffa05db4700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa000b0d70 0x7ffa001f1b10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000c4d0 tx=0x7ff9f000c890 comp rx=0 tx=0).stop 2023-11-28T13:00:41.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.242+0000 7ffa05db4700 1 -- 172.21.15.111:0/767838786 shutdown_connections 2023-11-28T13:00:41.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.242+0000 7ffa05db4700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9ec0682f0 0x7ff9ec06a7a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.242+0000 7ffa05db4700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa001604b0 0x7ffa000cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.242+0000 7ffa05db4700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa000b0d70 0x7ffa001f1b10 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.242+0000 7ffa05db4700 1 --2- 172.21.15.111:0/767838786 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa000cc550 0x7ffa000ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.242+0000 7ffa05db4700 1 -- 172.21.15.111:0/767838786 >> 172.21.15.111:0/767838786 conn(0x7ffa00098ee0 msgr2=0x7ffa0015c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:41.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.242+0000 7ffa05db4700 1 -- 172.21.15.111:0/767838786 shutdown_connections 2023-11-28T13:00:41.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.242+0000 7ffa05db4700 1 -- 172.21.15.111:0/767838786 wait complete. 2023-11-28T13:00:41.255 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T13:00:41.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.378+0000 7f424cd7c700 1 Processor -- start 2023-11-28T13:00:41.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.378+0000 7f424cd7c700 1 -- start start 2023-11-28T13:00:41.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f424cd7c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f42481596e0 0x7f424815cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f424cd7c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f424815d510 0x7f42480cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f424cd7c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424815de70 con 0x7f4248159310 2023-11-28T13:00:41.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f424cd7c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424815d980 con 0x7f42481596e0 2023-11-28T13:00:41.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f424cd7c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42480ced10 con 0x7f424815d510 2023-11-28T13:00:41.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f4245d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f42481596e0 0x7f424815cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f4245d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f42481596e0 0x7f424815cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35734/0 (socket says 172.21.15.111:35734) 2023-11-28T13:00:41.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f4245d9b700 1 -- 172.21.15.111:0/596390194 learned_addr learned my addr 172.21.15.111:0/596390194 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:41.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f4245d9b700 1 -- 172.21.15.111:0/596390194 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f424815d510 msgr2=0x7f42480cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f4246d9d700 1 --2- 172.21.15.111:0/596390194 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f424815d510 0x7f42480cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f4245d9b700 1 --2- 172.21.15.111:0/596390194 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f424815d510 0x7f42480cb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f4245d9b700 1 -- 172.21.15.111:0/596390194 >> v1:172.21.15.149:6789/0 conn(0x7f4248159310 legacy=0x7f42480ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f4245d9b700 1 -- 172.21.15.111:0/596390194 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42480ceff0 con 0x7f42481596e0 2023-11-28T13:00:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f4245d9b700 1 --2- 172.21.15.111:0/596390194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42481596e0 0x7f424815cfd0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f42380098f0 tx=0x7f4238009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da19067bd99561c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f424559a700 1 -- 172.21.15.111:0/596390194 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238004230 con 0x7f42481596e0 2023-11-28T13:00:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f424559a700 1 -- 172.21.15.111:0/596390194 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4238004390 con 0x7f42481596e0 2023-11-28T13:00:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.380+0000 7f424559a700 1 -- 172.21.15.111:0/596390194 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42380036a0 con 0x7f42481596e0 2023-11-28T13:00:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.381+0000 7f424cd7c700 1 -- 172.21.15.111:0/596390194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42481596e0 msgr2=0x7f424815cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.381+0000 7f424cd7c700 1 --2- 172.21.15.111:0/596390194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42481596e0 0x7f424815cfd0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f42380098f0 tx=0x7f4238009c00 comp rx=0 tx=0).stop 2023-11-28T13:00:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.381+0000 7f424cd7c700 1 -- 172.21.15.111:0/596390194 shutdown_connections 2023-11-28T13:00:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.381+0000 7f424cd7c700 1 --2- 172.21.15.111:0/596390194 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f424815d510 0x7f42480cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.381+0000 7f424cd7c700 1 --2- 172.21.15.111:0/596390194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42481596e0 0x7f424815cfd0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.381+0000 7f424cd7c700 1 -- 172.21.15.111:0/596390194 >> 172.21.15.111:0/596390194 conn(0x7f424806db10 msgr2=0x7f424806df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:41.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.381+0000 7f424cd7c700 1 -- 172.21.15.111:0/596390194 shutdown_connections 2023-11-28T13:00:41.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.381+0000 7f424cd7c700 1 -- 172.21.15.111:0/596390194 wait complete. 2023-11-28T13:00:41.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.383+0000 7f424cd7c700 1 Processor -- start 2023-11-28T13:00:41.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.383+0000 7f424cd7c700 1 -- start start 2023-11-28T13:00:41.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f424cd7c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4248159310 0x7f424814e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f424cd7c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42481596e0 0x7f42480c5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f424cd7c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f424815d510 0x7f42480c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f424cd7c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424814e8a0 con 0x7f424815d510 2023-11-28T13:00:41.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f424cd7c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42480c6c90 con 0x7f42481596e0 2023-11-28T13:00:41.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f424cd7c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42480c6f70 con 0x7f4248159310 2023-11-28T13:00:41.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f4246d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f424815d510 0x7f42480c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f4246d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f424815d510 0x7f42480c5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35742/0 (socket says 172.21.15.111:35742) 2023-11-28T13:00:41.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f4246d9d700 1 -- 172.21.15.111:0/2962818408 learned_addr learned my addr 172.21.15.111:0/2962818408 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:41.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f4245d9b700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42481596e0 0x7f42480c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f424659c700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4248159310 0x7f424814e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f4246d9d700 1 -- 172.21.15.111:0/2962818408 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4248159310 msgr2=0x7f424814e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f4246d9d700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4248159310 0x7f424814e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f4246d9d700 1 -- 172.21.15.111:0/2962818408 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42481596e0 msgr2=0x7f42480c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f4246d9d700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42481596e0 0x7f42480c5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f4246d9d700 1 -- 172.21.15.111:0/2962818408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4238009550 con 0x7f424815d510 2023-11-28T13:00:41.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f4246d9d700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f424815d510 0x7f42480c5e40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f424000cd70 tx=0x7f424000a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:41.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f4236ffd700 1 -- 172.21.15.111:0/2962818408 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4240003a20 con 0x7f424815d510 2023-11-28T13:00:41.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.384+0000 7f4236ffd700 1 -- 172.21.15.111:0/2962818408 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4240003b80 con 0x7f424815d510 2023-11-28T13:00:41.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.385+0000 7f424cd7c700 1 -- 172.21.15.111:0/2962818408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42480cb0f0 con 0x7f424815d510 2023-11-28T13:00:41.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.385+0000 7f4236ffd700 1 -- 172.21.15.111:0/2962818408 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4240004020 con 0x7f424815d510 2023-11-28T13:00:41.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.385+0000 7f424cd7c700 1 -- 172.21.15.111:0/2962818408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42480cf2d0 con 0x7f424815d510 2023-11-28T13:00:41.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.385+0000 7f424cd7c700 1 -- 172.21.15.111:0/2962818408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42480cb3d0 con 0x7f424815d510 2023-11-28T13:00:41.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.386+0000 7f4236ffd700 1 -- 172.21.15.111:0/2962818408 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f424001c020 con 0x7f424815d510 2023-11-28T13:00:41.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.386+0000 7f4236ffd700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42280682f0 0x7f422806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.386+0000 7f4236ffd700 1 -- 172.21.15.111:0/2962818408 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4240012070 con 0x7f424815d510 2023-11-28T13:00:41.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.386+0000 7f424659c700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42280682f0 0x7f422806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.387+0000 7f424659c700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42280682f0 0x7f422806a7a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f423c009960 tx=0x7f423c0090e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:41.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.391+0000 7f4236ffd700 1 -- 172.21.15.111:0/2962818408 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4240052bc0 con 0x7f424815d510 2023-11-28T13:00:41.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.575+0000 7f424cd7c700 1 -- 172.21.15.111:0/2962818408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f42480cd3c0 con 0x7f424815d510 2023-11-28T13:00:41.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.575+0000 7f4236ffd700 1 -- 172.21.15.111:0/2962818408 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) v1 ==== 100+0+11 (secure 0 0 0) 0x7f42400561e0 con 0x7f424815d510 2023-11-28T13:00:41.577 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T13:00:41.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.577+0000 7f424cd7c700 1 -- 172.21.15.111:0/2962818408 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42280682f0 msgr2=0x7f422806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.577+0000 7f424cd7c700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42280682f0 0x7f422806a7a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f423c009960 tx=0x7f423c0090e0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.577+0000 7f424cd7c700 1 -- 172.21.15.111:0/2962818408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f424815d510 msgr2=0x7f42480c5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.577+0000 7f424cd7c700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f424815d510 0x7f42480c5e40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f424000cd70 tx=0x7f424000a570 comp rx=0 tx=0).stop 2023-11-28T13:00:41.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.577+0000 7f424cd7c700 1 -- 172.21.15.111:0/2962818408 shutdown_connections 2023-11-28T13:00:41.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.577+0000 7f424cd7c700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f424815d510 0x7f42480c5e40 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.577+0000 7f424cd7c700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4248159310 0x7f424814e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.577+0000 7f424cd7c700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f42280682f0 0x7f422806a7a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.577+0000 7f424cd7c700 1 --2- 172.21.15.111:0/2962818408 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f42481596e0 0x7f42480c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.577+0000 7f424cd7c700 1 -- 172.21.15.111:0/2962818408 >> 172.21.15.111:0/2962818408 conn(0x7f424806db10 msgr2=0x7f4248156170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:41.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.577+0000 7f424cd7c700 1 -- 172.21.15.111:0/2962818408 shutdown_connections 2023-11-28T13:00:41.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.577+0000 7f424cd7c700 1 -- 172.21.15.111:0/2962818408 wait complete. 2023-11-28T13:00:41.591 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:00:41.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.711+0000 7fa9eda9c700 1 Processor -- start 2023-11-28T13:00:41.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.711+0000 7fa9eda9c700 1 -- start start 2023-11-28T13:00:41.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.712+0000 7fa9eda9c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fa9e81604b0 0x7fa9e80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.712+0000 7fa9eda9c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e815cd90 con 0x7fa9e80b0d70 2023-11-28T13:00:41.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.712+0000 7fa9eda9c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e81613e0 con 0x7fa9e80cc550 2023-11-28T13:00:41.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.712+0000 7fa9eda9c700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e8160950 con 0x7fa9e81604b0 2023-11-28T13:00:41.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.712+0000 7fa9e77fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa9e80b0d70 0x7fa9e80b1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36524/0 (socket says 172.21.15.111:36524) 2023-11-28T13:00:41.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.712+0000 7fa9e77fe700 1 -- 172.21.15.111:0/1741400379 learned_addr learned my addr 172.21.15.111:0/1741400379 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:41.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e6ffd700 1 --2- 172.21.15.111:0/1741400379 >> v2:172.21.15.149:3300/0 conn(0x7fa9e81604b0 0x7fa9e80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199100065 0 0) 0x7fa9dc001290 con 0x7fa9e80b0d70 2023-11-28T13:00:41.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d00035b0 con 0x7fa9e80b0d70 2023-11-28T13:00:41.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488700108 0 0) 0x7fa9e81613e0 con 0x7fa9e80cc550 2023-11-28T13:00:41.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9dc001290 con 0x7fa9e80cc550 2023-11-28T13:00:41.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4267598705 0 0) 0x7fa9d00035b0 con 0x7fa9e80b0d70 2023-11-28T13:00:41.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 >> v2:172.21.15.149:3300/0 conn(0x7fa9e81604b0 msgr2=0x7fa9e80ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:41.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 --2- 172.21.15.111:0/1741400379 >> v2:172.21.15.149:3300/0 conn(0x7fa9e81604b0 0x7fa9e80ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 >> v1:172.21.15.149:6789/0 conn(0x7fa9e80cc550 legacy=0x7fa9e80af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8160b40 con 0x7fa9e80b0d70 2023-11-28T13:00:41.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fa9dc001980 con 0x7fa9e80b0d70 2023-11-28T13:00:41.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 >> v1:172.21.15.111:6789/0 conn(0x7fa9e80b0d70 legacy=0x7fa9e80b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 --2- 172.21.15.111:0/1741400379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d0003be0 0x7fa9d0003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 --2- 172.21.15.111:0/1741400379 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d0004850 0x7fa9d0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 --2- 172.21.15.111:0/1741400379 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d0005200 0x7fa9d000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d000c0e0 con 0x7fa9d0003be0 2023-11-28T13:00:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d000c220 con 0x7fa9d0005200 2023-11-28T13:00:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d000c3b0 con 0x7fa9d0004850 2023-11-28T13:00:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e77fe700 1 --2- 172.21.15.111:0/1741400379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d0003be0 0x7fa9d0003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e7fff700 1 --2- 172.21.15.111:0/1741400379 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d0005200 0x7fa9d000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e6ffd700 1 --2- 172.21.15.111:0/1741400379 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d0004850 0x7fa9d0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e77fe700 1 -- 172.21.15.111:0/1741400379 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d0004850 msgr2=0x7fa9d0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e77fe700 1 --2- 172.21.15.111:0/1741400379 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d0004850 0x7fa9d0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e77fe700 1 -- 172.21.15.111:0/1741400379 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d0005200 msgr2=0x7fa9d000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e77fe700 1 --2- 172.21.15.111:0/1741400379 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d0005200 0x7fa9d000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e77fe700 1 -- 172.21.15.111:0/1741400379 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d00035b0 con 0x7fa9d0003be0 2023-11-28T13:00:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e7fff700 1 -- 172.21.15.111:0/1741400379 reap_dead start 2023-11-28T13:00:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e6ffd700 1 --2- 172.21.15.111:0/1741400379 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9d0004850 0x7fa9d0004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.713+0000 7fa9e77fe700 1 --2- 172.21.15.111:0/1741400379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d0003be0 0x7fa9d0003fd0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00c0e0 tx=0x7fa9d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f04063e465d9419 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.714+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc00cc20 con 0x7fa9d0003be0 2023-11-28T13:00:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.714+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9dc00cda0 con 0x7fa9d0003be0 2023-11-28T13:00:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.714+0000 7fa9e67fc700 1 -- 172.21.15.111:0/1741400379 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc007310 con 0x7fa9d0003be0 2023-11-28T13:00:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.714+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1741400379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d0003be0 msgr2=0x7fa9d0003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.714+0000 7fa9eda9c700 1 --2- 172.21.15.111:0/1741400379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d0003be0 0x7fa9d0003fd0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc00c0e0 tx=0x7fa9d0005880 comp rx=0 tx=0).stop 2023-11-28T13:00:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.714+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1741400379 shutdown_connections 2023-11-28T13:00:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.714+0000 7fa9eda9c700 1 --2- 172.21.15.111:0/1741400379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d0003be0 0x7fa9d0003fd0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.714+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1741400379 >> 172.21.15.111:0/1741400379 conn(0x7fa9e8098ee0 msgr2=0x7fa9e806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.714+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1741400379 shutdown_connections 2023-11-28T13:00:41.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.714+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1741400379 wait complete. 2023-11-28T13:00:41.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.715+0000 7fa9eda9c700 1 Processor -- start 2023-11-28T13:00:41.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.716+0000 7fa9eda9c700 1 -- start start 2023-11-28T13:00:41.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.716+0000 7fa9eda9c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d0003be0 0x7fa9e80b5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.716+0000 7fa9eda9c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9e8156350 0x7fa9e80c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.716+0000 7fa9eda9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9e80ad6c0 0x7fa9e80adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.716+0000 7fa9eda9c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80ae070 con 0x7fa9e80ad6c0 2023-11-28T13:00:41.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.716+0000 7fa9eda9c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80c4a80 con 0x7fa9d0003be0 2023-11-28T13:00:41.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.716+0000 7fa9eda9c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80c3e30 con 0x7fa9e8156350 2023-11-28T13:00:41.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.716+0000 7fa9e7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9e80ad6c0 0x7fa9e80adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.716+0000 7fa9e7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9e80ad6c0 0x7fa9e80adb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35762/0 (socket says 172.21.15.111:35762) 2023-11-28T13:00:41.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.716+0000 7fa9e7fff700 1 -- 172.21.15.111:0/1860522854 learned_addr learned my addr 172.21.15.111:0/1860522854 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:41.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.716+0000 7fa9e77fe700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d0003be0 0x7fa9e80b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.716+0000 7fa9e6ffd700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9e8156350 0x7fa9e80c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.717+0000 7fa9e7fff700 1 -- 172.21.15.111:0/1860522854 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9e8156350 msgr2=0x7fa9e80c35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.717+0000 7fa9e7fff700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9e8156350 0x7fa9e80c35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.717+0000 7fa9e7fff700 1 -- 172.21.15.111:0/1860522854 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d0003be0 msgr2=0x7fa9e80b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.717+0000 7fa9e7fff700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d0003be0 0x7fa9e80b5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.717+0000 7fa9e7fff700 1 -- 172.21.15.111:0/1860522854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8161100 con 0x7fa9e80ad6c0 2023-11-28T13:00:41.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.717+0000 7fa9e7fff700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9e80ad6c0 0x7fa9e80adb30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80b1ee0 tx=0x7fa9e000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:41.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.717+0000 7fa9cffff700 1 -- 172.21.15.111:0/1860522854 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0003c10 con 0x7fa9e80ad6c0 2023-11-28T13:00:41.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.717+0000 7fa9cffff700 1 -- 172.21.15.111:0/1860522854 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e0003d90 con 0x7fa9e80ad6c0 2023-11-28T13:00:41.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.717+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1860522854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e8160e20 con 0x7fa9e80ad6c0 2023-11-28T13:00:41.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.717+0000 7fa9cffff700 1 -- 172.21.15.111:0/1860522854 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e000db20 con 0x7fa9e80ad6c0 2023-11-28T13:00:41.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.717+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1860522854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e80af340 con 0x7fa9e80ad6c0 2023-11-28T13:00:41.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.718+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1860522854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e80c4bf0 con 0x7fa9e80ad6c0 2023-11-28T13:00:41.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.718+0000 7fa9cffff700 1 -- 172.21.15.111:0/1860522854 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa9e000dc80 con 0x7fa9e80ad6c0 2023-11-28T13:00:41.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.719+0000 7fa9cffff700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9d40682f0 0x7fa9d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:41.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.719+0000 7fa9cffff700 1 -- 172.21.15.111:0/1860522854 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fa9e0086220 con 0x7fa9e80ad6c0 2023-11-28T13:00:41.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.719+0000 7fa9e77fe700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9d40682f0 0x7fa9d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:41.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.719+0000 7fa9e77fe700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9d40682f0 0x7fa9d406a7a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc015040 tx=0x7fa9dc007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:41.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.723+0000 7fa9cffff700 1 -- 172.21.15.111:0/1860522854 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa9e00529b0 con 0x7fa9e80ad6c0 2023-11-28T13:00:41.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.946+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1860522854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9e8067170 con 0x7fa9e80ad6c0 2023-11-28T13:00:41.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.947+0000 7fa9cffff700 1 -- 172.21.15.111:0/1860522854 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4660 (secure 0 0 0) 0x7fa9e0055fd0 con 0x7fa9e80ad6c0 2023-11-28T13:00:41.949 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3254653481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3254653481},{"type":"v1","addr":"172.21.15.149:6835","nonce":3254653481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1890904311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1890904311},{"type":"v1","addr":"172.21.15.111:6836","nonce":1890904311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1592804351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1592804351},{"type":"v1","addr":"172.21.15.111:6837","nonce":1592804351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:12.011963+0000","modified":"2023-11-28T13:00:15.034385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5352":{"gid":5352,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.149:6834/3579364312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":3579364312},{"type":"v1","addr":"172.21.15.149:6834","nonce":3579364312}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T13:00:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.948+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1860522854 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9d40682f0 msgr2=0x7fa9d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.948+0000 7fa9eda9c700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9d40682f0 0x7fa9d406a7a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc015040 tx=0x7fa9dc007000 comp rx=0 tx=0).stop 2023-11-28T13:00:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.949+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1860522854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9e80ad6c0 msgr2=0x7fa9e80adb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.949+0000 7fa9eda9c700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9e80ad6c0 0x7fa9e80adb30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80b1ee0 tx=0x7fa9e000d430 comp rx=0 tx=0).stop 2023-11-28T13:00:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.949+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1860522854 shutdown_connections 2023-11-28T13:00:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.949+0000 7fa9eda9c700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9e80ad6c0 0x7fa9e80adb30 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.949+0000 7fa9eda9c700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa9d40682f0 0x7fa9d406a7a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.949+0000 7fa9eda9c700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa9d0003be0 0x7fa9e80b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.949+0000 7fa9eda9c700 1 --2- 172.21.15.111:0/1860522854 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa9e8156350 0x7fa9e80c35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:41.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.949+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1860522854 >> 172.21.15.111:0/1860522854 conn(0x7fa9e8098ee0 msgr2=0x7fa9e80af6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:41.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.949+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1860522854 shutdown_connections 2023-11-28T13:00:41.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:41.949+0000 7fa9eda9c700 1 -- 172.21.15.111:0/1860522854 wait complete. 2023-11-28T13:00:41.952 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 17 2023-11-28T13:00:41.962 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:00:42.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.083+0000 7f9fb8766700 1 Processor -- start 2023-11-28T13:00:42.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.083+0000 7f9fb8766700 1 -- start start 2023-11-28T13:00:42.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb8766700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fb0159960 0x7f9fb00b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb8766700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9fb00aecc0 0x7f9fb00cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb8766700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb00ad420 con 0x7f9fb00cdbc0 2023-11-28T13:00:42.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb8766700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb00ce040 con 0x7f9fb00aecc0 2023-11-28T13:00:42.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb8766700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb0159d30 con 0x7f9fb0159960 2023-11-28T13:00:42.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb6502700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fb0159960 0x7f9fb00b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb6502700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fb0159960 0x7f9fb00b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:57514/0 (socket says 172.21.15.111:57514) 2023-11-28T13:00:42.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb5d01700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9fb00aecc0 0x7f9fb00cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb6502700 1 -- 172.21.15.111:0/3892337208 learned_addr learned my addr 172.21.15.111:0/3892337208 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:42.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb5500700 1 -- 172.21.15.111:0/3892337208 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69914334 0 0) 0x7f9fb00ad420 con 0x7f9fb00cdbc0 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb5500700 1 -- 172.21.15.111:0/3892337208 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f9c0035b0 con 0x7f9fb00cdbc0 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb5d01700 1 -- 172.21.15.111:0/3892337208 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fb0159960 msgr2=0x7f9fb00b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb5d01700 1 --2- 172.21.15.111:0/3892337208 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fb0159960 0x7f9fb00b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb5d01700 1 -- 172.21.15.111:0/3892337208 >> v1:172.21.15.149:6789/0 conn(0x7f9fb00cdbc0 legacy=0x7f9fb00afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb5d01700 1 -- 172.21.15.111:0/3892337208 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb00ac810 con 0x7f9fb00aecc0 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.084+0000 7f9fb6502700 1 --2- 172.21.15.111:0/3892337208 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fb0159960 0x7f9fb00b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.085+0000 7f9fb5d01700 1 --2- 172.21.15.111:0/3892337208 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9fb00aecc0 0x7f9fb00cd680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0009840 tx=0x7f9fa0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26d518e4c556caae server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.085+0000 7f9fb5500700 1 -- 172.21.15.111:0/3892337208 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0011830 con 0x7f9fb00aecc0 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.085+0000 7f9fb5500700 1 -- 172.21.15.111:0/3892337208 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa0011990 con 0x7f9fb00aecc0 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.085+0000 7f9fb5500700 1 -- 172.21.15.111:0/3892337208 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0011d00 con 0x7f9fb00aecc0 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.085+0000 7f9fb8766700 1 -- 172.21.15.111:0/3892337208 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9fb00aecc0 msgr2=0x7f9fb00cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.085+0000 7f9fb8766700 1 --2- 172.21.15.111:0/3892337208 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9fb00aecc0 0x7f9fb00cd680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0009840 tx=0x7f9fa0009b50 comp rx=0 tx=0).stop 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.085+0000 7f9fb8766700 1 -- 172.21.15.111:0/3892337208 shutdown_connections 2023-11-28T13:00:42.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.085+0000 7f9fb8766700 1 --2- 172.21.15.111:0/3892337208 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fb0159960 0x7f9fb00b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.085+0000 7f9fb8766700 1 --2- 172.21.15.111:0/3892337208 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9fb00aecc0 0x7f9fb00cd680 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.085+0000 7f9fb8766700 1 -- 172.21.15.111:0/3892337208 >> 172.21.15.111:0/3892337208 conn(0x7f9fb0098ee0 msgr2=0x7f9fb006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:42.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.085+0000 7f9fb8766700 1 -- 172.21.15.111:0/3892337208 shutdown_connections 2023-11-28T13:00:42.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.085+0000 7f9fb8766700 1 -- 172.21.15.111:0/3892337208 wait complete. 2023-11-28T13:00:42.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.087+0000 7f9fb8766700 1 Processor -- start 2023-11-28T13:00:42.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.087+0000 7f9fb8766700 1 -- start start 2023-11-28T13:00:42.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.087+0000 7f9fb8766700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fb00aecc0 0x7f9fb00b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.087+0000 7f9fb8766700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9fb00cdbc0 0x7f9fb01e46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.087+0000 7f9fb8766700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fb0159960 0x7f9fb01e26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.087+0000 7f9fb8766700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb00b5720 con 0x7f9fb0159960 2023-11-28T13:00:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.087+0000 7f9fb8766700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb00cb710 con 0x7f9fb00cdbc0 2023-11-28T13:00:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.087+0000 7f9fb8766700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb00b5440 con 0x7f9fb00aecc0 2023-11-28T13:00:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.087+0000 7f9fb6d03700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fb0159960 0x7f9fb01e26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.087+0000 7f9fb6d03700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fb0159960 0x7f9fb01e26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35776/0 (socket says 172.21.15.111:35776) 2023-11-28T13:00:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.087+0000 7f9fb6d03700 1 -- 172.21.15.111:0/257987465 learned_addr learned my addr 172.21.15.111:0/257987465 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.088+0000 7f9fb5d01700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9fb00cdbc0 0x7f9fb01e46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.088+0000 7f9fb6502700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fb00aecc0 0x7f9fb00b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.088+0000 7f9fb6d03700 1 -- 172.21.15.111:0/257987465 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fb00aecc0 msgr2=0x7f9fb00b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.088+0000 7f9fb6d03700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fb00aecc0 0x7f9fb00b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.088+0000 7f9fb6d03700 1 -- 172.21.15.111:0/257987465 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9fb00cdbc0 msgr2=0x7f9fb01e46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.088+0000 7f9fb6d03700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9fb00cdbc0 0x7f9fb01e46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.088+0000 7f9fb6d03700 1 -- 172.21.15.111:0/257987465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fac009500 con 0x7f9fb0159960 2023-11-28T13:00:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.088+0000 7f9fb6d03700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fb0159960 0x7f9fb01e26f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800cc20 tx=0x7f9fa800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.088+0000 7f9fa6ffd700 1 -- 172.21.15.111:0/257987465 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8003c00 con 0x7f9fb0159960 2023-11-28T13:00:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.088+0000 7f9fa6ffd700 1 -- 172.21.15.111:0/257987465 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa8003d60 con 0x7f9fb0159960 2023-11-28T13:00:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.088+0000 7f9fb8766700 1 -- 172.21.15.111:0/257987465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa00094a0 con 0x7f9fb0159960 2023-11-28T13:00:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.088+0000 7f9fa6ffd700 1 -- 172.21.15.111:0/257987465 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa800aae0 con 0x7f9fb0159960 2023-11-28T13:00:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.088+0000 7f9fb8766700 1 -- 172.21.15.111:0/257987465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb00acdd0 con 0x7f9fb0159960 2023-11-28T13:00:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.089+0000 7f9fb8766700 1 -- 172.21.15.111:0/257987465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb00acaf0 con 0x7f9fb0159960 2023-11-28T13:00:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.090+0000 7f9fa6ffd700 1 -- 172.21.15.111:0/257987465 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9fa8010050 con 0x7f9fb0159960 2023-11-28T13:00:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.090+0000 7f9fa6ffd700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9f940682f0 0x7f9f9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.090+0000 7f9fa6ffd700 1 -- 172.21.15.111:0/257987465 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f9fa8085e70 con 0x7f9fb0159960 2023-11-28T13:00:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.090+0000 7f9fb6502700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9f940682f0 0x7f9f9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.090+0000 7f9fb6502700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9f940682f0 0x7f9f9406a7a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9fb01f5460 tx=0x7f9fac009290 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:42.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.094+0000 7f9fa6ffd700 1 -- 172.21.15.111:0/257987465 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9fa80523e0 con 0x7f9fb0159960 2023-11-28T13:00:42.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.279+0000 7f9fb8766700 1 -- 172.21.15.111:0/257987465 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fb00ad0b0 con 0x7f9fb0159960 2023-11-28T13:00:42.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.280+0000 7f9fa6ffd700 1 -- 172.21.15.111:0/257987465 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+18271 (secure 0 0 0) 0x7f9fa8055a00 con 0x7f9fb0159960 2023-11-28T13:00:42.282 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:42.282 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":42,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:41.660524+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:09.793782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:10.979525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.7","mappings":[{"from":0,"to":4}]},{"pgid":"7.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:00:42.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.282+0000 7f9fb8766700 1 -- 172.21.15.111:0/257987465 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9f940682f0 msgr2=0x7f9f9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.282+0000 7f9fb8766700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9f940682f0 0x7f9f9406a7a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9fb01f5460 tx=0x7f9fac009290 comp rx=0 tx=0).stop 2023-11-28T13:00:42.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.282+0000 7f9fb8766700 1 -- 172.21.15.111:0/257987465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fb0159960 msgr2=0x7f9fb01e26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.282+0000 7f9fb8766700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fb0159960 0x7f9fb01e26f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800cc20 tx=0x7f9fa800cf30 comp rx=0 tx=0).stop 2023-11-28T13:00:42.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.282+0000 7f9fb8766700 1 -- 172.21.15.111:0/257987465 shutdown_connections 2023-11-28T13:00:42.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.282+0000 7f9fb8766700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fb0159960 0x7f9fb01e26f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.282+0000 7f9fb8766700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9fb00aecc0 0x7f9fb00b4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.282+0000 7f9fb8766700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9f940682f0 0x7f9f9406a7a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.282+0000 7f9fb8766700 1 --2- 172.21.15.111:0/257987465 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9fb00cdbc0 0x7f9fb01e46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.282+0000 7f9fb8766700 1 -- 172.21.15.111:0/257987465 >> 172.21.15.111:0/257987465 conn(0x7f9fb0098ee0 msgr2=0x7f9fb00afa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:42.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.283+0000 7f9fb8766700 1 -- 172.21.15.111:0/257987465 shutdown_connections 2023-11-28T13:00:42.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.283+0000 7f9fb8766700 1 -- 172.21.15.111:0/257987465 wait complete. 2023-11-28T13:00:42.296 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T13:00:42.297 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:00:42.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.415+0000 7f2818ae7700 1 Processor -- start 2023-11-28T13:00:42.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.415+0000 7f2818ae7700 1 -- start start 2023-11-28T13:00:42.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.416+0000 7f2818ae7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28140b15c0 0x7f28140ac670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.416+0000 7f2818ae7700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28140af650 con 0x7f28140b0250 2023-11-28T13:00:42.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.416+0000 7f2818ae7700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28140cb6d0 con 0x7f281415a9e0 2023-11-28T13:00:42.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.416+0000 7f2818ae7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28140cb9b0 con 0x7f28140b15c0 2023-11-28T13:00:42.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.416+0000 7f2812d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f28140b0250 0x7f28140cc770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36562/0 (socket says 172.21.15.111:36562) 2023-11-28T13:00:42.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.416+0000 7f2811d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f28140b15c0 0x7f28140ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.416+0000 7f2812d9d700 1 -- 172.21.15.111:0/3016486888 learned_addr learned my addr 172.21.15.111:0/3016486888 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:42.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.416+0000 7f2811d9b700 1 -- 172.21.15.111:0/3016486888 >> v1:172.21.15.149:6789/0 conn(0x7f281415a9e0 legacy=0x7f28140caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.416+0000 7f2811d9b700 1 -- 172.21.15.111:0/3016486888 >> v1:172.21.15.111:6789/0 conn(0x7f28140b0250 legacy=0x7f28140cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.416+0000 7f281159a700 1 -- 172.21.15.111:0/3016486888 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100946412 0 0) 0x7f28140af650 con 0x7f28140b0250 2023-11-28T13:00:42.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.416+0000 7f2811d9b700 1 -- 172.21.15.111:0/3016486888 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28140cbc90 con 0x7f28140b15c0 2023-11-28T13:00:42.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.416+0000 7f2811d9b700 1 --2- 172.21.15.111:0/3016486888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28140b15c0 0x7f28140ac670 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2808009840 tx=0x7f2808009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbad900610e28807 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:42.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.417+0000 7f281159a700 1 -- 172.21.15.111:0/3016486888 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808011830 con 0x7f28140b15c0 2023-11-28T13:00:42.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.417+0000 7f281159a700 1 -- 172.21.15.111:0/3016486888 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2808011990 con 0x7f28140b15c0 2023-11-28T13:00:42.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.417+0000 7f281159a700 1 -- 172.21.15.111:0/3016486888 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808011d20 con 0x7f28140b15c0 2023-11-28T13:00:42.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.417+0000 7f2818ae7700 1 -- 172.21.15.111:0/3016486888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28140b15c0 msgr2=0x7f28140ac670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.417+0000 7f2818ae7700 1 --2- 172.21.15.111:0/3016486888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28140b15c0 0x7f28140ac670 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2808009840 tx=0x7f2808009b50 comp rx=0 tx=0).stop 2023-11-28T13:00:42.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.417+0000 7f2818ae7700 1 -- 172.21.15.111:0/3016486888 shutdown_connections 2023-11-28T13:00:42.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.417+0000 7f2818ae7700 1 --2- 172.21.15.111:0/3016486888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28140b15c0 0x7f28140ac670 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.417+0000 7f2818ae7700 1 -- 172.21.15.111:0/3016486888 >> 172.21.15.111:0/3016486888 conn(0x7f2814098ee0 msgr2=0x7f281406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:42.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.417+0000 7f2818ae7700 1 -- 172.21.15.111:0/3016486888 shutdown_connections 2023-11-28T13:00:42.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.417+0000 7f2818ae7700 1 -- 172.21.15.111:0/3016486888 wait complete. 2023-11-28T13:00:42.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.418+0000 7f2818ae7700 1 Processor -- start 2023-11-28T13:00:42.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.419+0000 7f2818ae7700 1 -- start start 2023-11-28T13:00:42.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.419+0000 7f2818ae7700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28140b0250 0x7f2814151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.419+0000 7f2818ae7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28140b15c0 0x7f28141523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.419+0000 7f2818ae7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f281415a9e0 0x7f28140b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.419+0000 7f2818ae7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28141547b0 con 0x7f28140b15c0 2023-11-28T13:00:42.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.419+0000 7f2818ae7700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281415cc50 con 0x7f28140b0250 2023-11-28T13:00:42.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.419+0000 7f2818ae7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281415c3b0 con 0x7f281415a9e0 2023-11-28T13:00:42.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.419+0000 7f2811d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28140b15c0 0x7f28141523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.419+0000 7f2811d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28140b15c0 0x7f28141523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35788/0 (socket says 172.21.15.111:35788) 2023-11-28T13:00:42.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.419+0000 7f2811d9b700 1 -- 172.21.15.111:0/1893891212 learned_addr learned my addr 172.21.15.111:0/1893891212 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:42.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f2812d9d700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f281415a9e0 0x7f28140b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f2811d9b700 1 -- 172.21.15.111:0/1893891212 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f281415a9e0 msgr2=0x7f28140b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f2811d9b700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f281415a9e0 0x7f28140b3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f2811d9b700 1 -- 172.21.15.111:0/1893891212 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28140b0250 msgr2=0x7f2814151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f281259c700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28140b0250 0x7f2814151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f2811d9b700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28140b0250 0x7f2814151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f2811d9b700 1 -- 172.21.15.111:0/1893891212 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28080094a0 con 0x7f28140b15c0 2023-11-28T13:00:42.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f281259c700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28140b0250 0x7f2814151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:42.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f2811d9b700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28140b15c0 0x7f28141523a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f28080090b0 tx=0x7f2808004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:42.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f2802ffd700 1 -- 172.21.15.111:0/1893891212 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280801a900 con 0x7f28140b15c0 2023-11-28T13:00:42.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f2802ffd700 1 -- 172.21.15.111:0/1893891212 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280801aa60 con 0x7f28140b15c0 2023-11-28T13:00:42.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f2802ffd700 1 -- 172.21.15.111:0/1893891212 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280801add0 con 0x7f28140b15c0 2023-11-28T13:00:42.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f2818ae7700 1 -- 172.21.15.111:0/1893891212 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28140cc250 con 0x7f28140b15c0 2023-11-28T13:00:42.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.420+0000 7f2818ae7700 1 -- 172.21.15.111:0/1893891212 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28140cbf70 con 0x7f28140b15c0 2023-11-28T13:00:42.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.421+0000 7f2818ae7700 1 -- 172.21.15.111:0/1893891212 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28140b06d0 con 0x7f28140b15c0 2023-11-28T13:00:42.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.421+0000 7f2802ffd700 1 -- 172.21.15.111:0/1893891212 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f280802b030 con 0x7f28140b15c0 2023-11-28T13:00:42.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.422+0000 7f2802ffd700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f27f40682f0 0x7f27f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.422+0000 7f2802ffd700 1 -- 172.21.15.111:0/1893891212 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f28080871a0 con 0x7f28140b15c0 2023-11-28T13:00:42.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.422+0000 7f281259c700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f27f40682f0 0x7f27f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.422+0000 7f281259c700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f27f40682f0 0x7f27f406a7a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2804000fb0 tx=0x7f2804009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:42.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.426+0000 7f2802ffd700 1 -- 172.21.15.111:0/1893891212 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2808053930 con 0x7f28140b15c0 2023-11-28T13:00:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.648+0000 7f2818ae7700 1 -- 172.21.15.111:0/1893891212 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28140b4380 con 0x7f28140b15c0 2023-11-28T13:00:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.649+0000 7f2802ffd700 1 -- 172.21.15.111:0/1893891212 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f28140b4380 con 0x7f28140b15c0 2023-11-28T13:00:42.650 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:42.650 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3254653481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3254653481},{"type":"v1","addr":"172.21.15.149:6835","nonce":3254653481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1890904311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1890904311},{"type":"v1","addr":"172.21.15.111:6836","nonce":1890904311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1592804351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1592804351},{"type":"v1","addr":"172.21.15.111:6837","nonce":1592804351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:12.011963+0000","modified":"2023-11-28T13:00:15.034385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5352":{"gid":5352,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.149:6834/3579364312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":3579364312},{"type":"v1","addr":"172.21.15.149:6834","nonce":3579364312}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T13:00:42.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.650+0000 7f2818ae7700 1 -- 172.21.15.111:0/1893891212 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f27f40682f0 msgr2=0x7f27f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.650+0000 7f2818ae7700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f27f40682f0 0x7f27f406a7a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2804000fb0 tx=0x7f2804009040 comp rx=0 tx=0).stop 2023-11-28T13:00:42.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.651+0000 7f2818ae7700 1 -- 172.21.15.111:0/1893891212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28140b15c0 msgr2=0x7f28141523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.651+0000 7f2818ae7700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28140b15c0 0x7f28141523a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f28080090b0 tx=0x7f2808004750 comp rx=0 tx=0).stop 2023-11-28T13:00:42.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.651+0000 7f2818ae7700 1 -- 172.21.15.111:0/1893891212 shutdown_connections 2023-11-28T13:00:42.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.651+0000 7f2818ae7700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f281415a9e0 0x7f28140b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.651+0000 7f2818ae7700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f27f40682f0 0x7f27f406a7a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.651+0000 7f2818ae7700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f28140b0250 0x7f2814151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.651+0000 7f2818ae7700 1 --2- 172.21.15.111:0/1893891212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28140b15c0 0x7f28141523a0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.651+0000 7f2818ae7700 1 -- 172.21.15.111:0/1893891212 >> 172.21.15.111:0/1893891212 conn(0x7f2814098ee0 msgr2=0x7f28140c8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:42.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.651+0000 7f2818ae7700 1 -- 172.21.15.111:0/1893891212 shutdown_connections 2023-11-28T13:00:42.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.651+0000 7f2818ae7700 1 -- 172.21.15.111:0/1893891212 wait complete. 2023-11-28T13:00:42.653 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 17 2023-11-28T13:00:42.664 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:00:42.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.782+0000 7f95facb3700 1 Processor -- start 2023-11-28T13:00:42.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.782+0000 7f95facb3700 1 -- start start 2023-11-28T13:00:42.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95facb3700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95f415a5d0 0x7f95f40ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95facb3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f95f40b0160 0x7f95f40ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95facb3700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f40ae8b0 con 0x7f95f40cb2d0 2023-11-28T13:00:42.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95facb3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95f40cc1b0 con 0x7f95f40b0160 2023-11-28T13:00:42.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95facb3700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f40cc390 con 0x7f95f415a5d0 2023-11-28T13:00:42.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95f3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f95f40b0160 0x7f95f40ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95f3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f95f40b0160 0x7f95f40ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35796/0 (socket says 172.21.15.111:35796) 2023-11-28T13:00:42.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95f3fff700 1 -- 172.21.15.111:0/771768375 learned_addr learned my addr 172.21.15.111:0/771768375 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:42.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95f8a4f700 1 --2- 172.21.15.111:0/771768375 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95f415a5d0 0x7f95f40ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95f3fff700 1 -- 172.21.15.111:0/771768375 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95f415a5d0 msgr2=0x7f95f40ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:42.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95f3fff700 1 --2- 172.21.15.111:0/771768375 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95f415a5d0 0x7f95f40ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95f3fff700 1 -- 172.21.15.111:0/771768375 >> v1:172.21.15.149:6789/0 conn(0x7f95f40cb2d0 legacy=0x7f95f415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95f3fff700 1 -- 172.21.15.111:0/771768375 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f40cc670 con 0x7f95f40b0160 2023-11-28T13:00:42.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95f3fff700 1 --2- 172.21.15.111:0/771768375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f40b0160 0x7f95f40ced90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f95e00098f0 tx=0x7f95e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d32c03089297b5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:42.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.783+0000 7f95f37fe700 1 -- 172.21.15.111:0/771768375 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e000b8a0 con 0x7f95f40b0160 2023-11-28T13:00:42.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.784+0000 7f95f37fe700 1 -- 172.21.15.111:0/771768375 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e000ba00 con 0x7f95f40b0160 2023-11-28T13:00:42.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.784+0000 7f95f37fe700 1 -- 172.21.15.111:0/771768375 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e000bd90 con 0x7f95f40b0160 2023-11-28T13:00:42.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.784+0000 7f95facb3700 1 -- 172.21.15.111:0/771768375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f40b0160 msgr2=0x7f95f40ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.784+0000 7f95facb3700 1 --2- 172.21.15.111:0/771768375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f40b0160 0x7f95f40ced90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f95e00098f0 tx=0x7f95e0009c00 comp rx=0 tx=0).stop 2023-11-28T13:00:42.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.784+0000 7f95facb3700 1 -- 172.21.15.111:0/771768375 shutdown_connections 2023-11-28T13:00:42.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.784+0000 7f95facb3700 1 --2- 172.21.15.111:0/771768375 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95f415a5d0 0x7f95f40ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.784+0000 7f95facb3700 1 --2- 172.21.15.111:0/771768375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f40b0160 0x7f95f40ced90 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.784+0000 7f95facb3700 1 -- 172.21.15.111:0/771768375 >> 172.21.15.111:0/771768375 conn(0x7f95f409a040 msgr2=0x7f95f40d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:42.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.784+0000 7f95facb3700 1 -- 172.21.15.111:0/771768375 shutdown_connections 2023-11-28T13:00:42.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.784+0000 7f95facb3700 1 -- 172.21.15.111:0/771768375 wait complete. 2023-11-28T13:00:42.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.786+0000 7f95facb3700 1 Processor -- start 2023-11-28T13:00:42.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.786+0000 7f95facb3700 1 -- start start 2023-11-28T13:00:42.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95facb3700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95f40b0160 0x7f95f415b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95facb3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f40cb2d0 0x7f95f414c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95facb3700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f95f415a5d0 0x7f95f4154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95facb3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f415cb60 con 0x7f95f40cb2d0 2023-11-28T13:00:42.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95facb3700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f41f4170 con 0x7f95f415a5d0 2023-11-28T13:00:42.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95facb3700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f415d700 con 0x7f95f40b0160 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f40cb2d0 0x7f95f414c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f40cb2d0 0x7f95f414c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35802/0 (socket says 172.21.15.111:35802) 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f3fff700 1 -- 172.21.15.111:0/2401273905 learned_addr learned my addr 172.21.15.111:0/2401273905 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f8a4f700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95f40b0160 0x7f95f415b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f9250700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f95f415a5d0 0x7f95f4154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f3fff700 1 -- 172.21.15.111:0/2401273905 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95f40b0160 msgr2=0x7f95f415b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f3fff700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95f40b0160 0x7f95f415b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f3fff700 1 -- 172.21.15.111:0/2401273905 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f95f415a5d0 msgr2=0x7f95f4154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f3fff700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f95f415a5d0 0x7f95f4154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f3fff700 1 -- 172.21.15.111:0/2401273905 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e0009550 con 0x7f95f40cb2d0 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f9250700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f95f415a5d0 0x7f95f4154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f3fff700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f40cb2d0 0x7f95f414c160 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f95e0009c40 tx=0x7f95e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f17fa700 1 -- 172.21.15.111:0/2401273905 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e00045d0 con 0x7f95f40cb2d0 2023-11-28T13:00:42.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f17fa700 1 -- 172.21.15.111:0/2401273905 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e001e070 con 0x7f95f40cb2d0 2023-11-28T13:00:42.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95facb3700 1 -- 172.21.15.111:0/2401273905 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f40ccc30 con 0x7f95f40cb2d0 2023-11-28T13:00:42.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95f17fa700 1 -- 172.21.15.111:0/2401273905 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e00038e0 con 0x7f95f40cb2d0 2023-11-28T13:00:42.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.787+0000 7f95facb3700 1 -- 172.21.15.111:0/2401273905 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f40cc950 con 0x7f95f40cb2d0 2023-11-28T13:00:42.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.789+0000 7f95facb3700 1 -- 172.21.15.111:0/2401273905 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f40ccf10 con 0x7f95f40cb2d0 2023-11-28T13:00:42.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.789+0000 7f95f17fa700 1 -- 172.21.15.111:0/2401273905 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f95e000f050 con 0x7f95f40cb2d0 2023-11-28T13:00:42.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.789+0000 7f95f17fa700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95d80682f0 0x7f95d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:42.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.789+0000 7f95f17fa700 1 -- 172.21.15.111:0/2401273905 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f95e0020070 con 0x7f95f40cb2d0 2023-11-28T13:00:42.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.789+0000 7f95f8a4f700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95d80682f0 0x7f95d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:42.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.790+0000 7f95f8a4f700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95d80682f0 0x7f95d806a7a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f95e8006fd0 tx=0x7f95e8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:42.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:42.793+0000 7f95f17fa700 1 -- 172.21.15.111:0/2401273905 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f95e00539c0 con 0x7f95f40cb2d0 2023-11-28T13:00:43.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.023+0000 7f95facb3700 1 -- 172.21.15.111:0/2401273905 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95f4067170 con 0x7f95f40cb2d0 2023-11-28T13:00:43.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.023+0000 7f95f17fa700 1 -- 172.21.15.111:0/2401273905 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f95e0056fe0 con 0x7f95f40cb2d0 2023-11-28T13:00:43.025 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:43.025 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3254653481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3254653481},{"type":"v1","addr":"172.21.15.149:6835","nonce":3254653481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1890904311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1890904311},{"type":"v1","addr":"172.21.15.111:6836","nonce":1890904311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1592804351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1592804351},{"type":"v1","addr":"172.21.15.111:6837","nonce":1592804351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:12.011963+0000","modified":"2023-11-28T13:00:15.034385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5352},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5352":{"gid":5352,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.149:6834/3579364312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":3579364312},{"type":"v1","addr":"172.21.15.149:6834","nonce":3579364312}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T13:00:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.025+0000 7f95facb3700 1 -- 172.21.15.111:0/2401273905 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95d80682f0 msgr2=0x7f95d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.025+0000 7f95facb3700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95d80682f0 0x7f95d806a7a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f95e8006fd0 tx=0x7f95e8008040 comp rx=0 tx=0).stop 2023-11-28T13:00:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.025+0000 7f95facb3700 1 -- 172.21.15.111:0/2401273905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f40cb2d0 msgr2=0x7f95f414c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.025+0000 7f95facb3700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f40cb2d0 0x7f95f414c160 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f95e0009c40 tx=0x7f95e0004750 comp rx=0 tx=0).stop 2023-11-28T13:00:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.025+0000 7f95facb3700 1 -- 172.21.15.111:0/2401273905 shutdown_connections 2023-11-28T13:00:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.025+0000 7f95facb3700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95d80682f0 0x7f95d806a7a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.025+0000 7f95facb3700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f95f415a5d0 0x7f95f4154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.025+0000 7f95facb3700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95f40b0160 0x7f95f415b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.025+0000 7f95facb3700 1 --2- 172.21.15.111:0/2401273905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95f40cb2d0 0x7f95f414c160 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.025+0000 7f95facb3700 1 -- 172.21.15.111:0/2401273905 >> 172.21.15.111:0/2401273905 conn(0x7f95f409a040 msgr2=0x7f95f406ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:43.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.025+0000 7f95facb3700 1 -- 172.21.15.111:0/2401273905 shutdown_connections 2023-11-28T13:00:43.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.025+0000 7f95facb3700 1 -- 172.21.15.111:0/2401273905 wait complete. 2023-11-28T13:00:43.028 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 17 2023-11-28T13:00:43.039 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:00:43.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.156+0000 7f0208062700 1 Processor -- start 2023-11-28T13:00:43.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.156+0000 7f0208062700 1 -- start start 2023-11-28T13:00:43.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.157+0000 7f0208062700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f02000cd680 0x7f02000cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:43.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.157+0000 7f0208062700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f020006ade0 0x7f020006b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:43.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.157+0000 7f0208062700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020006a950 con 0x7f020015c710 2023-11-28T13:00:43.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.157+0000 7f0208062700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02000ce030 con 0x7f020006ade0 2023-11-28T13:00:43.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.157+0000 7f0208062700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02000b1980 con 0x7f02000cd680 2023-11-28T13:00:43.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.157+0000 7f02065ff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f020006ade0 0x7f020006b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:43.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.157+0000 7f02065ff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f020006ade0 0x7f020006b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35812/0 (socket says 172.21.15.111:35812) 2023-11-28T13:00:43.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.157+0000 7f02065ff700 1 -- 172.21.15.111:0/3163777475 learned_addr learned my addr 172.21.15.111:0/3163777475 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:43.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.157+0000 7f02055fd700 1 --2- 172.21.15.111:0/3163777475 >> v2:172.21.15.149:3300/0 conn(0x7f02000cd680 0x7f02000cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:43.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.157+0000 7f02065ff700 1 -- 172.21.15.111:0/3163777475 >> v2:172.21.15.149:3300/0 conn(0x7f02000cd680 msgr2=0x7f02000cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:43.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.158+0000 7f02065ff700 1 --2- 172.21.15.111:0/3163777475 >> v2:172.21.15.149:3300/0 conn(0x7f02000cd680 0x7f02000cdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.158+0000 7f02065ff700 1 -- 172.21.15.111:0/3163777475 >> v1:172.21.15.149:6789/0 conn(0x7f020015c710 legacy=0x7f0200160cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:43.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.158+0000 7f02065ff700 1 -- 172.21.15.111:0/3163777475 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02000b1c60 con 0x7f020006ade0 2023-11-28T13:00:43.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.158+0000 7f02065ff700 1 --2- 172.21.15.111:0/3163777475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f020006ade0 0x7f020006b200 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f02000cfb10 tx=0x7f01f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6c56363d71eec2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:43.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.158+0000 7f0204dfc700 1 -- 172.21.15.111:0/3163777475 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f800e070 con 0x7f020006ade0 2023-11-28T13:00:43.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.158+0000 7f0204dfc700 1 -- 172.21.15.111:0/3163777475 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f800b9c0 con 0x7f020006ade0 2023-11-28T13:00:43.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.158+0000 7f0204dfc700 1 -- 172.21.15.111:0/3163777475 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f800bd70 con 0x7f020006ade0 2023-11-28T13:00:43.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.158+0000 7f0208062700 1 -- 172.21.15.111:0/3163777475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f020006ade0 msgr2=0x7f020006b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:43.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.158+0000 7f0208062700 1 --2- 172.21.15.111:0/3163777475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f020006ade0 0x7f020006b200 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f02000cfb10 tx=0x7f01f800b330 comp rx=0 tx=0).stop 2023-11-28T13:00:43.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.158+0000 7f0208062700 1 -- 172.21.15.111:0/3163777475 shutdown_connections 2023-11-28T13:00:43.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.158+0000 7f0208062700 1 --2- 172.21.15.111:0/3163777475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f020006ade0 0x7f020006b200 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.158+0000 7f0208062700 1 --2- 172.21.15.111:0/3163777475 >> v2:172.21.15.149:3300/0 conn(0x7f02000cd680 0x7f02000cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.158+0000 7f0208062700 1 -- 172.21.15.111:0/3163777475 >> 172.21.15.111:0/3163777475 conn(0x7f02000d07d0 msgr2=0x7f02000cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:43.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.159+0000 7f0208062700 1 -- 172.21.15.111:0/3163777475 shutdown_connections 2023-11-28T13:00:43.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.159+0000 7f0208062700 1 -- 172.21.15.111:0/3163777475 wait complete. 2023-11-28T13:00:43.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.160+0000 7f0208062700 1 Processor -- start 2023-11-28T13:00:43.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.160+0000 7f0208062700 1 -- start start 2023-11-28T13:00:43.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.160+0000 7f0208062700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f020006ade0 0x7f0200156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:43.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.160+0000 7f0208062700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f02000cd680 0x7f02000b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:43.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.160+0000 7f0208062700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f020015c710 0x7f02000b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:43.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f0208062700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020015e680 con 0x7f020015c710 2023-11-28T13:00:43.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f0208062700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02001f6120 con 0x7f020006ade0 2023-11-28T13:00:43.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f0208062700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02001f6400 con 0x7f02000cd680 2023-11-28T13:00:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f02065ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f020015c710 0x7f02000b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f02065ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f020015c710 0x7f02000b0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35818/0 (socket says 172.21.15.111:35818) 2023-11-28T13:00:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f02065ff700 1 -- 172.21.15.111:0/3854111369 learned_addr learned my addr 172.21.15.111:0/3854111369 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f02065ff700 1 -- 172.21.15.111:0/3854111369 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f02000cd680 msgr2=0x7f02000b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f02065ff700 1 --2- 172.21.15.111:0/3854111369 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f02000cd680 0x7f02000b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f02065ff700 1 -- 172.21.15.111:0/3854111369 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f020006ade0 msgr2=0x7f0200156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f02065ff700 1 --2- 172.21.15.111:0/3854111369 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f020006ade0 0x7f0200156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f02065ff700 1 -- 172.21.15.111:0/3854111369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f8009b30 con 0x7f020015c710 2023-11-28T13:00:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f02065ff700 1 --2- 172.21.15.111:0/3854111369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f020015c710 0x7f02000b0f70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f02000717e0 tx=0x7f01f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f01f27fc700 1 -- 172.21.15.111:0/3854111369 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f80099b0 con 0x7f020015c710 2023-11-28T13:00:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f01f27fc700 1 -- 172.21.15.111:0/3854111369 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f80092e0 con 0x7f020015c710 2023-11-28T13:00:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f0208062700 1 -- 172.21.15.111:0/3854111369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020015cae0 con 0x7f020015c710 2023-11-28T13:00:43.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f01f27fc700 1 -- 172.21.15.111:0/3854111369 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f801c640 con 0x7f020015c710 2023-11-28T13:00:43.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.161+0000 7f0208062700 1 -- 172.21.15.111:0/3854111369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02000b1f40 con 0x7f020015c710 2023-11-28T13:00:43.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.162+0000 7f0208062700 1 -- 172.21.15.111:0/3854111369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020006a400 con 0x7f020015c710 2023-11-28T13:00:43.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.163+0000 7f01f27fc700 1 -- 172.21.15.111:0/3854111369 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f01f8022030 con 0x7f020015c710 2023-11-28T13:00:43.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.163+0000 7f01f27fc700 1 --2- 172.21.15.111:0/3854111369 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01e40682f0 0x7f01e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:43.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.163+0000 7f01f27fc700 1 -- 172.21.15.111:0/3854111369 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f01f80876f0 con 0x7f020015c710 2023-11-28T13:00:43.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.163+0000 7f0205dfe700 1 --2- 172.21.15.111:0/3854111369 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01e40682f0 0x7f01e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:43.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.163+0000 7f0205dfe700 1 --2- 172.21.15.111:0/3854111369 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01e40682f0 0x7f01e406a7a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f01f4000f60 tx=0x7f01f4007a40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:43.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.167+0000 7f01f27fc700 1 -- 172.21.15.111:0/3854111369 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f01f8053dd0 con 0x7f020015c710 2023-11-28T13:00:43.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.351+0000 7f0208062700 1 -- 172.21.15.111:0/3854111369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f020015e450 con 0x7f020015c710 2023-11-28T13:00:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.352+0000 7f01f27fc700 1 -- 172.21.15.111:0/3854111369 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+18271 (secure 0 0 0) 0x7f020015e450 con 0x7f020015c710 2023-11-28T13:00:43.354 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:43.355 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":43,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:42.666575+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:09.793782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:10.979525+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.7","mappings":[{"from":0,"to":4}]},{"pgid":"7.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:00:43.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.354+0000 7f0208062700 1 -- 172.21.15.111:0/3854111369 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01e40682f0 msgr2=0x7f01e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:43.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.354+0000 7f0208062700 1 --2- 172.21.15.111:0/3854111369 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01e40682f0 0x7f01e406a7a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f01f4000f60 tx=0x7f01f4007a40 comp rx=0 tx=0).stop 2023-11-28T13:00:43.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.354+0000 7f0208062700 1 -- 172.21.15.111:0/3854111369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f020015c710 msgr2=0x7f02000b0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:43.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.354+0000 7f0208062700 1 --2- 172.21.15.111:0/3854111369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f020015c710 0x7f02000b0f70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f02000717e0 tx=0x7f01f8004ad0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.354+0000 7f0208062700 1 -- 172.21.15.111:0/3854111369 shutdown_connections 2023-11-28T13:00:43.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.354+0000 7f0208062700 1 --2- 172.21.15.111:0/3854111369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f020015c710 0x7f02000b0f70 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.354+0000 7f0208062700 1 --2- 172.21.15.111:0/3854111369 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01e40682f0 0x7f01e406a7a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.354+0000 7f0208062700 1 --2- 172.21.15.111:0/3854111369 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f020006ade0 0x7f0200156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.354+0000 7f0208062700 1 --2- 172.21.15.111:0/3854111369 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f02000cd680 0x7f02000b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.354+0000 7f0208062700 1 -- 172.21.15.111:0/3854111369 >> 172.21.15.111:0/3854111369 conn(0x7f02000d07d0 msgr2=0x7f02000cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:43.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.354+0000 7f0208062700 1 -- 172.21.15.111:0/3854111369 shutdown_connections 2023-11-28T13:00:43.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.354+0000 7f0208062700 1 -- 172.21.15.111:0/3854111369 wait complete. 2023-11-28T13:00:43.369 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T13:00:43.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.488+0000 7fa5c3228700 1 Processor -- start 2023-11-28T13:00:43.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.488+0000 7fa5c3228700 1 -- start start 2023-11-28T13:00:43.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5c3228700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa5bc0cf860 0x7fa5bc06b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:43.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5c3228700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc0cfdb0 con 0x7fa5bc15cbf0 2023-11-28T13:00:43.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5c3228700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc0cc740 con 0x7fa5bc0cffd0 2023-11-28T13:00:43.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5c3228700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5bc0b35e0 con 0x7fa5bc0cf860 2023-11-28T13:00:43.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5bbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa5bc0cf860 0x7fa5bc06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:43.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5c0fc4700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa5bc15cbf0 0x7fa5bc0b3b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36600/0 (socket says 172.21.15.111:36600) 2023-11-28T13:00:43.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5bbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa5bc0cf860 0x7fa5bc06b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35824/0 (socket says 172.21.15.111:35824) 2023-11-28T13:00:43.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5c0fc4700 1 -- 172.21.15.111:0/2897891638 learned_addr learned my addr 172.21.15.111:0/2897891638 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:43.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5bbfff700 1 -- 172.21.15.111:0/2897891638 >> v1:172.21.15.149:6789/0 conn(0x7fa5bc0cffd0 legacy=0x7fa5bc0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:43.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5bbfff700 1 -- 172.21.15.111:0/2897891638 >> v1:172.21.15.111:6789/0 conn(0x7fa5bc15cbf0 legacy=0x7fa5bc0b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:43.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5bb7fe700 1 -- 172.21.15.111:0/2897891638 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980074939 0 0) 0x7fa5bc0cfdb0 con 0x7fa5bc15cbf0 2023-11-28T13:00:43.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5bbfff700 1 -- 172.21.15.111:0/2897891638 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc0cbe10 con 0x7fa5bc0cf860 2023-11-28T13:00:43.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5bbfff700 1 --2- 172.21.15.111:0/2897891638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5bc0cf860 0x7fa5bc06b380 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009840 tx=0x7fa5a8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25931ae523499f09 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:43.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5bb7fe700 1 -- 172.21.15.111:0/2897891638 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8011830 con 0x7fa5bc0cf860 2023-11-28T13:00:43.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5bb7fe700 1 -- 172.21.15.111:0/2897891638 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a8011990 con 0x7fa5bc0cf860 2023-11-28T13:00:43.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.489+0000 7fa5bb7fe700 1 -- 172.21.15.111:0/2897891638 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8011d20 con 0x7fa5bc0cf860 2023-11-28T13:00:43.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.490+0000 7fa5c3228700 1 -- 172.21.15.111:0/2897891638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5bc0cf860 msgr2=0x7fa5bc06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:43.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.490+0000 7fa5c3228700 1 --2- 172.21.15.111:0/2897891638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5bc0cf860 0x7fa5bc06b380 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009840 tx=0x7fa5a8009b50 comp rx=0 tx=0).stop 2023-11-28T13:00:43.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.490+0000 7fa5c3228700 1 -- 172.21.15.111:0/2897891638 shutdown_connections 2023-11-28T13:00:43.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.490+0000 7fa5c3228700 1 --2- 172.21.15.111:0/2897891638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5bc0cf860 0x7fa5bc06b380 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.490+0000 7fa5c3228700 1 -- 172.21.15.111:0/2897891638 >> 172.21.15.111:0/2897891638 conn(0x7fa5bc09a040 msgr2=0x7fa5bc0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:43.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.490+0000 7fa5c3228700 1 -- 172.21.15.111:0/2897891638 shutdown_connections 2023-11-28T13:00:43.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.490+0000 7fa5c3228700 1 -- 172.21.15.111:0/2897891638 wait complete. 2023-11-28T13:00:43.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.491+0000 7fa5c3228700 1 Processor -- start 2023-11-28T13:00:43.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.491+0000 7fa5c3228700 1 -- start start 2023-11-28T13:00:43.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c3228700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5bc0cf860 0x7fa5bc15cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:43.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c3228700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa5bc0cffd0 0x7fa5bc15fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:43.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c3228700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5bc15cbf0 0x7fa5bc151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:43.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c3228700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc0ca360 con 0x7fa5bc15cbf0 2023-11-28T13:00:43.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c3228700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc06fe50 con 0x7fa5bc0cffd0 2023-11-28T13:00:43.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c3228700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc0c9770 con 0x7fa5bc0cf860 2023-11-28T13:00:43.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c17c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5bc15cbf0 0x7fa5bc151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:43.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c17c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5bc15cbf0 0x7fa5bc151140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35832/0 (socket says 172.21.15.111:35832) 2023-11-28T13:00:43.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c17c5700 1 -- 172.21.15.111:0/3281452388 learned_addr learned my addr 172.21.15.111:0/3281452388 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:43.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c0fc4700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5bc0cf860 0x7fa5bc15cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:43.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5bbfff700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa5bc0cffd0 0x7fa5bc15fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:43.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c17c5700 1 -- 172.21.15.111:0/3281452388 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5bc0cf860 msgr2=0x7fa5bc15cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:43.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c17c5700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5bc0cf860 0x7fa5bc15cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c17c5700 1 -- 172.21.15.111:0/3281452388 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa5bc0cffd0 msgr2=0x7fa5bc15fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:43.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c17c5700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa5bc0cffd0 0x7fa5bc15fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:43.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.492+0000 7fa5c17c5700 1 -- 172.21.15.111:0/3281452388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a80094a0 con 0x7fa5bc15cbf0 2023-11-28T13:00:43.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.493+0000 7fa5bbfff700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa5bc0cffd0 0x7fa5bc15fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:43.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.493+0000 7fa5c17c5700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5bc15cbf0 0x7fa5bc151140 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400cc80 tx=0x7fa5b400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:43.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.493+0000 7fa5b97fa700 1 -- 172.21.15.111:0/3281452388 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b400ad40 con 0x7fa5bc15cbf0 2023-11-28T13:00:43.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.493+0000 7fa5b97fa700 1 -- 172.21.15.111:0/3281452388 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b400ce80 con 0x7fa5bc15cbf0 2023-11-28T13:00:43.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.493+0000 7fa5c3228700 1 -- 172.21.15.111:0/3281452388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5bc0cc3d0 con 0x7fa5bc15cbf0 2023-11-28T13:00:43.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.493+0000 7fa5b97fa700 1 -- 172.21.15.111:0/3281452388 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b400aea0 con 0x7fa5bc15cbf0 2023-11-28T13:00:43.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.493+0000 7fa5c3228700 1 -- 172.21.15.111:0/3281452388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc0cc0f0 con 0x7fa5bc15cbf0 2023-11-28T13:00:43.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.494+0000 7fa5c3228700 1 -- 172.21.15.111:0/3281452388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5bc0b4270 con 0x7fa5bc15cbf0 2023-11-28T13:00:43.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.494+0000 7fa5b97fa700 1 -- 172.21.15.111:0/3281452388 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa5b4010050 con 0x7fa5bc15cbf0 2023-11-28T13:00:43.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.494+0000 7fa5b97fa700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5a00682f0 0x7fa5a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:43.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.495+0000 7fa5b97fa700 1 -- 172.21.15.111:0/3281452388 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fa5b4085ef0 con 0x7fa5bc15cbf0 2023-11-28T13:00:43.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.495+0000 7fa5c0fc4700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5a00682f0 0x7fa5a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:43.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.495+0000 7fa5c0fc4700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5a00682f0 0x7fa5a006a7a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0000f60 tx=0x7fa5b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:43.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.499+0000 7fa5b97fa700 1 -- 172.21.15.111:0/3281452388 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa5b40524e0 con 0x7fa5bc15cbf0 2023-11-28T13:00:43.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:43.727+0000 7fa5c3228700 1 -- 172.21.15.111:0/3281452388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa5bc067880 con 0x7fa5bc15cbf0 2023-11-28T13:00:44.675 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-11-28T13:00:44.673+0000 7fb379352700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T13:00:44.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.678+0000 7fa5b97fa700 1 -- 172.21.15.111:0/3281452388 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa5b4055b00 con 0x7fa5bc15cbf0 2023-11-28T13:00:44.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.681+0000 7fa5c3228700 1 -- 172.21.15.111:0/3281452388 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5a00682f0 msgr2=0x7fa5a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.681+0000 7fa5c3228700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5a00682f0 0x7fa5a006a7a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0000f60 tx=0x7fa5b0009040 comp rx=0 tx=0).stop 2023-11-28T13:00:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.681+0000 7fa5c3228700 1 -- 172.21.15.111:0/3281452388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5bc15cbf0 msgr2=0x7fa5bc151140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.681+0000 7fa5c3228700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5bc15cbf0 0x7fa5bc151140 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400cc80 tx=0x7fa5b400a340 comp rx=0 tx=0).stop 2023-11-28T13:00:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.681+0000 7fa5c3228700 1 -- 172.21.15.111:0/3281452388 shutdown_connections 2023-11-28T13:00:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.681+0000 7fa5c3228700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5bc15cbf0 0x7fa5bc151140 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.681+0000 7fa5c3228700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa5bc0cf860 0x7fa5bc15cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.681+0000 7fa5c3228700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5a00682f0 0x7fa5a006a7a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.681+0000 7fa5c3228700 1 --2- 172.21.15.111:0/3281452388 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa5bc0cffd0 0x7fa5bc15fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.681+0000 7fa5c3228700 1 -- 172.21.15.111:0/3281452388 >> 172.21.15.111:0/3281452388 conn(0x7fa5bc09a040 msgr2=0x7fa5bc0b4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.681+0000 7fa5c3228700 1 -- 172.21.15.111:0/3281452388 shutdown_connections 2023-11-28T13:00:44.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.681+0000 7fa5c3228700 1 -- 172.21.15.111:0/3281452388 wait complete. 2023-11-28T13:00:44.684 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T13:00:44.684 INFO:tasks.ceph.mds.d.smithi149.stderr: -7409> 2023-11-28T13:00:15.067+0000 7f4308044700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T13:00:44.696 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T13:00:44.718 INFO:tasks.ceph.mds.d.smithi149.stdout:starting mds.d at 2023-11-28T13:00:44.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.819+0000 7faf3572f700 1 Processor -- start 2023-11-28T13:00:44.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.819+0000 7faf3572f700 1 -- start start 2023-11-28T13:00:44.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.820+0000 7faf3572f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faf300cb830 0x7faf3006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:44.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.820+0000 7faf3572f700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7faf3006a7b0 0x7faf3015df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:44.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.820+0000 7faf3572f700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf3006b490 con 0x7faf30154b60 2023-11-28T13:00:44.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.820+0000 7faf3572f700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf300c9930 con 0x7faf3006a7b0 2023-11-28T13:00:44.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.820+0000 7faf3572f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf300c9e80 con 0x7faf300cb830 2023-11-28T13:00:44.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.820+0000 7faf2e7fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faf300cb830 0x7faf3006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:44.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.820+0000 7faf2f7fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7faf3006a7b0 0x7faf3015df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:44.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.820+0000 7faf2e7fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faf300cb830 0x7faf3006a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:57586/0 (socket says 172.21.15.111:57586) 2023-11-28T13:00:44.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.820+0000 7faf2e7fc700 1 -- 172.21.15.111:0/1329081291 learned_addr learned my addr 172.21.15.111:0/1329081291 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:44.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.821+0000 7faf2dffb700 1 -- 172.21.15.111:0/1329081291 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570035646 0 0) 0x7faf3006b490 con 0x7faf30154b60 2023-11-28T13:00:44.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.821+0000 7faf2dffb700 1 -- 172.21.15.111:0/1329081291 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf180035b0 con 0x7faf30154b60 2023-11-28T13:00:44.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.821+0000 7faf2e7fc700 1 -- 172.21.15.111:0/1329081291 >> v2:172.21.15.149:3300/0 conn(0x7faf3006a7b0 msgr2=0x7faf3015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:44.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.821+0000 7faf2e7fc700 1 --2- 172.21.15.111:0/1329081291 >> v2:172.21.15.149:3300/0 conn(0x7faf3006a7b0 0x7faf3015df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:44.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.821+0000 7faf2e7fc700 1 -- 172.21.15.111:0/1329081291 >> v1:172.21.15.149:6789/0 conn(0x7faf30154b60 legacy=0x7faf3015b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:44.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.821+0000 7faf2e7fc700 1 -- 172.21.15.111:0/1329081291 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3006abf0 con 0x7faf300cb830 2023-11-28T13:00:44.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.821+0000 7faf2f7fe700 1 --2- 172.21.15.111:0/1329081291 >> v2:172.21.15.149:3300/0 conn(0x7faf3006a7b0 0x7faf3015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:44.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.821+0000 7faf2e7fc700 1 --2- 172.21.15.111:0/1329081291 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faf300cb830 0x7faf3006a270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faf240098f0 tx=0x7faf24009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb2c7aae5da695b8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:44.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.821+0000 7faf2dffb700 1 -- 172.21.15.111:0/1329081291 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24011830 con 0x7faf300cb830 2023-11-28T13:00:44.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.821+0000 7faf2dffb700 1 -- 172.21.15.111:0/1329081291 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf24011990 con 0x7faf300cb830 2023-11-28T13:00:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.821+0000 7faf2dffb700 1 -- 172.21.15.111:0/1329081291 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24011d20 con 0x7faf300cb830 2023-11-28T13:00:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.822+0000 7faf3572f700 1 -- 172.21.15.111:0/1329081291 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faf300cb830 msgr2=0x7faf3006a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.822+0000 7faf3572f700 1 --2- 172.21.15.111:0/1329081291 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faf300cb830 0x7faf3006a270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faf240098f0 tx=0x7faf24009c00 comp rx=0 tx=0).stop 2023-11-28T13:00:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.822+0000 7faf3572f700 1 -- 172.21.15.111:0/1329081291 shutdown_connections 2023-11-28T13:00:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.822+0000 7faf3572f700 1 --2- 172.21.15.111:0/1329081291 >> v2:172.21.15.149:3300/0 conn(0x7faf3006a7b0 0x7faf3015df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.822+0000 7faf3572f700 1 --2- 172.21.15.111:0/1329081291 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faf300cb830 0x7faf3006a270 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.822+0000 7faf3572f700 1 -- 172.21.15.111:0/1329081291 >> 172.21.15.111:0/1329081291 conn(0x7faf300cf2e0 msgr2=0x7faf300cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.822+0000 7faf3572f700 1 -- 172.21.15.111:0/1329081291 shutdown_connections 2023-11-28T13:00:44.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.822+0000 7faf3572f700 1 -- 172.21.15.111:0/1329081291 wait complete. 2023-11-28T13:00:44.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.823+0000 7faf3572f700 1 Processor -- start 2023-11-28T13:00:44.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.823+0000 7faf3572f700 1 -- start start 2023-11-28T13:00:44.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.824+0000 7faf3572f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faf3006a7b0 0x7faf301f1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:44.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.824+0000 7faf3572f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faf300cb830 0x7faf300b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:44.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.824+0000 7faf3572f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf30154b60 0x7faf3014d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:44.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.824+0000 7faf3572f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf300b0f40 con 0x7faf30154b60 2023-11-28T13:00:44.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.824+0000 7faf3572f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf300b0c90 con 0x7faf3006a7b0 2023-11-28T13:00:44.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.824+0000 7faf3572f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf300aecc0 con 0x7faf300cb830 2023-11-28T13:00:44.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf2f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf30154b60 0x7faf3014d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf2f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf30154b60 0x7faf3014d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35846/0 (socket says 172.21.15.111:35846) 2023-11-28T13:00:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf2f7fe700 1 -- 172.21.15.111:0/3432183694 learned_addr learned my addr 172.21.15.111:0/3432183694 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf2effd700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faf3006a7b0 0x7faf301f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf2e7fc700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faf300cb830 0x7faf300b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf2f7fe700 1 -- 172.21.15.111:0/3432183694 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faf300cb830 msgr2=0x7faf300b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf2f7fe700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faf300cb830 0x7faf300b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf2f7fe700 1 -- 172.21.15.111:0/3432183694 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faf3006a7b0 msgr2=0x7faf301f1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf2f7fe700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faf3006a7b0 0x7faf301f1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf2f7fe700 1 -- 172.21.15.111:0/3432183694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf28009a70 con 0x7faf30154b60 2023-11-28T13:00:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf2effd700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faf3006a7b0 0x7faf301f1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf2f7fe700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf30154b60 0x7faf3014d6d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7faf2800efc0 tx=0x7faf2800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:44.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf177fe700 1 -- 172.21.15.111:0/3432183694 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf28009d70 con 0x7faf30154b60 2023-11-28T13:00:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf177fe700 1 -- 172.21.15.111:0/3432183694 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2800f040 con 0x7faf30154b60 2023-11-28T13:00:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf3572f700 1 -- 172.21.15.111:0/3432183694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf24009550 con 0x7faf30154b60 2023-11-28T13:00:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf177fe700 1 -- 172.21.15.111:0/3432183694 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2800ce00 con 0x7faf30154b60 2023-11-28T13:00:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.825+0000 7faf3572f700 1 -- 172.21.15.111:0/3432183694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3006b9c0 con 0x7faf30154b60 2023-11-28T13:00:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.826+0000 7faf3572f700 1 -- 172.21.15.111:0/3432183694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3006b6e0 con 0x7faf30154b60 2023-11-28T13:00:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.827+0000 7faf177fe700 1 -- 172.21.15.111:0/3432183694 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faf28014480 con 0x7faf30154b60 2023-11-28T13:00:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.827+0000 7faf177fe700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faf1c0682f0 0x7faf1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.827+0000 7faf177fe700 1 -- 172.21.15.111:0/3432183694 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6526+0+0 (secure 0 0 0) 0x7faf28086790 con 0x7faf30154b60 2023-11-28T13:00:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.827+0000 7faf2effd700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faf1c0682f0 0x7faf1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:44.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.827+0000 7faf2effd700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faf1c0682f0 0x7faf1c06a7a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faf200011e0 tx=0x7faf20009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:44.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:44.831+0000 7faf177fe700 1 -- 172.21.15.111:0/3432183694 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faf28052dc0 con 0x7faf30154b60 2023-11-28T13:00:45.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.060+0000 7faf3572f700 1 -- 172.21.15.111:0/3432183694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7faf300c9b10 con 0x7faf30154b60 2023-11-28T13:00:45.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.735+0000 7faf177fe700 1 -- 172.21.15.111:0/3432183694 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7faf280563e0 con 0x7faf30154b60 2023-11-28T13:00:45.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.738+0000 7faf3572f700 1 -- 172.21.15.111:0/3432183694 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faf1c0682f0 msgr2=0x7faf1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:45.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.738+0000 7faf3572f700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faf1c0682f0 0x7faf1c06a7a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faf200011e0 tx=0x7faf20009040 comp rx=0 tx=0).stop 2023-11-28T13:00:45.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.738+0000 7faf3572f700 1 -- 172.21.15.111:0/3432183694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf30154b60 msgr2=0x7faf3014d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:45.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.738+0000 7faf3572f700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf30154b60 0x7faf3014d6d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7faf2800efc0 tx=0x7faf2800c3a0 comp rx=0 tx=0).stop 2023-11-28T13:00:45.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.738+0000 7faf3572f700 1 -- 172.21.15.111:0/3432183694 shutdown_connections 2023-11-28T13:00:45.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.738+0000 7faf3572f700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf30154b60 0x7faf3014d6d0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:45.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.738+0000 7faf3572f700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faf1c0682f0 0x7faf1c06a7a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:45.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.738+0000 7faf3572f700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faf3006a7b0 0x7faf301f1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:45.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.738+0000 7faf3572f700 1 --2- 172.21.15.111:0/3432183694 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faf300cb830 0x7faf300b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:45.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.738+0000 7faf3572f700 1 -- 172.21.15.111:0/3432183694 >> 172.21.15.111:0/3432183694 conn(0x7faf300cf2e0 msgr2=0x7faf3006e990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:45.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.738+0000 7faf3572f700 1 -- 172.21.15.111:0/3432183694 shutdown_connections 2023-11-28T13:00:45.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.738+0000 7faf3572f700 1 -- 172.21.15.111:0/3432183694 wait complete. 2023-11-28T13:00:45.754 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T13:00:45.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.882+0000 7f8ca1516700 1 Processor -- start 2023-11-28T13:00:45.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.882+0000 7f8ca1516700 1 -- start start 2023-11-28T13:00:45.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.883+0000 7f8ca1516700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8c9c1594f0 0x7f8c9c15a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:45.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.883+0000 7f8ca1516700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8c9c0712b0 0x7f8c9c071720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:45.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.883+0000 7f8ca1516700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f8c9c15cf50 0x7f8c9c15d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:45.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.883+0000 7f8ca1516700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c9c15a1c0 con 0x7f8c9c0712b0 2023-11-28T13:00:45.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.883+0000 7f8ca1516700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c9c0725c0 con 0x7f8c9c15cf50 2023-11-28T13:00:45.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.883+0000 7f8ca1516700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9c071c60 con 0x7f8c9c1594f0 2023-11-28T13:00:45.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.883+0000 7f8c9a7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8c9c0712b0 0x7f8c9c071720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:45.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.883+0000 7f8c9a7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8c9c0712b0 0x7f8c9c071720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35852/0 (socket says 172.21.15.111:35852) 2023-11-28T13:00:45.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.883+0000 7f8c9a7fc700 1 -- 172.21.15.111:0/2517829445 learned_addr learned my addr 172.21.15.111:0/2517829445 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.883+0000 7f8c9affd700 1 --2- 172.21.15.111:0/2517829445 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8c9c1594f0 0x7f8c9c15a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.883+0000 7f8c9b7fe700 1 --2- 172.21.15.111:0/2517829445 >> v2:172.21.15.149:3300/0 conn(0x7f8c9c15cf50 0x7f8c9c15d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8c9a7fc700 1 -- 172.21.15.111:0/2517829445 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8c9c1594f0 msgr2=0x7f8c9c15a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8c9a7fc700 1 --2- 172.21.15.111:0/2517829445 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8c9c1594f0 0x7f8c9c15a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8c9a7fc700 1 -- 172.21.15.111:0/2517829445 >> v2:172.21.15.149:3300/0 conn(0x7f8c9c15cf50 msgr2=0x7f8c9c15d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8c9a7fc700 1 --2- 172.21.15.111:0/2517829445 >> v2:172.21.15.149:3300/0 conn(0x7f8c9c15cf50 0x7f8c9c15d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8c9a7fc700 1 -- 172.21.15.111:0/2517829445 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c9c1598c0 con 0x7f8c9c0712b0 2023-11-28T13:00:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8c9affd700 1 --2- 172.21.15.111:0/2517829445 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8c9c1594f0 0x7f8c9c15a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8c9b7fe700 1 --2- 172.21.15.111:0/2517829445 >> v2:172.21.15.149:3300/0 conn(0x7f8c9c15cf50 0x7f8c9c15d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:45.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8c9a7fc700 1 --2- 172.21.15.111:0/2517829445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c9c0712b0 0x7f8c9c071720 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0cf190 tx=0x7f8c8c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e302a898fb0ec238 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:45.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8c99ffb700 1 -- 172.21.15.111:0/2517829445 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c00b860 con 0x7f8c9c0712b0 2023-11-28T13:00:45.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8c99ffb700 1 -- 172.21.15.111:0/2517829445 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8c00b9c0 con 0x7f8c9c0712b0 2023-11-28T13:00:45.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8c99ffb700 1 -- 172.21.15.111:0/2517829445 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c00bd50 con 0x7f8c9c0712b0 2023-11-28T13:00:45.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8ca1516700 1 -- 172.21.15.111:0/2517829445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c9c0712b0 msgr2=0x7f8c9c071720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:45.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8ca1516700 1 --2- 172.21.15.111:0/2517829445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c9c0712b0 0x7f8c9c071720 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0cf190 tx=0x7f8c8c009c00 comp rx=0 tx=0).stop 2023-11-28T13:00:45.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8ca1516700 1 -- 172.21.15.111:0/2517829445 shutdown_connections 2023-11-28T13:00:45.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8ca1516700 1 --2- 172.21.15.111:0/2517829445 >> v2:172.21.15.149:3300/0 conn(0x7f8c9c15cf50 0x7f8c9c15d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:45.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8ca1516700 1 --2- 172.21.15.111:0/2517829445 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8c9c1594f0 0x7f8c9c15a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:45.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8ca1516700 1 --2- 172.21.15.111:0/2517829445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c9c0712b0 0x7f8c9c071720 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:45.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.884+0000 7f8ca1516700 1 -- 172.21.15.111:0/2517829445 >> 172.21.15.111:0/2517829445 conn(0x7f8c9c0cf2e0 msgr2=0x7f8c9c0cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:45.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.885+0000 7f8ca1516700 1 -- 172.21.15.111:0/2517829445 shutdown_connections 2023-11-28T13:00:45.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.885+0000 7f8ca1516700 1 -- 172.21.15.111:0/2517829445 wait complete. 2023-11-28T13:00:45.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.886+0000 7f8ca1516700 1 Processor -- start 2023-11-28T13:00:45.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.886+0000 7f8ca1516700 1 -- start start 2023-11-28T13:00:45.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8ca1516700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8c9c0712b0 0x7f8c9c14ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:45.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8ca1516700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c9c1594f0 0x7f8c9c1e4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:45.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8ca1516700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8c9c15cf50 0x7f8c9c1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:45.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8ca1516700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9c1e3c90 con 0x7f8c9c1594f0 2023-11-28T13:00:45.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8ca1516700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9c14ef40 con 0x7f8c9c0712b0 2023-11-28T13:00:45.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8ca1516700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9c1e3010 con 0x7f8c9c15cf50 2023-11-28T13:00:45.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8c9a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c9c1594f0 0x7f8c9c1e4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:45.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8c9a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c9c1594f0 0x7f8c9c1e4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35858/0 (socket says 172.21.15.111:35858) 2023-11-28T13:00:45.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8c9a7fc700 1 -- 172.21.15.111:0/1572236000 learned_addr learned my addr 172.21.15.111:0/1572236000 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:45.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8c9a7fc700 1 -- 172.21.15.111:0/1572236000 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8c9c15cf50 msgr2=0x7f8c9c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8c9affd700 1 --2- 172.21.15.111:0/1572236000 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8c9c0712b0 0x7f8c9c14ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8c9a7fc700 1 --2- 172.21.15.111:0/1572236000 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8c9c15cf50 0x7f8c9c1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8c9a7fc700 1 -- 172.21.15.111:0/1572236000 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8c9c0712b0 msgr2=0x7f8c9c14ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8c9a7fc700 1 --2- 172.21.15.111:0/1572236000 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8c9c0712b0 0x7f8c9c14ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8c9a7fc700 1 -- 172.21.15.111:0/1572236000 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c009550 con 0x7f8c9c1594f0 2023-11-28T13:00:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8c9a7fc700 1 --2- 172.21.15.111:0/1572236000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c9c1594f0 0x7f8c9c1e4550 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c009130 tx=0x7f8c8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8c837fe700 1 -- 172.21.15.111:0/1572236000 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c0044d0 con 0x7f8c9c1594f0 2023-11-28T13:00:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8c837fe700 1 -- 172.21.15.111:0/1572236000 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8c01e070 con 0x7f8c9c1594f0 2023-11-28T13:00:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8c837fe700 1 -- 172.21.15.111:0/1572236000 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c0038a0 con 0x7f8c9c1594f0 2023-11-28T13:00:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8ca1516700 1 -- 172.21.15.111:0/1572236000 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c9c15d910 con 0x7f8c9c1594f0 2023-11-28T13:00:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.887+0000 7f8ca1516700 1 -- 172.21.15.111:0/1572236000 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c9c15de40 con 0x7f8c9c1594f0 2023-11-28T13:00:45.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.888+0000 7f8ca1516700 1 -- 172.21.15.111:0/1572236000 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c9c15db60 con 0x7f8c9c1594f0 2023-11-28T13:00:45.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.889+0000 7f8c837fe700 1 -- 172.21.15.111:0/1572236000 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8c8c00f050 con 0x7f8c9c1594f0 2023-11-28T13:00:45.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.889+0000 7f8c837fe700 1 --2- 172.21.15.111:0/1572236000 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8c880682f0 0x7f8c8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:45.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.889+0000 7f8c9affd700 1 --2- 172.21.15.111:0/1572236000 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8c880682f0 0x7f8c8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:45.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.889+0000 7f8c837fe700 1 -- 172.21.15.111:0/1572236000 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6526+0+0 (secure 0 0 0) 0x7f8c8c020070 con 0x7f8c9c1594f0 2023-11-28T13:00:45.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.889+0000 7f8c9affd700 1 --2- 172.21.15.111:0/1572236000 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8c880682f0 0x7f8c8806a7a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009910 tx=0x7f8c90008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:45.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:45.893+0000 7f8c837fe700 1 -- 172.21.15.111:0/1572236000 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8c8c053810 con 0x7f8c9c1594f0 2023-11-28T13:00:46.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.074+0000 7f8ca1516700 1 -- 172.21.15.111:0/1572236000 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8c9c1598c0 con 0x7f8c9c1594f0 2023-11-28T13:00:46.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.738+0000 7f8c837fe700 1 -- 172.21.15.111:0/1572236000 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v45) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8c9c1598c0 con 0x7f8c9c1594f0 2023-11-28T13:00:46.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.741+0000 7f8ca1516700 1 -- 172.21.15.111:0/1572236000 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8c880682f0 msgr2=0x7f8c8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:46.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.741+0000 7f8ca1516700 1 --2- 172.21.15.111:0/1572236000 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8c880682f0 0x7f8c8806a7a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009910 tx=0x7f8c90008040 comp rx=0 tx=0).stop 2023-11-28T13:00:46.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.741+0000 7f8ca1516700 1 -- 172.21.15.111:0/1572236000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c9c1594f0 msgr2=0x7f8c9c1e4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:46.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.741+0000 7f8ca1516700 1 --2- 172.21.15.111:0/1572236000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c9c1594f0 0x7f8c9c1e4550 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c009130 tx=0x7f8c8c004750 comp rx=0 tx=0).stop 2023-11-28T13:00:46.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.741+0000 7f8ca1516700 1 -- 172.21.15.111:0/1572236000 shutdown_connections 2023-11-28T13:00:46.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.741+0000 7f8ca1516700 1 --2- 172.21.15.111:0/1572236000 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8c9c15cf50 0x7f8c9c1536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:46.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.741+0000 7f8ca1516700 1 --2- 172.21.15.111:0/1572236000 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8c880682f0 0x7f8c8806a7a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:46.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.741+0000 7f8ca1516700 1 --2- 172.21.15.111:0/1572236000 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8c9c0712b0 0x7f8c9c14ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:46.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.741+0000 7f8ca1516700 1 --2- 172.21.15.111:0/1572236000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c9c1594f0 0x7f8c9c1e4550 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:46.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.741+0000 7f8ca1516700 1 -- 172.21.15.111:0/1572236000 >> 172.21.15.111:0/1572236000 conn(0x7f8c9c0cf2e0 msgr2=0x7f8c9c15b190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:46.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.741+0000 7f8ca1516700 1 -- 172.21.15.111:0/1572236000 shutdown_connections 2023-11-28T13:00:46.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.741+0000 7f8ca1516700 1 -- 172.21.15.111:0/1572236000 wait complete. 2023-11-28T13:00:46.744 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-11-28T13:00:46.757 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T13:00:46.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.885+0000 7fade84eb700 1 Processor -- start 2023-11-28T13:00:46.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.886+0000 7fade84eb700 1 -- start start 2023-11-28T13:00:46.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.886+0000 7fade84eb700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fade006a980 0x7fade00b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:46.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.886+0000 7fade84eb700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade00af8a0 con 0x7fade015c650 2023-11-28T13:00:46.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.886+0000 7fade84eb700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade00ca400 con 0x7fade00afb10 2023-11-28T13:00:46.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.886+0000 7fade84eb700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade00b00d0 con 0x7fade006a980 2023-11-28T13:00:46.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade6287700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fade015c650 0x7fade006a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36642/0 (socket says 172.21.15.111:36642) 2023-11-28T13:00:46.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade6287700 1 -- 172.21.15.111:0/65432377 learned_addr learned my addr 172.21.15.111:0/65432377 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:46.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5a86700 1 --2- 172.21.15.111:0/65432377 >> v2:172.21.15.149:3300/0 conn(0x7fade006a980 0x7fade00b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:46.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539004040 0 0) 0x7fade00af8a0 con 0x7fade015c650 2023-11-28T13:00:46.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadcc0035b0 con 0x7fade015c650 2023-11-28T13:00:46.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677601304 0 0) 0x7fade00ca400 con 0x7fade00afb10 2023-11-28T13:00:46.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade00af8a0 con 0x7fade00afb10 2023-11-28T13:00:46.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3105551289 0 0) 0x7fadcc0035b0 con 0x7fade015c650 2023-11-28T13:00:46.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 >> v2:172.21.15.149:3300/0 conn(0x7fade006a980 msgr2=0x7fade00b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:46.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 --2- 172.21.15.111:0/65432377 >> v2:172.21.15.149:3300/0 conn(0x7fade006a980 0x7fade00b0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:46.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 >> v1:172.21.15.149:6789/0 conn(0x7fade00afb10 legacy=0x7fade00cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:46.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade00ca6e0 con 0x7fade015c650 2023-11-28T13:00:46.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fadd0001670 con 0x7fade015c650 2023-11-28T13:00:46.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 >> v1:172.21.15.111:6789/0 conn(0x7fade015c650 legacy=0x7fade006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:46.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 --2- 172.21.15.111:0/65432377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadcc003b90 0x7fadcc003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:46.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 --2- 172.21.15.111:0/65432377 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fadcc004850 0x7fadcc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:46.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 --2- 172.21.15.111:0/65432377 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fadcc005200 0x7fadcc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:46.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade00b00d0 con 0x7fadcc003b90 2023-11-28T13:00:46.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc00c110 con 0x7fadcc004850 2023-11-28T13:00:46.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc00c2d0 con 0x7fadcc005200 2023-11-28T13:00:46.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade6287700 1 --2- 172.21.15.111:0/65432377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadcc003b90 0x7fadcc003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:46.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.887+0000 7fade5a86700 1 --2- 172.21.15.111:0/65432377 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fadcc004850 0x7fadcc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:46.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade6a88700 1 --2- 172.21.15.111:0/65432377 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fadcc005200 0x7fadcc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:46.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade6287700 1 -- 172.21.15.111:0/65432377 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fadcc005200 msgr2=0x7fadcc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:46.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade6287700 1 --2- 172.21.15.111:0/65432377 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fadcc005200 0x7fadcc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:46.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade6287700 1 -- 172.21.15.111:0/65432377 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fadcc004850 msgr2=0x7fadcc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:46.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade6287700 1 --2- 172.21.15.111:0/65432377 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fadcc004850 0x7fadcc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:46.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade6a88700 1 -- 172.21.15.111:0/65432377 reap_dead start 2023-11-28T13:00:46.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade6287700 1 -- 172.21.15.111:0/65432377 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc0035b0 con 0x7fadcc003b90 2023-11-28T13:00:46.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade6287700 1 --2- 172.21.15.111:0/65432377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadcc003b90 0x7fadcc003f80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fadd0001eb0 tx=0x7fadcc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acee084bcb0fc6e5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:46.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd000cd80 con 0x7fadcc003b90 2023-11-28T13:00:46.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd0001af0 con 0x7fadcc003b90 2023-11-28T13:00:46.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade5285700 1 -- 172.21.15.111:0/65432377 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0005ba0 con 0x7fadcc003b90 2023-11-28T13:00:46.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade84eb700 1 -- 172.21.15.111:0/65432377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadcc003b90 msgr2=0x7fadcc003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:46.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade84eb700 1 --2- 172.21.15.111:0/65432377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadcc003b90 0x7fadcc003f80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fadd0001eb0 tx=0x7fadcc005880 comp rx=0 tx=0).stop 2023-11-28T13:00:46.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade84eb700 1 -- 172.21.15.111:0/65432377 shutdown_connections 2023-11-28T13:00:46.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade84eb700 1 --2- 172.21.15.111:0/65432377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadcc003b90 0x7fadcc003f80 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:46.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.888+0000 7fade84eb700 1 -- 172.21.15.111:0/65432377 >> 172.21.15.111:0/65432377 conn(0x7fade0098ee0 msgr2=0x7fade006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:46.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.889+0000 7fade84eb700 1 -- 172.21.15.111:0/65432377 shutdown_connections 2023-11-28T13:00:46.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.889+0000 7fade84eb700 1 -- 172.21.15.111:0/65432377 wait complete. 2023-11-28T13:00:46.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.890+0000 7fade84eb700 1 Processor -- start 2023-11-28T13:00:46.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.890+0000 7fade84eb700 1 -- start start 2023-11-28T13:00:46.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade84eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadcc003b90 0x7fade00b4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:46.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade84eb700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fade0151a00 0x7fade0151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:46.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade84eb700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fade01570f0 0x7fade00cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:46.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade84eb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade00b4ec0 con 0x7fadcc003b90 2023-11-28T13:00:46.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade84eb700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade01566a0 con 0x7fade01570f0 2023-11-28T13:00:46.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade84eb700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade00b4660 con 0x7fade0151a00 2023-11-28T13:00:46.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade6287700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadcc003b90 0x7fade00b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:46.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade6287700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadcc003b90 0x7fade00b4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35874/0 (socket says 172.21.15.111:35874) 2023-11-28T13:00:46.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade6287700 1 -- 172.21.15.111:0/1704804026 learned_addr learned my addr 172.21.15.111:0/1704804026 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:46.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade5a86700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fade0151a00 0x7fade0151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:46.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade6a88700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fade01570f0 0x7fade00cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:46.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade6287700 1 -- 172.21.15.111:0/1704804026 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fade0151a00 msgr2=0x7fade0151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:46.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade6287700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fade0151a00 0x7fade0151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:46.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade6287700 1 -- 172.21.15.111:0/1704804026 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fade01570f0 msgr2=0x7fade00cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:46.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade6287700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fade01570f0 0x7fade00cb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:46.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade6287700 1 -- 172.21.15.111:0/1704804026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade00caca0 con 0x7fadcc003b90 2023-11-28T13:00:46.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.891+0000 7fade6a88700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fade01570f0 0x7fade00cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:46.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.892+0000 7fade6287700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadcc003b90 0x7fade00b4120 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fadd00018b0 tx=0x7fadd0001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:46.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.892+0000 7fadd6ffd700 1 -- 172.21.15.111:0/1704804026 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd00070d0 con 0x7fadcc003b90 2023-11-28T13:00:46.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.892+0000 7fadd6ffd700 1 -- 172.21.15.111:0/1704804026 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd0007230 con 0x7fadcc003b90 2023-11-28T13:00:46.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.892+0000 7fade84eb700 1 -- 172.21.15.111:0/1704804026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade00ca9c0 con 0x7fadcc003b90 2023-11-28T13:00:46.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.892+0000 7fadd6ffd700 1 -- 172.21.15.111:0/1704804026 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd00075e0 con 0x7fadcc003b90 2023-11-28T13:00:46.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.892+0000 7fade84eb700 1 -- 172.21.15.111:0/1704804026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade015ca20 con 0x7fadcc003b90 2023-11-28T13:00:46.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.893+0000 7fade84eb700 1 -- 172.21.15.111:0/1704804026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade0067170 con 0x7fadcc003b90 2023-11-28T13:00:46.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.893+0000 7fadd6ffd700 1 -- 172.21.15.111:0/1704804026 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fadd0023030 con 0x7fadcc003b90 2023-11-28T13:00:46.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.894+0000 7fadd6ffd700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fadc40682f0 0x7fadc406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:46.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.894+0000 7fadd6ffd700 1 -- 172.21.15.111:0/1704804026 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6088+0+0 (secure 0 0 0) 0x7fadd0086740 con 0x7fadcc003b90 2023-11-28T13:00:46.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.894+0000 7fade5a86700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fadc40682f0 0x7fadc406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:46.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.894+0000 7fade5a86700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fadc40682f0 0x7fadc406a7a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fade00b5c50 tx=0x7faddc009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:46.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:46.898+0000 7fadd6ffd700 1 -- 172.21.15.111:0/1704804026 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fadd0052f80 con 0x7fadcc003b90 2023-11-28T13:00:47.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.079+0000 7fade84eb700 1 -- 172.21.15.111:0/1704804026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fade0159ec0 con 0x7fadcc003b90 2023-11-28T13:00:47.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.746+0000 7fadd6ffd700 1 -- 172.21.15.111:0/1704804026 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v46) v1 ==== 169+0+0 (secure 0 0 0) 0x7fadd00565a0 con 0x7fadcc003b90 2023-11-28T13:00:47.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.748+0000 7fade84eb700 1 -- 172.21.15.111:0/1704804026 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fadc40682f0 msgr2=0x7fadc406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:47.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.748+0000 7fade84eb700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fadc40682f0 0x7fadc406a7a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fade00b5c50 tx=0x7faddc009210 comp rx=0 tx=0).stop 2023-11-28T13:00:47.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.748+0000 7fade84eb700 1 -- 172.21.15.111:0/1704804026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadcc003b90 msgr2=0x7fade00b4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:47.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.748+0000 7fade84eb700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadcc003b90 0x7fade00b4120 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fadd00018b0 tx=0x7fadd0001980 comp rx=0 tx=0).stop 2023-11-28T13:00:47.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.749+0000 7fade84eb700 1 -- 172.21.15.111:0/1704804026 shutdown_connections 2023-11-28T13:00:47.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.749+0000 7fade84eb700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fadc40682f0 0x7fadc406a7a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:47.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.749+0000 7fade84eb700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fade01570f0 0x7fade00cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:47.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.749+0000 7fade84eb700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadcc003b90 0x7fade00b4120 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:47.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.749+0000 7fade84eb700 1 --2- 172.21.15.111:0/1704804026 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fade0151a00 0x7fade0151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:47.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.749+0000 7fade84eb700 1 -- 172.21.15.111:0/1704804026 >> 172.21.15.111:0/1704804026 conn(0x7fade0098ee0 msgr2=0x7fade00ce690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:47.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.749+0000 7fade84eb700 1 -- 172.21.15.111:0/1704804026 shutdown_connections 2023-11-28T13:00:47.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.749+0000 7fade84eb700 1 -- 172.21.15.111:0/1704804026 wait complete. 2023-11-28T13:00:47.752 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-11-28T13:00:47.764 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T13:00:47.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.889+0000 7f72a4f36700 1 Processor -- start 2023-11-28T13:00:47.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.889+0000 7f72a4f36700 1 -- start start 2023-11-28T13:00:47.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.890+0000 7f72a4f36700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72a00b0d70 0x7f72a01604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:47.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.890+0000 7f72a4f36700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72a00b1140 0x7f72a00ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:47.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.890+0000 7f72a4f36700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a015cd90 con 0x7f72a00b15b0 2023-11-28T13:00:47.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.890+0000 7f72a4f36700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a01613e0 con 0x7f72a00b0d70 2023-11-28T13:00:47.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.890+0000 7f72a4f36700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a0160b40 con 0x7f72a00b1140 2023-11-28T13:00:47.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.890+0000 7f729e59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72a00b0d70 0x7f72a01604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:47.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.890+0000 7f729ed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f72a00b15b0 0x7f72a00cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36664/0 (socket says 172.21.15.111:36664) 2023-11-28T13:00:47.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.890+0000 7f729ed9d700 1 -- 172.21.15.111:0/2236529311 learned_addr learned my addr 172.21.15.111:0/2236529311 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:47.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f729dd9b700 1 --2- 172.21.15.111:0/2236529311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72a00b1140 0x7f72a00ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:47.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f729e59c700 1 -- 172.21.15.111:0/2236529311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72a00b1140 msgr2=0x7f72a00ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:47.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f729e59c700 1 --2- 172.21.15.111:0/2236529311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72a00b1140 0x7f72a00ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:47.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f729e59c700 1 -- 172.21.15.111:0/2236529311 >> v1:172.21.15.111:6789/0 conn(0x7f72a00b15b0 legacy=0x7f72a00cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:47.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f729d59a700 1 -- 172.21.15.111:0/2236529311 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201344837 0 0) 0x7f7298001490 con 0x7f72a00b15b0 2023-11-28T13:00:47.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f729e59c700 1 -- 172.21.15.111:0/2236529311 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a0160e20 con 0x7f72a00b0d70 2023-11-28T13:00:47.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f729dd9b700 1 --2- 172.21.15.111:0/2236529311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72a00b1140 0x7f72a00ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:47.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f729e59c700 1 --2- 172.21.15.111:0/2236529311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a00b0d70 0x7f72a01604b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7294009820 tx=0x7f7294009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c3dc65ba695238 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:47.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f729d59a700 1 -- 172.21.15.111:0/2236529311 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7294005e30 con 0x7f72a00b0d70 2023-11-28T13:00:47.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f729d59a700 1 -- 172.21.15.111:0/2236529311 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7294006030 con 0x7f72a00b0d70 2023-11-28T13:00:47.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f729d59a700 1 -- 172.21.15.111:0/2236529311 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72940063e0 con 0x7f72a00b0d70 2023-11-28T13:00:47.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f72a4f36700 1 -- 172.21.15.111:0/2236529311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a00b0d70 msgr2=0x7f72a01604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:47.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f72a4f36700 1 --2- 172.21.15.111:0/2236529311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a00b0d70 0x7f72a01604b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f7294009820 tx=0x7f7294009b30 comp rx=0 tx=0).stop 2023-11-28T13:00:47.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f72a4f36700 1 -- 172.21.15.111:0/2236529311 shutdown_connections 2023-11-28T13:00:47.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f72a4f36700 1 --2- 172.21.15.111:0/2236529311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a00b0d70 0x7f72a01604b0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:47.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f72a4f36700 1 --2- 172.21.15.111:0/2236529311 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72a00b1140 0x7f72a00ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:47.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.891+0000 7f72a4f36700 1 -- 172.21.15.111:0/2236529311 >> 172.21.15.111:0/2236529311 conn(0x7f72a0098ee0 msgr2=0x7f72a006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:47.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.892+0000 7f72a4f36700 1 -- 172.21.15.111:0/2236529311 shutdown_connections 2023-11-28T13:00:47.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.892+0000 7f72a4f36700 1 -- 172.21.15.111:0/2236529311 wait complete. 2023-11-28T13:00:47.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.893+0000 7f72a4f36700 1 Processor -- start 2023-11-28T13:00:47.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.893+0000 7f72a4f36700 1 -- start start 2023-11-28T13:00:47.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f72a4f36700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a00b0d70 0x7f72a015bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:47.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f72a4f36700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72a00b1140 0x7f72a015df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:47.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f72a4f36700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72a00b15b0 0x7f72a00b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:47.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f72a4f36700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a00b4420 con 0x7f72a00b0d70 2023-11-28T13:00:47.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f72a4f36700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a015e7d0 con 0x7f72a00b1140 2023-11-28T13:00:47.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f72a4f36700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a015eab0 con 0x7f72a00b15b0 2023-11-28T13:00:47.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f729e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a00b0d70 0x7f72a015bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:47.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f729e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a00b0d70 0x7f72a015bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35888/0 (socket says 172.21.15.111:35888) 2023-11-28T13:00:47.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f729e59c700 1 -- 172.21.15.111:0/746365283 learned_addr learned my addr 172.21.15.111:0/746365283 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:47.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f729e59c700 1 -- 172.21.15.111:0/746365283 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72a00b15b0 msgr2=0x7f72a00b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:47.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f729ed9d700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72a00b15b0 0x7f72a00b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:47.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f729dd9b700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72a00b1140 0x7f72a015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:47.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f729e59c700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72a00b15b0 0x7f72a00b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:47.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f729e59c700 1 -- 172.21.15.111:0/746365283 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72a00b1140 msgr2=0x7f72a015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:47.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f729e59c700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72a00b1140 0x7f72a015df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:47.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f729e59c700 1 -- 172.21.15.111:0/746365283 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7294009430 con 0x7f72a00b0d70 2023-11-28T13:00:47.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f729e59c700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a00b0d70 0x7f72a015bff0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7294000c00 tx=0x7f7294004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:47.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f728effd700 1 -- 172.21.15.111:0/746365283 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7294004240 con 0x7f72a00b0d70 2023-11-28T13:00:47.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f728effd700 1 -- 172.21.15.111:0/746365283 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7294020070 con 0x7f72a00b0d70 2023-11-28T13:00:47.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f72a4f36700 1 -- 172.21.15.111:0/746365283 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a00af340 con 0x7f72a00b0d70 2023-11-28T13:00:47.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f728effd700 1 -- 172.21.15.111:0/746365283 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72940045e0 con 0x7f72a00b0d70 2023-11-28T13:00:47.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.894+0000 7f72a4f36700 1 -- 172.21.15.111:0/746365283 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a0161100 con 0x7f72a00b0d70 2023-11-28T13:00:47.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.895+0000 7f72a4f36700 1 -- 172.21.15.111:0/746365283 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a00cec00 con 0x7f72a00b0d70 2023-11-28T13:00:47.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.896+0000 7f728effd700 1 -- 172.21.15.111:0/746365283 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f729402b020 con 0x7f72a00b0d70 2023-11-28T13:00:47.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.896+0000 7f728effd700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f72800682f0 0x7f728006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:47.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.896+0000 7f728effd700 1 -- 172.21.15.111:0/746365283 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f7294087450 con 0x7f72a00b0d70 2023-11-28T13:00:47.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.896+0000 7f729dd9b700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f72800682f0 0x7f728006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:47.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.897+0000 7f729dd9b700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f72800682f0 0x7f728006a7a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7290006fd0 tx=0x7f7290008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:47.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:47.900+0000 7f728effd700 1 -- 172.21.15.111:0/746365283 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7294053df0 con 0x7f72a00b0d70 2023-11-28T13:00:48.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.083+0000 7f72a4f36700 1 -- 172.21.15.111:0/746365283 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f72a0067170 con 0x7f72a00b0d70 2023-11-28T13:00:48.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.089+0000 7f728effd700 1 -- 172.21.15.111:0/746365283 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7294057410 con 0x7f72a00b0d70 2023-11-28T13:00:48.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.091+0000 7f72a4f36700 1 -- 172.21.15.111:0/746365283 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f72800682f0 msgr2=0x7f728006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:48.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.091+0000 7f72a4f36700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f72800682f0 0x7f728006a7a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7290006fd0 tx=0x7f7290008040 comp rx=0 tx=0).stop 2023-11-28T13:00:48.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.091+0000 7f72a4f36700 1 -- 172.21.15.111:0/746365283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a00b0d70 msgr2=0x7f72a015bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:48.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.091+0000 7f72a4f36700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a00b0d70 0x7f72a015bff0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7294000c00 tx=0x7f7294004750 comp rx=0 tx=0).stop 2023-11-28T13:00:48.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.091+0000 7f72a4f36700 1 -- 172.21.15.111:0/746365283 shutdown_connections 2023-11-28T13:00:48.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.091+0000 7f72a4f36700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72a00b15b0 0x7f72a00b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.091+0000 7f72a4f36700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a00b0d70 0x7f72a015bff0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.091+0000 7f72a4f36700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f72800682f0 0x7f728006a7a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.091+0000 7f72a4f36700 1 --2- 172.21.15.111:0/746365283 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72a00b1140 0x7f72a015df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.091+0000 7f72a4f36700 1 -- 172.21.15.111:0/746365283 >> 172.21.15.111:0/746365283 conn(0x7f72a0098ee0 msgr2=0x7f72a00ac670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:48.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.091+0000 7f72a4f36700 1 -- 172.21.15.111:0/746365283 shutdown_connections 2023-11-28T13:00:48.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.091+0000 7f72a4f36700 1 -- 172.21.15.111:0/746365283 wait complete. 2023-11-28T13:00:48.105 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2023-11-28T13:00:48.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.233+0000 7fa5a2505700 1 Processor -- start 2023-11-28T13:00:48.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.233+0000 7fa5a2505700 1 -- start start 2023-11-28T13:00:48.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.233+0000 7fa5a2505700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa59c15a9e0 0x7fa59c0ac670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.233+0000 7fa5a2505700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fa59c0b0250 0x7fa59c0b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.233+0000 7fa5a2505700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa59c0af650 con 0x7fa59c0b15c0 2023-11-28T13:00:48.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.233+0000 7fa5a2505700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa59c0b0c30 con 0x7fa59c15a9e0 2023-11-28T13:00:48.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.233+0000 7fa5a2505700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa59c0cb6d0 con 0x7fa59c0b0250 2023-11-28T13:00:48.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.234+0000 7fa59bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa59c15a9e0 0x7fa59c0ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.234+0000 7fa59bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa59c15a9e0 0x7fa59c0ac670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35896/0 (socket says 172.21.15.111:35896) 2023-11-28T13:00:48.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.234+0000 7fa59bfff700 1 -- 172.21.15.111:0/1322914036 learned_addr learned my addr 172.21.15.111:0/1322914036 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:48.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.234+0000 7fa59bfff700 1 -- 172.21.15.111:0/1322914036 >> v2:172.21.15.149:3300/0 conn(0x7fa59c0b0250 msgr2=0x7fa59c0b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:48.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.234+0000 7fa5a0aa2700 1 --2- 172.21.15.111:0/1322914036 >> v2:172.21.15.149:3300/0 conn(0x7fa59c0b0250 0x7fa59c0b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.234+0000 7fa59bfff700 1 --2- 172.21.15.111:0/1322914036 >> v2:172.21.15.149:3300/0 conn(0x7fa59c0b0250 0x7fa59c0b06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.234+0000 7fa59bfff700 1 -- 172.21.15.111:0/1322914036 >> v1:172.21.15.149:6789/0 conn(0x7fa59c0b15c0 legacy=0x7fa59c0caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:48.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.234+0000 7fa59bfff700 1 -- 172.21.15.111:0/1322914036 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59c0cb9b0 con 0x7fa59c15a9e0 2023-11-28T13:00:48.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.234+0000 7fa59bfff700 1 --2- 172.21.15.111:0/1322914036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa59c15a9e0 0x7fa59c0ac670 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa59c06c450 tx=0x7fa58c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f658938a891e545c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:48.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.234+0000 7fa59affd700 1 -- 172.21.15.111:0/1322914036 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c005e30 con 0x7fa59c15a9e0 2023-11-28T13:00:48.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.234+0000 7fa59affd700 1 -- 172.21.15.111:0/1322914036 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58c006030 con 0x7fa59c15a9e0 2023-11-28T13:00:48.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.234+0000 7fa59affd700 1 -- 172.21.15.111:0/1322914036 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c0063e0 con 0x7fa59c15a9e0 2023-11-28T13:00:48.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.235+0000 7fa5a2505700 1 -- 172.21.15.111:0/1322914036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa59c15a9e0 msgr2=0x7fa59c0ac670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:48.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.235+0000 7fa5a2505700 1 --2- 172.21.15.111:0/1322914036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa59c15a9e0 0x7fa59c0ac670 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa59c06c450 tx=0x7fa58c009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.235+0000 7fa5a2505700 1 -- 172.21.15.111:0/1322914036 shutdown_connections 2023-11-28T13:00:48.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.235+0000 7fa5a2505700 1 --2- 172.21.15.111:0/1322914036 >> v2:172.21.15.149:3300/0 conn(0x7fa59c0b0250 0x7fa59c0b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.235+0000 7fa5a2505700 1 --2- 172.21.15.111:0/1322914036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa59c15a9e0 0x7fa59c0ac670 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.235+0000 7fa5a2505700 1 -- 172.21.15.111:0/1322914036 >> 172.21.15.111:0/1322914036 conn(0x7fa59c098ee0 msgr2=0x7fa59c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:48.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.235+0000 7fa5a2505700 1 -- 172.21.15.111:0/1322914036 shutdown_connections 2023-11-28T13:00:48.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.235+0000 7fa5a2505700 1 -- 172.21.15.111:0/1322914036 wait complete. 2023-11-28T13:00:48.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.236+0000 7fa5a2505700 1 Processor -- start 2023-11-28T13:00:48.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.236+0000 7fa5a2505700 1 -- start start 2023-11-28T13:00:48.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa5a2505700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa59c0b0250 0x7fa59c156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa5a2505700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa59c0b15c0 0x7fa59c0c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa5a2505700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa59c15a9e0 0x7fa59c06a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa5a2505700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c157120 con 0x7fa59c0b15c0 2023-11-28T13:00:48.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa5a2505700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c1f25e0 con 0x7fa59c0b0250 2023-11-28T13:00:48.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa5a2505700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c1f28c0 con 0x7fa59c15a9e0 2023-11-28T13:00:48.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa59b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa59c0b15c0 0x7fa59c0c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa59b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa59c0b15c0 0x7fa59c0c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35898/0 (socket says 172.21.15.111:35898) 2023-11-28T13:00:48.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa59b7fe700 1 -- 172.21.15.111:0/3548730617 learned_addr learned my addr 172.21.15.111:0/3548730617 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:48.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa59bfff700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa59c0b0250 0x7fa59c156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa5a0aa2700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa59c15a9e0 0x7fa59c06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa59b7fe700 1 -- 172.21.15.111:0/3548730617 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa59c15a9e0 msgr2=0x7fa59c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:48.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa59b7fe700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa59c15a9e0 0x7fa59c06a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa59b7fe700 1 -- 172.21.15.111:0/3548730617 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa59c0b0250 msgr2=0x7fa59c156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:48.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa59b7fe700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa59c0b0250 0x7fa59c156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.237+0000 7fa59b7fe700 1 -- 172.21.15.111:0/3548730617 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c009500 con 0x7fa59c0b15c0 2023-11-28T13:00:48.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.238+0000 7fa59b7fe700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa59c0b15c0 0x7fa59c0c5770 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa59000c4a0 tx=0x7fa59000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:48.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.238+0000 7fa598ff9700 1 -- 172.21.15.111:0/3548730617 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5900054e0 con 0x7fa59c0b15c0 2023-11-28T13:00:48.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.238+0000 7fa598ff9700 1 -- 172.21.15.111:0/3548730617 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa590003680 con 0x7fa59c0b15c0 2023-11-28T13:00:48.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.238+0000 7fa598ff9700 1 -- 172.21.15.111:0/3548730617 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa590003a10 con 0x7fa59c0b15c0 2023-11-28T13:00:48.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.238+0000 7fa5a2505700 1 -- 172.21.15.111:0/3548730617 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa59c0cbf70 con 0x7fa59c0b15c0 2023-11-28T13:00:48.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.238+0000 7fa5a2505700 1 -- 172.21.15.111:0/3548730617 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59c0cbc90 con 0x7fa59c0b15c0 2023-11-28T13:00:48.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.239+0000 7fa5a2505700 1 -- 172.21.15.111:0/3548730617 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59c0cc250 con 0x7fa59c0b15c0 2023-11-28T13:00:48.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.239+0000 7fa598ff9700 1 -- 172.21.15.111:0/3548730617 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa590010070 con 0x7fa59c0b15c0 2023-11-28T13:00:48.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.239+0000 7fa598ff9700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5880682f0 0x7fa58806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.240+0000 7fa598ff9700 1 -- 172.21.15.111:0/3548730617 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fa590084a30 con 0x7fa59c0b15c0 2023-11-28T13:00:48.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.240+0000 7fa59bfff700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5880682f0 0x7fa58806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.240+0000 7fa59bfff700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5880682f0 0x7fa58806a7a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa59c06c450 tx=0x7fa58c008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:48.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.244+0000 7fa598ff9700 1 -- 172.21.15.111:0/3548730617 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa590086e60 con 0x7fa59c0b15c0 2023-11-28T13:00:48.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.498+0000 7fa5a2505700 1 -- 172.21.15.111:0/3548730617 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) v1 -- 0x7fa59c0cd7a0 con 0x7fa59c0b15c0 2023-11-28T13:00:48.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.748+0000 7fa598ff9700 1 -- 172.21.15.111:0/3548730617 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v117) v1 ==== 142+0+0 (secure 0 0 0) 0x7fa59c0cd7a0 con 0x7fa59c0b15c0 2023-11-28T13:00:48.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.750+0000 7fa5a2505700 1 -- 172.21.15.111:0/3548730617 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5880682f0 msgr2=0x7fa58806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:48.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.750+0000 7fa5a2505700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5880682f0 0x7fa58806a7a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa59c06c450 tx=0x7fa58c008070 comp rx=0 tx=0).stop 2023-11-28T13:00:48.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.750+0000 7fa5a2505700 1 -- 172.21.15.111:0/3548730617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa59c0b15c0 msgr2=0x7fa59c0c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:48.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.750+0000 7fa5a2505700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa59c0b15c0 0x7fa59c0c5770 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa59000c4a0 tx=0x7fa59000c7b0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.750+0000 7fa5a2505700 1 -- 172.21.15.111:0/3548730617 shutdown_connections 2023-11-28T13:00:48.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.750+0000 7fa5a2505700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa59c15a9e0 0x7fa59c06a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.750+0000 7fa5a2505700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa5880682f0 0x7fa58806a7a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.750+0000 7fa5a2505700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa59c0b0250 0x7fa59c156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.750+0000 7fa5a2505700 1 --2- 172.21.15.111:0/3548730617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa59c0b15c0 0x7fa59c0c5770 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.750+0000 7fa5a2505700 1 -- 172.21.15.111:0/3548730617 >> 172.21.15.111:0/3548730617 conn(0x7fa59c098ee0 msgr2=0x7fa59c158060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:48.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.750+0000 7fa5a2505700 1 -- 172.21.15.111:0/3548730617 shutdown_connections 2023-11-28T13:00:48.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.750+0000 7fa5a2505700 1 -- 172.21.15.111:0/3548730617 wait complete. 2023-11-28T13:00:48.764 INFO:tasks.cephfs_test_runner:test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2023-11-28T13:00:48.764 INFO:tasks.cephfs_test_runner:Starting test: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) That the MDS can repair an inodes backtrace in the data pool 2023-11-28T13:00:48.765 INFO:tasks.cephfs_test_runner:test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-28T13:00:48.765 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2023-11-28T13:00:48.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.884+0000 7fbc850e0700 1 Processor -- start 2023-11-28T13:00:48.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.884+0000 7fbc850e0700 1 -- start start 2023-11-28T13:00:48.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc850e0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc800b0160 0x7fbc800adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc850e0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc800cc390 con 0x7fbc8015a5d0 2023-11-28T13:00:48.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc850e0700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc800cc670 con 0x7fbc800cef60 2023-11-28T13:00:48.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc850e0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800cc1b0 con 0x7fbc800b0160 2023-11-28T13:00:48.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc7ed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbc8015a5d0 0x7fbc800ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36684/0 (socket says 172.21.15.111:36684) 2023-11-28T13:00:48.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc7ed9d700 1 -- 172.21.15.111:0/280298719 learned_addr learned my addr 172.21.15.111:0/280298719 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:48.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733591841 0 0) 0x7fbc800cc390 con 0x7fbc8015a5d0 2023-11-28T13:00:48.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc7f59e700 1 --2- 172.21.15.111:0/280298719 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc800b0160 0x7fbc800adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc680035b0 con 0x7fbc8015a5d0 2023-11-28T13:00:48.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3462265238 0 0) 0x7fbc680035b0 con 0x7fbc8015a5d0 2023-11-28T13:00:48.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc800b0160 msgr2=0x7fbc800adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:48.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc7dd9b700 1 --2- 172.21.15.111:0/280298719 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc800b0160 0x7fbc800adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 >> v1:172.21.15.149:6789/0 conn(0x7fbc800cef60 legacy=0x7fbc800ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:48.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc800cc950 con 0x7fbc8015a5d0 2023-11-28T13:00:48.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fbc740017f0 con 0x7fbc8015a5d0 2023-11-28T13:00:48.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 >> v1:172.21.15.111:6789/0 conn(0x7fbc8015a5d0 legacy=0x7fbc800ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:48.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.885+0000 7fbc7dd9b700 1 --2- 172.21.15.111:0/280298719 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc68003a30 0x7fbc68003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7dd9b700 1 --2- 172.21.15.111:0/280298719 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc68004850 0x7fbc68004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7dd9b700 1 --2- 172.21.15.111:0/280298719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc68005200 0x7fbc6800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800cc1b0 con 0x7fbc68005200 2023-11-28T13:00:48.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6800c0e0 con 0x7fbc68004850 2023-11-28T13:00:48.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6800c250 con 0x7fbc68003a30 2023-11-28T13:00:48.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7f59e700 1 --2- 172.21.15.111:0/280298719 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc800b0160 0x7fbc800adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:48.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7ed9d700 1 --2- 172.21.15.111:0/280298719 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc68003a30 0x7fbc68003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7f59e700 1 --2- 172.21.15.111:0/280298719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc68005200 0x7fbc6800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7e59c700 1 --2- 172.21.15.111:0/280298719 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc68004850 0x7fbc68004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7f59e700 1 -- 172.21.15.111:0/280298719 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc68003a30 msgr2=0x7fbc68003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:48.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7f59e700 1 --2- 172.21.15.111:0/280298719 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc68003a30 0x7fbc68003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7f59e700 1 -- 172.21.15.111:0/280298719 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc68004850 msgr2=0x7fbc68004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:48.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7f59e700 1 --2- 172.21.15.111:0/280298719 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc68004850 0x7fbc68004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7f59e700 1 -- 172.21.15.111:0/280298719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc680035b0 con 0x7fbc68005200 2023-11-28T13:00:48.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7f59e700 1 --2- 172.21.15.111:0/280298719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc68005200 0x7fbc6800ba10 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fbc800b0530 tx=0x7fbc68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f1eb1ced413a5a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:48.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78007dc0 con 0x7fbc68005200 2023-11-28T13:00:48.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7800f040 con 0x7fbc68005200 2023-11-28T13:00:48.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.886+0000 7fbc7dd9b700 1 -- 172.21.15.111:0/280298719 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78003c60 con 0x7fbc68005200 2023-11-28T13:00:48.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.887+0000 7fbc850e0700 1 -- 172.21.15.111:0/280298719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc68005200 msgr2=0x7fbc6800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:48.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.887+0000 7fbc850e0700 1 --2- 172.21.15.111:0/280298719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc68005200 0x7fbc6800ba10 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fbc800b0530 tx=0x7fbc68005880 comp rx=0 tx=0).stop 2023-11-28T13:00:48.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.887+0000 7fbc7f59e700 1 -- 172.21.15.111:0/280298719 reap_dead start 2023-11-28T13:00:48.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.887+0000 7fbc850e0700 1 -- 172.21.15.111:0/280298719 shutdown_connections 2023-11-28T13:00:48.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.887+0000 7fbc850e0700 1 -- 172.21.15.111:0/280298719 >> 172.21.15.111:0/280298719 conn(0x7fbc8009a040 msgr2=0x7fbc800d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:48.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.887+0000 7fbc850e0700 1 -- 172.21.15.111:0/280298719 shutdown_connections 2023-11-28T13:00:48.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.887+0000 7fbc850e0700 1 -- 172.21.15.111:0/280298719 wait complete. 2023-11-28T13:00:48.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.889+0000 7fbc850e0700 1 Processor -- start 2023-11-28T13:00:48.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.889+0000 7fbc850e0700 1 -- start start 2023-11-28T13:00:48.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc850e0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc800b1e10 0x7fbc800b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc850e0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc800b4ff0 0x7fbc80159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc850e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc800b5460 0x7fbc80159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc850e0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800b6190 con 0x7fbc800b5460 2023-11-28T13:00:48.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc850e0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800b58f0 con 0x7fbc800b1e10 2023-11-28T13:00:48.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc850e0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800b5bd0 con 0x7fbc800b4ff0 2023-11-28T13:00:48.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc7f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc800b5460 0x7fbc80159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc7f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc800b5460 0x7fbc80159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35922/0 (socket says 172.21.15.111:35922) 2023-11-28T13:00:48.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc7f59e700 1 -- 172.21.15.111:0/809375363 learned_addr learned my addr 172.21.15.111:0/809375363 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:48.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc7e59c700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc800b4ff0 0x7fbc80159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc7ed9d700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc800b1e10 0x7fbc800b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc7f59e700 1 -- 172.21.15.111:0/809375363 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc800b4ff0 msgr2=0x7fbc80159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:48.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc7f59e700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc800b4ff0 0x7fbc80159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc7f59e700 1 -- 172.21.15.111:0/809375363 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc800b1e10 msgr2=0x7fbc800b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:48.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc7f59e700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc800b1e10 0x7fbc800b21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:48.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc7f59e700 1 -- 172.21.15.111:0/809375363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc800ccf10 con 0x7fbc800b5460 2023-11-28T13:00:48.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc7f59e700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc800b5460 0x7fbc80159740 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbc78013000 tx=0x7fbc78004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:48.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc677fe700 1 -- 172.21.15.111:0/809375363 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc780043b0 con 0x7fbc800b5460 2023-11-28T13:00:48.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc677fe700 1 -- 172.21.15.111:0/809375363 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc78004530 con 0x7fbc800b5460 2023-11-28T13:00:48.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc850e0700 1 -- 172.21.15.111:0/809375363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc800ccc30 con 0x7fbc800b5460 2023-11-28T13:00:48.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc677fe700 1 -- 172.21.15.111:0/809375363 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc780048e0 con 0x7fbc800b5460 2023-11-28T13:00:48.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.890+0000 7fbc850e0700 1 -- 172.21.15.111:0/809375363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8015a9a0 con 0x7fbc800b5460 2023-11-28T13:00:48.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.891+0000 7fbc850e0700 1 -- 172.21.15.111:0/809375363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8006f690 con 0x7fbc800b5460 2023-11-28T13:00:48.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.892+0000 7fbc677fe700 1 -- 172.21.15.111:0/809375363 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbc78028460 con 0x7fbc800b5460 2023-11-28T13:00:48.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.892+0000 7fbc677fe700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbc6c0682f0 0x7fbc6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:48.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.892+0000 7fbc7ed9d700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbc6c0682f0 0x7fbc6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:48.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.892+0000 7fbc677fe700 1 -- 172.21.15.111:0/809375363 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fbc78086c50 con 0x7fbc800b5460 2023-11-28T13:00:48.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.893+0000 7fbc7ed9d700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbc6c0682f0 0x7fbc6c06a7a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbc74000fd0 tx=0x7fbc74009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:48.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:48.897+0000 7fbc677fe700 1 -- 172.21.15.111:0/809375363 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbc78053780 con 0x7fbc800b5460 2023-11-28T13:00:49.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.153+0000 7fbc850e0700 1 -- 172.21.15.111:0/809375363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) v1 -- 0x7fbc80067170 con 0x7fbc800b5460 2023-11-28T13:00:49.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.750+0000 7fbc677fe700 1 -- 172.21.15.111:0/809375363 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v118) v1 ==== 152+0+0 (secure 0 0 0) 0x7fbc78056da0 con 0x7fbc800b5460 2023-11-28T13:00:49.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.752+0000 7fbc850e0700 1 -- 172.21.15.111:0/809375363 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbc6c0682f0 msgr2=0x7fbc6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:49.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.752+0000 7fbc850e0700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbc6c0682f0 0x7fbc6c06a7a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbc74000fd0 tx=0x7fbc74009040 comp rx=0 tx=0).stop 2023-11-28T13:00:49.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.752+0000 7fbc850e0700 1 -- 172.21.15.111:0/809375363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc800b5460 msgr2=0x7fbc80159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:49.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.752+0000 7fbc850e0700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc800b5460 0x7fbc80159740 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbc78013000 tx=0x7fbc78004b40 comp rx=0 tx=0).stop 2023-11-28T13:00:49.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.752+0000 7fbc850e0700 1 -- 172.21.15.111:0/809375363 shutdown_connections 2023-11-28T13:00:49.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.752+0000 7fbc850e0700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc800b5460 0x7fbc80159740 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:49.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.753+0000 7fbc850e0700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbc6c0682f0 0x7fbc6c06a7a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:49.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.753+0000 7fbc850e0700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbc800b1e10 0x7fbc800b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:49.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.753+0000 7fbc850e0700 1 --2- 172.21.15.111:0/809375363 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbc800b4ff0 0x7fbc80159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:49.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.753+0000 7fbc850e0700 1 -- 172.21.15.111:0/809375363 >> 172.21.15.111:0/809375363 conn(0x7fbc8009a040 msgr2=0x7fbc800af8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:49.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.753+0000 7fbc850e0700 1 -- 172.21.15.111:0/809375363 shutdown_connections 2023-11-28T13:00:49.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.753+0000 7fbc850e0700 1 -- 172.21.15.111:0/809375363 wait complete. 2023-11-28T13:00:49.767 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T13:00:49.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.890+0000 7f391806c700 1 Processor -- start 2023-11-28T13:00:49.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.890+0000 7f391806c700 1 -- start start 2023-11-28T13:00:49.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f391806c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f391015a5d0 0x7f39100ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:49.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f391806c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39100cb2d0 0x7f39100ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:49.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f391806c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39100ae8b0 con 0x7f39100b0160 2023-11-28T13:00:49.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f391806c700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39100cc1b0 con 0x7f391015a5d0 2023-11-28T13:00:49.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f391806c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39100cc390 con 0x7f39100cb2d0 2023-11-28T13:00:49.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f3916609700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39100cb2d0 0x7f39100ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:49.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f3915e08700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f391015a5d0 0x7f39100ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:49.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f3915e08700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f391015a5d0 0x7f39100ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:53826/0 (socket says 172.21.15.111:53826) 2023-11-28T13:00:49.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f3915607700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f39100b0160 0x7f391015e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:45246/0 (socket says 172.21.15.111:45246) 2023-11-28T13:00:49.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f3916609700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39100cb2d0 0x7f39100ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:57678/0 (socket says 172.21.15.111:57678) 2023-11-28T13:00:49.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f3915e08700 1 -- 172.21.15.111:0/661847618 learned_addr learned my addr 172.21.15.111:0/661847618 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:49.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f3915e08700 1 -- 172.21.15.111:0/661847618 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39100cb2d0 msgr2=0x7f39100ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:49.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f3915e08700 1 --2- 172.21.15.111:0/661847618 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39100cb2d0 0x7f39100ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:49.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f3915e08700 1 -- 172.21.15.111:0/661847618 >> v1:172.21.15.149:6789/0 conn(0x7f39100b0160 legacy=0x7f391015e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:49.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f3915e08700 1 -- 172.21.15.111:0/661847618 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39100cc670 con 0x7f391015a5d0 2023-11-28T13:00:49.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f3916609700 1 --2- 172.21.15.111:0/661847618 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39100cb2d0 0x7f39100ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:49.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.891+0000 7f3915e08700 1 --2- 172.21.15.111:0/661847618 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f391015a5d0 0x7f39100ce850 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f390c006fd0 tx=0x7f390c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e40bbccded090f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:49.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.892+0000 7f3914e06700 1 -- 172.21.15.111:0/661847618 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c005e30 con 0x7f391015a5d0 2023-11-28T13:00:49.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.892+0000 7f3914e06700 1 -- 172.21.15.111:0/661847618 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390c004030 con 0x7f391015a5d0 2023-11-28T13:00:49.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.892+0000 7f3914e06700 1 -- 172.21.15.111:0/661847618 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c0043e0 con 0x7f391015a5d0 2023-11-28T13:00:49.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.892+0000 7f391806c700 1 -- 172.21.15.111:0/661847618 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f391015a5d0 msgr2=0x7f39100ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:49.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.892+0000 7f391806c700 1 --2- 172.21.15.111:0/661847618 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f391015a5d0 0x7f39100ce850 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f390c006fd0 tx=0x7f390c009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:49.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.892+0000 7f391806c700 1 -- 172.21.15.111:0/661847618 shutdown_connections 2023-11-28T13:00:49.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.892+0000 7f391806c700 1 --2- 172.21.15.111:0/661847618 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39100cb2d0 0x7f39100ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:49.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.892+0000 7f391806c700 1 --2- 172.21.15.111:0/661847618 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f391015a5d0 0x7f39100ce850 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:49.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.892+0000 7f391806c700 1 -- 172.21.15.111:0/661847618 >> 172.21.15.111:0/661847618 conn(0x7f391009a040 msgr2=0x7f39100d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:49.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.892+0000 7f391806c700 1 -- 172.21.15.111:0/661847618 shutdown_connections 2023-11-28T13:00:49.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.892+0000 7f391806c700 1 -- 172.21.15.111:0/661847618 wait complete. 2023-11-28T13:00:49.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.893+0000 7f391806c700 1 Processor -- start 2023-11-28T13:00:49.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.893+0000 7f391806c700 1 -- start start 2023-11-28T13:00:49.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.894+0000 7f391806c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39100b0160 0x7f391015d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.894+0000 7f391806c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39100cb2d0 0x7f39100b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.894+0000 7f391806c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f391015a5d0 0x7f39100b0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.894+0000 7f391806c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39101540d0 con 0x7f39100b0160 2023-11-28T13:00:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.894+0000 7f391806c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391015df00 con 0x7f391015a5d0 2023-11-28T13:00:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.894+0000 7f391806c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3910153aa0 con 0x7f39100cb2d0 2023-11-28T13:00:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.894+0000 7f3915e08700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39100b0160 0x7f391015d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.894+0000 7f3915e08700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39100b0160 0x7f391015d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35932/0 (socket says 172.21.15.111:35932) 2023-11-28T13:00:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.894+0000 7f3915e08700 1 -- 172.21.15.111:0/2333997021 learned_addr learned my addr 172.21.15.111:0/2333997021 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.894+0000 7f3915607700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39100cb2d0 0x7f39100b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.894+0000 7f3916609700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f391015a5d0 0x7f39100b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.895+0000 7f3915e08700 1 -- 172.21.15.111:0/2333997021 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39100cb2d0 msgr2=0x7f39100b5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:49.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.895+0000 7f3915e08700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39100cb2d0 0x7f39100b5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.895+0000 7f3915e08700 1 -- 172.21.15.111:0/2333997021 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f391015a5d0 msgr2=0x7f39100b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.895+0000 7f3915e08700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f391015a5d0 0x7f39100b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.895+0000 7f3915e08700 1 -- 172.21.15.111:0/2333997021 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3908009a70 con 0x7f39100b0160 2023-11-28T13:00:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.895+0000 7f3916609700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f391015a5d0 0x7f39100b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.895+0000 7f3915e08700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39100b0160 0x7f391015d9c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f390c009c10 tx=0x7f390c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.895+0000 7f39067fc700 1 -- 172.21.15.111:0/2333997021 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c003d90 con 0x7f39100b0160 2023-11-28T13:00:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.895+0000 7f39067fc700 1 -- 172.21.15.111:0/2333997021 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390c01c420 con 0x7f39100b0160 2023-11-28T13:00:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.895+0000 7f391806c700 1 -- 172.21.15.111:0/2333997021 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390c009500 con 0x7f39100b0160 2023-11-28T13:00:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.895+0000 7f39067fc700 1 -- 172.21.15.111:0/2333997021 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c01c7d0 con 0x7f39100b0160 2023-11-28T13:00:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.895+0000 7f391806c700 1 -- 172.21.15.111:0/2333997021 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39100ccc30 con 0x7f39100b0160 2023-11-28T13:00:49.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.896+0000 7f391806c700 1 -- 172.21.15.111:0/2333997021 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39100cc950 con 0x7f39100b0160 2023-11-28T13:00:49.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.896+0000 7f39067fc700 1 -- 172.21.15.111:0/2333997021 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f390c02c070 con 0x7f39100b0160 2023-11-28T13:00:49.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.897+0000 7f39067fc700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38f40682f0 0x7f38f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:49.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.897+0000 7f39067fc700 1 -- 172.21.15.111:0/2333997021 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f390c086c00 con 0x7f39100b0160 2023-11-28T13:00:49.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.897+0000 7f3915607700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38f40682f0 0x7f38f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:49.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.897+0000 7f3915607700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38f40682f0 0x7f38f406a7a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f39100b59a0 tx=0x7f3900007d10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:49.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:49.901+0000 7f39067fc700 1 -- 172.21.15.111:0/2333997021 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f390c053550 con 0x7f39100b0160 2023-11-28T13:00:50.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.086+0000 7f391806c700 1 -- 172.21.15.111:0/2333997021 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f39100ccf10 con 0x7f39100b0160 2023-11-28T13:00:50.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.092+0000 7f39067fc700 1 -- 172.21.15.111:0/2333997021 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f390c05c0c0 con 0x7f39100b0160 2023-11-28T13:00:50.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.094+0000 7f391806c700 1 -- 172.21.15.111:0/2333997021 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38f40682f0 msgr2=0x7f38f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:50.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.094+0000 7f391806c700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38f40682f0 0x7f38f406a7a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f39100b59a0 tx=0x7f3900007d10 comp rx=0 tx=0).stop 2023-11-28T13:00:50.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.094+0000 7f391806c700 1 -- 172.21.15.111:0/2333997021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39100b0160 msgr2=0x7f391015d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:50.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.094+0000 7f391806c700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39100b0160 0x7f391015d9c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f390c009c10 tx=0x7f390c004750 comp rx=0 tx=0).stop 2023-11-28T13:00:50.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.094+0000 7f391806c700 1 -- 172.21.15.111:0/2333997021 shutdown_connections 2023-11-28T13:00:50.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.094+0000 7f391806c700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38f40682f0 0x7f38f406a7a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.094+0000 7f391806c700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f391015a5d0 0x7f39100b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.094+0000 7f391806c700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39100b0160 0x7f391015d9c0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.094+0000 7f391806c700 1 --2- 172.21.15.111:0/2333997021 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39100cb2d0 0x7f39100b5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.094+0000 7f391806c700 1 -- 172.21.15.111:0/2333997021 >> 172.21.15.111:0/2333997021 conn(0x7f391009a040 msgr2=0x7f391006ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:50.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.094+0000 7f391806c700 1 -- 172.21.15.111:0/2333997021 shutdown_connections 2023-11-28T13:00:50.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.094+0000 7f391806c700 1 -- 172.21.15.111:0/2333997021 wait complete. 2023-11-28T13:00:50.107 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T13:00:50.107 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T13:00:50.146 DEBUG:teuthology.orchestra.run.smithi149:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T13:00:50.194 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T13:00:50.195 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:00:50.195 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:00:50.212 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284268k,nr_inodes=4071067,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18839 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-28T13:00:50.212 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T13:00:50.212 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:00:50.231 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8026740k,nr_inodes=2006685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16040 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=1609048k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-28T13:00:50.231 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:00:50.231 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:00:50.271 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284268k,nr_inodes=4071067,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18839 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-28T13:00:50.271 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T13:00:50.271 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:00:50.290 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8026740k,nr_inodes=2006685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16040 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=1609048k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-28T13:00:50.290 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T13:00:50.291 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:00:50.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.429+0000 7fd931784700 1 Processor -- start 2023-11-28T13:00:50.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.429+0000 7fd931784700 1 -- start start 2023-11-28T13:00:50.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.430+0000 7fd931784700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd92c0b0d70 0x7fd92c1604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.430+0000 7fd931784700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd92c0b1140 0x7fd92c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.430+0000 7fd931784700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92c15cd90 con 0x7fd92c0b15b0 2023-11-28T13:00:50.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.430+0000 7fd931784700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd92c1613e0 con 0x7fd92c0b0d70 2023-11-28T13:00:50.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.430+0000 7fd931784700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92c160b40 con 0x7fd92c0b1140 2023-11-28T13:00:50.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.430+0000 7fd92b7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd92c0b15b0 0x7fd92c0cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36722/0 (socket says 172.21.15.111:36722) 2023-11-28T13:00:50.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.430+0000 7fd92b7fe700 1 -- 172.21.15.111:0/3704918366 learned_addr learned my addr 172.21.15.111:0/3704918366 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:50.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.430+0000 7fd92a7fc700 1 --2- 172.21.15.111:0/3704918366 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd92c0b1140 0x7fd92c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.430+0000 7fd92affd700 1 --2- 172.21.15.111:0/3704918366 >> v2:172.21.15.149:3300/0 conn(0x7fd92c0b0d70 0x7fd92c1604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.431+0000 7fd929ffb700 1 -- 172.21.15.111:0/3704918366 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612780459 0 0) 0x7fd924001490 con 0x7fd92c0b15b0 2023-11-28T13:00:50.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.431+0000 7fd929ffb700 1 -- 172.21.15.111:0/3704918366 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9140035b0 con 0x7fd92c0b15b0 2023-11-28T13:00:50.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.431+0000 7fd92a7fc700 1 -- 172.21.15.111:0/3704918366 >> v2:172.21.15.149:3300/0 conn(0x7fd92c0b0d70 msgr2=0x7fd92c1604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:50.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.431+0000 7fd92a7fc700 1 --2- 172.21.15.111:0/3704918366 >> v2:172.21.15.149:3300/0 conn(0x7fd92c0b0d70 0x7fd92c1604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.431+0000 7fd92a7fc700 1 -- 172.21.15.111:0/3704918366 >> v1:172.21.15.111:6789/0 conn(0x7fd92c0b15b0 legacy=0x7fd92c0cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:50.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.431+0000 7fd92a7fc700 1 -- 172.21.15.111:0/3704918366 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c160e20 con 0x7fd92c0b1140 2023-11-28T13:00:50.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.431+0000 7fd92a7fc700 1 --2- 172.21.15.111:0/3704918366 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd92c0b1140 0x7fd92c0ce6c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd91c009840 tx=0x7fd91c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbe5f3689c0a3758 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.431+0000 7fd929ffb700 1 -- 172.21.15.111:0/3704918366 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91c011830 con 0x7fd92c0b1140 2023-11-28T13:00:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.431+0000 7fd929ffb700 1 -- 172.21.15.111:0/3704918366 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91c011990 con 0x7fd92c0b1140 2023-11-28T13:00:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.431+0000 7fd929ffb700 1 -- 172.21.15.111:0/3704918366 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91c011d00 con 0x7fd92c0b1140 2023-11-28T13:00:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.432+0000 7fd931784700 1 -- 172.21.15.111:0/3704918366 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd92c0b1140 msgr2=0x7fd92c0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.432+0000 7fd931784700 1 --2- 172.21.15.111:0/3704918366 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd92c0b1140 0x7fd92c0ce6c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd91c009840 tx=0x7fd91c009b50 comp rx=0 tx=0).stop 2023-11-28T13:00:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.432+0000 7fd931784700 1 -- 172.21.15.111:0/3704918366 shutdown_connections 2023-11-28T13:00:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.432+0000 7fd931784700 1 --2- 172.21.15.111:0/3704918366 >> v2:172.21.15.149:3300/0 conn(0x7fd92c0b0d70 0x7fd92c1604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.432+0000 7fd931784700 1 --2- 172.21.15.111:0/3704918366 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd92c0b1140 0x7fd92c0ce6c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.432+0000 7fd931784700 1 -- 172.21.15.111:0/3704918366 >> 172.21.15.111:0/3704918366 conn(0x7fd92c098ee0 msgr2=0x7fd92c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.432+0000 7fd931784700 1 -- 172.21.15.111:0/3704918366 shutdown_connections 2023-11-28T13:00:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.432+0000 7fd931784700 1 -- 172.21.15.111:0/3704918366 wait complete. 2023-11-28T13:00:50.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.433+0000 7fd931784700 1 Processor -- start 2023-11-28T13:00:50.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.433+0000 7fd931784700 1 -- start start 2023-11-28T13:00:50.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd931784700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd92c0b0d70 0x7fd92c0b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd931784700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd92c0b1140 0x7fd92c0b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd931784700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd92c0b15b0 0x7fd92c1e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd931784700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92c1e8ae0 con 0x7fd92c0b0d70 2023-11-28T13:00:50.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd931784700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92c1e7ed0 con 0x7fd92c0b1140 2023-11-28T13:00:50.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd931784700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92c1e81b0 con 0x7fd92c0b15b0 2023-11-28T13:00:50.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd92affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd92c0b0d70 0x7fd92c0b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd92affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd92c0b0d70 0x7fd92c0b4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35946/0 (socket says 172.21.15.111:35946) 2023-11-28T13:00:50.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd92affd700 1 -- 172.21.15.111:0/1198383670 learned_addr learned my addr 172.21.15.111:0/1198383670 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:50.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd92b7fe700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd92c0b15b0 0x7fd92c1e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd92a7fc700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd92c0b1140 0x7fd92c0b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd92affd700 1 -- 172.21.15.111:0/1198383670 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd92c0b15b0 msgr2=0x7fd92c1e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:50.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd92affd700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd92c0b15b0 0x7fd92c1e7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd92affd700 1 -- 172.21.15.111:0/1198383670 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd92c0b1140 msgr2=0x7fd92c0b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:50.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd92affd700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd92c0b1140 0x7fd92c0b5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd92affd700 1 -- 172.21.15.111:0/1198383670 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c0094a0 con 0x7fd92c0b0d70 2023-11-28T13:00:50.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.434+0000 7fd92affd700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd92c0b0d70 0x7fd92c0b4f50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd92000e990 tx=0x7fd92000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:50.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.435+0000 7fd9137fe700 1 -- 172.21.15.111:0/1198383670 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9200096f0 con 0x7fd92c0b0d70 2023-11-28T13:00:50.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.435+0000 7fd9137fe700 1 -- 172.21.15.111:0/1198383670 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92000c690 con 0x7fd92c0b0d70 2023-11-28T13:00:50.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.435+0000 7fd931784700 1 -- 172.21.15.111:0/1198383670 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd92c0af340 con 0x7fd92c0b0d70 2023-11-28T13:00:50.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.435+0000 7fd9137fe700 1 -- 172.21.15.111:0/1198383670 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92000ca00 con 0x7fd92c0b0d70 2023-11-28T13:00:50.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.435+0000 7fd931784700 1 -- 172.21.15.111:0/1198383670 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd92c161100 con 0x7fd92c0b0d70 2023-11-28T13:00:50.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.435+0000 7fd931784700 1 -- 172.21.15.111:0/1198383670 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd92c0cec00 con 0x7fd92c0b0d70 2023-11-28T13:00:50.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.436+0000 7fd9137fe700 1 -- 172.21.15.111:0/1198383670 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd920023070 con 0x7fd92c0b0d70 2023-11-28T13:00:50.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.436+0000 7fd9137fe700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9180682f0 0x7fd91806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.436+0000 7fd9137fe700 1 -- 172.21.15.111:0/1198383670 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fd920085d20 con 0x7fd92c0b0d70 2023-11-28T13:00:50.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.436+0000 7fd92a7fc700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9180682f0 0x7fd91806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.437+0000 7fd92a7fc700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9180682f0 0x7fd91806a7a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd91c000c00 tx=0x7fd91c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.441+0000 7fd9137fe700 1 -- 172.21.15.111:0/1198383670 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd9200526c0 con 0x7fd92c0b0d70 2023-11-28T13:00:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.666+0000 7fd931784700 1 -- 172.21.15.111:0/1198383670 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd92c067170 con 0x7fd92c0b0d70 2023-11-28T13:00:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.666+0000 7fd9137fe700 1 -- 172.21.15.111:0/1198383670 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3541 (secure 0 0 0) 0x7fd920017df0 con 0x7fd92c0b0d70 2023-11-28T13:00:50.669 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:50.669 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3254653481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3254653481},{"type":"v1","addr":"172.21.15.149:6835","nonce":3254653481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5057,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/1792916529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1792916529},{"type":"v1","addr":"172.21.15.149:6834","nonce":1792916529}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5337,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1890904311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1890904311},{"type":"v1","addr":"172.21.15.111:6836","nonce":1890904311}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1592804351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1592804351},{"type":"v1","addr":"172.21.15.111:6837","nonce":1592804351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[]} 2023-11-28T13:00:50.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.668+0000 7fd931784700 1 -- 172.21.15.111:0/1198383670 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9180682f0 msgr2=0x7fd91806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:50.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.668+0000 7fd931784700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9180682f0 0x7fd91806a7a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd91c000c00 tx=0x7fd91c008030 comp rx=0 tx=0).stop 2023-11-28T13:00:50.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.668+0000 7fd931784700 1 -- 172.21.15.111:0/1198383670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd92c0b0d70 msgr2=0x7fd92c0b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:50.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.668+0000 7fd931784700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd92c0b0d70 0x7fd92c0b4f50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd92000e990 tx=0x7fd92000eca0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.668+0000 7fd931784700 1 -- 172.21.15.111:0/1198383670 shutdown_connections 2023-11-28T13:00:50.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.669+0000 7fd931784700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd92c0b15b0 0x7fd92c1e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.669+0000 7fd931784700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd92c0b0d70 0x7fd92c0b4f50 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.669+0000 7fd931784700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9180682f0 0x7fd91806a7a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.669+0000 7fd931784700 1 --2- 172.21.15.111:0/1198383670 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd92c0b1140 0x7fd92c0b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.669+0000 7fd931784700 1 -- 172.21.15.111:0/1198383670 >> 172.21.15.111:0/1198383670 conn(0x7fd92c098ee0 msgr2=0x7fd92c06ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:50.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.669+0000 7fd931784700 1 -- 172.21.15.111:0/1198383670 shutdown_connections 2023-11-28T13:00:50.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.669+0000 7fd931784700 1 -- 172.21.15.111:0/1198383670 wait complete. 2023-11-28T13:00:50.673 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 19 2023-11-28T13:00:50.682 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T13:00:50.683 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T13:00:50.683 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T13:00:50.683 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:00:50.683 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T13:00:50.684 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T13:00:50.684 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T13:00:50.684 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:00:50.685 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T13:00:50.685 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T13:00:50.685 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T13:00:50.685 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:00:50.686 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T13:00:50.686 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T13:00:50.686 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T13:00:50.686 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:00:50.693 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T13:00:50.693 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T13:00:50.694 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T13:00:50.694 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T13:00:50.696 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T13:00:50.696 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T13:00:50.697 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T13:00:50.697 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T13:00:50.736 INFO:tasks.ceph.mds.a:Started 2023-11-28T13:00:50.736 DEBUG:teuthology.parallel:result is None 2023-11-28T13:00:50.737 INFO:tasks.ceph.mds.c:Started 2023-11-28T13:00:50.737 DEBUG:teuthology.parallel:result is None 2023-11-28T13:00:50.739 INFO:tasks.ceph.mds.b:Started 2023-11-28T13:00:50.740 DEBUG:teuthology.parallel:result is None 2023-11-28T13:00:50.741 INFO:tasks.ceph.mds.d:Started 2023-11-28T13:00:50.741 DEBUG:teuthology.parallel:result is None 2023-11-28T13:00:50.741 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T13:00:50.815 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-11-28T13:00:50.815 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-11-28T13:00:50.815 INFO:tasks.ceph.mds.c.smithi111.stderr:2023-11-28T13:00:50.814+0000 7fa2fdc87700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-11-28T13:00:50.820 INFO:tasks.ceph.mds.b.smithi149.stdout:starting mds.b at 2023-11-28T13:00:50.820 INFO:tasks.ceph.mds.d.smithi149.stdout:starting mds.d at 2023-11-28T13:00:50.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.871+0000 7f4ee4736700 1 Processor -- start 2023-11-28T13:00:50.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.871+0000 7f4ee4736700 1 -- start start 2023-11-28T13:00:50.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee4736700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4edc0cd460 0x7f4edc06c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee4736700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4edc0b88e0 con 0x7f4edc0b8510 2023-11-28T13:00:50.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee4736700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4edc0cd8f0 con 0x7f4edc0d0d60 2023-11-28T13:00:50.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee4736700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4edc0cdbd0 con 0x7f4edc0cd460 2023-11-28T13:00:50.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee24d2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4edc0b8510 0x7f4edc06b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36756/0 (socket says 172.21.15.111:36756) 2023-11-28T13:00:50.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee24d2700 1 -- 172.21.15.111:0/4253046596 learned_addr learned my addr 172.21.15.111:0/4253046596 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:50.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270922567 0 0) 0x7f4edc0b88e0 con 0x7f4edc0b8510 2023-11-28T13:00:50.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee1cd1700 1 --2- 172.21.15.111:0/4253046596 >> v2:172.21.15.149:3300/0 conn(0x7f4edc0cd460 0x7f4edc06c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec40035b0 con 0x7f4edc0b8510 2023-11-28T13:00:50.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 210715361 0 0) 0x7f4ec40035b0 con 0x7f4edc0b8510 2023-11-28T13:00:50.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 >> v2:172.21.15.149:3300/0 conn(0x7f4edc0cd460 msgr2=0x7f4edc06c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:50.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 --2- 172.21.15.111:0/4253046596 >> v2:172.21.15.149:3300/0 conn(0x7f4edc0cd460 0x7f4edc06c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 >> v1:172.21.15.149:6789/0 conn(0x7f4edc0d0d60 legacy=0x7f4edc0d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:50.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4edc0cfd10 con 0x7f4edc0b8510 2023-11-28T13:00:50.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f4ed4001670 con 0x7f4edc0b8510 2023-11-28T13:00:50.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 >> v1:172.21.15.111:6789/0 conn(0x7f4edc0b8510 legacy=0x7f4edc06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:50.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 --2- 172.21.15.111:0/4253046596 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ec4003a00 0x7f4ec4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 --2- 172.21.15.111:0/4253046596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec4004850 0x7f4ec4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 --2- 172.21.15.111:0/4253046596 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ec4005200 0x7f4ec400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4edc0cdbd0 con 0x7f4ec4004850 2023-11-28T13:00:50.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec400c080 con 0x7f4ec4003a00 2023-11-28T13:00:50.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.872+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec400c240 con 0x7f4ec4005200 2023-11-28T13:00:50.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.873+0000 7f4ee1cd1700 1 --2- 172.21.15.111:0/4253046596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec4004850 0x7f4ec4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.873+0000 7f4ee24d2700 1 --2- 172.21.15.111:0/4253046596 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ec4003a00 0x7f4ec4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.873+0000 7f4ee2cd3700 1 --2- 172.21.15.111:0/4253046596 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ec4005200 0x7f4ec400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.873+0000 7f4ee1cd1700 1 -- 172.21.15.111:0/4253046596 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ec4005200 msgr2=0x7f4ec400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:50.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.873+0000 7f4ee1cd1700 1 --2- 172.21.15.111:0/4253046596 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ec4005200 0x7f4ec400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.873+0000 7f4ee1cd1700 1 -- 172.21.15.111:0/4253046596 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ec4003a00 msgr2=0x7f4ec4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:50.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.873+0000 7f4ee1cd1700 1 --2- 172.21.15.111:0/4253046596 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ec4003a00 0x7f4ec4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.873+0000 7f4ee2cd3700 1 -- 172.21.15.111:0/4253046596 reap_dead start 2023-11-28T13:00:50.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.873+0000 7f4ee1cd1700 1 -- 172.21.15.111:0/4253046596 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec40035b0 con 0x7f4ec4004850 2023-11-28T13:00:50.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.873+0000 7f4ee1cd1700 1 --2- 172.21.15.111:0/4253046596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec4004850 0x7f4ec4004cc0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0b1de0 tx=0x7f4ec4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6443881b8a2d198a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:50.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.873+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed000e6c0 con 0x7f4ec4004850 2023-11-28T13:00:50.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.873+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed000e820 con 0x7f4ec4004850 2023-11-28T13:00:50.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.873+0000 7f4ee14d0700 1 -- 172.21.15.111:0/4253046596 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed000ebb0 con 0x7f4ec4004850 2023-11-28T13:00:50.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.874+0000 7f4ee4736700 1 -- 172.21.15.111:0/4253046596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec4004850 msgr2=0x7f4ec4004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:50.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.874+0000 7f4ee4736700 1 --2- 172.21.15.111:0/4253046596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec4004850 0x7f4ec4004cc0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0b1de0 tx=0x7f4ec4005880 comp rx=0 tx=0).stop 2023-11-28T13:00:50.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.874+0000 7f4ee4736700 1 -- 172.21.15.111:0/4253046596 shutdown_connections 2023-11-28T13:00:50.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.874+0000 7f4ee4736700 1 --2- 172.21.15.111:0/4253046596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec4004850 0x7f4ec4004cc0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.874+0000 7f4ee4736700 1 -- 172.21.15.111:0/4253046596 >> 172.21.15.111:0/4253046596 conn(0x7f4edc0719a0 msgr2=0x7f4edc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:50.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.874+0000 7f4ee4736700 1 -- 172.21.15.111:0/4253046596 shutdown_connections 2023-11-28T13:00:50.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.874+0000 7f4ee4736700 1 -- 172.21.15.111:0/4253046596 wait complete. 2023-11-28T13:00:50.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.875+0000 7f4ee4736700 1 Processor -- start 2023-11-28T13:00:50.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.875+0000 7f4ee4736700 1 -- start start 2023-11-28T13:00:50.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.876+0000 7f4ee4736700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec4004850 0x7f4edc1ec020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.876+0000 7f4ee4736700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4edc0cbef0 0x7f4edc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.876+0000 7f4ee4736700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4edc0b66f0 0x7f4edc159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.876+0000 7f4ee4736700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4edc0b6b90 con 0x7f4ec4004850 2023-11-28T13:00:50.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.876+0000 7f4ee4736700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4edc154c90 con 0x7f4edc0cbef0 2023-11-28T13:00:50.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.876+0000 7f4ee4736700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4edc1ec560 con 0x7f4edc0b66f0 2023-11-28T13:00:50.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.876+0000 7f4ee24d2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec4004850 0x7f4edc1ec020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.876+0000 7f4ee24d2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec4004850 0x7f4edc1ec020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35990/0 (socket says 172.21.15.111:35990) 2023-11-28T13:00:50.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.876+0000 7f4ee24d2700 1 -- 172.21.15.111:0/2511290523 learned_addr learned my addr 172.21.15.111:0/2511290523 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:50.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.877+0000 7f4ee1cd1700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4edc0cbef0 0x7f4edc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.877+0000 7f4ee2cd3700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4edc0b66f0 0x7f4edc159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.877+0000 7f4ee24d2700 1 -- 172.21.15.111:0/2511290523 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4edc0b66f0 msgr2=0x7f4edc159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:50.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.877+0000 7f4ee24d2700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4edc0b66f0 0x7f4edc159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.877+0000 7f4ee24d2700 1 -- 172.21.15.111:0/2511290523 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4edc0cbef0 msgr2=0x7f4edc0b61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:50.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.877+0000 7f4ee24d2700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4edc0cbef0 0x7f4edc0b61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:50.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.877+0000 7f4ee24d2700 1 -- 172.21.15.111:0/2511290523 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4edc0d29c0 con 0x7f4ec4004850 2023-11-28T13:00:50.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.877+0000 7f4ee24d2700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec4004850 0x7f4edc1ec020 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40072f0 tx=0x7f4ed4007600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:50.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.877+0000 7f4edaffd700 1 -- 172.21.15.111:0/2511290523 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed4011580 con 0x7f4ec4004850 2023-11-28T13:00:50.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.877+0000 7f4edaffd700 1 -- 172.21.15.111:0/2511290523 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed40116e0 con 0x7f4ec4004850 2023-11-28T13:00:50.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.877+0000 7f4ee4736700 1 -- 172.21.15.111:0/2511290523 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4edc0cfff0 con 0x7f4ec4004850 2023-11-28T13:00:50.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.877+0000 7f4edaffd700 1 -- 172.21.15.111:0/2511290523 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed4011a50 con 0x7f4ec4004850 2023-11-28T13:00:50.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.877+0000 7f4ee4736700 1 -- 172.21.15.111:0/2511290523 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4edc0d2b50 con 0x7f4ec4004850 2023-11-28T13:00:50.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.878+0000 7f4ee4736700 1 -- 172.21.15.111:0/2511290523 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4edc067880 con 0x7f4ec4004850 2023-11-28T13:00:50.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.878+0000 7f4edaffd700 1 -- 172.21.15.111:0/2511290523 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ed401b030 con 0x7f4ec4004850 2023-11-28T13:00:50.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.879+0000 7f4edaffd700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4ec80682f0 0x7f4ec806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:50.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.879+0000 7f4edaffd700 1 -- 172.21.15.111:0/2511290523 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f4ed40852f0 con 0x7f4ec4004850 2023-11-28T13:00:50.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.879+0000 7f4ee1cd1700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4ec80682f0 0x7f4ec806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:50.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.879+0000 7f4ee1cd1700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4ec80682f0 0x7f4ec806a7a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00090b0 tx=0x7f4ed001e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:50.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:50.883+0000 7f4edaffd700 1 -- 172.21.15.111:0/2511290523 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ed404d5e0 con 0x7f4ec4004850 2023-11-28T13:00:51.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.081+0000 7f4ee4736700 1 -- 172.21.15.111:0/2511290523 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4edc0b52c0 con 0x7f4ec4004850 2023-11-28T13:00:51.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.863+0000 7f4edaffd700 1 -- 172.21.15.111:0/2511290523 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v51) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4ed4019080 con 0x7f4ec4004850 2023-11-28T13:00:51.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.865+0000 7f4ee4736700 1 -- 172.21.15.111:0/2511290523 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4ec80682f0 msgr2=0x7f4ec806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:51.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.865+0000 7f4ee4736700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4ec80682f0 0x7f4ec806a7a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00090b0 tx=0x7f4ed001e040 comp rx=0 tx=0).stop 2023-11-28T13:00:51.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.865+0000 7f4ee4736700 1 -- 172.21.15.111:0/2511290523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec4004850 msgr2=0x7f4edc1ec020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:51.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.865+0000 7f4ee4736700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec4004850 0x7f4edc1ec020 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40072f0 tx=0x7f4ed4007600 comp rx=0 tx=0).stop 2023-11-28T13:00:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.865+0000 7f4ee4736700 1 -- 172.21.15.111:0/2511290523 shutdown_connections 2023-11-28T13:00:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.865+0000 7f4ee4736700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4edc0b66f0 0x7f4edc159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.865+0000 7f4ee4736700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec4004850 0x7f4edc1ec020 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.865+0000 7f4ee4736700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4ec80682f0 0x7f4ec806a7a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.865+0000 7f4ee4736700 1 --2- 172.21.15.111:0/2511290523 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4edc0cbef0 0x7f4edc0b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.865+0000 7f4ee4736700 1 -- 172.21.15.111:0/2511290523 >> 172.21.15.111:0/2511290523 conn(0x7f4edc0719a0 msgr2=0x7f4edc0b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:51.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.865+0000 7f4ee4736700 1 -- 172.21.15.111:0/2511290523 shutdown_connections 2023-11-28T13:00:51.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:51.865+0000 7f4ee4736700 1 -- 172.21.15.111:0/2511290523 wait complete. 2023-11-28T13:00:51.869 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-11-28T13:00:51.881 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T13:00:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.004+0000 7f6cc7f3f700 1 Processor -- start 2023-11-28T13:00:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.004+0000 7f6cc7f3f700 1 -- start start 2023-11-28T13:00:52.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.005+0000 7f6cc7f3f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6cc00b0d70 0x7f6cc01604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.005+0000 7f6cc7f3f700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f6cc00b1140 0x7f6cc00ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.005+0000 7f6cc7f3f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc015cd90 con 0x7f6cc00cc550 2023-11-28T13:00:52.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.005+0000 7f6cc7f3f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc01613e0 con 0x7f6cc00b0d70 2023-11-28T13:00:52.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.005+0000 7f6cc7f3f700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc0160b40 con 0x7f6cc00b1140 2023-11-28T13:00:52.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.005+0000 7f6cc5cdb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6cc00b0d70 0x7f6cc01604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.005+0000 7f6cc64dc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6cc00cc550 0x7f6cc00af860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36784/0 (socket says 172.21.15.111:36784) 2023-11-28T13:00:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.005+0000 7f6cc5cdb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6cc00b0d70 0x7f6cc01604b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36002/0 (socket says 172.21.15.111:36002) 2023-11-28T13:00:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.005+0000 7f6cc64dc700 1 -- 172.21.15.111:0/820136157 learned_addr learned my addr 172.21.15.111:0/820136157 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.005+0000 7f6cc5cdb700 1 -- 172.21.15.111:0/820136157 >> v2:172.21.15.149:3300/0 conn(0x7f6cc00b1140 msgr2=0x7f6cc00ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.005+0000 7f6cc54da700 1 --2- 172.21.15.111:0/820136157 >> v2:172.21.15.149:3300/0 conn(0x7f6cc00b1140 0x7f6cc00ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc5cdb700 1 --2- 172.21.15.111:0/820136157 >> v2:172.21.15.149:3300/0 conn(0x7f6cc00b1140 0x7f6cc00ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc5cdb700 1 -- 172.21.15.111:0/820136157 >> v1:172.21.15.111:6789/0 conn(0x7f6cc00cc550 legacy=0x7f6cc00af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc5cdb700 1 -- 172.21.15.111:0/820136157 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc0160e20 con 0x7f6cc00b0d70 2023-11-28T13:00:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc4cd9700 1 -- 172.21.15.111:0/820136157 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672458842 0 0) 0x7f6cb8001490 con 0x7f6cc00cc550 2023-11-28T13:00:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc5cdb700 1 --2- 172.21.15.111:0/820136157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc00b0d70 0x7f6cc01604b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6cc006b310 tx=0x7f6cb0009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8218eec05d63947d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc4cd9700 1 -- 172.21.15.111:0/820136157 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb0005e30 con 0x7f6cc00b0d70 2023-11-28T13:00:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc4cd9700 1 -- 172.21.15.111:0/820136157 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb0006030 con 0x7f6cc00b0d70 2023-11-28T13:00:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc4cd9700 1 -- 172.21.15.111:0/820136157 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb00063c0 con 0x7f6cc00b0d70 2023-11-28T13:00:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/820136157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc00b0d70 msgr2=0x7f6cc01604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc7f3f700 1 --2- 172.21.15.111:0/820136157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc00b0d70 0x7f6cc01604b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6cc006b310 tx=0x7f6cb0009b00 comp rx=0 tx=0).stop 2023-11-28T13:00:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/820136157 shutdown_connections 2023-11-28T13:00:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc7f3f700 1 --2- 172.21.15.111:0/820136157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc00b0d70 0x7f6cc01604b0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc7f3f700 1 --2- 172.21.15.111:0/820136157 >> v2:172.21.15.149:3300/0 conn(0x7f6cc00b1140 0x7f6cc00ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/820136157 >> 172.21.15.111:0/820136157 conn(0x7f6cc0098ee0 msgr2=0x7f6cc006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/820136157 shutdown_connections 2023-11-28T13:00:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.006+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/820136157 wait complete. 2023-11-28T13:00:52.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.008+0000 7f6cc7f3f700 1 Processor -- start 2023-11-28T13:00:52.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.008+0000 7f6cc7f3f700 1 -- start start 2023-11-28T13:00:52.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.008+0000 7f6cc7f3f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6cc00b0d70 0x7f6cc015c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.008+0000 7f6cc7f3f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6cc00b1140 0x7f6cc00c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.008+0000 7f6cc7f3f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc00cc550 0x7f6cc00c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.008+0000 7f6cc7f3f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc00b50f0 con 0x7f6cc00cc550 2023-11-28T13:00:52.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.008+0000 7f6cc7f3f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc015c760 con 0x7f6cc00b1140 2023-11-28T13:00:52.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.008+0000 7f6cc7f3f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc00b4470 con 0x7f6cc00b0d70 2023-11-28T13:00:52.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.008+0000 7f6cc64dc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc00cc550 0x7f6cc00c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cc64dc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc00cc550 0x7f6cc00c5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36012/0 (socket says 172.21.15.111:36012) 2023-11-28T13:00:52.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cc64dc700 1 -- 172.21.15.111:0/2594121096 learned_addr learned my addr 172.21.15.111:0/2594121096 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:52.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cc54da700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6cc00b1140 0x7f6cc00c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cc5cdb700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6cc00b0d70 0x7f6cc015c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cc64dc700 1 -- 172.21.15.111:0/2594121096 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6cc00b0d70 msgr2=0x7f6cc015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cc64dc700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6cc00b0d70 0x7f6cc015c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cc64dc700 1 -- 172.21.15.111:0/2594121096 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6cc00b1140 msgr2=0x7f6cc00c35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cc64dc700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6cc00b1140 0x7f6cc00c35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cc64dc700 1 -- 172.21.15.111:0/2594121096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb0009430 con 0x7f6cc00cc550 2023-11-28T13:00:52.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cc5cdb700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6cc00b0d70 0x7f6cc015c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:52.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cc64dc700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc00cc550 0x7f6cc00c5a70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800cba0 tx=0x7f6cb800ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:52.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cb67fc700 1 -- 172.21.15.111:0/2594121096 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb800aad0 con 0x7f6cc00cc550 2023-11-28T13:00:52.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cb67fc700 1 -- 172.21.15.111:0/2594121096 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb800ac50 con 0x7f6cc00cc550 2023-11-28T13:00:52.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/2594121096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc00af340 con 0x7f6cc00cc550 2023-11-28T13:00:52.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cb67fc700 1 -- 172.21.15.111:0/2594121096 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb80070d0 con 0x7f6cc00cc550 2023-11-28T13:00:52.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.009+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/2594121096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc0161100 con 0x7f6cc00cc550 2023-11-28T13:00:52.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.010+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/2594121096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc00cec00 con 0x7f6cc00cc550 2023-11-28T13:00:52.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.010+0000 7f6cb67fc700 1 -- 172.21.15.111:0/2594121096 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6cb801a030 con 0x7f6cc00cc550 2023-11-28T13:00:52.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.011+0000 7f6cb67fc700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ca40682f0 0x7f6ca406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.011+0000 7f6cb67fc700 1 -- 172.21.15.111:0/2594121096 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f6cb8085570 con 0x7f6cc00cc550 2023-11-28T13:00:52.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.011+0000 7f6cc5cdb700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ca40682f0 0x7f6ca406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.011+0000 7f6cc5cdb700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ca40682f0 0x7f6ca406a7a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0000c00 tx=0x7f6cb0008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:52.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.015+0000 7f6cb67fc700 1 -- 172.21.15.111:0/2594121096 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6cb8051f80 con 0x7f6cc00cc550 2023-11-28T13:00:52.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.273+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/2594121096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6cc00b5230 con 0x7f6cc00cc550 2023-11-28T13:00:52.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.274+0000 7f6cb67fc700 1 -- 172.21.15.111:0/2594121096 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5298 (secure 0 0 0) 0x7f6cc00b5230 con 0x7f6cc00cc550 2023-11-28T13:00:52.275 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:52.276 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:00:52.276 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-11-28T13:00:52.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.276 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-11-28T13:00:52.276 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlH823BRAArK8kE5qUgPoqFZPxIudN+w==", 2023-11-28T13:00:52.276 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.276 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:52.276 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:00:52.276 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:00:52.276 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.276 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.277 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-11-28T13:00:52.277 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlIo9ZEhAA5NmyiPE68OmMcsG1hTiTgw==", 2023-11-28T13:00:52.277 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.277 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:52.277 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:00:52.277 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:00:52.277 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.277 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.278 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-11-28T13:00:52.278 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlQqHxDBAA8RDJ2PETpTHNjnn1i1T3Ow==", 2023-11-28T13:00:52.278 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.278 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:52.278 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:00:52.278 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:00:52.278 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.278 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.278 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-11-28T13:00:52.279 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42Vl8RDmGRAAkiGKEQMr9rHahKJ/3lw9BA==", 2023-11-28T13:00:52.279 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.279 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:52.279 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:00:52.279 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:00:52.279 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.279 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.279 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-11-28T13:00:52.279 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDc42VlfB0eNRAAJEa0AL3iDLArHc4n6CWK9g==", 2023-11-28T13:00:52.280 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.280 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:52.280 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:52.280 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.280 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.280 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-11-28T13:00:52.280 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDf42VlnrbLNBAAU+sVJ/6EsadNo3BXpyrbSw==", 2023-11-28T13:00:52.280 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.281 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:52.281 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:52.281 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.281 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.281 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.281 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-11-28T13:00:52.281 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDi42VlwCUsNRAAwZKJ/o1qTBtWwR35B1r2zQ==", 2023-11-28T13:00:52.281 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.281 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:52.282 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:52.282 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.282 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.282 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-11-28T13:00:52.282 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDl42Vl3/t2NBAAMPuhlpUi37Fpd/6NclsErw==", 2023-11-28T13:00:52.282 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.282 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:52.282 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:52.282 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.283 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.283 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-11-28T13:00:52.283 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDr42VlkrWgNxAAuu2dXZZ3U/XRrq3cWCEa8w==", 2023-11-28T13:00:52.283 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.283 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:52.283 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:52.283 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.283 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.284 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-11-28T13:00:52.284 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDu42VldDFENxAAYKD2Dqgka5G+XxGjOR3M6w==", 2023-11-28T13:00:52.284 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.284 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:52.284 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:52.284 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.284 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.285 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-11-28T13:00:52.285 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDx42VlvjotNhAAWD2E1sgAZLdxb5CG1Rg26g==", 2023-11-28T13:00:52.285 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.285 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:52.285 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:52.285 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.285 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.285 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-11-28T13:00:52.285 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQD042VlZheVNBAAlMIKaDmgUMmt/hA+vx2/PA==", 2023-11-28T13:00:52.286 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.286 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:00:52.286 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:00:52.286 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.286 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.286 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-11-28T13:00:52.287 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42Vl3MswHhAA/oDIgj9lzhcnU7f0ksZ9cw==", 2023-11-28T13:00:52.287 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.287 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:52.287 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow r", 2023-11-28T13:00:52.287 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rw pool=cephfs_data" 2023-11-28T13:00:52.287 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.287 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-11-28T13:00:52.287 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlV3dQJxAAdzQk1q+DoxFDEr/TYJi/6A==", 2023-11-28T13:00:52.287 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.288 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:52.288 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-11-28T13:00:52.288 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-11-28T13:00:52.288 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-11-28T13:00:52.288 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.288 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-11-28T13:00:52.288 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlJiXlIxAAYuFMFLUqIDN5f0E008GKZw==", 2023-11-28T13:00:52.288 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.288 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:52.289 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-11-28T13:00:52.289 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-11-28T13:00:52.289 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-11-28T13:00:52.289 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.289 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-11-28T13:00:52.289 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlagSZKhAAkLS8RwTZZTDyO2slb0MzIQ==", 2023-11-28T13:00:52.289 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.289 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:00:52.289 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-11-28T13:00:52.290 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-11-28T13:00:52.290 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-11-28T13:00:52.290 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.290 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-11-28T13:00:52.290 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDY42Vl74R5IhAADPhHAlk7DX23esg8UFYW/Q==", 2023-11-28T13:00:52.290 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.290 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-11-28T13:00:52.291 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:00:52.291 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:00:52.291 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.291 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.291 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-11-28T13:00:52.291 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42Vl+AxXAhAADTJ8Jt/WNo8qr5zAyQJ4rw==", 2023-11-28T13:00:52.291 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.292 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-11-28T13:00:52.292 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-11-28T13:00:52.292 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.292 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:52.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:52.292 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-11-28T13:00:52.292 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDY42Vl6WCiOBAALRDd8ZjOOlxxLKQsgXP9Jg==", 2023-11-28T13:00:52.292 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:00:52.292 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-11-28T13:00:52.293 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-11-28T13:00:52.293 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:00:52.293 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.293 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:52.293 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-11-28T13:00:52.293 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:00:52.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.276+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/2594121096 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ca40682f0 msgr2=0x7f6ca406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.276+0000 7f6cc7f3f700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ca40682f0 0x7f6ca406a7a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0000c00 tx=0x7f6cb0008070 comp rx=0 tx=0).stop 2023-11-28T13:00:52.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.276+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/2594121096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc00cc550 msgr2=0x7f6cc00c5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.276+0000 7f6cc7f3f700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc00cc550 0x7f6cc00c5a70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800cba0 tx=0x7f6cb800ceb0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.276+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/2594121096 shutdown_connections 2023-11-28T13:00:52.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.276+0000 7f6cc7f3f700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc00cc550 0x7f6cc00c5a70 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.276+0000 7f6cc7f3f700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6cc00b0d70 0x7f6cc015c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.276+0000 7f6cc7f3f700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ca40682f0 0x7f6ca406a7a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.276+0000 7f6cc7f3f700 1 --2- 172.21.15.111:0/2594121096 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6cc00b1140 0x7f6cc00c35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.276+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/2594121096 >> 172.21.15.111:0/2594121096 conn(0x7f6cc0098ee0 msgr2=0x7f6cc00af6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:52.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.276+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/2594121096 shutdown_connections 2023-11-28T13:00:52.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.276+0000 7f6cc7f3f700 1 -- 172.21.15.111:0/2594121096 wait complete. 2023-11-28T13:00:52.298 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:00:52.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.416+0000 7eff03749700 1 Processor -- start 2023-11-28T13:00:52.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.416+0000 7eff03749700 1 -- start start 2023-11-28T13:00:52.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.417+0000 7eff03749700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc15c6c0 0x7efefc0afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.417+0000 7eff03749700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc1610c0 con 0x7efefc0cb300 2023-11-28T13:00:52.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.417+0000 7eff03749700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc0cc750 con 0x7efefc0b0440 2023-11-28T13:00:52.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.417+0000 7eff03749700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0cca30 con 0x7efefc15c6c0 2023-11-28T13:00:52.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.417+0000 7eff01ce6700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efefc0cb300 0x7efefc1609b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36798/0 (socket says 172.21.15.111:36798) 2023-11-28T13:00:52.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.417+0000 7eff01ce6700 1 -- 172.21.15.111:0/4221367061 learned_addr learned my addr 172.21.15.111:0/4221367061 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:52.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.417+0000 7eff014e5700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc15c6c0 0x7efefc0afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594138596 0 0) 0x7efefc1610c0 con 0x7efefc0cb300 2023-11-28T13:00:52.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee80035b0 con 0x7efefc0cb300 2023-11-28T13:00:52.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776825291 0 0) 0x7efefc0cc750 con 0x7efefc0b0440 2023-11-28T13:00:52.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efefc1610c0 con 0x7efefc0b0440 2023-11-28T13:00:52.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 628187099 0 0) 0x7efee80035b0 con 0x7efefc0cb300 2023-11-28T13:00:52.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc15c6c0 msgr2=0x7efefc0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:52.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc15c6c0 0x7efefc0afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 >> v1:172.21.15.149:6789/0 conn(0x7efefc0b0440 legacy=0x7efefc1602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc0ccd10 con 0x7efefc0cb300 2023-11-28T13:00:52.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7efef40019f0 con 0x7efefc0cb300 2023-11-28T13:00:52.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 >> v1:172.21.15.111:6789/0 conn(0x7efefc0cb300 legacy=0x7efefc1609b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc15c6c0 0x7efee8003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efee8004120 0x7efee8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efee8004d90 0x7efee8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0cca30 con 0x7efee8004d90 2023-11-28T13:00:52.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee800bac0 con 0x7efee8004120 2023-11-28T13:00:52.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee800bc80 con 0x7efefc15c6c0 2023-11-28T13:00:52.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7eff01ce6700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efee8004d90 0x7efee8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7eff014e5700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc15c6c0 0x7efee8003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7eff00ce4700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efee8004120 0x7efee8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7eff01ce6700 1 -- 172.21.15.111:0/4221367061 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc15c6c0 msgr2=0x7efee8003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:52.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7eff01ce6700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc15c6c0 0x7efee8003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7eff01ce6700 1 -- 172.21.15.111:0/4221367061 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efee8004120 msgr2=0x7efee8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:52.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7eff01ce6700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efee8004120 0x7efee8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7eff01ce6700 1 -- 172.21.15.111:0/4221367061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee80035b0 con 0x7efee8004d90 2023-11-28T13:00:52.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7eff014e5700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc15c6c0 0x7efee8003bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:52.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7eff00ce4700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efee8004120 0x7efee8004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:52.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.418+0000 7eff01ce6700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efee8004d90 0x7efee8005200 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7efef4002230 tx=0x7efee8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219565e2a85300b8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:52.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.419+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4012600 con 0x7efee8004d90 2023-11-28T13:00:52.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.419+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef4012760 con 0x7efee8004d90 2023-11-28T13:00:52.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.419+0000 7efef3fff700 1 -- 172.21.15.111:0/4221367061 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4012b10 con 0x7efee8004d90 2023-11-28T13:00:52.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.419+0000 7eff03749700 1 -- 172.21.15.111:0/4221367061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efee8004d90 msgr2=0x7efee8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.419+0000 7eff03749700 1 --2- 172.21.15.111:0/4221367061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efee8004d90 0x7efee8005200 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7efef4002230 tx=0x7efee8005880 comp rx=0 tx=0).stop 2023-11-28T13:00:52.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.419+0000 7eff01ce6700 1 -- 172.21.15.111:0/4221367061 reap_dead start 2023-11-28T13:00:52.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.419+0000 7eff03749700 1 -- 172.21.15.111:0/4221367061 shutdown_connections 2023-11-28T13:00:52.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.419+0000 7eff03749700 1 -- 172.21.15.111:0/4221367061 >> 172.21.15.111:0/4221367061 conn(0x7efefc098ee0 msgr2=0x7efefc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:52.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.419+0000 7eff03749700 1 -- 172.21.15.111:0/4221367061 shutdown_connections 2023-11-28T13:00:52.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.419+0000 7eff03749700 1 -- 172.21.15.111:0/4221367061 wait complete. 2023-11-28T13:00:52.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.421+0000 7eff03749700 1 Processor -- start 2023-11-28T13:00:52.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.421+0000 7eff03749700 1 -- start start 2023-11-28T13:00:52.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff03749700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efefc14d420 0x7efefc0c4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff03749700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc0c4590 0x7efefc0c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff03749700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc0c6290 0x7efefc0b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff03749700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0c7fc0 con 0x7efefc0c6290 2023-11-28T13:00:52.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff03749700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0c82a0 con 0x7efefc14d420 2023-11-28T13:00:52.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff03749700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0c8580 con 0x7efefc0c4590 2023-11-28T13:00:52.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff01ce6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc0c6290 0x7efefc0b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff01ce6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc0c6290 0x7efefc0b3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36032/0 (socket says 172.21.15.111:36032) 2023-11-28T13:00:52.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff01ce6700 1 -- 172.21.15.111:0/4168627028 learned_addr learned my addr 172.21.15.111:0/4168627028 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:52.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff014e5700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efefc14d420 0x7efefc0c4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff00ce4700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc0c4590 0x7efefc0c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff01ce6700 1 -- 172.21.15.111:0/4168627028 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc0c4590 msgr2=0x7efefc0c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff01ce6700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc0c4590 0x7efefc0c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff01ce6700 1 -- 172.21.15.111:0/4168627028 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efefc14d420 msgr2=0x7efefc0c4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff01ce6700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efefc14d420 0x7efefc0c4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff01ce6700 1 -- 172.21.15.111:0/4168627028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc0cd2d0 con 0x7efefc0c6290 2023-11-28T13:00:52.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff014e5700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efefc14d420 0x7efefc0c4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:52.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff01ce6700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc0c6290 0x7efefc0b3f50 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7efef4000f80 tx=0x7efef4001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:52.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7efef1ffb700 1 -- 172.21.15.111:0/4168627028 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4001f30 con 0x7efefc0c6290 2023-11-28T13:00:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7efef1ffb700 1 -- 172.21.15.111:0/4168627028 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef4012ae0 con 0x7efefc0c6290 2023-11-28T13:00:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7efef1ffb700 1 -- 172.21.15.111:0/4168627028 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4012e90 con 0x7efefc0c6290 2023-11-28T13:00:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.422+0000 7eff03749700 1 -- 172.21.15.111:0/4168627028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efefc0ccff0 con 0x7efefc0c6290 2023-11-28T13:00:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.423+0000 7eff03749700 1 -- 172.21.15.111:0/4168627028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efefc15ca90 con 0x7efefc0c6290 2023-11-28T13:00:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.424+0000 7eff03749700 1 -- 172.21.15.111:0/4168627028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efefc067170 con 0x7efefc0c6290 2023-11-28T13:00:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.424+0000 7efef1ffb700 1 -- 172.21.15.111:0/4168627028 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efef402e070 con 0x7efefc0c6290 2023-11-28T13:00:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.424+0000 7efef1ffb700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efee00682f0 0x7efee006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.424+0000 7efef1ffb700 1 -- 172.21.15.111:0/4168627028 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5530+0+0 (secure 0 0 0) 0x7efef4086e10 con 0x7efefc0c6290 2023-11-28T13:00:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.424+0000 7eff014e5700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efee00682f0 0x7efee006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.425+0000 7eff014e5700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efee00682f0 0x7efee006a7a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7efeec009bf0 tx=0x7efeec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:52.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.429+0000 7efef1ffb700 1 -- 172.21.15.111:0/4168627028 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efef4053880 con 0x7efefc0c6290 2023-11-28T13:00:52.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.613+0000 7eff03749700 1 -- 172.21.15.111:0/4168627028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efefc0b1b10 con 0x7efefc0c6290 2023-11-28T13:00:52.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.614+0000 7efef1ffb700 1 -- 172.21.15.111:0/4168627028 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+14520 (secure 0 0 0) 0x7efef4017090 con 0x7efefc0c6290 2023-11-28T13:00:52.615 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:52.615 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":51,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:51.859829+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:00:52.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.616+0000 7eff03749700 1 -- 172.21.15.111:0/4168627028 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efee00682f0 msgr2=0x7efee006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.616+0000 7eff03749700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efee00682f0 0x7efee006a7a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7efeec009bf0 tx=0x7efeec008040 comp rx=0 tx=0).stop 2023-11-28T13:00:52.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.616+0000 7eff03749700 1 -- 172.21.15.111:0/4168627028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc0c6290 msgr2=0x7efefc0b3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.616+0000 7eff03749700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc0c6290 0x7efefc0b3f50 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7efef4000f80 tx=0x7efef4001d00 comp rx=0 tx=0).stop 2023-11-28T13:00:52.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.616+0000 7eff03749700 1 -- 172.21.15.111:0/4168627028 shutdown_connections 2023-11-28T13:00:52.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.616+0000 7eff03749700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efefc0c6290 0x7efefc0b3f50 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.616+0000 7eff03749700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efee00682f0 0x7efee006a7a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.616+0000 7eff03749700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efefc14d420 0x7efefc0c4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.616+0000 7eff03749700 1 --2- 172.21.15.111:0/4168627028 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efefc0c4590 0x7efefc0c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.616+0000 7eff03749700 1 -- 172.21.15.111:0/4168627028 >> 172.21.15.111:0/4168627028 conn(0x7efefc098ee0 msgr2=0x7efefc158040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:52.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.616+0000 7eff03749700 1 -- 172.21.15.111:0/4168627028 shutdown_connections 2023-11-28T13:00:52.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.616+0000 7eff03749700 1 -- 172.21.15.111:0/4168627028 wait complete. 2023-11-28T13:00:52.629 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T13:00:52.630 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T13:00:52.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.751+0000 7f907eba1700 1 Processor -- start 2023-11-28T13:00:52.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.751+0000 7f907eba1700 1 -- start start 2023-11-28T13:00:52.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.752+0000 7f907eba1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90780b0160 0x7f90780adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.752+0000 7f907eba1700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90780cc390 con 0x7f90780cef60 2023-11-28T13:00:52.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.752+0000 7f907eba1700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90780cc670 con 0x7f907815a5d0 2023-11-28T13:00:52.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.752+0000 7f907eba1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90780cc1b0 con 0x7f90780b0160 2023-11-28T13:00:52.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.752+0000 7f9077fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f90780cef60 0x7f90780ad8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36814/0 (socket says 172.21.15.111:36814) 2023-11-28T13:00:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.752+0000 7f9077fff700 1 -- 172.21.15.111:0/1140218059 learned_addr learned my addr 172.21.15.111:0/1140218059 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.752+0000 7f907d13e700 1 --2- 172.21.15.111:0/1140218059 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90780b0160 0x7f90780adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.752+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938918248 0 0) 0x7f90780cc390 con 0x7f90780cef60 2023-11-28T13:00:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.752+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90680035b0 con 0x7f90780cef60 2023-11-28T13:00:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542430637 0 0) 0x7f90780cc670 con 0x7f907815a5d0 2023-11-28T13:00:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90780cc390 con 0x7f907815a5d0 2023-11-28T13:00:52.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1783817023 0 0) 0x7f90680035b0 con 0x7f90780cef60 2023-11-28T13:00:52.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90780b0160 msgr2=0x7f90780adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:52.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 --2- 172.21.15.111:0/1140218059 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90780b0160 0x7f90780adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 >> v1:172.21.15.149:6789/0 conn(0x7f907815a5d0 legacy=0x7f90780ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90780cc950 con 0x7f90780cef60 2023-11-28T13:00:52.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f9064001670 con 0x7f90780cef60 2023-11-28T13:00:52.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 >> v1:172.21.15.111:6789/0 conn(0x7f90780cef60 legacy=0x7f90780ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 --2- 172.21.15.111:0/1140218059 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90780b0160 0x7f9068003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 --2- 172.21.15.111:0/1140218059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9068004120 0x7f9068004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 --2- 172.21.15.111:0/1140218059 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9068004d90 0x7f9068005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90780cc1b0 con 0x7f9068004120 2023-11-28T13:00:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906800bac0 con 0x7f9068004d90 2023-11-28T13:00:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906800bc80 con 0x7f90780b0160 2023-11-28T13:00:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f9077fff700 1 --2- 172.21.15.111:0/1140218059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9068004120 0x7f9068004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f907d13e700 1 --2- 172.21.15.111:0/1140218059 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9068004d90 0x7f9068005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f907c93d700 1 --2- 172.21.15.111:0/1140218059 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90780b0160 0x7f9068003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f9077fff700 1 -- 172.21.15.111:0/1140218059 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90780b0160 msgr2=0x7f9068003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f9077fff700 1 --2- 172.21.15.111:0/1140218059 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90780b0160 0x7f9068003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f9077fff700 1 -- 172.21.15.111:0/1140218059 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9068004d90 msgr2=0x7f9068005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f9077fff700 1 --2- 172.21.15.111:0/1140218059 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9068004d90 0x7f9068005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f9077fff700 1 -- 172.21.15.111:0/1140218059 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90680035b0 con 0x7f9068004120 2023-11-28T13:00:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f907c93d700 1 --2- 172.21.15.111:0/1140218059 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90780b0160 0x7f9068003bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f9077fff700 1 --2- 172.21.15.111:0/1140218059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9068004120 0x7f9068004850 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9064001eb0 tx=0x7f9068005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647b6b7eb3b07d61 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906400cd50 con 0x7f9068004120 2023-11-28T13:00:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9064001af0 con 0x7f9068004120 2023-11-28T13:00:52.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.753+0000 7f90777fe700 1 -- 172.21.15.111:0/1140218059 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064005ba0 con 0x7f9068004120 2023-11-28T13:00:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.754+0000 7f907eba1700 1 -- 172.21.15.111:0/1140218059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9068004120 msgr2=0x7f9068004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.754+0000 7f907eba1700 1 --2- 172.21.15.111:0/1140218059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9068004120 0x7f9068004850 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9064001eb0 tx=0x7f9068005880 comp rx=0 tx=0).stop 2023-11-28T13:00:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.754+0000 7f907d13e700 1 -- 172.21.15.111:0/1140218059 reap_dead start 2023-11-28T13:00:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.754+0000 7f907eba1700 1 -- 172.21.15.111:0/1140218059 shutdown_connections 2023-11-28T13:00:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.754+0000 7f907eba1700 1 -- 172.21.15.111:0/1140218059 >> 172.21.15.111:0/1140218059 conn(0x7f907809a040 msgr2=0x7f90780d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:52.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.754+0000 7f907eba1700 1 -- 172.21.15.111:0/1140218059 shutdown_connections 2023-11-28T13:00:52.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.754+0000 7f907eba1700 1 -- 172.21.15.111:0/1140218059 wait complete. 2023-11-28T13:00:52.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.755+0000 7f907eba1700 1 Processor -- start 2023-11-28T13:00:52.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.755+0000 7f907eba1700 1 -- start start 2023-11-28T13:00:52.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.756+0000 7f907eba1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9068004120 0x7f90780b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.756+0000 7f907eba1700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9078159660 0x7f90781492c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.756+0000 7f907eba1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90781514c0 0x7f9078151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.756+0000 7f907eba1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9078151e80 con 0x7f9068004120 2023-11-28T13:00:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.756+0000 7f907eba1700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90780b2ab0 con 0x7f9078159660 2023-11-28T13:00:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.756+0000 7f907eba1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9078155400 con 0x7f90781514c0 2023-11-28T13:00:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.756+0000 7f907c93d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9068004120 0x7f90780b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.756+0000 7f907c93d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9068004120 0x7f90780b2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36046/0 (socket says 172.21.15.111:36046) 2023-11-28T13:00:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.756+0000 7f907c93d700 1 -- 172.21.15.111:0/2175880688 learned_addr learned my addr 172.21.15.111:0/2175880688 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.756+0000 7f907d13e700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90781514c0 0x7f9078151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.756+0000 7f9077fff700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9078159660 0x7f90781492c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.757+0000 7f907c93d700 1 -- 172.21.15.111:0/2175880688 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90781514c0 msgr2=0x7f9078151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.757+0000 7f907c93d700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90781514c0 0x7f9078151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.757+0000 7f907c93d700 1 -- 172.21.15.111:0/2175880688 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9078159660 msgr2=0x7f90781492c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.757+0000 7f907c93d700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9078159660 0x7f90781492c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.757+0000 7f907c93d700 1 -- 172.21.15.111:0/2175880688 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90780ccf10 con 0x7f9068004120 2023-11-28T13:00:52.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.757+0000 7f9077fff700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9078159660 0x7f90781492c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:52.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.757+0000 7f907c93d700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9068004120 0x7f90780b2cb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f906c00e7b0 tx=0x7f906c00eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:52.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.759+0000 7f90757fa700 1 -- 172.21.15.111:0/2175880688 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c0052e0 con 0x7f9068004120 2023-11-28T13:00:52.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.759+0000 7f907eba1700 1 -- 172.21.15.111:0/2175880688 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90780ccc30 con 0x7f9068004120 2023-11-28T13:00:52.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.759+0000 7f907eba1700 1 -- 172.21.15.111:0/2175880688 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907815a9a0 con 0x7f9068004120 2023-11-28T13:00:52.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.759+0000 7f90757fa700 1 -- 172.21.15.111:0/2175880688 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906c005440 con 0x7f9068004120 2023-11-28T13:00:52.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.759+0000 7f90757fa700 1 -- 172.21.15.111:0/2175880688 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c0038d0 con 0x7f9068004120 2023-11-28T13:00:52.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.760+0000 7f907eba1700 1 -- 172.21.15.111:0/2175880688 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90580052f0 con 0x7f9068004120 2023-11-28T13:00:52.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.760+0000 7f90757fa700 1 -- 172.21.15.111:0/2175880688 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f906c01b030 con 0x7f9068004120 2023-11-28T13:00:52.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.761+0000 7f90757fa700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f905c0682f0 0x7f905c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:52.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.761+0000 7f90757fa700 1 -- 172.21.15.111:0/2175880688 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f906c085200 con 0x7f9068004120 2023-11-28T13:00:52.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.761+0000 7f9077fff700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f905c0682f0 0x7f905c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:52.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.761+0000 7f9077fff700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f905c0682f0 0x7f905c06a7a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9064000c00 tx=0x7f906400a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:52.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.765+0000 7f90757fa700 1 -- 172.21.15.111:0/2175880688 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f906c051b40 con 0x7f9068004120 2023-11-28T13:00:52.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.947+0000 7f907eba1700 1 -- 172.21.15.111:0/2175880688 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9058005b20 con 0x7f9068004120 2023-11-28T13:00:52.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.947+0000 7f90757fa700 1 -- 172.21.15.111:0/2175880688 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7f906c055160 con 0x7f9068004120 2023-11-28T13:00:52.949 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T13:00:52.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.949+0000 7f907eba1700 1 -- 172.21.15.111:0/2175880688 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f905c0682f0 msgr2=0x7f905c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.949+0000 7f907eba1700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f905c0682f0 0x7f905c06a7a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9064000c00 tx=0x7f906400a000 comp rx=0 tx=0).stop 2023-11-28T13:00:52.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.949+0000 7f907eba1700 1 -- 172.21.15.111:0/2175880688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9068004120 msgr2=0x7f90780b2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:52.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.949+0000 7f907eba1700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9068004120 0x7f90780b2cb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f906c00e7b0 tx=0x7f906c00eac0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.949+0000 7f907eba1700 1 -- 172.21.15.111:0/2175880688 shutdown_connections 2023-11-28T13:00:52.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.949+0000 7f907eba1700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f90781514c0 0x7f9078151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.949+0000 7f907eba1700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9068004120 0x7f90780b2cb0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.949+0000 7f907eba1700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f905c0682f0 0x7f905c06a7a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.949+0000 7f907eba1700 1 --2- 172.21.15.111:0/2175880688 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9078159660 0x7f90781492c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:52.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.949+0000 7f907eba1700 1 -- 172.21.15.111:0/2175880688 >> 172.21.15.111:0/2175880688 conn(0x7f907809a040 msgr2=0x7f907806c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:52.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.949+0000 7f907eba1700 1 -- 172.21.15.111:0/2175880688 shutdown_connections 2023-11-28T13:00:52.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:52.949+0000 7f907eba1700 1 -- 172.21.15.111:0/2175880688 wait complete. 2023-11-28T13:00:52.963 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T13:00:53.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.086+0000 7f4b09e08700 1 Processor -- start 2023-11-28T13:00:53.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.086+0000 7f4b09e08700 1 -- start start 2023-11-28T13:00:53.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.087+0000 7f4b09e08700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b040b0440 0x7f4b040b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.087+0000 7f4b09e08700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4b040ca2f0 0x7f4b040ca770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.087+0000 7f4b09e08700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b040cc750 con 0x7f4b0415c6c0 2023-11-28T13:00:53.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.087+0000 7f4b09e08700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b040b0260 con 0x7f4b040ca2f0 2023-11-28T13:00:53.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.087+0000 7f4b09e08700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b040b0df0 con 0x7f4b040b0440 2023-11-28T13:00:53.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.087+0000 7f4b03fff700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4b040ca2f0 0x7f4b040ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.087+0000 7f4b02ffd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b040b0440 0x7f4b040b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.087+0000 7f4b037fe700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f4b0415c6c0 0x7f4b040afb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:45374/0 (socket says 172.21.15.111:45374) 2023-11-28T13:00:53.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.087+0000 7f4b037fe700 1 -- 172.21.15.111:0/478526531 learned_addr learned my addr 172.21.15.111:0/478526531 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:53.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.088+0000 7f4b02ffd700 1 -- 172.21.15.111:0/478526531 >> v2:172.21.15.149:3300/0 conn(0x7f4b040ca2f0 msgr2=0x7f4b040ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:53.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.088+0000 7f4b02ffd700 1 --2- 172.21.15.111:0/478526531 >> v2:172.21.15.149:3300/0 conn(0x7f4b040ca2f0 0x7f4b040ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.088+0000 7f4b02ffd700 1 -- 172.21.15.111:0/478526531 >> v1:172.21.15.149:6789/0 conn(0x7f4b0415c6c0 legacy=0x7f4b040afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.088+0000 7f4b02ffd700 1 -- 172.21.15.111:0/478526531 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b040cca30 con 0x7f4b040b0440 2023-11-28T13:00:53.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.088+0000 7f4b03fff700 1 --2- 172.21.15.111:0/478526531 >> v2:172.21.15.149:3300/0 conn(0x7f4b040ca2f0 0x7f4b040ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:53.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.088+0000 7f4b02ffd700 1 --2- 172.21.15.111:0/478526531 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b040b0440 0x7f4b040b08b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009840 tx=0x7f4af4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae08173c723d6cb1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:53.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.088+0000 7f4b027fc700 1 -- 172.21.15.111:0/478526531 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4011830 con 0x7f4b040b0440 2023-11-28T13:00:53.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.088+0000 7f4b027fc700 1 -- 172.21.15.111:0/478526531 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af4011990 con 0x7f4b040b0440 2023-11-28T13:00:53.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.088+0000 7f4b027fc700 1 -- 172.21.15.111:0/478526531 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4011d20 con 0x7f4b040b0440 2023-11-28T13:00:53.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.089+0000 7f4b09e08700 1 -- 172.21.15.111:0/478526531 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b040b0440 msgr2=0x7f4b040b08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.089+0000 7f4b09e08700 1 --2- 172.21.15.111:0/478526531 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b040b0440 0x7f4b040b08b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009840 tx=0x7f4af4009b50 comp rx=0 tx=0).stop 2023-11-28T13:00:53.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.089+0000 7f4b09e08700 1 -- 172.21.15.111:0/478526531 shutdown_connections 2023-11-28T13:00:53.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.089+0000 7f4b09e08700 1 --2- 172.21.15.111:0/478526531 >> v2:172.21.15.149:3300/0 conn(0x7f4b040ca2f0 0x7f4b040ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.089+0000 7f4b09e08700 1 --2- 172.21.15.111:0/478526531 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b040b0440 0x7f4b040b08b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.089+0000 7f4b09e08700 1 -- 172.21.15.111:0/478526531 >> 172.21.15.111:0/478526531 conn(0x7f4b04098ee0 msgr2=0x7f4b0406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:53.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.089+0000 7f4b09e08700 1 -- 172.21.15.111:0/478526531 shutdown_connections 2023-11-28T13:00:53.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.089+0000 7f4b09e08700 1 -- 172.21.15.111:0/478526531 wait complete. 2023-11-28T13:00:53.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.090+0000 7f4b09e08700 1 Processor -- start 2023-11-28T13:00:53.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.090+0000 7f4b09e08700 1 -- start start 2023-11-28T13:00:53.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b09e08700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b040b0440 0x7f4b041554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b09e08700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b040ca2f0 0x7f4b040b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b09e08700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4b0415c6c0 0x7f4b040c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b09e08700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b040c86c0 con 0x7f4b040b0440 2023-11-28T13:00:53.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b09e08700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b040b3ce0 con 0x7f4b0415c6c0 2023-11-28T13:00:53.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b09e08700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b040b30d0 con 0x7f4b040ca2f0 2023-11-28T13:00:53.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b037fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b040b0440 0x7f4b041554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b037fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b040b0440 0x7f4b041554b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36060/0 (socket says 172.21.15.111:36060) 2023-11-28T13:00:53.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b037fe700 1 -- 172.21.15.111:0/2793976567 learned_addr learned my addr 172.21.15.111:0/2793976567 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:53.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b03fff700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4b0415c6c0 0x7f4b040c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b02ffd700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b040ca2f0 0x7f4b040b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b037fe700 1 -- 172.21.15.111:0/2793976567 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b040ca2f0 msgr2=0x7f4b040b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b037fe700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b040ca2f0 0x7f4b040b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b037fe700 1 -- 172.21.15.111:0/2793976567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4b0415c6c0 msgr2=0x7f4b040c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b037fe700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4b0415c6c0 0x7f4b040c8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b037fe700 1 -- 172.21.15.111:0/2793976567 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc009a70 con 0x7f4b040b0440 2023-11-28T13:00:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4b037fe700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b040b0440 0x7f4b041554b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4af800c520 tx=0x7f4af800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.091+0000 7f4aebfff700 1 -- 172.21.15.111:0/2793976567 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8005d70 con 0x7f4b040b0440 2023-11-28T13:00:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.092+0000 7f4aebfff700 1 -- 172.21.15.111:0/2793976567 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af80070d0 con 0x7f4b040b0440 2023-11-28T13:00:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.092+0000 7f4aebfff700 1 -- 172.21.15.111:0/2793976567 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8007440 con 0x7f4b040b0440 2023-11-28T13:00:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.092+0000 7f4b09e08700 1 -- 172.21.15.111:0/2793976567 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af40094a0 con 0x7f4b040b0440 2023-11-28T13:00:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.092+0000 7f4b09e08700 1 -- 172.21.15.111:0/2793976567 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b040ccff0 con 0x7f4b040b0440 2023-11-28T13:00:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.092+0000 7f4b09e08700 1 -- 172.21.15.111:0/2793976567 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b040ccd10 con 0x7f4b040b0440 2023-11-28T13:00:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.093+0000 7f4aebfff700 1 -- 172.21.15.111:0/2793976567 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4af8019030 con 0x7f4b040b0440 2023-11-28T13:00:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.093+0000 7f4aebfff700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4af00682f0 0x7f4af006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.093+0000 7f4aebfff700 1 -- 172.21.15.111:0/2793976567 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f4af8084bd0 con 0x7f4b040b0440 2023-11-28T13:00:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.093+0000 7f4b02ffd700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4af00682f0 0x7f4af006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.094+0000 7f4b02ffd700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4af00682f0 0x7f4af006a7a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009fd0 tx=0x7f4af401c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.098+0000 7f4aebfff700 1 -- 172.21.15.111:0/2793976567 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4af804b5e0 con 0x7f4b040b0440 2023-11-28T13:00:53.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.279+0000 7f4b09e08700 1 -- 172.21.15.111:0/2793976567 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4b040cdf80 con 0x7f4b040b0440 2023-11-28T13:00:53.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.279+0000 7f4aebfff700 1 -- 172.21.15.111:0/2793976567 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v51) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4af80170a0 con 0x7f4b040b0440 2023-11-28T13:00:53.281 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T13:00:53.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.282+0000 7f4b09e08700 1 -- 172.21.15.111:0/2793976567 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4af00682f0 msgr2=0x7f4af006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.282+0000 7f4b09e08700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4af00682f0 0x7f4af006a7a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009fd0 tx=0x7f4af401c040 comp rx=0 tx=0).stop 2023-11-28T13:00:53.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.282+0000 7f4b09e08700 1 -- 172.21.15.111:0/2793976567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b040b0440 msgr2=0x7f4b041554b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.282+0000 7f4b09e08700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b040b0440 0x7f4b041554b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4af800c520 tx=0x7f4af800c830 comp rx=0 tx=0).stop 2023-11-28T13:00:53.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.282+0000 7f4b09e08700 1 -- 172.21.15.111:0/2793976567 shutdown_connections 2023-11-28T13:00:53.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.282+0000 7f4b09e08700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4af00682f0 0x7f4af006a7a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.282+0000 7f4b09e08700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4b0415c6c0 0x7f4b040c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.282+0000 7f4b09e08700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b040b0440 0x7f4b041554b0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.282+0000 7f4b09e08700 1 --2- 172.21.15.111:0/2793976567 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4b040ca2f0 0x7f4b040b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.282+0000 7f4b09e08700 1 -- 172.21.15.111:0/2793976567 >> 172.21.15.111:0/2793976567 conn(0x7f4b04098ee0 msgr2=0x7f4b040af550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:53.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.282+0000 7f4b09e08700 1 -- 172.21.15.111:0/2793976567 shutdown_connections 2023-11-28T13:00:53.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.282+0000 7f4b09e08700 1 -- 172.21.15.111:0/2793976567 wait complete. 2023-11-28T13:00:53.295 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T13:00:53.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.413+0000 7f3479697700 1 Processor -- start 2023-11-28T13:00:53.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.414+0000 7f3479697700 1 -- start start 2023-11-28T13:00:53.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.414+0000 7f3479697700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f347415cbf0 0x7f34740b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.414+0000 7f3479697700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f347406ba90 0x7f34740ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.414+0000 7f3479697700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34740cfdb0 con 0x7f34740b40a0 2023-11-28T13:00:53.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.414+0000 7f3479697700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34740cc740 con 0x7f347406ba90 2023-11-28T13:00:53.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.414+0000 7f3479697700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34740b35e0 con 0x7f347415cbf0 2023-11-28T13:00:53.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f34737fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f347406ba90 0x7f34740ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f3472ffd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f347415cbf0 0x7f34740b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f34727fc700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f34740b40a0 0x7f347406b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:45388/0 (socket says 172.21.15.111:45388) 2023-11-28T13:00:53.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f34727fc700 1 -- 172.21.15.111:0/4443269 learned_addr learned my addr 172.21.15.111:0/4443269 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:53.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f3472ffd700 1 -- 172.21.15.111:0/4443269 >> v2:172.21.15.149:3300/0 conn(0x7f347406ba90 msgr2=0x7f34740ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:53.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f3472ffd700 1 --2- 172.21.15.111:0/4443269 >> v2:172.21.15.149:3300/0 conn(0x7f347406ba90 0x7f34740ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f3472ffd700 1 -- 172.21.15.111:0/4443269 >> v1:172.21.15.149:6789/0 conn(0x7f34740b40a0 legacy=0x7f347406b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f3472ffd700 1 -- 172.21.15.111:0/4443269 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34740cbe10 con 0x7f347415cbf0 2023-11-28T13:00:53.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f3471ffb700 1 -- 172.21.15.111:0/4443269 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196568288 0 0) 0x7f34740cfdb0 con 0x7f34740b40a0 2023-11-28T13:00:53.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f34737fe700 1 --2- 172.21.15.111:0/4443269 >> v2:172.21.15.149:3300/0 conn(0x7f347406ba90 0x7f34740ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:53.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f3472ffd700 1 --2- 172.21.15.111:0/4443269 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f347415cbf0 0x7f34740b3b60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3468009840 tx=0x7f3468009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b527ca33ba12672 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:53.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f3471ffb700 1 -- 172.21.15.111:0/4443269 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3468011830 con 0x7f347415cbf0 2023-11-28T13:00:53.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f3471ffb700 1 -- 172.21.15.111:0/4443269 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3468011990 con 0x7f347415cbf0 2023-11-28T13:00:53.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.415+0000 7f3471ffb700 1 -- 172.21.15.111:0/4443269 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3468011d20 con 0x7f347415cbf0 2023-11-28T13:00:53.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.416+0000 7f3479697700 1 -- 172.21.15.111:0/4443269 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f347415cbf0 msgr2=0x7f34740b3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.416+0000 7f3479697700 1 --2- 172.21.15.111:0/4443269 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f347415cbf0 0x7f34740b3b60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3468009840 tx=0x7f3468009b50 comp rx=0 tx=0).stop 2023-11-28T13:00:53.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.416+0000 7f3479697700 1 -- 172.21.15.111:0/4443269 shutdown_connections 2023-11-28T13:00:53.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.416+0000 7f3479697700 1 --2- 172.21.15.111:0/4443269 >> v2:172.21.15.149:3300/0 conn(0x7f347406ba90 0x7f34740ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.416+0000 7f3479697700 1 --2- 172.21.15.111:0/4443269 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f347415cbf0 0x7f34740b3b60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.416+0000 7f3479697700 1 -- 172.21.15.111:0/4443269 >> 172.21.15.111:0/4443269 conn(0x7f347409a040 msgr2=0x7f34740d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:53.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.416+0000 7f3479697700 1 -- 172.21.15.111:0/4443269 shutdown_connections 2023-11-28T13:00:53.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.416+0000 7f3479697700 1 -- 172.21.15.111:0/4443269 wait complete. 2023-11-28T13:00:53.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.417+0000 7f3479697700 1 Processor -- start 2023-11-28T13:00:53.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.417+0000 7f3479697700 1 -- start start 2023-11-28T13:00:53.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.418+0000 7f3479697700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f347406ba90 0x7f34740c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.418+0000 7f3479697700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34740b40a0 0x7f347415a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.418+0000 7f3479697700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f347415cbf0 0x7f34740c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.418+0000 7f3479697700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34740c5710 con 0x7f34740b40a0 2023-11-28T13:00:53.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.418+0000 7f3479697700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34740c47b0 con 0x7f347406ba90 2023-11-28T13:00:53.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.418+0000 7f3479697700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34740c4a90 con 0x7f347415cbf0 2023-11-28T13:00:53.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.418+0000 7f34727fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34740b40a0 0x7f347415a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.418+0000 7f34727fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34740b40a0 0x7f347415a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36078/0 (socket says 172.21.15.111:36078) 2023-11-28T13:00:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.418+0000 7f34727fc700 1 -- 172.21.15.111:0/2514085457 learned_addr learned my addr 172.21.15.111:0/2514085457 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.418+0000 7f3472ffd700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f347406ba90 0x7f34740c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.418+0000 7f34737fe700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f347415cbf0 0x7f34740c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.419+0000 7f34727fc700 1 -- 172.21.15.111:0/2514085457 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f347415cbf0 msgr2=0x7f34740c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.419+0000 7f34727fc700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f347415cbf0 0x7f34740c5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.419+0000 7f34727fc700 1 -- 172.21.15.111:0/2514085457 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f347406ba90 msgr2=0x7f34740c7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.419+0000 7f34727fc700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f347406ba90 0x7f34740c7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.419+0000 7f34727fc700 1 -- 172.21.15.111:0/2514085457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c009a70 con 0x7f34740b40a0 2023-11-28T13:00:53.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.419+0000 7f34727fc700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34740b40a0 0x7f347415a3d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f34640011e0 tx=0x7f346400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:53.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.419+0000 7f345b7fe700 1 -- 172.21.15.111:0/2514085457 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3464004c80 con 0x7f34740b40a0 2023-11-28T13:00:53.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.419+0000 7f345b7fe700 1 -- 172.21.15.111:0/2514085457 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3464004e00 con 0x7f34740b40a0 2023-11-28T13:00:53.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.419+0000 7f3479697700 1 -- 172.21.15.111:0/2514085457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34680094a0 con 0x7f34740b40a0 2023-11-28T13:00:53.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.419+0000 7f345b7fe700 1 -- 172.21.15.111:0/2514085457 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3464007310 con 0x7f34740b40a0 2023-11-28T13:00:53.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.419+0000 7f3479697700 1 -- 172.21.15.111:0/2514085457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34740cc3d0 con 0x7f34740b40a0 2023-11-28T13:00:53.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.420+0000 7f3479697700 1 -- 172.21.15.111:0/2514085457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34740cc0f0 con 0x7f34740b40a0 2023-11-28T13:00:53.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.420+0000 7f345b7fe700 1 -- 172.21.15.111:0/2514085457 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f346401a020 con 0x7f34740b40a0 2023-11-28T13:00:53.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.421+0000 7f345b7fe700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f34600682f0 0x7f346006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.421+0000 7f345b7fe700 1 -- 172.21.15.111:0/2514085457 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f3464085610 con 0x7f34740b40a0 2023-11-28T13:00:53.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.421+0000 7f3472ffd700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f34600682f0 0x7f346006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.421+0000 7f3472ffd700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f34600682f0 0x7f346006a7a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3468009fd0 tx=0x7f346801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:53.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.425+0000 7f345b7fe700 1 -- 172.21.15.111:0/2514085457 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3464052000 con 0x7f34740b40a0 2023-11-28T13:00:53.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.610+0000 7f3479697700 1 -- 172.21.15.111:0/2514085457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f34740d06f0 con 0x7f34740b40a0 2023-11-28T13:00:53.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.610+0000 7f345b7fe700 1 -- 172.21.15.111:0/2514085457 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v51) v1 ==== 84+0+126 (secure 0 0 0) 0x7f3464055620 con 0x7f34740b40a0 2023-11-28T13:00:53.612 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:53.612 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-11-28T13:00:53.612 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:53.612 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-11-28T13:00:53.612 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-11-28T13:00:53.613 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:00:53.613 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:00:53.613 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 2, 2023-11-28T13:00:53.613 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": "rbd" 2023-11-28T13:00:53.613 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:00:53.613 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-11-28T13:00:53.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.612+0000 7f3479697700 1 -- 172.21.15.111:0/2514085457 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f34600682f0 msgr2=0x7f346006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.612+0000 7f3479697700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f34600682f0 0x7f346006a7a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3468009fd0 tx=0x7f346801c040 comp rx=0 tx=0).stop 2023-11-28T13:00:53.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.612+0000 7f3479697700 1 -- 172.21.15.111:0/2514085457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34740b40a0 msgr2=0x7f347415a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.612+0000 7f3479697700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34740b40a0 0x7f347415a3d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f34640011e0 tx=0x7f346400ca30 comp rx=0 tx=0).stop 2023-11-28T13:00:53.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.612+0000 7f3479697700 1 -- 172.21.15.111:0/2514085457 shutdown_connections 2023-11-28T13:00:53.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.612+0000 7f3479697700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f347415cbf0 0x7f34740c5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.612+0000 7f3479697700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f34600682f0 0x7f346006a7a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.612+0000 7f3479697700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f347406ba90 0x7f34740c7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.612+0000 7f3479697700 1 --2- 172.21.15.111:0/2514085457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34740b40a0 0x7f347415a3d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.612+0000 7f3479697700 1 -- 172.21.15.111:0/2514085457 >> 172.21.15.111:0/2514085457 conn(0x7f347409a040 msgr2=0x7f34741e51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:53.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.612+0000 7f3479697700 1 -- 172.21.15.111:0/2514085457 shutdown_connections 2023-11-28T13:00:53.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.612+0000 7f3479697700 1 -- 172.21.15.111:0/2514085457 wait complete. 2023-11-28T13:00:53.626 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T13:00:53.626 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T13:00:53.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.747+0000 7f0870ef8700 1 Processor -- start 2023-11-28T13:00:53.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.747+0000 7f0870ef8700 1 -- start start 2023-11-28T13:00:53.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.748+0000 7f0870ef8700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f086c15a5d0 0x7f086c0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.748+0000 7f0870ef8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f086c0cb2d0 0x7f086c0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.748+0000 7f0870ef8700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086c0ae8b0 con 0x7f086c0b0160 2023-11-28T13:00:53.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.748+0000 7f0870ef8700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f086c0cc1b0 con 0x7f086c15a5d0 2023-11-28T13:00:53.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.748+0000 7f0870ef8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086c0cc390 con 0x7f086c0cb2d0 2023-11-28T13:00:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.748+0000 7f086ad9d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f086c0cb2d0 0x7f086c0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.748+0000 7f086a59c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f086c15a5d0 0x7f086c0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.748+0000 7f0869d9b700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f086c0b0160 0x7f086c15e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:45402/0 (socket says 172.21.15.111:45402) 2023-11-28T13:00:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.749+0000 7f086a59c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f086c15a5d0 0x7f086c0ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:53982/0 (socket says 172.21.15.111:53982) 2023-11-28T13:00:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.749+0000 7f0869d9b700 1 -- 172.21.15.111:0/393101490 learned_addr learned my addr 172.21.15.111:0/393101490 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.749+0000 7f086ad9d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f086c0cb2d0 0x7f086c0ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:57830/0 (socket says 172.21.15.111:57830) 2023-11-28T13:00:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.749+0000 7f086ad9d700 1 -- 172.21.15.111:0/393101490 >> v2:172.21.15.149:3300/0 conn(0x7f086c15a5d0 msgr2=0x7f086c0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.749+0000 7f086ad9d700 1 --2- 172.21.15.111:0/393101490 >> v2:172.21.15.149:3300/0 conn(0x7f086c15a5d0 0x7f086c0ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.749+0000 7f086ad9d700 1 -- 172.21.15.111:0/393101490 >> v1:172.21.15.149:6789/0 conn(0x7f086c0b0160 legacy=0x7f086c15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.749+0000 7f086ad9d700 1 -- 172.21.15.111:0/393101490 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c0cc670 con 0x7f086c0cb2d0 2023-11-28T13:00:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.749+0000 7f086a59c700 1 --2- 172.21.15.111:0/393101490 >> v2:172.21.15.149:3300/0 conn(0x7f086c15a5d0 0x7f086c0ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:53.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.749+0000 7f086ad9d700 1 --2- 172.21.15.111:0/393101490 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f086c0cb2d0 0x7f086c0ced90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0864009d90 tx=0x7f086400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16fad8d701790c5f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:53.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.749+0000 7f086959a700 1 -- 172.21.15.111:0/393101490 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086400e070 con 0x7f086c0cb2d0 2023-11-28T13:00:53.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.749+0000 7f086959a700 1 -- 172.21.15.111:0/393101490 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086400bb20 con 0x7f086c0cb2d0 2023-11-28T13:00:53.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.749+0000 7f086959a700 1 -- 172.21.15.111:0/393101490 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0864012430 con 0x7f086c0cb2d0 2023-11-28T13:00:53.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.750+0000 7f0870ef8700 1 -- 172.21.15.111:0/393101490 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f086c0cb2d0 msgr2=0x7f086c0ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.750+0000 7f0870ef8700 1 --2- 172.21.15.111:0/393101490 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f086c0cb2d0 0x7f086c0ced90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0864009d90 tx=0x7f086400b330 comp rx=0 tx=0).stop 2023-11-28T13:00:53.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.750+0000 7f0870ef8700 1 -- 172.21.15.111:0/393101490 shutdown_connections 2023-11-28T13:00:53.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.750+0000 7f0870ef8700 1 --2- 172.21.15.111:0/393101490 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f086c0cb2d0 0x7f086c0ced90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.750+0000 7f0870ef8700 1 --2- 172.21.15.111:0/393101490 >> v2:172.21.15.149:3300/0 conn(0x7f086c15a5d0 0x7f086c0ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.750+0000 7f0870ef8700 1 -- 172.21.15.111:0/393101490 >> 172.21.15.111:0/393101490 conn(0x7f086c09a040 msgr2=0x7f086c0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:53.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.750+0000 7f0870ef8700 1 -- 172.21.15.111:0/393101490 shutdown_connections 2023-11-28T13:00:53.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.750+0000 7f0870ef8700 1 -- 172.21.15.111:0/393101490 wait complete. 2023-11-28T13:00:53.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.751+0000 7f0870ef8700 1 Processor -- start 2023-11-28T13:00:53.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.752+0000 7f0870ef8700 1 -- start start 2023-11-28T13:00:53.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.752+0000 7f0870ef8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c0b0160 0x7f086c14c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.752+0000 7f0870ef8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f086c0cb2d0 0x7f086c157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.752+0000 7f0870ef8700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f086c15a5d0 0x7f086c158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.752+0000 7f0870ef8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c14c830 con 0x7f086c0b0160 2023-11-28T13:00:53.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.752+0000 7f0870ef8700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086c0b6940 con 0x7f086c15a5d0 2023-11-28T13:00:53.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.752+0000 7f0870ef8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086c0b6380 con 0x7f086c0cb2d0 2023-11-28T13:00:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f086a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c0b0160 0x7f086c14c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f086a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c0b0160 0x7f086c14c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36088/0 (socket says 172.21.15.111:36088) 2023-11-28T13:00:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f086a59c700 1 -- 172.21.15.111:0/2686915041 learned_addr learned my addr 172.21.15.111:0/2686915041 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f086ad9d700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f086c15a5d0 0x7f086c158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f0869d9b700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f086c0cb2d0 0x7f086c157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f086a59c700 1 -- 172.21.15.111:0/2686915041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f086c0cb2d0 msgr2=0x7f086c157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f086a59c700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f086c0cb2d0 0x7f086c157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f086a59c700 1 -- 172.21.15.111:0/2686915041 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f086c15a5d0 msgr2=0x7f086c158370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f086a59c700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f086c15a5d0 0x7f086c158370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f086a59c700 1 -- 172.21.15.111:0/2686915041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0860009430 con 0x7f086c0b0160 2023-11-28T13:00:53.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f086a59c700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c0b0160 0x7f086c14c2f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f086000ebe0 tx=0x7f086000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:53.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f085affd700 1 -- 172.21.15.111:0/2686915041 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0860009740 con 0x7f086c0b0160 2023-11-28T13:00:53.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f085affd700 1 -- 172.21.15.111:0/2686915041 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0860005a20 con 0x7f086c0b0160 2023-11-28T13:00:53.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f0870ef8700 1 -- 172.21.15.111:0/2686915041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0864009a70 con 0x7f086c0b0160 2023-11-28T13:00:53.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f085affd700 1 -- 172.21.15.111:0/2686915041 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0860005db0 con 0x7f086c0b0160 2023-11-28T13:00:53.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.753+0000 7f0870ef8700 1 -- 172.21.15.111:0/2686915041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086c0ccc30 con 0x7f086c0b0160 2023-11-28T13:00:53.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.754+0000 7f0870ef8700 1 -- 172.21.15.111:0/2686915041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f086c0cc950 con 0x7f086c0b0160 2023-11-28T13:00:53.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.755+0000 7f085affd700 1 -- 172.21.15.111:0/2686915041 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0860022070 con 0x7f086c0b0160 2023-11-28T13:00:53.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.755+0000 7f085affd700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f084c0682f0 0x7f084c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:53.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.755+0000 7f085affd700 1 -- 172.21.15.111:0/2686915041 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f0860085b20 con 0x7f086c0b0160 2023-11-28T13:00:53.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.755+0000 7f0869d9b700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f084c0682f0 0x7f084c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:53.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.755+0000 7f0869d9b700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f084c0682f0 0x7f084c06a7a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f086c0c7c80 tx=0x7f085c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:53.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.759+0000 7f085affd700 1 -- 172.21.15.111:0/2686915041 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0860052530 con 0x7f086c0b0160 2023-11-28T13:00:53.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:53.946+0000 7f0870ef8700 1 -- 172.21.15.111:0/2686915041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f086c0ccf10 con 0x7f086c0b0160 2023-11-28T13:00:54.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:54.875+0000 7f085affd700 1 -- 172.21.15.111:0/2686915041 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v52) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0860020020 con 0x7f086c0b0160 2023-11-28T13:00:54.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:54.879+0000 7f0870ef8700 1 -- 172.21.15.111:0/2686915041 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f084c0682f0 msgr2=0x7f084c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:54.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:54.879+0000 7f0870ef8700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f084c0682f0 0x7f084c06a7a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f086c0c7c80 tx=0x7f085c009040 comp rx=0 tx=0).stop 2023-11-28T13:00:54.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:54.879+0000 7f0870ef8700 1 -- 172.21.15.111:0/2686915041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c0b0160 msgr2=0x7f086c14c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:54.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:54.879+0000 7f0870ef8700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c0b0160 0x7f086c14c2f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f086000ebe0 tx=0x7f086000eef0 comp rx=0 tx=0).stop 2023-11-28T13:00:54.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:54.879+0000 7f0870ef8700 1 -- 172.21.15.111:0/2686915041 shutdown_connections 2023-11-28T13:00:54.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:54.879+0000 7f0870ef8700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f084c0682f0 0x7f084c06a7a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:54.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:54.879+0000 7f0870ef8700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f086c15a5d0 0x7f086c158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:54.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:54.879+0000 7f0870ef8700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f086c0b0160 0x7f086c14c2f0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:54.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:54.879+0000 7f0870ef8700 1 --2- 172.21.15.111:0/2686915041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f086c0cb2d0 0x7f086c157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:54.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:54.879+0000 7f0870ef8700 1 -- 172.21.15.111:0/2686915041 >> 172.21.15.111:0/2686915041 conn(0x7f086c09a040 msgr2=0x7f086c159340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:54.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:54.879+0000 7f0870ef8700 1 -- 172.21.15.111:0/2686915041 shutdown_connections 2023-11-28T13:00:54.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:54.879+0000 7f0870ef8700 1 -- 172.21.15.111:0/2686915041 wait complete. 2023-11-28T13:00:54.883 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-11-28T13:00:54.898 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T13:00:55.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.030+0000 7fd667204700 1 Processor -- start 2023-11-28T13:00:55.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.030+0000 7fd667204700 1 -- start start 2023-11-28T13:00:55.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.031+0000 7fd667204700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd6600ca440 0x7fd66006b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:55.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.031+0000 7fd667204700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd6600aee50 0x7fd66015a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:55.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.031+0000 7fd667204700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66006a600 con 0x7fd6600ca810 2023-11-28T13:00:55.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.031+0000 7fd667204700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6600cd160 con 0x7fd6600ca440 2023-11-28T13:00:55.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.031+0000 7fd667204700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66006a8e0 con 0x7fd6600aee50 2023-11-28T13:00:55.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.031+0000 7fd6657a1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd6600aee50 0x7fd66015a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:55.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.031+0000 7fd664fa0700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd6600ca440 0x7fd66006b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:55.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.031+0000 7fd65ffff700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7fd6600ca810 0x7fd6600ae740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:45414/0 (socket says 172.21.15.111:45414) 2023-11-28T13:00:55.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.031+0000 7fd6657a1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd6600aee50 0x7fd66015a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:57846/0 (socket says 172.21.15.111:57846) 2023-11-28T13:00:55.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.031+0000 7fd65ffff700 1 -- 172.21.15.111:0/2456288447 learned_addr learned my addr 172.21.15.111:0/2456288447 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:55.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.032+0000 7fd6657a1700 1 -- 172.21.15.111:0/2456288447 >> v2:172.21.15.149:3300/0 conn(0x7fd6600ca440 msgr2=0x7fd66006b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.032+0000 7fd6657a1700 1 --2- 172.21.15.111:0/2456288447 >> v2:172.21.15.149:3300/0 conn(0x7fd6600ca440 0x7fd66006b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.032+0000 7fd6657a1700 1 -- 172.21.15.111:0/2456288447 >> v1:172.21.15.149:6789/0 conn(0x7fd6600ca810 legacy=0x7fd6600ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.032+0000 7fd6657a1700 1 -- 172.21.15.111:0/2456288447 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66006abc0 con 0x7fd6600aee50 2023-11-28T13:00:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.032+0000 7fd65f7fe700 1 -- 172.21.15.111:0/2456288447 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640326241 0 0) 0x7fd66006a600 con 0x7fd6600ca810 2023-11-28T13:00:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.032+0000 7fd664fa0700 1 --2- 172.21.15.111:0/2456288447 >> v2:172.21.15.149:3300/0 conn(0x7fd6600ca440 0x7fd66006b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.032+0000 7fd6657a1700 1 --2- 172.21.15.111:0/2456288447 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd6600aee50 0x7fd66015a0e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd658009d90 tx=0x7fd65800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c089ae311bf2c326 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.032+0000 7fd65f7fe700 1 -- 172.21.15.111:0/2456288447 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65800e070 con 0x7fd6600aee50 2023-11-28T13:00:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.032+0000 7fd65f7fe700 1 -- 172.21.15.111:0/2456288447 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65800bb20 con 0x7fd6600aee50 2023-11-28T13:00:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.032+0000 7fd65f7fe700 1 -- 172.21.15.111:0/2456288447 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd658012430 con 0x7fd6600aee50 2023-11-28T13:00:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.033+0000 7fd667204700 1 -- 172.21.15.111:0/2456288447 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd6600aee50 msgr2=0x7fd66015a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.033+0000 7fd667204700 1 --2- 172.21.15.111:0/2456288447 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd6600aee50 0x7fd66015a0e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd658009d90 tx=0x7fd65800b330 comp rx=0 tx=0).stop 2023-11-28T13:00:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.033+0000 7fd667204700 1 -- 172.21.15.111:0/2456288447 shutdown_connections 2023-11-28T13:00:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.033+0000 7fd667204700 1 --2- 172.21.15.111:0/2456288447 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd6600aee50 0x7fd66015a0e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:55.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.033+0000 7fd667204700 1 --2- 172.21.15.111:0/2456288447 >> v2:172.21.15.149:3300/0 conn(0x7fd6600ca440 0x7fd66006b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:55.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.033+0000 7fd667204700 1 -- 172.21.15.111:0/2456288447 >> 172.21.15.111:0/2456288447 conn(0x7fd66006e790 msgr2=0x7fd6600cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:55.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.033+0000 7fd667204700 1 -- 172.21.15.111:0/2456288447 shutdown_connections 2023-11-28T13:00:55.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.033+0000 7fd667204700 1 -- 172.21.15.111:0/2456288447 wait complete. 2023-11-28T13:00:55.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.034+0000 7fd667204700 1 Processor -- start 2023-11-28T13:00:55.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.034+0000 7fd667204700 1 -- start start 2023-11-28T13:00:55.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd667204700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6600aee50 0x7fd660155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:55.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd667204700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd6600ca440 0x7fd6600b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:55.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd667204700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd6600ca810 0x7fd6600b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:55.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd667204700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd660156990 con 0x7fd6600aee50 2023-11-28T13:00:55.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd667204700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6600cd750 con 0x7fd6600ca440 2023-11-28T13:00:55.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd667204700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6600cd9a0 con 0x7fd6600ca810 2023-11-28T13:00:55.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd664fa0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6600aee50 0x7fd660155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:55.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd664fa0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6600aee50 0x7fd660155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36100/0 (socket says 172.21.15.111:36100) 2023-11-28T13:00:55.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd664fa0700 1 -- 172.21.15.111:0/236514280 learned_addr learned my addr 172.21.15.111:0/236514280 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:55.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd6657a1700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd6600ca810 0x7fd6600b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:55.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd65ffff700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd6600ca440 0x7fd6600b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:55.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd664fa0700 1 -- 172.21.15.111:0/236514280 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd6600ca810 msgr2=0x7fd6600b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:55.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd664fa0700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd6600ca810 0x7fd6600b47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:55.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd664fa0700 1 -- 172.21.15.111:0/236514280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd6600ca440 msgr2=0x7fd6600b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:55.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd664fa0700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd6600ca440 0x7fd6600b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:55.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd664fa0700 1 -- 172.21.15.111:0/236514280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6540094a0 con 0x7fd6600aee50 2023-11-28T13:00:55.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd664fa0700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6600aee50 0x7fd660155e50 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd65400ebe0 tx=0x7fd65400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:55.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.035+0000 7fd65d7fa700 1 -- 172.21.15.111:0/236514280 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65400c850 con 0x7fd6600aee50 2023-11-28T13:00:55.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.036+0000 7fd65d7fa700 1 -- 172.21.15.111:0/236514280 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65400c9b0 con 0x7fd6600aee50 2023-11-28T13:00:55.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.036+0000 7fd667204700 1 -- 172.21.15.111:0/236514280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd658009a70 con 0x7fd6600aee50 2023-11-28T13:00:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.036+0000 7fd65d7fa700 1 -- 172.21.15.111:0/236514280 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65400cd40 con 0x7fd6600aee50 2023-11-28T13:00:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.036+0000 7fd667204700 1 -- 172.21.15.111:0/236514280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66006b900 con 0x7fd6600aee50 2023-11-28T13:00:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.036+0000 7fd667204700 1 -- 172.21.15.111:0/236514280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66006aea0 con 0x7fd6600aee50 2023-11-28T13:00:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.037+0000 7fd65d7fa700 1 -- 172.21.15.111:0/236514280 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd654019030 con 0x7fd6600aee50 2023-11-28T13:00:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.037+0000 7fd65d7fa700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6440682f0 0x7fd64406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.037+0000 7fd65d7fa700 1 -- 172.21.15.111:0/236514280 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5900+0+0 (secure 0 0 0) 0x7fd654085b20 con 0x7fd6600aee50 2023-11-28T13:00:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.037+0000 7fd65ffff700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6440682f0 0x7fd64406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:55.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.038+0000 7fd65ffff700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6440682f0 0x7fd64406a7a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd6601f5470 tx=0x7fd64c007e70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:55.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.042+0000 7fd65d7fa700 1 -- 172.21.15.111:0/236514280 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd654052420 con 0x7fd6600aee50 2023-11-28T13:00:55.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.224+0000 7fd667204700 1 -- 172.21.15.111:0/236514280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd6600ce5b0 con 0x7fd6600aee50 2023-11-28T13:00:55.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.883+0000 7fd65d7fa700 1 -- 172.21.15.111:0/236514280 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v53) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd6540170d0 con 0x7fd6600aee50 2023-11-28T13:00:55.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.887+0000 7fd667204700 1 -- 172.21.15.111:0/236514280 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6440682f0 msgr2=0x7fd64406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:55.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.887+0000 7fd667204700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6440682f0 0x7fd64406a7a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd6601f5470 tx=0x7fd64c007e70 comp rx=0 tx=0).stop 2023-11-28T13:00:55.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.887+0000 7fd667204700 1 -- 172.21.15.111:0/236514280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6600aee50 msgr2=0x7fd660155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:55.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.887+0000 7fd667204700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6600aee50 0x7fd660155e50 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd65400ebe0 tx=0x7fd65400eef0 comp rx=0 tx=0).stop 2023-11-28T13:00:55.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.891+0000 7fd667204700 1 -- 172.21.15.111:0/236514280 shutdown_connections 2023-11-28T13:00:55.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.891+0000 7fd667204700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd6600ca810 0x7fd6600b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:55.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.891+0000 7fd667204700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6600aee50 0x7fd660155e50 secure :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd65400ebe0 tx=0x7fd65400eef0 comp rx=0 tx=0).stop 2023-11-28T13:00:55.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.891+0000 7fd667204700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6440682f0 0x7fd64406a7a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:55.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.891+0000 7fd667204700 1 --2- 172.21.15.111:0/236514280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd6600ca440 0x7fd6600b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:55.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.891+0000 7fd667204700 1 -- 172.21.15.111:0/236514280 >> 172.21.15.111:0/236514280 conn(0x7fd66006e790 msgr2=0x7fd6600cfa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:55.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.892+0000 7fd667204700 1 -- 172.21.15.111:0/236514280 shutdown_connections 2023-11-28T13:00:55.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:55.892+0000 7fd667204700 1 -- 172.21.15.111:0/236514280 wait complete. 2023-11-28T13:00:55.897 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-11-28T13:00:55.911 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T13:00:56.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.033+0000 7f1b2a217700 1 Processor -- start 2023-11-28T13:00:56.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.033+0000 7f1b2a217700 1 -- start start 2023-11-28T13:00:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b2a217700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f1b2415a6e0 0x7f1b2406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b2a217700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1b240ca210 0x7f1b240ca680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b2a217700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b240cc330 0x7f1b240cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b2a217700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b2406ba40 con 0x7f1b240ca210 2023-11-28T13:00:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b2a217700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b2415aab0 con 0x7f1b2415a6e0 2023-11-28T13:00:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b2a217700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2406a480 con 0x7f1b240cc330 2023-11-28T13:00:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b22ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1b240ca210 0x7f1b240ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b22ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1b240ca210 0x7f1b240ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36108/0 (socket says 172.21.15.111:36108) 2023-11-28T13:00:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b22ffd700 1 -- 172.21.15.111:0/2562734508 learned_addr learned my addr 172.21.15.111:0/2562734508 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b22ffd700 1 -- 172.21.15.111:0/2562734508 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b240cc330 msgr2=0x7f1b240cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b23fff700 1 --2- 172.21.15.111:0/2562734508 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b240cc330 0x7f1b240cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b237fe700 1 --2- 172.21.15.111:0/2562734508 >> v2:172.21.15.149:3300/0 conn(0x7f1b2415a6e0 0x7f1b2406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b22ffd700 1 --2- 172.21.15.111:0/2562734508 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b240cc330 0x7f1b240cc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b22ffd700 1 -- 172.21.15.111:0/2562734508 >> v2:172.21.15.149:3300/0 conn(0x7f1b2415a6e0 msgr2=0x7f1b2406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b22ffd700 1 --2- 172.21.15.111:0/2562734508 >> v2:172.21.15.149:3300/0 conn(0x7f1b2415a6e0 0x7f1b2406b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b22ffd700 1 -- 172.21.15.111:0/2562734508 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2406a740 con 0x7f1b240ca210 2023-11-28T13:00:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b237fe700 1 --2- 172.21.15.111:0/2562734508 >> v2:172.21.15.149:3300/0 conn(0x7f1b2415a6e0 0x7f1b2406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.034+0000 7f1b22ffd700 1 --2- 172.21.15.111:0/2562734508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b240ca210 0x7f1b240ca680 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1b240704a0 tx=0x7f1b14009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b971dd6adf8d21 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.035+0000 7f1b227fc700 1 -- 172.21.15.111:0/2562734508 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1400b870 con 0x7f1b240ca210 2023-11-28T13:00:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.035+0000 7f1b227fc700 1 -- 172.21.15.111:0/2562734508 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1400b9d0 con 0x7f1b240ca210 2023-11-28T13:00:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.035+0000 7f1b227fc700 1 -- 172.21.15.111:0/2562734508 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1400bd60 con 0x7f1b240ca210 2023-11-28T13:00:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.035+0000 7f1b2a217700 1 -- 172.21.15.111:0/2562734508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b240ca210 msgr2=0x7f1b240ca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.035+0000 7f1b2a217700 1 --2- 172.21.15.111:0/2562734508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b240ca210 0x7f1b240ca680 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1b240704a0 tx=0x7f1b14009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.035+0000 7f1b2a217700 1 -- 172.21.15.111:0/2562734508 shutdown_connections 2023-11-28T13:00:56.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.035+0000 7f1b2a217700 1 --2- 172.21.15.111:0/2562734508 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b240cc330 0x7f1b240cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.035+0000 7f1b2a217700 1 --2- 172.21.15.111:0/2562734508 >> v2:172.21.15.149:3300/0 conn(0x7f1b2415a6e0 0x7f1b2406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.035+0000 7f1b2a217700 1 --2- 172.21.15.111:0/2562734508 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b240ca210 0x7f1b240ca680 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.035+0000 7f1b2a217700 1 -- 172.21.15.111:0/2562734508 >> 172.21.15.111:0/2562734508 conn(0x7f1b2406db10 msgr2=0x7f1b2406df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:56.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.035+0000 7f1b2a217700 1 -- 172.21.15.111:0/2562734508 shutdown_connections 2023-11-28T13:00:56.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.035+0000 7f1b2a217700 1 -- 172.21.15.111:0/2562734508 wait complete. 2023-11-28T13:00:56.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.036+0000 7f1b2a217700 1 Processor -- start 2023-11-28T13:00:56.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.037+0000 7f1b2a217700 1 -- start start 2023-11-28T13:00:56.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.037+0000 7f1b2a217700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b240ca210 0x7f1b240cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.037+0000 7f1b2a217700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b240cc330 0x7f1b2415c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.037+0000 7f1b2a217700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b2415a6e0 0x7f1b240b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.037+0000 7f1b2a217700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2415cca0 con 0x7f1b2415a6e0 2023-11-28T13:00:56.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.037+0000 7f1b2a217700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2415cf80 con 0x7f1b240cc330 2023-11-28T13:00:56.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.037+0000 7f1b2a217700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b240aca40 con 0x7f1b240ca210 2023-11-28T13:00:56.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.037+0000 7f1b23fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b2415a6e0 0x7f1b240b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.037+0000 7f1b23fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b2415a6e0 0x7f1b240b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36118/0 (socket says 172.21.15.111:36118) 2023-11-28T13:00:56.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.037+0000 7f1b23fff700 1 -- 172.21.15.111:0/2376103295 learned_addr learned my addr 172.21.15.111:0/2376103295 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:56.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b237fe700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b240ca210 0x7f1b240cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b22ffd700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b240cc330 0x7f1b2415c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b23fff700 1 -- 172.21.15.111:0/2376103295 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b240ca210 msgr2=0x7f1b240cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b23fff700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b240ca210 0x7f1b240cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b23fff700 1 -- 172.21.15.111:0/2376103295 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b240cc330 msgr2=0x7f1b2415c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b23fff700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b240cc330 0x7f1b2415c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b23fff700 1 -- 172.21.15.111:0/2376103295 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b14009550 con 0x7f1b2415a6e0 2023-11-28T13:00:56.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b237fe700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b240ca210 0x7f1b240cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:56.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b22ffd700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b240cc330 0x7f1b2415c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:56.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b23fff700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b2415a6e0 0x7f1b240b4080 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00cc30 tx=0x7f1b1c00a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:56.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b0bfff700 1 -- 172.21.15.111:0/2376103295 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c00ab70 con 0x7f1b2415a6e0 2023-11-28T13:00:56.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b0bfff700 1 -- 172.21.15.111:0/2376103295 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1c00acd0 con 0x7f1b2415a6e0 2023-11-28T13:00:56.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b2a217700 1 -- 172.21.15.111:0/2376103295 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2406ad00 con 0x7f1b2415a6e0 2023-11-28T13:00:56.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b0bfff700 1 -- 172.21.15.111:0/2376103295 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c003c40 con 0x7f1b2415a6e0 2023-11-28T13:00:56.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.038+0000 7f1b2a217700 1 -- 172.21.15.111:0/2376103295 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2406aa20 con 0x7f1b2415a6e0 2023-11-28T13:00:56.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.039+0000 7f1b2a217700 1 -- 172.21.15.111:0/2376103295 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2406afe0 con 0x7f1b2415a6e0 2023-11-28T13:00:56.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.040+0000 7f1b0bfff700 1 -- 172.21.15.111:0/2376103295 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1b1c01f070 con 0x7f1b2415a6e0 2023-11-28T13:00:56.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.040+0000 7f1b0bfff700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b100682f0 0x7f1b1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.040+0000 7f1b0bfff700 1 -- 172.21.15.111:0/2376103295 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6282+0+0 (secure 0 0 0) 0x7f1b1c085e50 con 0x7f1b2415a6e0 2023-11-28T13:00:56.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.040+0000 7f1b237fe700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b100682f0 0x7f1b1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.040+0000 7f1b237fe700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b100682f0 0x7f1b1006a7a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1b18006fd0 tx=0x7f1b18008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:56.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.044+0000 7f1b0bfff700 1 -- 172.21.15.111:0/2376103295 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1b1c0525d0 con 0x7f1b2415a6e0 2023-11-28T13:00:56.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.270+0000 7f1b2a217700 1 -- 172.21.15.111:0/2376103295 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1b240acf30 con 0x7f1b2415a6e0 2023-11-28T13:00:56.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.276+0000 7f1b0bfff700 1 -- 172.21.15.111:0/2376103295 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7f1b1c055bf0 con 0x7f1b2415a6e0 2023-11-28T13:00:56.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.280+0000 7f1b2a217700 1 -- 172.21.15.111:0/2376103295 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b100682f0 msgr2=0x7f1b1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.280+0000 7f1b2a217700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b100682f0 0x7f1b1006a7a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1b18006fd0 tx=0x7f1b18008040 comp rx=0 tx=0).stop 2023-11-28T13:00:56.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.280+0000 7f1b2a217700 1 -- 172.21.15.111:0/2376103295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b2415a6e0 msgr2=0x7f1b240b4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.280+0000 7f1b2a217700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b2415a6e0 0x7f1b240b4080 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00cc30 tx=0x7f1b1c00a1f0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.281+0000 7f1b2a217700 1 -- 172.21.15.111:0/2376103295 shutdown_connections 2023-11-28T13:00:56.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.281+0000 7f1b2a217700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b2415a6e0 0x7f1b240b4080 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.281+0000 7f1b2a217700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b240ca210 0x7f1b240cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.281+0000 7f1b2a217700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b100682f0 0x7f1b1006a7a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.281+0000 7f1b2a217700 1 --2- 172.21.15.111:0/2376103295 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b240cc330 0x7f1b2415c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.281+0000 7f1b2a217700 1 -- 172.21.15.111:0/2376103295 >> 172.21.15.111:0/2376103295 conn(0x7f1b2406db10 msgr2=0x7f1b24158ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:56.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.281+0000 7f1b2a217700 1 -- 172.21.15.111:0/2376103295 shutdown_connections 2023-11-28T13:00:56.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.282+0000 7f1b2a217700 1 -- 172.21.15.111:0/2376103295 wait complete. 2023-11-28T13:00:56.286 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 8 and data pool 9 2023-11-28T13:00:56.308 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:00:56.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.427+0000 7fd7104ab700 1 Processor -- start 2023-11-28T13:00:56.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.427+0000 7fd7104ab700 1 -- start start 2023-11-28T13:00:56.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.428+0000 7fd7104ab700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd7080b0d70 0x7fd7081604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.428+0000 7fd7104ab700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7080b1140 0x7fd7080ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.428+0000 7fd7104ab700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70815cd90 con 0x7fd7080b15b0 2023-11-28T13:00:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.428+0000 7fd7104ab700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7081613e0 con 0x7fd7080b0d70 2023-11-28T13:00:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.428+0000 7fd7104ab700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd708160b40 con 0x7fd7080b1140 2023-11-28T13:00:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.429+0000 7fd70e247700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd7080b0d70 0x7fd7081604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.429+0000 7fd70e247700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd7080b0d70 0x7fd7081604b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36134/0 (socket says 172.21.15.111:36134) 2023-11-28T13:00:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.429+0000 7fd70e247700 1 -- 172.21.15.111:0/3751804989 learned_addr learned my addr 172.21.15.111:0/3751804989 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.429+0000 7fd70da46700 1 --2- 172.21.15.111:0/3751804989 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7080b1140 0x7fd7080ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.429+0000 7fd70e247700 1 -- 172.21.15.111:0/3751804989 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7080b1140 msgr2=0x7fd7080ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.429+0000 7fd70e247700 1 --2- 172.21.15.111:0/3751804989 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7080b1140 0x7fd7080ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.429+0000 7fd70e247700 1 -- 172.21.15.111:0/3751804989 >> v1:172.21.15.149:6789/0 conn(0x7fd7080b15b0 legacy=0x7fd7080cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.429+0000 7fd70e247700 1 -- 172.21.15.111:0/3751804989 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd708160e20 con 0x7fd7080b0d70 2023-11-28T13:00:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.429+0000 7fd70da46700 1 --2- 172.21.15.111:0/3751804989 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7080b1140 0x7fd7080ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.429+0000 7fd70e247700 1 --2- 172.21.15.111:0/3751804989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080b0d70 0x7fd7081604b0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8006fd0 tx=0x7fd6f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3545c4e01bc092 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.429+0000 7fd70d245700 1 -- 172.21.15.111:0/3751804989 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f8005e30 con 0x7fd7080b0d70 2023-11-28T13:00:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.429+0000 7fd70d245700 1 -- 172.21.15.111:0/3751804989 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f8004030 con 0x7fd7080b0d70 2023-11-28T13:00:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.429+0000 7fd70d245700 1 -- 172.21.15.111:0/3751804989 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f80043e0 con 0x7fd7080b0d70 2023-11-28T13:00:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.430+0000 7fd7104ab700 1 -- 172.21.15.111:0/3751804989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080b0d70 msgr2=0x7fd7081604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.430+0000 7fd7104ab700 1 --2- 172.21.15.111:0/3751804989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080b0d70 0x7fd7081604b0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8006fd0 tx=0x7fd6f8009bd0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.430+0000 7fd7104ab700 1 -- 172.21.15.111:0/3751804989 shutdown_connections 2023-11-28T13:00:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.430+0000 7fd7104ab700 1 --2- 172.21.15.111:0/3751804989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080b0d70 0x7fd7081604b0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.430+0000 7fd7104ab700 1 --2- 172.21.15.111:0/3751804989 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7080b1140 0x7fd7080ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.430+0000 7fd7104ab700 1 -- 172.21.15.111:0/3751804989 >> 172.21.15.111:0/3751804989 conn(0x7fd708098ee0 msgr2=0x7fd70806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.430+0000 7fd7104ab700 1 -- 172.21.15.111:0/3751804989 shutdown_connections 2023-11-28T13:00:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.430+0000 7fd7104ab700 1 -- 172.21.15.111:0/3751804989 wait complete. 2023-11-28T13:00:56.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.431+0000 7fd7104ab700 1 Processor -- start 2023-11-28T13:00:56.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.431+0000 7fd7104ab700 1 -- start start 2023-11-28T13:00:56.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd7104ab700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7080b0d70 0x7fd7080c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd7104ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080b1140 0x7fd708155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd7104ab700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7080b15b0 0x7fd708155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd7104ab700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7081e6230 con 0x7fd7080b1140 2023-11-28T13:00:56.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd7104ab700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7080c4d10 con 0x7fd7080b15b0 2023-11-28T13:00:56.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd7104ab700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7080c4ff0 con 0x7fd7080b0d70 2023-11-28T13:00:56.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd70da46700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080b1140 0x7fd708155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd70da46700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080b1140 0x7fd708155fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36142/0 (socket says 172.21.15.111:36142) 2023-11-28T13:00:56.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd70da46700 1 -- 172.21.15.111:0/1033874899 learned_addr learned my addr 172.21.15.111:0/1033874899 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:56.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd70e247700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7080b0d70 0x7fd7080c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd70ea48700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7080b15b0 0x7fd708155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd70da46700 1 -- 172.21.15.111:0/1033874899 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7080b0d70 msgr2=0x7fd7080c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd70da46700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7080b0d70 0x7fd7080c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd70da46700 1 -- 172.21.15.111:0/1033874899 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7080b15b0 msgr2=0x7fd708155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd70da46700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7080b15b0 0x7fd708155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd70da46700 1 -- 172.21.15.111:0/1033874899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f8009500 con 0x7fd7080b1140 2023-11-28T13:00:56.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd70ea48700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7080b15b0 0x7fd708155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:56.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd70da46700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080b1140 0x7fd708155fd0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd70400c750 tx=0x7fd70400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:56.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd6feffd700 1 -- 172.21.15.111:0/1033874899 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd704011800 con 0x7fd7080b1140 2023-11-28T13:00:56.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd6feffd700 1 -- 172.21.15.111:0/1033874899 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd704011960 con 0x7fd7080b1140 2023-11-28T13:00:56.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd7104ab700 1 -- 172.21.15.111:0/1033874899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7080af340 con 0x7fd7080b1140 2023-11-28T13:00:56.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd6feffd700 1 -- 172.21.15.111:0/1033874899 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd704011cd0 con 0x7fd7080b1140 2023-11-28T13:00:56.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.432+0000 7fd7104ab700 1 -- 172.21.15.111:0/1033874899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd708161100 con 0x7fd7080b1140 2023-11-28T13:00:56.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.433+0000 7fd7104ab700 1 -- 172.21.15.111:0/1033874899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7080cec00 con 0x7fd7080b1140 2023-11-28T13:00:56.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.434+0000 7fd6feffd700 1 -- 172.21.15.111:0/1033874899 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd704020070 con 0x7fd7080b1140 2023-11-28T13:00:56.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.434+0000 7fd6feffd700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6ec0682f0 0x7fd6ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.434+0000 7fd70e247700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6ec0682f0 0x7fd6ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.434+0000 7fd6feffd700 1 -- 172.21.15.111:0/1033874899 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fd704085c00 con 0x7fd7080b1140 2023-11-28T13:00:56.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.435+0000 7fd70e247700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6ec0682f0 0x7fd6ec06a7a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8009c10 tx=0x7fd6f8008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:56.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.439+0000 7fd6feffd700 1 -- 172.21.15.111:0/1033874899 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd70408b020 con 0x7fd7080b1140 2023-11-28T13:00:56.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.624+0000 7fd7104ab700 1 -- 172.21.15.111:0/1033874899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7080c54e0 con 0x7fd7080b1140 2023-11-28T13:00:56.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.625+0000 7fd6feffd700 1 -- 172.21.15.111:0/1033874899 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+18241 (secure 0 0 0) 0x7fd7080c54e0 con 0x7fd7080b1140 2023-11-28T13:00:56.627 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:56.627 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":54,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:56.271962+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:53.965825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:55.242768+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:00:56.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.627+0000 7fd7104ab700 1 -- 172.21.15.111:0/1033874899 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6ec0682f0 msgr2=0x7fd6ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.627+0000 7fd7104ab700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6ec0682f0 0x7fd6ec06a7a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8009c10 tx=0x7fd6f8008070 comp rx=0 tx=0).stop 2023-11-28T13:00:56.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.627+0000 7fd7104ab700 1 -- 172.21.15.111:0/1033874899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080b1140 msgr2=0x7fd708155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.627+0000 7fd7104ab700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080b1140 0x7fd708155fd0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd70400c750 tx=0x7fd70400ca60 comp rx=0 tx=0).stop 2023-11-28T13:00:56.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.627+0000 7fd7104ab700 1 -- 172.21.15.111:0/1033874899 shutdown_connections 2023-11-28T13:00:56.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.627+0000 7fd7104ab700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd6ec0682f0 0x7fd6ec06a7a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.627+0000 7fd7104ab700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd7080b15b0 0x7fd708155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.627+0000 7fd7104ab700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd7080b0d70 0x7fd7080c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.627+0000 7fd7104ab700 1 --2- 172.21.15.111:0/1033874899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080b1140 0x7fd708155fd0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.627+0000 7fd7104ab700 1 -- 172.21.15.111:0/1033874899 >> 172.21.15.111:0/1033874899 conn(0x7fd708098ee0 msgr2=0x7fd70806f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:56.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.627+0000 7fd7104ab700 1 -- 172.21.15.111:0/1033874899 shutdown_connections 2023-11-28T13:00:56.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.627+0000 7fd7104ab700 1 -- 172.21.15.111:0/1033874899 wait complete. 2023-11-28T13:00:56.642 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:00:56.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.760+0000 7f0b386e2700 1 Processor -- start 2023-11-28T13:00:56.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.760+0000 7f0b386e2700 1 -- start start 2023-11-28T13:00:56.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b386e2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300d1d70 0x7f0b30167270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b386e2700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3006c790 con 0x7f0b300cd460 2023-11-28T13:00:56.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b386e2700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b300b88e0 con 0x7f0b300b8510 2023-11-28T13:00:56.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b386e2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300cdbd0 con 0x7f0b300d1d70 2023-11-28T13:00:56.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b35c7d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0b300cd460 0x7f0b3006c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36928/0 (socket says 172.21.15.111:36928) 2023-11-28T13:00:56.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b35c7d700 1 -- 172.21.15.111:0/2387320998 learned_addr learned my addr 172.21.15.111:0/2387320998 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:56.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805163388 0 0) 0x7f0b3006c790 con 0x7f0b300cd460 2023-11-28T13:00:56.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b1c0035b0 con 0x7f0b300cd460 2023-11-28T13:00:56.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b36c7f700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300d1d70 0x7f0b30167270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3123658512 0 0) 0x7f0b1c0035b0 con 0x7f0b300cd460 2023-11-28T13:00:56.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300d1d70 msgr2=0x7f0b30167270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:56.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300d1d70 0x7f0b30167270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 >> v1:172.21.15.149:6789/0 conn(0x7f0b300b8510 legacy=0x7f0b3006b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b300cd8f0 con 0x7f0b300cd460 2023-11-28T13:00:56.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f0b2c001670 con 0x7f0b300cd460 2023-11-28T13:00:56.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 >> v1:172.21.15.111:6789/0 conn(0x7f0b300cd460 legacy=0x7f0b3006c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300d1d70 0x7f0b1c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b1c003f70 0x7f0b1c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b1c004d90 0x7f0b1c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300cdbd0 con 0x7f0b1c003f70 2023-11-28T13:00:56.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c00ba40 con 0x7f0b1c004d90 2023-11-28T13:00:56.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c00bc00 con 0x7f0b300d1d70 2023-11-28T13:00:56.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b35c7d700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b1c003f70 0x7f0b1c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.761+0000 7f0b3647e700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300d1d70 0x7f0b1c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b36c7f700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b1c004d90 0x7f0b1c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b35c7d700 1 -- 172.21.15.111:0/2387320998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300d1d70 msgr2=0x7f0b1c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:56.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b35c7d700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300d1d70 0x7f0b1c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b35c7d700 1 -- 172.21.15.111:0/2387320998 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b1c004d90 msgr2=0x7f0b1c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:56.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b35c7d700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b1c004d90 0x7f0b1c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b35c7d700 1 -- 172.21.15.111:0/2387320998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c0035b0 con 0x7f0b1c003f70 2023-11-28T13:00:56.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b36c7f700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b1c004d90 0x7f0b1c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:56.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b3647e700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300d1d70 0x7f0b1c003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:56.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b35c7d700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b1c003f70 0x7f0b1c004850 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0b3008af10 tx=0x7f0b1c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f1da84a2bd50d7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:56.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c00cdd0 con 0x7f0b1c003f70 2023-11-28T13:00:56.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2c005960 con 0x7f0b1c003f70 2023-11-28T13:00:56.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b3547c700 1 -- 172.21.15.111:0/2387320998 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c005d10 con 0x7f0b1c003f70 2023-11-28T13:00:56.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b386e2700 1 -- 172.21.15.111:0/2387320998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b1c003f70 msgr2=0x7f0b1c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b386e2700 1 --2- 172.21.15.111:0/2387320998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b1c003f70 0x7f0b1c004850 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0b3008af10 tx=0x7f0b1c005880 comp rx=0 tx=0).stop 2023-11-28T13:00:56.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.762+0000 7f0b36c7f700 1 -- 172.21.15.111:0/2387320998 reap_dead start 2023-11-28T13:00:56.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.763+0000 7f0b386e2700 1 -- 172.21.15.111:0/2387320998 shutdown_connections 2023-11-28T13:00:56.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.763+0000 7f0b386e2700 1 -- 172.21.15.111:0/2387320998 >> 172.21.15.111:0/2387320998 conn(0x7f0b300719a0 msgr2=0x7f0b30071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:56.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.763+0000 7f0b386e2700 1 -- 172.21.15.111:0/2387320998 shutdown_connections 2023-11-28T13:00:56.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.763+0000 7f0b386e2700 1 -- 172.21.15.111:0/2387320998 wait complete. 2023-11-28T13:00:56.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.764+0000 7f0b386e2700 1 Processor -- start 2023-11-28T13:00:56.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.764+0000 7f0b386e2700 1 -- start start 2023-11-28T13:00:56.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.765+0000 7f0b386e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b1c003f70 0x7f0b30154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.765+0000 7f0b386e2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300b5cd0 0x7f0b300b6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.765+0000 7f0b386e2700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b300cc760 0x7f0b300ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.765+0000 7f0b386e2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300b6680 con 0x7f0b1c003f70 2023-11-28T13:00:56.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.765+0000 7f0b386e2700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300cd140 con 0x7f0b300cc760 2023-11-28T13:00:56.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.765+0000 7f0b386e2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300ca1e0 con 0x7f0b300b5cd0 2023-11-28T13:00:56.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.765+0000 7f0b3647e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b1c003f70 0x7f0b30154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.765+0000 7f0b3647e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b1c003f70 0x7f0b30154670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36160/0 (socket says 172.21.15.111:36160) 2023-11-28T13:00:56.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.765+0000 7f0b3647e700 1 -- 172.21.15.111:0/3409175440 learned_addr learned my addr 172.21.15.111:0/3409175440 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:56.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.765+0000 7f0b35c7d700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300b5cd0 0x7f0b300b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.765+0000 7f0b36c7f700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b300cc760 0x7f0b300ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.766+0000 7f0b3647e700 1 -- 172.21.15.111:0/3409175440 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300b5cd0 msgr2=0x7f0b300b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.766+0000 7f0b3647e700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300b5cd0 0x7f0b300b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.766+0000 7f0b3647e700 1 -- 172.21.15.111:0/3409175440 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b300cc760 msgr2=0x7f0b300ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.766+0000 7f0b3647e700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b300cc760 0x7f0b300ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.766+0000 7f0b3647e700 1 -- 172.21.15.111:0/3409175440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b300cfff0 con 0x7f0b1c003f70 2023-11-28T13:00:56.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.766+0000 7f0b35c7d700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300b5cd0 0x7f0b300b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:56.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.766+0000 7f0b36c7f700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b300cc760 0x7f0b300ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:00:56.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.766+0000 7f0b3647e700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b1c003f70 0x7f0b30154670 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400e800 tx=0x7f0b2400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:56.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.766+0000 7f0b22ffd700 1 -- 172.21.15.111:0/3409175440 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b240052e0 con 0x7f0b1c003f70 2023-11-28T13:00:56.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.766+0000 7f0b22ffd700 1 -- 172.21.15.111:0/3409175440 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b24005440 con 0x7f0b1c003f70 2023-11-28T13:00:56.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.766+0000 7f0b22ffd700 1 -- 172.21.15.111:0/3409175440 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b240038c0 con 0x7f0b1c003f70 2023-11-28T13:00:56.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.766+0000 7f0b386e2700 1 -- 172.21.15.111:0/3409175440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b300cfd10 con 0x7f0b1c003f70 2023-11-28T13:00:56.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.766+0000 7f0b386e2700 1 -- 172.21.15.111:0/3409175440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b300d29c0 con 0x7f0b1c003f70 2023-11-28T13:00:56.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.767+0000 7f0b386e2700 1 -- 172.21.15.111:0/3409175440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b30067880 con 0x7f0b1c003f70 2023-11-28T13:00:56.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.767+0000 7f0b22ffd700 1 -- 172.21.15.111:0/3409175440 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0b24003a20 con 0x7f0b1c003f70 2023-11-28T13:00:56.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.768+0000 7f0b22ffd700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0b140682f0 0x7f0b1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:56.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.768+0000 7f0b35c7d700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0b140682f0 0x7f0b1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:56.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.768+0000 7f0b22ffd700 1 -- 172.21.15.111:0/3409175440 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0b24085cf0 con 0x7f0b1c003f70 2023-11-28T13:00:56.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.768+0000 7f0b35c7d700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0b140682f0 0x7f0b1406a7a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0b300b6930 tx=0x7f0b2c007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:56.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.772+0000 7f0b22ffd700 1 -- 172.21.15.111:0/3409175440 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0b24052410 con 0x7f0b1c003f70 2023-11-28T13:00:56.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.960+0000 7f0b386e2700 1 -- 172.21.15.111:0/3409175440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b300d2420 con 0x7f0b1c003f70 2023-11-28T13:00:56.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.961+0000 7f0b22ffd700 1 -- 172.21.15.111:0/3409175440 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+18241 (secure 0 0 0) 0x7f0b24003cb0 con 0x7f0b1c003f70 2023-11-28T13:00:56.963 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:56.963 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":54,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:56.271962+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:53.965825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:55.242768+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:00:56.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.963+0000 7f0b386e2700 1 -- 172.21.15.111:0/3409175440 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0b140682f0 msgr2=0x7f0b1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.963+0000 7f0b386e2700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0b140682f0 0x7f0b1406a7a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0b300b6930 tx=0x7f0b2c007000 comp rx=0 tx=0).stop 2023-11-28T13:00:56.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.963+0000 7f0b386e2700 1 -- 172.21.15.111:0/3409175440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b1c003f70 msgr2=0x7f0b30154670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:56.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.963+0000 7f0b386e2700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b1c003f70 0x7f0b30154670 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400e800 tx=0x7f0b2400ebc0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.963+0000 7f0b386e2700 1 -- 172.21.15.111:0/3409175440 shutdown_connections 2023-11-28T13:00:56.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.963+0000 7f0b386e2700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0b140682f0 0x7f0b1406a7a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.963+0000 7f0b386e2700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b300cc760 0x7f0b300ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.963+0000 7f0b386e2700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b1c003f70 0x7f0b30154670 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.963+0000 7f0b386e2700 1 --2- 172.21.15.111:0/3409175440 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b300b5cd0 0x7f0b300b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:56.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.963+0000 7f0b386e2700 1 -- 172.21.15.111:0/3409175440 >> 172.21.15.111:0/3409175440 conn(0x7f0b300719a0 msgr2=0x7f0b300b3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:56.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.963+0000 7f0b386e2700 1 -- 172.21.15.111:0/3409175440 shutdown_connections 2023-11-28T13:00:56.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:56.963+0000 7f0b386e2700 1 -- 172.21.15.111:0/3409175440 wait complete. 2023-11-28T13:00:56.977 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T13:00:57.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.099+0000 7fbfaa952700 1 Processor -- start 2023-11-28T13:00:57.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.099+0000 7fbfaa952700 1 -- start start 2023-11-28T13:00:57.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfaa952700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fbfa4159960 0x7fbfa40b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:57.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfaa952700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfa40aecc0 0x7fbfa40cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:57.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfaa952700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa40ad420 con 0x7fbfa40cdbc0 2023-11-28T13:00:57.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfaa952700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa40ce040 con 0x7fbfa4159960 2023-11-28T13:00:57.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfaa952700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4159d30 con 0x7fbfa40aecc0 2023-11-28T13:00:57.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa8eef700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbfa40cdbc0 0x7fbfa40afa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36950/0 (socket says 172.21.15.111:36950) 2023-11-28T13:00:57.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa8eef700 1 -- 172.21.15.111:0/2057097487 learned_addr learned my addr 172.21.15.111:0/2057097487 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:57.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566064381 0 0) 0x7fbfa40ad420 con 0x7fbfa40cdbc0 2023-11-28T13:00:57.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf8c0035b0 con 0x7fbfa40cdbc0 2023-11-28T13:00:57.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa3fff700 1 --2- 172.21.15.111:0/2057097487 >> v2:172.21.15.149:3300/0 conn(0x7fbfa4159960 0x7fbfa40b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:57.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa37fe700 1 --2- 172.21.15.111:0/2057097487 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfa40aecc0 0x7fbfa40cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:57.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 940713125 0 0) 0x7fbf8c0035b0 con 0x7fbfa40cdbc0 2023-11-28T13:00:57.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfa40aecc0 msgr2=0x7fbfa40cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:57.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 --2- 172.21.15.111:0/2057097487 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfa40aecc0 0x7fbfa40cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:57.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 >> v2:172.21.15.149:3300/0 conn(0x7fbfa4159960 msgr2=0x7fbfa40b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:57.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 --2- 172.21.15.111:0/2057097487 >> v2:172.21.15.149:3300/0 conn(0x7fbfa4159960 0x7fbfa40b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:57.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa40ac810 con 0x7fbfa40cdbc0 2023-11-28T13:00:57.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fbf9c0019f0 con 0x7fbfa40cdbc0 2023-11-28T13:00:57.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 >> v1:172.21.15.111:6789/0 conn(0x7fbfa40cdbc0 legacy=0x7fbfa40afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:57.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 --2- 172.21.15.111:0/2057097487 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbf8c003a00 0x7fbf8c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:57.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 --2- 172.21.15.111:0/2057097487 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfa40aecc0 0x7fbf8c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:57.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 --2- 172.21.15.111:0/2057097487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf8c004d90 0x7fbf8c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:57.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa40ce040 con 0x7fbf8c004d90 2023-11-28T13:00:57.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa4159d30 con 0x7fbf8c003a00 2023-11-28T13:00:57.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.100+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c00ba10 con 0x7fbfa40aecc0 2023-11-28T13:00:57.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.101+0000 7fbfa8eef700 1 --2- 172.21.15.111:0/2057097487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf8c004d90 0x7fbf8c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:57.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.101+0000 7fbfa37fe700 1 --2- 172.21.15.111:0/2057097487 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfa40aecc0 0x7fbf8c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:57.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.101+0000 7fbfa3fff700 1 --2- 172.21.15.111:0/2057097487 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbf8c003a00 0x7fbf8c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:57.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.101+0000 7fbfa8eef700 1 -- 172.21.15.111:0/2057097487 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfa40aecc0 msgr2=0x7fbf8c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:57.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.101+0000 7fbfa8eef700 1 --2- 172.21.15.111:0/2057097487 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbfa40aecc0 0x7fbf8c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:57.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.101+0000 7fbfa8eef700 1 -- 172.21.15.111:0/2057097487 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbf8c003a00 msgr2=0x7fbf8c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:57.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.101+0000 7fbfa8eef700 1 --2- 172.21.15.111:0/2057097487 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbf8c003a00 0x7fbf8c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:57.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.101+0000 7fbfa8eef700 1 -- 172.21.15.111:0/2057097487 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c0035b0 con 0x7fbf8c004d90 2023-11-28T13:00:57.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.101+0000 7fbfa8eef700 1 --2- 172.21.15.111:0/2057097487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf8c004d90 0x7fbf8c005200 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00a8d0 tx=0x7fbf8c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36afb34737882490 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.101+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c00a340 con 0x7fbf8c004d90 2023-11-28T13:00:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.101+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c012600 con 0x7fbf8c004d90 2023-11-28T13:00:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.101+0000 7fbfa2ffd700 1 -- 172.21.15.111:0/2057097487 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c0129b0 con 0x7fbf8c004d90 2023-11-28T13:00:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.102+0000 7fbfaa952700 1 -- 172.21.15.111:0/2057097487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf8c004d90 msgr2=0x7fbf8c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.102+0000 7fbfaa952700 1 --2- 172.21.15.111:0/2057097487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbf8c004d90 0x7fbf8c005200 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00a8d0 tx=0x7fbf8c005880 comp rx=0 tx=0).stop 2023-11-28T13:00:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.102+0000 7fbfa8eef700 1 -- 172.21.15.111:0/2057097487 reap_dead start 2023-11-28T13:00:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.102+0000 7fbfaa952700 1 -- 172.21.15.111:0/2057097487 shutdown_connections 2023-11-28T13:00:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.102+0000 7fbfaa952700 1 -- 172.21.15.111:0/2057097487 >> 172.21.15.111:0/2057097487 conn(0x7fbfa4098ee0 msgr2=0x7fbfa406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.102+0000 7fbfaa952700 1 -- 172.21.15.111:0/2057097487 shutdown_connections 2023-11-28T13:00:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.102+0000 7fbfaa952700 1 -- 172.21.15.111:0/2057097487 wait complete. 2023-11-28T13:00:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.103+0000 7fbfaa952700 1 Processor -- start 2023-11-28T13:00:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.103+0000 7fbfaa952700 1 -- start start 2023-11-28T13:00:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfaa952700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbf8c004d90 0x7fbfa41500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfaa952700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbfa4152280 0x7fbfa40cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfaa952700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfa40c57c0 0x7fbfa40cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfaa952700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa40c5330 con 0x7fbfa40c57c0 2023-11-28T13:00:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfaa952700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa40cc340 con 0x7fbfa4152280 2023-11-28T13:00:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfaa952700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa40c4ce0 con 0x7fbf8c004d90 2023-11-28T13:00:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfa8eef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfa40c57c0 0x7fbfa40cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfa8eef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfa40c57c0 0x7fbfa40cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36184/0 (socket says 172.21.15.111:36184) 2023-11-28T13:00:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfa8eef700 1 -- 172.21.15.111:0/2673610979 learned_addr learned my addr 172.21.15.111:0/2673610979 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfa37fe700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbfa4152280 0x7fbfa40cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfa3fff700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbf8c004d90 0x7fbfa41500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfa8eef700 1 -- 172.21.15.111:0/2673610979 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbf8c004d90 msgr2=0x7fbfa41500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfa8eef700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbf8c004d90 0x7fbfa41500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfa8eef700 1 -- 172.21.15.111:0/2673610979 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbfa4152280 msgr2=0x7fbfa40cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfa8eef700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbfa4152280 0x7fbfa40cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfa8eef700 1 -- 172.21.15.111:0/2673610979 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa40acdd0 con 0x7fbfa40c57c0 2023-11-28T13:00:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.104+0000 7fbfa8eef700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfa40c57c0 0x7fbfa40cc6e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0046d0 tx=0x7fbf9c001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.105+0000 7fbfa0ff9700 1 -- 172.21.15.111:0/2673610979 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c00a340 con 0x7fbfa40c57c0 2023-11-28T13:00:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.105+0000 7fbfa0ff9700 1 -- 172.21.15.111:0/2673610979 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c0019c0 con 0x7fbfa40c57c0 2023-11-28T13:00:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.105+0000 7fbfaa952700 1 -- 172.21.15.111:0/2673610979 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa40acaf0 con 0x7fbfa40c57c0 2023-11-28T13:00:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.105+0000 7fbfa0ff9700 1 -- 172.21.15.111:0/2673610979 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c01e640 con 0x7fbfa40c57c0 2023-11-28T13:00:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.105+0000 7fbfaa952700 1 -- 172.21.15.111:0/2673610979 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa40ad0b0 con 0x7fbfa40c57c0 2023-11-28T13:00:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.106+0000 7fbfaa952700 1 -- 172.21.15.111:0/2673610979 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa40d0c90 con 0x7fbfa40c57c0 2023-11-28T13:00:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.106+0000 7fbfa0ff9700 1 -- 172.21.15.111:0/2673610979 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbf9c010070 con 0x7fbfa40c57c0 2023-11-28T13:00:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.106+0000 7fbfa0ff9700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbf900682f0 0x7fbf9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.106+0000 7fbfa0ff9700 1 -- 172.21.15.111:0/2673610979 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fbf9c087810 con 0x7fbfa40c57c0 2023-11-28T13:00:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.106+0000 7fbfa3fff700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbf900682f0 0x7fbf9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.107+0000 7fbfa3fff700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbf900682f0 0x7fbf9006a7a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbf98009c40 tx=0x7fbf980091c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.111+0000 7fbfa0ff9700 1 -- 172.21.15.111:0/2673610979 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbf9c053ff0 con 0x7fbfa40c57c0 2023-11-28T13:00:57.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:57.338+0000 7fbfaa952700 1 -- 172.21.15.111:0/2673610979 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbfa406c4a0 con 0x7fbfa40c57c0 2023-11-28T13:00:58.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.284+0000 7fbfa0ff9700 1 -- 172.21.15.111:0/2673610979 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbf9c0170b0 con 0x7fbfa40c57c0 2023-11-28T13:00:58.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.287+0000 7fbfaa952700 1 -- 172.21.15.111:0/2673610979 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbf900682f0 msgr2=0x7fbf9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:58.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.287+0000 7fbfaa952700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbf900682f0 0x7fbf9006a7a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbf98009c40 tx=0x7fbf980091c0 comp rx=0 tx=0).stop 2023-11-28T13:00:58.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.287+0000 7fbfaa952700 1 -- 172.21.15.111:0/2673610979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfa40c57c0 msgr2=0x7fbfa40cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:58.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.287+0000 7fbfaa952700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfa40c57c0 0x7fbfa40cc6e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0046d0 tx=0x7fbf9c001360 comp rx=0 tx=0).stop 2023-11-28T13:00:58.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.287+0000 7fbfaa952700 1 -- 172.21.15.111:0/2673610979 shutdown_connections 2023-11-28T13:00:58.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.287+0000 7fbfaa952700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfa40c57c0 0x7fbfa40cc6e0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:58.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.287+0000 7fbfaa952700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbf8c004d90 0x7fbfa41500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:58.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.287+0000 7fbfaa952700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbf900682f0 0x7fbf9006a7a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:58.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.287+0000 7fbfaa952700 1 --2- 172.21.15.111:0/2673610979 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbfa4152280 0x7fbfa40cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:58.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.287+0000 7fbfaa952700 1 -- 172.21.15.111:0/2673610979 >> 172.21.15.111:0/2673610979 conn(0x7fbfa4098ee0 msgr2=0x7fbfa406dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:58.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.287+0000 7fbfaa952700 1 -- 172.21.15.111:0/2673610979 shutdown_connections 2023-11-28T13:00:58.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.287+0000 7fbfaa952700 1 -- 172.21.15.111:0/2673610979 wait complete. 2023-11-28T13:00:58.303 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T13:00:58.303 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T13:00:58.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.430+0000 7ff9475b8700 1 Processor -- start 2023-11-28T13:00:58.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.430+0000 7ff9475b8700 1 -- start start 2023-11-28T13:00:58.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.431+0000 7ff9475b8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9400cdef0 0x7ff9400d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:58.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.431+0000 7ff9475b8700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff9400cc5a0 0x7ff9400ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:58.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.431+0000 7ff9475b8700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9400d05a0 con 0x7ff9400cfc90 2023-11-28T13:00:58.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.431+0000 7ff9475b8700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9400ca260 con 0x7ff9400cc5a0 2023-11-28T13:00:58.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.431+0000 7ff9475b8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9400ccaa0 con 0x7ff9400cdef0 2023-11-28T13:00:58.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.431+0000 7ff944b53700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9400cdef0 0x7ff9400d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:58.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.431+0000 7ff944b53700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9400cdef0 0x7ff9400d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:57938/0 (socket says 172.21.15.111:57938) 2023-11-28T13:00:58.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.431+0000 7ff944b53700 1 -- 172.21.15.111:0/1369560077 learned_addr learned my addr 172.21.15.111:0/1369560077 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:58.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.431+0000 7ff945b55700 1 --2- 172.21.15.111:0/1369560077 >> v2:172.21.15.149:3300/0 conn(0x7ff9400cc5a0 0x7ff9400ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:58.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.431+0000 7ff945354700 1 --1- 172.21.15.111:0/1369560077 >> v1:172.21.15.149:6789/0 conn(0x7ff9400cfc90 0x7ff9400cd7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:45508/0 (socket says 172.21.15.111:45508) 2023-11-28T13:00:58.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.432+0000 7ff944b53700 1 -- 172.21.15.111:0/1369560077 >> v2:172.21.15.149:3300/0 conn(0x7ff9400cc5a0 msgr2=0x7ff9400ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:58.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.432+0000 7ff944b53700 1 --2- 172.21.15.111:0/1369560077 >> v2:172.21.15.149:3300/0 conn(0x7ff9400cc5a0 0x7ff9400ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:58.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.432+0000 7ff944b53700 1 -- 172.21.15.111:0/1369560077 >> v1:172.21.15.149:6789/0 conn(0x7ff9400cfc90 legacy=0x7ff9400cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:58.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.432+0000 7ff944b53700 1 -- 172.21.15.111:0/1369560077 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9400ccd80 con 0x7ff9400cdef0 2023-11-28T13:00:58.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.432+0000 7ff945b55700 1 --2- 172.21.15.111:0/1369560077 >> v2:172.21.15.149:3300/0 conn(0x7ff9400cc5a0 0x7ff9400ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:00:58.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.432+0000 7ff944b53700 1 --2- 172.21.15.111:0/1369560077 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9400cdef0 0x7ff9400d0060 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00bb90 tx=0x7ff93c00bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72f922ddb8dd6c5a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:58.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.432+0000 7ff937fff700 1 -- 172.21.15.111:0/1369560077 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c00dd30 con 0x7ff9400cdef0 2023-11-28T13:00:58.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.432+0000 7ff937fff700 1 -- 172.21.15.111:0/1369560077 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93c00de90 con 0x7ff9400cdef0 2023-11-28T13:00:58.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.432+0000 7ff937fff700 1 -- 172.21.15.111:0/1369560077 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c010690 con 0x7ff9400cdef0 2023-11-28T13:00:58.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.433+0000 7ff9475b8700 1 -- 172.21.15.111:0/1369560077 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9400cdef0 msgr2=0x7ff9400d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:58.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.433+0000 7ff9475b8700 1 --2- 172.21.15.111:0/1369560077 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9400cdef0 0x7ff9400d0060 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00bb90 tx=0x7ff93c00bea0 comp rx=0 tx=0).stop 2023-11-28T13:00:58.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.433+0000 7ff9475b8700 1 -- 172.21.15.111:0/1369560077 shutdown_connections 2023-11-28T13:00:58.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.433+0000 7ff9475b8700 1 --2- 172.21.15.111:0/1369560077 >> v2:172.21.15.149:3300/0 conn(0x7ff9400cc5a0 0x7ff9400ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:58.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.433+0000 7ff9475b8700 1 --2- 172.21.15.111:0/1369560077 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9400cdef0 0x7ff9400d0060 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:58.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.433+0000 7ff9475b8700 1 -- 172.21.15.111:0/1369560077 >> 172.21.15.111:0/1369560077 conn(0x7ff940098ee0 msgr2=0x7ff94006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:58.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.433+0000 7ff9475b8700 1 -- 172.21.15.111:0/1369560077 shutdown_connections 2023-11-28T13:00:58.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.433+0000 7ff9475b8700 1 -- 172.21.15.111:0/1369560077 wait complete. 2023-11-28T13:00:58.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.434+0000 7ff9475b8700 1 Processor -- start 2023-11-28T13:00:58.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.434+0000 7ff9475b8700 1 -- start start 2023-11-28T13:00:58.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff9475b8700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9400cc5a0 0x7ff94015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:58.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff9475b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9400cdef0 0x7ff9400af450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:58.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff9475b8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9400cfc90 0x7ff9400b0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:58.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff9475b8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9401f5880 con 0x7ff9400cdef0 2023-11-28T13:00:58.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff9475b8700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9400af990 con 0x7ff9400cc5a0 2023-11-28T13:00:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff9475b8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9401e4500 con 0x7ff9400cfc90 2023-11-28T13:00:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff944b53700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9400cdef0 0x7ff9400af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff944b53700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9400cdef0 0x7ff9400af450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36194/0 (socket says 172.21.15.111:36194) 2023-11-28T13:00:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff944b53700 1 -- 172.21.15.111:0/1580180470 learned_addr learned my addr 172.21.15.111:0/1580180470 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff945354700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9400cc5a0 0x7ff94015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff945b55700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9400cfc90 0x7ff9400b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff944b53700 1 -- 172.21.15.111:0/1580180470 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9400cfc90 msgr2=0x7ff9400b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff944b53700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9400cfc90 0x7ff9400b0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff944b53700 1 -- 172.21.15.111:0/1580180470 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9400cc5a0 msgr2=0x7ff94015b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff944b53700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9400cc5a0 0x7ff94015b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff944b53700 1 -- 172.21.15.111:0/1580180470 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff938009a70 con 0x7ff9400cdef0 2023-11-28T13:00:58.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff944b53700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9400cdef0 0x7ff9400af450 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff93c000c00 tx=0x7ff93c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:58.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.435+0000 7ff935ffb700 1 -- 172.21.15.111:0/1580180470 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c003b30 con 0x7ff9400cdef0 2023-11-28T13:00:58.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.436+0000 7ff935ffb700 1 -- 172.21.15.111:0/1580180470 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93c003c90 con 0x7ff9400cdef0 2023-11-28T13:00:58.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.436+0000 7ff9475b8700 1 -- 172.21.15.111:0/1580180470 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff93c00b7f0 con 0x7ff9400cdef0 2023-11-28T13:00:58.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.436+0000 7ff935ffb700 1 -- 172.21.15.111:0/1580180470 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c010470 con 0x7ff9400cdef0 2023-11-28T13:00:58.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.436+0000 7ff9475b8700 1 -- 172.21.15.111:0/1580180470 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9400cd340 con 0x7ff9400cdef0 2023-11-28T13:00:58.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.437+0000 7ff9475b8700 1 -- 172.21.15.111:0/1580180470 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9400cd060 con 0x7ff9400cdef0 2023-11-28T13:00:58.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.437+0000 7ff935ffb700 1 -- 172.21.15.111:0/1580180470 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff93c019400 con 0x7ff9400cdef0 2023-11-28T13:00:58.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.437+0000 7ff935ffb700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9240682f0 0x7ff92406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:58.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.437+0000 7ff935ffb700 1 -- 172.21.15.111:0/1580180470 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff93c087250 con 0x7ff9400cdef0 2023-11-28T13:00:58.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.438+0000 7ff945354700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9240682f0 0x7ff92406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:58.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.438+0000 7ff945354700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9240682f0 0x7ff92406a7a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff930000f60 tx=0x7ff930007ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:58.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.442+0000 7ff935ffb700 1 -- 172.21.15.111:0/1580180470 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff93c053890 con 0x7ff9400cdef0 2023-11-28T13:00:58.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:58.668+0000 7ff9475b8700 1 -- 172.21.15.111:0/1580180470 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff940143280 con 0x7ff9400cdef0 2023-11-28T13:00:58.751 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:00:58.753 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:00:59.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.291+0000 7ff935ffb700 1 -- 172.21.15.111:0/1580180470 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff93c056eb0 con 0x7ff9400cdef0 2023-11-28T13:00:59.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.293+0000 7ff9475b8700 1 -- 172.21.15.111:0/1580180470 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9240682f0 msgr2=0x7ff92406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.293+0000 7ff9475b8700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9240682f0 0x7ff92406a7a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff930000f60 tx=0x7ff930007ed0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.293+0000 7ff9475b8700 1 -- 172.21.15.111:0/1580180470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9400cdef0 msgr2=0x7ff9400af450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.293+0000 7ff9475b8700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9400cdef0 0x7ff9400af450 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff93c000c00 tx=0x7ff93c004750 comp rx=0 tx=0).stop 2023-11-28T13:00:59.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.293+0000 7ff9475b8700 1 -- 172.21.15.111:0/1580180470 shutdown_connections 2023-11-28T13:00:59.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.293+0000 7ff9475b8700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff9400cfc90 0x7ff9400b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.293+0000 7ff9475b8700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff9240682f0 0x7ff92406a7a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.293+0000 7ff9475b8700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff9400cc5a0 0x7ff94015b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.293+0000 7ff9475b8700 1 --2- 172.21.15.111:0/1580180470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9400cdef0 0x7ff9400af450 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.293+0000 7ff9475b8700 1 -- 172.21.15.111:0/1580180470 >> 172.21.15.111:0/1580180470 conn(0x7ff940098ee0 msgr2=0x7ff9400cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.293+0000 7ff9475b8700 1 -- 172.21.15.111:0/1580180470 shutdown_connections 2023-11-28T13:00:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.293+0000 7ff9475b8700 1 -- 172.21.15.111:0/1580180470 wait complete. 2023-11-28T13:00:59.307 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T13:00:59.308 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:00:59.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.427+0000 7ff15c882700 1 Processor -- start 2023-11-28T13:00:59.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.427+0000 7ff15c882700 1 -- start start 2023-11-28T13:00:59.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.428+0000 7ff15c882700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff15815c5e0 0x7ff15806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.428+0000 7ff15c882700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15815c9b0 con 0x7ff15806b720 2023-11-28T13:00:59.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.428+0000 7ff15c882700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff158160e60 con 0x7ff15806e970 2023-11-28T13:00:59.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.428+0000 7ff15c882700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1580cce90 con 0x7ff15815c5e0 2023-11-28T13:00:59.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.428+0000 7ff155d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff15806b720 0x7ff15806a7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:36980/0 (socket says 172.21.15.111:36980) 2023-11-28T13:00:59.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.428+0000 7ff155d9b700 1 -- 172.21.15.111:0/4149108721 learned_addr learned my addr 172.21.15.111:0/4149108721 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:59.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.428+0000 7ff15659c700 1 --2- 172.21.15.111:0/4149108721 >> v2:172.21.15.149:3300/0 conn(0x7ff15815c5e0 0x7ff15806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.428+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135012106 0 0) 0x7ff15815c9b0 con 0x7ff15806b720 2023-11-28T13:00:59.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.428+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1400035b0 con 0x7ff15806b720 2023-11-28T13:00:59.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2181418738 0 0) 0x7ff1400035b0 con 0x7ff15806b720 2023-11-28T13:00:59.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 >> v2:172.21.15.149:3300/0 conn(0x7ff15815c5e0 msgr2=0x7ff15806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:59.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 --2- 172.21.15.111:0/4149108721 >> v2:172.21.15.149:3300/0 conn(0x7ff15815c5e0 0x7ff15806a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 >> v1:172.21.15.149:6789/0 conn(0x7ff15806e970 legacy=0x7ff15806ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1580cd0a0 con 0x7ff15806b720 2023-11-28T13:00:59.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7ff148001670 con 0x7ff15806b720 2023-11-28T13:00:59.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 >> v1:172.21.15.111:6789/0 conn(0x7ff15806b720 legacy=0x7ff15806a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 --2- 172.21.15.111:0/4149108721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff140003a00 0x7ff140003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 --2- 172.21.15.111:0/4149108721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff140004850 0x7ff140004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 --2- 172.21.15.111:0/4149108721 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff140005200 0x7ff14000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1580cce90 con 0x7ff140003a00 2023-11-28T13:00:59.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14000c080 con 0x7ff140005200 2023-11-28T13:00:59.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14000c240 con 0x7ff140004850 2023-11-28T13:00:59.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15659c700 1 --2- 172.21.15.111:0/4149108721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff140003a00 0x7ff140003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff156d9d700 1 --2- 172.21.15.111:0/4149108721 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff140005200 0x7ff14000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff155d9b700 1 --2- 172.21.15.111:0/4149108721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff140004850 0x7ff140004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15659c700 1 -- 172.21.15.111:0/4149108721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff140004850 msgr2=0x7ff140004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:59.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15659c700 1 --2- 172.21.15.111:0/4149108721 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff140004850 0x7ff140004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15659c700 1 -- 172.21.15.111:0/4149108721 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff140005200 msgr2=0x7ff14000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:59.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15659c700 1 --2- 172.21.15.111:0/4149108721 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff140005200 0x7ff14000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff156d9d700 1 -- 172.21.15.111:0/4149108721 reap_dead start 2023-11-28T13:00:59.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15659c700 1 -- 172.21.15.111:0/4149108721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1400035b0 con 0x7ff140003a00 2023-11-28T13:00:59.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15659c700 1 --2- 172.21.15.111:0/4149108721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff140003a00 0x7ff140003e10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff14c009b60 tx=0x7ff140005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5e431b599830e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:59.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c00e6c0 con 0x7ff140003a00 2023-11-28T13:00:59.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c00e820 con 0x7ff140003a00 2023-11-28T13:00:59.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.429+0000 7ff15559a700 1 -- 172.21.15.111:0/4149108721 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c00eb90 con 0x7ff140003a00 2023-11-28T13:00:59.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.430+0000 7ff15c882700 1 -- 172.21.15.111:0/4149108721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff140003a00 msgr2=0x7ff140003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.430+0000 7ff15c882700 1 --2- 172.21.15.111:0/4149108721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff140003a00 0x7ff140003e10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff14c009b60 tx=0x7ff140005880 comp rx=0 tx=0).stop 2023-11-28T13:00:59.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.430+0000 7ff15c882700 1 -- 172.21.15.111:0/4149108721 shutdown_connections 2023-11-28T13:00:59.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.430+0000 7ff15c882700 1 --2- 172.21.15.111:0/4149108721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff140003a00 0x7ff140003e10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.430+0000 7ff15c882700 1 -- 172.21.15.111:0/4149108721 >> 172.21.15.111:0/4149108721 conn(0x7ff1580b0db0 msgr2=0x7ff1580b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:59.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.430+0000 7ff15c882700 1 -- 172.21.15.111:0/4149108721 shutdown_connections 2023-11-28T13:00:59.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.430+0000 7ff15c882700 1 -- 172.21.15.111:0/4149108721 wait complete. 2023-11-28T13:00:59.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.431+0000 7ff15c882700 1 Processor -- start 2023-11-28T13:00:59.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.431+0000 7ff15c882700 1 -- start start 2023-11-28T13:00:59.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff15c882700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff140003a00 0x7ff1580b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff15c882700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff158150a20 0x7ff1580b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff15c882700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1581e7aa0 0x7ff1581e7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff15c882700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1581e8450 con 0x7ff1581e7aa0 2023-11-28T13:00:59.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff15c882700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1580b3350 con 0x7ff158150a20 2023-11-28T13:00:59.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff15c882700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1580b3630 con 0x7ff140003a00 2023-11-28T13:00:59.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff156d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1581e7aa0 0x7ff1581e7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff156d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1581e7aa0 0x7ff1581e7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36216/0 (socket says 172.21.15.111:36216) 2023-11-28T13:00:59.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff156d9d700 1 -- 172.21.15.111:0/853354181 learned_addr learned my addr 172.21.15.111:0/853354181 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:59.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff155d9b700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff158150a20 0x7ff1580b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff15659c700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff140003a00 0x7ff1580b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff156d9d700 1 -- 172.21.15.111:0/853354181 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff140003a00 msgr2=0x7ff1580b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff156d9d700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff140003a00 0x7ff1580b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff156d9d700 1 -- 172.21.15.111:0/853354181 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff158150a20 msgr2=0x7ff1580b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff156d9d700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff158150a20 0x7ff1580b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff156d9d700 1 -- 172.21.15.111:0/853354181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1580cd660 con 0x7ff1581e7aa0 2023-11-28T13:00:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.432+0000 7ff155d9b700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff158150a20 0x7ff1580b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.433+0000 7ff156d9d700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1581e7aa0 0x7ff1581e7f10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff1580b1920 tx=0x7ff15000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.433+0000 7ff146ffd700 1 -- 172.21.15.111:0/853354181 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff150003c10 con 0x7ff1581e7aa0 2023-11-28T13:00:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.433+0000 7ff146ffd700 1 -- 172.21.15.111:0/853354181 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff150003d90 con 0x7ff1581e7aa0 2023-11-28T13:00:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.433+0000 7ff15c882700 1 -- 172.21.15.111:0/853354181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1580cd380 con 0x7ff1581e7aa0 2023-11-28T13:00:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.433+0000 7ff146ffd700 1 -- 172.21.15.111:0/853354181 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff150005310 con 0x7ff1581e7aa0 2023-11-28T13:00:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.433+0000 7ff15c882700 1 -- 172.21.15.111:0/853354181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff158160af0 con 0x7ff1581e7aa0 2023-11-28T13:00:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.434+0000 7ff15c882700 1 -- 172.21.15.111:0/853354181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff158067170 con 0x7ff1581e7aa0 2023-11-28T13:00:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.434+0000 7ff146ffd700 1 -- 172.21.15.111:0/853354181 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff150005470 con 0x7ff1581e7aa0 2023-11-28T13:00:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.434+0000 7ff146ffd700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff1380682f0 0x7ff13806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.435+0000 7ff146ffd700 1 -- 172.21.15.111:0/853354181 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff150085f30 con 0x7ff1581e7aa0 2023-11-28T13:00:59.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.435+0000 7ff15659c700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff1380682f0 0x7ff13806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.435+0000 7ff15659c700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff1380682f0 0x7ff13806a7a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff14c0090b0 tx=0x7ff14c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:59.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.439+0000 7ff146ffd700 1 -- 172.21.15.111:0/853354181 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff150052650 con 0x7ff1581e7aa0 2023-11-28T13:00:59.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.663+0000 7ff15c882700 1 -- 172.21.15.111:0/853354181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1580b5240 con 0x7ff1581e7aa0 2023-11-28T13:00:59.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.663+0000 7ff146ffd700 1 -- 172.21.15.111:0/853354181 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4660 (secure 0 0 0) 0x7ff150055c70 con 0x7ff1581e7aa0 2023-11-28T13:00:59.665 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:59.665 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2072621958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2072621958},{"type":"v1","addr":"172.21.15.149:6834","nonce":2072621958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3348427170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3348427170},{"type":"v1","addr":"172.21.15.149:6835","nonce":3348427170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1745443298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1745443298},{"type":"v1","addr":"172.21.15.111:6836","nonce":1745443298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:56.272312+0000","modified":"2023-11-28T13:00:59.284265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5814":{"gid":5814,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/4146274267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":4146274267},{"type":"v1","addr":"172.21.15.111:6837","nonce":4146274267}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T13:00:59.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.665+0000 7ff15c882700 1 -- 172.21.15.111:0/853354181 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff1380682f0 msgr2=0x7ff13806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.665+0000 7ff15c882700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff1380682f0 0x7ff13806a7a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff14c0090b0 tx=0x7ff14c008030 comp rx=0 tx=0).stop 2023-11-28T13:00:59.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.665+0000 7ff15c882700 1 -- 172.21.15.111:0/853354181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1581e7aa0 msgr2=0x7ff1581e7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.665+0000 7ff15c882700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1581e7aa0 0x7ff1581e7f10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff1580b1920 tx=0x7ff15000ee00 comp rx=0 tx=0).stop 2023-11-28T13:00:59.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.665+0000 7ff15c882700 1 -- 172.21.15.111:0/853354181 shutdown_connections 2023-11-28T13:00:59.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.665+0000 7ff15c882700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1581e7aa0 0x7ff1581e7f10 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.665+0000 7ff15c882700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff140003a00 0x7ff1580b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.665+0000 7ff15c882700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff1380682f0 0x7ff13806a7a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.665+0000 7ff15c882700 1 --2- 172.21.15.111:0/853354181 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff158150a20 0x7ff1580b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.665+0000 7ff15c882700 1 -- 172.21.15.111:0/853354181 >> 172.21.15.111:0/853354181 conn(0x7ff1580b0db0 msgr2=0x7ff1580b0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:59.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.665+0000 7ff15c882700 1 -- 172.21.15.111:0/853354181 shutdown_connections 2023-11-28T13:00:59.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.665+0000 7ff15c882700 1 -- 172.21.15.111:0/853354181 wait complete. 2023-11-28T13:00:59.668 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 25 2023-11-28T13:00:59.679 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:00:59.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.797+0000 7f4edb1a1700 1 Processor -- start 2023-11-28T13:00:59.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.797+0000 7f4edb1a1700 1 -- start start 2023-11-28T13:00:59.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.798+0000 7f4edb1a1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ed406a270 0x7f4ed406a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.798+0000 7f4edb1a1700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4ed415caf0 0x7f4ed415cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.798+0000 7f4edb1a1700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed4160940 con 0x7f4ed4160dd0 2023-11-28T13:00:59.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.798+0000 7f4edb1a1700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed41600e0 con 0x7f4ed415caf0 2023-11-28T13:00:59.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.798+0000 7f4edb1a1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed406ab80 con 0x7f4ed406a270 2023-11-28T13:00:59.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.798+0000 7f4ed999e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4ed4160dd0 0x7f4ed40b17b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37000/0 (socket says 172.21.15.111:37000) 2023-11-28T13:00:59.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.798+0000 7f4ed999e700 1 -- 172.21.15.111:0/104100011 learned_addr learned my addr 172.21.15.111:0/104100011 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:00:59.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.798+0000 7f4eda19f700 1 --2- 172.21.15.111:0/104100011 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ed406a270 0x7f4ed406a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.798+0000 7f4eda9a0700 1 --2- 172.21.15.111:0/104100011 >> v2:172.21.15.149:3300/0 conn(0x7f4ed415caf0 0x7f4ed415cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.798+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208339634 0 0) 0x7f4ed4160940 con 0x7f4ed4160dd0 2023-11-28T13:00:59.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.798+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec00035b0 con 0x7f4ed4160dd0 2023-11-28T13:00:59.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 866833871 0 0) 0x7f4ec00035b0 con 0x7f4ed4160dd0 2023-11-28T13:00:59.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ed406a270 msgr2=0x7f4ed406a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:59.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 --2- 172.21.15.111:0/104100011 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ed406a270 0x7f4ed406a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 >> v2:172.21.15.149:3300/0 conn(0x7f4ed415caf0 msgr2=0x7f4ed415cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:59.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 --2- 172.21.15.111:0/104100011 >> v2:172.21.15.149:3300/0 conn(0x7f4ed415caf0 0x7f4ed415cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed40b2270 con 0x7f4ed4160dd0 2023-11-28T13:00:59.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f4ed0001670 con 0x7f4ed4160dd0 2023-11-28T13:00:59.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 >> v1:172.21.15.111:6789/0 conn(0x7f4ed4160dd0 legacy=0x7f4ed40b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 --2- 172.21.15.111:0/104100011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec0003a00 0x7f4ec0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 --2- 172.21.15.111:0/104100011 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ed406a270 0x7f4ec0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 --2- 172.21.15.111:0/104100011 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ec0004d90 0x7f4ec0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed41600e0 con 0x7f4ec0003a00 2023-11-28T13:00:59.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed406ab80 con 0x7f4ec0004d90 2023-11-28T13:00:59.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec000ba10 con 0x7f4ed406a270 2023-11-28T13:00:59.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4eda19f700 1 --2- 172.21.15.111:0/104100011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec0003a00 0x7f4ec0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed999e700 1 --2- 172.21.15.111:0/104100011 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ed406a270 0x7f4ec0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4eda9a0700 1 --2- 172.21.15.111:0/104100011 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ec0004d90 0x7f4ec0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4eda19f700 1 -- 172.21.15.111:0/104100011 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ed406a270 msgr2=0x7f4ec0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:59.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4eda19f700 1 --2- 172.21.15.111:0/104100011 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ed406a270 0x7f4ec0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4eda19f700 1 -- 172.21.15.111:0/104100011 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ec0004d90 msgr2=0x7f4ec0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:00:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4eda19f700 1 --2- 172.21.15.111:0/104100011 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ec0004d90 0x7f4ec0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4eda19f700 1 -- 172.21.15.111:0/104100011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec00035b0 con 0x7f4ec0003a00 2023-11-28T13:00:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4eda19f700 1 --2- 172.21.15.111:0/104100011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec0003a00 0x7f4ec0003e10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4009b60 tx=0x7f4ec0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38d511d3761a5e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec400e660 con 0x7f4ec0003a00 2023-11-28T13:00:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec400e7c0 con 0x7f4ec0003a00 2023-11-28T13:00:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.799+0000 7f4ed919d700 1 -- 172.21.15.111:0/104100011 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec400eb30 con 0x7f4ec0003a00 2023-11-28T13:00:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.800+0000 7f4edb1a1700 1 -- 172.21.15.111:0/104100011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec0003a00 msgr2=0x7f4ec0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.800+0000 7f4edb1a1700 1 --2- 172.21.15.111:0/104100011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ec0003a00 0x7f4ec0003e10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4009b60 tx=0x7f4ec0005880 comp rx=0 tx=0).stop 2023-11-28T13:00:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.800+0000 7f4eda9a0700 1 -- 172.21.15.111:0/104100011 reap_dead start 2023-11-28T13:00:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.800+0000 7f4edb1a1700 1 -- 172.21.15.111:0/104100011 shutdown_connections 2023-11-28T13:00:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.800+0000 7f4edb1a1700 1 -- 172.21.15.111:0/104100011 >> 172.21.15.111:0/104100011 conn(0x7f4ed406bfb0 msgr2=0x7f4ed406d950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:59.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.800+0000 7f4edb1a1700 1 -- 172.21.15.111:0/104100011 shutdown_connections 2023-11-28T13:00:59.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.800+0000 7f4edb1a1700 1 -- 172.21.15.111:0/104100011 wait complete. 2023-11-28T13:00:59.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.801+0000 7f4edb1a1700 1 Processor -- start 2023-11-28T13:00:59.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.801+0000 7f4edb1a1700 1 -- start start 2023-11-28T13:00:59.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4edb1a1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ed40b4090 0x7f4ed40c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4edb1a1700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ed40c4690 0x7f4ed41e88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4edb1a1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed40b4460 0x7f4ed4155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4edb1a1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed41e8e20 con 0x7f4ed40b4460 2023-11-28T13:00:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4edb1a1700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40c4c00 con 0x7f4ed40c4690 2023-11-28T13:00:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4edb1a1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40c4ee0 con 0x7f4ed40b4090 2023-11-28T13:00:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4eda9a0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed40b4460 0x7f4ed4155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4eda9a0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed40b4460 0x7f4ed4155480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36238/0 (socket says 172.21.15.111:36238) 2023-11-28T13:00:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4eda9a0700 1 -- 172.21.15.111:0/3857113165 learned_addr learned my addr 172.21.15.111:0/3857113165 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:00:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4eda19f700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ed40b4090 0x7f4ed40c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4ed999e700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ed40c4690 0x7f4ed41e88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4eda9a0700 1 -- 172.21.15.111:0/3857113165 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ed40b4090 msgr2=0x7f4ed40c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4eda9a0700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ed40b4090 0x7f4ed40c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4eda9a0700 1 -- 172.21.15.111:0/3857113165 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ed40c4690 msgr2=0x7f4ed41e88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4eda9a0700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ed40c4690 0x7f4ed41e88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.802+0000 7f4eda9a0700 1 -- 172.21.15.111:0/3857113165 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed41605d0 con 0x7f4ed40b4460 2023-11-28T13:00:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.803+0000 7f4ed999e700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ed40c4690 0x7f4ed41e88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:00:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.803+0000 7f4eda9a0700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed40b4460 0x7f4ed4155480 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4071640 tx=0x7f4ecc011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.803+0000 7f4ecaffd700 1 -- 172.21.15.111:0/3857113165 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc00e040 con 0x7f4ed40b4460 2023-11-28T13:00:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.803+0000 7f4ecaffd700 1 -- 172.21.15.111:0/3857113165 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ecc00f040 con 0x7f4ed40b4460 2023-11-28T13:00:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.803+0000 7f4edb1a1700 1 -- 172.21.15.111:0/3857113165 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed41602f0 con 0x7f4ed40b4460 2023-11-28T13:00:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.803+0000 7f4ecaffd700 1 -- 172.21.15.111:0/3857113165 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc00fd00 con 0x7f4ed40b4460 2023-11-28T13:00:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.803+0000 7f4edb1a1700 1 -- 172.21.15.111:0/3857113165 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed40cf660 con 0x7f4ed40b4460 2023-11-28T13:00:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.804+0000 7f4edb1a1700 1 -- 172.21.15.111:0/3857113165 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed404e620 con 0x7f4ed40b4460 2023-11-28T13:00:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.804+0000 7f4ecaffd700 1 -- 172.21.15.111:0/3857113165 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ecc022460 con 0x7f4ed40b4460 2023-11-28T13:00:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.804+0000 7f4ecaffd700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4eb80682f0 0x7f4eb806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:00:59.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.805+0000 7f4ecaffd700 1 -- 172.21.15.111:0/3857113165 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4ecc0863f0 con 0x7f4ed40b4460 2023-11-28T13:00:59.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.805+0000 7f4eda19f700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4eb80682f0 0x7f4eb806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:00:59.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.805+0000 7f4eda19f700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4eb80682f0 0x7f4eb806a7a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4009a80 tx=0x7f4ec4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:00:59.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.809+0000 7f4ecaffd700 1 -- 172.21.15.111:0/3857113165 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ecc052ab0 con 0x7f4ed40b4460 2023-11-28T13:00:59.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.992+0000 7f4edb1a1700 1 -- 172.21.15.111:0/3857113165 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ed406e370 con 0x7f4ed40b4460 2023-11-28T13:00:59.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.993+0000 7f4ecaffd700 1 -- 172.21.15.111:0/3857113165 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+18228 (secure 0 0 0) 0x7f4ecc0560d0 con 0x7f4ed40b4460 2023-11-28T13:00:59.995 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:00:59.995 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":55,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:57.274026+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:53.965825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:55.242768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:00:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.995+0000 7f4edb1a1700 1 -- 172.21.15.111:0/3857113165 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4eb80682f0 msgr2=0x7f4eb806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.995+0000 7f4edb1a1700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4eb80682f0 0x7f4eb806a7a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4009a80 tx=0x7f4ec4008030 comp rx=0 tx=0).stop 2023-11-28T13:00:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.995+0000 7f4edb1a1700 1 -- 172.21.15.111:0/3857113165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed40b4460 msgr2=0x7f4ed4155480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:00:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.995+0000 7f4edb1a1700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed40b4460 0x7f4ed4155480 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4071640 tx=0x7f4ecc011ed0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.995+0000 7f4edb1a1700 1 -- 172.21.15.111:0/3857113165 shutdown_connections 2023-11-28T13:00:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.995+0000 7f4edb1a1700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed40b4460 0x7f4ed4155480 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.995+0000 7f4edb1a1700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ed40b4090 0x7f4ed40c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.995+0000 7f4edb1a1700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4eb80682f0 0x7f4eb806a7a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.995+0000 7f4edb1a1700 1 --2- 172.21.15.111:0/3857113165 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ed40c4690 0x7f4ed41e88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:00:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.995+0000 7f4edb1a1700 1 -- 172.21.15.111:0/3857113165 >> 172.21.15.111:0/3857113165 conn(0x7f4ec0003a00 msgr2=0x7f4ed415c370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:00:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.995+0000 7f4edb1a1700 1 -- 172.21.15.111:0/3857113165 shutdown_connections 2023-11-28T13:00:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:00:59.995+0000 7f4edb1a1700 1 -- 172.21.15.111:0/3857113165 wait complete. 2023-11-28T13:01:00.009 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T13:01:00.009 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:01:00.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.127+0000 7f0edd81b700 1 Processor -- start 2023-11-28T13:01:00.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.127+0000 7f0edd81b700 1 -- start start 2023-11-28T13:01:00.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0edd81b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0ed815cbf0 0x7f0ed80b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0edd81b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0ed806ba90 0x7f0ed80ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0edd81b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed80cfdb0 con 0x7f0ed80b40a0 2023-11-28T13:01:00.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0edd81b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed80cc740 con 0x7f0ed806ba90 2023-11-28T13:01:00.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0edd81b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed80b35e0 con 0x7f0ed815cbf0 2023-11-28T13:01:00.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0ed67fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f0ed80b40a0 0x7f0ed806b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37020/0 (socket says 172.21.15.111:37020) 2023-11-28T13:01:00.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0ed67fc700 1 -- 172.21.15.111:0/3917565930 learned_addr learned my addr 172.21.15.111:0/3917565930 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:00.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0ed77fe700 1 --2- 172.21.15.111:0/3917565930 >> v2:172.21.15.111:3300/0 conn(0x7f0ed806ba90 0x7f0ed80ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0ed5ffb700 1 -- 172.21.15.111:0/3917565930 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304906894 0 0) 0x7f0ed80cfdb0 con 0x7f0ed80b40a0 2023-11-28T13:01:00.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0ed5ffb700 1 -- 172.21.15.111:0/3917565930 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec00035b0 con 0x7f0ed80b40a0 2023-11-28T13:01:00.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0ed6ffd700 1 --2- 172.21.15.111:0/3917565930 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0ed815cbf0 0x7f0ed80b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0ed77fe700 1 -- 172.21.15.111:0/3917565930 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0ed815cbf0 msgr2=0x7f0ed80b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:00.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0ed77fe700 1 --2- 172.21.15.111:0/3917565930 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0ed815cbf0 0x7f0ed80b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0ed77fe700 1 -- 172.21.15.111:0/3917565930 >> v1:172.21.15.111:6789/0 conn(0x7f0ed80b40a0 legacy=0x7f0ed806b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0ed77fe700 1 -- 172.21.15.111:0/3917565930 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed80cbe10 con 0x7f0ed806ba90 2023-11-28T13:01:00.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0ed6ffd700 1 --2- 172.21.15.111:0/3917565930 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0ed815cbf0 0x7f0ed80b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:00.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0ed77fe700 1 --2- 172.21.15.111:0/3917565930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed806ba90 0x7f0ed80ccbd0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0009d90 tx=0x7f0ed000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f3837cdeaf6fed server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:00.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.128+0000 7f0ed5ffb700 1 -- 172.21.15.111:0/3917565930 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed000e070 con 0x7f0ed806ba90 2023-11-28T13:01:00.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.129+0000 7f0ed5ffb700 1 -- 172.21.15.111:0/3917565930 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed000bb20 con 0x7f0ed806ba90 2023-11-28T13:01:00.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.129+0000 7f0ed5ffb700 1 -- 172.21.15.111:0/3917565930 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0012430 con 0x7f0ed806ba90 2023-11-28T13:01:00.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.129+0000 7f0edd81b700 1 -- 172.21.15.111:0/3917565930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed806ba90 msgr2=0x7f0ed80ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.129+0000 7f0edd81b700 1 --2- 172.21.15.111:0/3917565930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed806ba90 0x7f0ed80ccbd0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0009d90 tx=0x7f0ed000b330 comp rx=0 tx=0).stop 2023-11-28T13:01:00.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.129+0000 7f0edd81b700 1 -- 172.21.15.111:0/3917565930 shutdown_connections 2023-11-28T13:01:00.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.129+0000 7f0edd81b700 1 --2- 172.21.15.111:0/3917565930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed806ba90 0x7f0ed80ccbd0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.129+0000 7f0edd81b700 1 --2- 172.21.15.111:0/3917565930 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0ed815cbf0 0x7f0ed80b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.129+0000 7f0edd81b700 1 -- 172.21.15.111:0/3917565930 >> 172.21.15.111:0/3917565930 conn(0x7f0ed809a040 msgr2=0x7f0ed80d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:00.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.129+0000 7f0edd81b700 1 -- 172.21.15.111:0/3917565930 shutdown_connections 2023-11-28T13:01:00.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.129+0000 7f0edd81b700 1 -- 172.21.15.111:0/3917565930 wait complete. 2023-11-28T13:01:00.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.130+0000 7f0edd81b700 1 Processor -- start 2023-11-28T13:01:00.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.130+0000 7f0edd81b700 1 -- start start 2023-11-28T13:01:00.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.131+0000 7f0edd81b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0ed806ba90 0x7f0ed8159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.131+0000 7f0edd81b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed80b40a0 0x7f0ed80c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.131+0000 7f0edd81b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0ed815cbf0 0x7f0ed81e66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.131+0000 7f0edd81b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed815e500 con 0x7f0ed80b40a0 2023-11-28T13:01:00.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.131+0000 7f0edd81b700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed815e920 con 0x7f0ed806ba90 2023-11-28T13:01:00.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.131+0000 7f0edd81b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed815d7a0 con 0x7f0ed815cbf0 2023-11-28T13:01:00.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.131+0000 7f0ed67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed80b40a0 0x7f0ed80c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.131+0000 7f0ed67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed80b40a0 0x7f0ed80c6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36250/0 (socket says 172.21.15.111:36250) 2023-11-28T13:01:00.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.131+0000 7f0ed67fc700 1 -- 172.21.15.111:0/2967521134 learned_addr learned my addr 172.21.15.111:0/2967521134 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:00.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.131+0000 7f0ed6ffd700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0ed806ba90 0x7f0ed8159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.132+0000 7f0ed67fc700 1 -- 172.21.15.111:0/2967521134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0ed815cbf0 msgr2=0x7f0ed81e66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.132+0000 7f0ed77fe700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0ed815cbf0 0x7f0ed81e66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.132+0000 7f0ed67fc700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0ed815cbf0 0x7f0ed81e66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.132+0000 7f0ed67fc700 1 -- 172.21.15.111:0/2967521134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0ed806ba90 msgr2=0x7f0ed8159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.132+0000 7f0ed67fc700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0ed806ba90 0x7f0ed8159f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.132+0000 7f0ed67fc700 1 -- 172.21.15.111:0/2967521134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed0009a70 con 0x7f0ed80b40a0 2023-11-28T13:01:00.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.132+0000 7f0ed6ffd700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0ed806ba90 0x7f0ed8159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:00.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.132+0000 7f0ed67fc700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed80b40a0 0x7f0ed80c6ee0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800c730 tx=0x7f0ec800caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:00.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.132+0000 7f0ebffff700 1 -- 172.21.15.111:0/2967521134 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8005d70 con 0x7f0ed80b40a0 2023-11-28T13:01:00.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.132+0000 7f0ebffff700 1 -- 172.21.15.111:0/2967521134 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec80070d0 con 0x7f0ed80b40a0 2023-11-28T13:01:00.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.132+0000 7f0edd81b700 1 -- 172.21.15.111:0/2967521134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed80cc3d0 con 0x7f0ed80b40a0 2023-11-28T13:01:00.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.132+0000 7f0ebffff700 1 -- 172.21.15.111:0/2967521134 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8007440 con 0x7f0ed80b40a0 2023-11-28T13:01:00.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.132+0000 7f0edd81b700 1 -- 172.21.15.111:0/2967521134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed80cc0f0 con 0x7f0ed80b40a0 2023-11-28T13:01:00.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.133+0000 7f0edd81b700 1 -- 172.21.15.111:0/2967521134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed80d06f0 con 0x7f0ed80b40a0 2023-11-28T13:01:00.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.133+0000 7f0ebffff700 1 -- 172.21.15.111:0/2967521134 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0ec8019030 con 0x7f0ed80b40a0 2023-11-28T13:01:00.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.133+0000 7f0ebffff700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0ec40682f0 0x7f0ec406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.134+0000 7f0ebffff700 1 -- 172.21.15.111:0/2967521134 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0ec80850d0 con 0x7f0ed80b40a0 2023-11-28T13:01:00.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.134+0000 7f0ed6ffd700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0ec40682f0 0x7f0ec406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.134+0000 7f0ed6ffd700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0ec40682f0 0x7f0ec406a7a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc008910 tx=0x7f0ecc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:00.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.138+0000 7f0ebffff700 1 -- 172.21.15.111:0/2967521134 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0ec804b5e0 con 0x7f0ed80b40a0 2023-11-28T13:01:00.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.319+0000 7f0edd81b700 1 -- 172.21.15.111:0/2967521134 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0ed8067880 con 0x7f0ec40682f0 2023-11-28T13:01:00.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.327+0000 7f0ebffff700 1 -- 172.21.15.111:0/2967521134 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311595 (secure 0 0 0) 0x7f0ed8067880 con 0x7f0ec40682f0 2023-11-28T13:01:00.333 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:00.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.330+0000 7f0edd81b700 1 -- 172.21.15.111:0/2967521134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0ec40682f0 msgr2=0x7f0ec406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.330+0000 7f0edd81b700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0ec40682f0 0x7f0ec406a7a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc008910 tx=0x7f0ecc008040 comp rx=0 tx=0).stop 2023-11-28T13:01:00.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.330+0000 7f0edd81b700 1 -- 172.21.15.111:0/2967521134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed80b40a0 msgr2=0x7f0ed80c6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.330+0000 7f0edd81b700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed80b40a0 0x7f0ed80c6ee0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800c730 tx=0x7f0ec800caf0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.331+0000 7f0edd81b700 1 -- 172.21.15.111:0/2967521134 shutdown_connections 2023-11-28T13:01:00.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.331+0000 7f0edd81b700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0ed815cbf0 0x7f0ed81e66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.331+0000 7f0edd81b700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0ec40682f0 0x7f0ec406a7a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.331+0000 7f0edd81b700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0ed806ba90 0x7f0ed8159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.331+0000 7f0edd81b700 1 --2- 172.21.15.111:0/2967521134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ed80b40a0 0x7f0ed80c6ee0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.331+0000 7f0edd81b700 1 -- 172.21.15.111:0/2967521134 >> 172.21.15.111:0/2967521134 conn(0x7f0ed809a040 msgr2=0x7f0ed806ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.331+0000 7f0edd81b700 1 -- 172.21.15.111:0/2967521134 shutdown_connections 2023-11-28T13:01:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.331+0000 7f0edd81b700 1 -- 172.21.15.111:0/2967521134 wait complete. 2023-11-28T13:01:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:01:00.344 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2023-11-28T13:00:58.360644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":75256,"kb_used_data":6072,"kb_used_omap":0,"kb_used_meta":69120,"kb_avail":754899464,"statfs":{"total":773094113280,"available":773017051136,"internally_reserved":0,"allocated":6217728,"data_stored":3483868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":70778880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000774"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.304647+0000","last_change":"2023-11-28T13:00:56.297180+0000","last_active":"2023-11-28T13:00:56.304647+0000","last_peered":"2023-11-28T13:00:56.304647+0000","last_clean":"2023-11-28T13:00:56.304647+0000","last_became_active":"2023-11-28T13:00:56.296928+0000","last_became_peered":"2023-11-28T13:00:56.296928+0000","last_unstale":"2023-11-28T13:00:56.304647+0000","last_undegraded":"2023-11-28T13:00:56.304647+0000","last_fullsized":"2023-11-28T13:00:56.304647+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:06:40.264221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.297885+0000","last_change":"2023-11-28T13:00:56.293874+0000","last_active":"2023-11-28T13:00:56.297885+0000","last_peered":"2023-11-28T13:00:56.297885+0000","last_clean":"2023-11-28T13:00:56.297885+0000","last_became_active":"2023-11-28T13:00:56.293612+0000","last_became_peered":"2023-11-28T13:00:56.293612+0000","last_unstale":"2023-11-28T13:00:56.297885+0000","last_undegraded":"2023-11-28T13:00:56.297885+0000","last_fullsized":"2023-11-28T13:00:56.297885+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:41:30.521782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301641+0000","last_change":"2023-11-28T13:00:56.299702+0000","last_active":"2023-11-28T13:00:56.301641+0000","last_peered":"2023-11-28T13:00:56.301641+0000","last_clean":"2023-11-28T13:00:56.301641+0000","last_became_active":"2023-11-28T13:00:56.298684+0000","last_became_peered":"2023-11-28T13:00:56.298684+0000","last_unstale":"2023-11-28T13:00:56.301641+0000","last_undegraded":"2023-11-28T13:00:56.301641+0000","last_fullsized":"2023-11-28T13:00:56.301641+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:36.003632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.292127+0000","last_change":"2023-11-28T13:00:56.284480+0000","last_active":"2023-11-28T13:00:56.292127+0000","last_peered":"2023-11-28T13:00:56.292127+0000","last_clean":"2023-11-28T13:00:56.292127+0000","last_became_active":"2023-11-28T13:00:56.284161+0000","last_became_peered":"2023-11-28T13:00:56.284161+0000","last_unstale":"2023-11-28T13:00:56.292127+0000","last_undegraded":"2023-11-28T13:00:56.292127+0000","last_fullsized":"2023-11-28T13:00:56.292127+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:36.003632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.287312+0000","last_change":"2023-11-28T13:00:56.284274+0000","last_active":"2023-11-28T13:00:56.287312+0000","last_peered":"2023-11-28T13:00:56.287312+0000","last_clean":"2023-11-28T13:00:56.287312+0000","last_became_active":"2023-11-28T13:00:56.284042+0000","last_became_peered":"2023-11-28T13:00:56.284042+0000","last_unstale":"2023-11-28T13:00:56.287312+0000","last_undegraded":"2023-11-28T13:00:56.287312+0000","last_fullsized":"2023-11-28T13:00:56.287312+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:06:57.178129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306439+0000","last_change":"2023-11-28T13:00:56.304342+0000","last_active":"2023-11-28T13:00:56.306439+0000","last_peered":"2023-11-28T13:00:56.306439+0000","last_clean":"2023-11-28T13:00:56.306439+0000","last_became_active":"2023-11-28T13:00:56.304070+0000","last_became_peered":"2023-11-28T13:00:56.304070+0000","last_unstale":"2023-11-28T13:00:56.306439+0000","last_undegraded":"2023-11-28T13:00:56.306439+0000","last_fullsized":"2023-11-28T13:00:56.306439+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:18.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.295655+0000","last_change":"2023-11-28T13:00:56.284160+0000","last_active":"2023-11-28T13:00:56.295655+0000","last_peered":"2023-11-28T13:00:56.295655+0000","last_clean":"2023-11-28T13:00:56.295655+0000","last_became_active":"2023-11-28T13:00:56.283893+0000","last_became_peered":"2023-11-28T13:00:56.283893+0000","last_unstale":"2023-11-28T13:00:56.295655+0000","last_undegraded":"2023-11-28T13:00:56.295655+0000","last_fullsized":"2023-11-28T13:00:56.295655+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:35:22.914408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307678+0000","last_change":"2023-11-28T13:00:56.303857+0000","last_active":"2023-11-28T13:00:56.307678+0000","last_peered":"2023-11-28T13:00:56.307678+0000","last_clean":"2023-11-28T13:00:56.307678+0000","last_became_active":"2023-11-28T13:00:56.303308+0000","last_became_peered":"2023-11-28T13:00:56.303308+0000","last_unstale":"2023-11-28T13:00:56.307678+0000","last_undegraded":"2023-11-28T13:00:56.307678+0000","last_fullsized":"2023-11-28T13:00:56.307678+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:42:06.976808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.303888+0000","last_change":"2023-11-28T13:00:56.300909+0000","last_active":"2023-11-28T13:00:56.303888+0000","last_peered":"2023-11-28T13:00:56.303888+0000","last_clean":"2023-11-28T13:00:56.303888+0000","last_became_active":"2023-11-28T13:00:56.300658+0000","last_became_peered":"2023-11-28T13:00:56.300658+0000","last_unstale":"2023-11-28T13:00:56.303888+0000","last_undegraded":"2023-11-28T13:00:56.303888+0000","last_fullsized":"2023-11-28T13:00:56.303888+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:18.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.287343+0000","last_change":"2023-11-28T13:00:56.282305+0000","last_active":"2023-11-28T13:00:56.287343+0000","last_peered":"2023-11-28T13:00:56.287343+0000","last_clean":"2023-11-28T13:00:56.287343+0000","last_became_active":"2023-11-28T13:00:56.282022+0000","last_became_peered":"2023-11-28T13:00:56.282022+0000","last_unstale":"2023-11-28T13:00:56.287343+0000","last_undegraded":"2023-11-28T13:00:56.287343+0000","last_fullsized":"2023-11-28T13:00:56.287343+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:09:20.885653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298210+0000","last_change":"2023-11-28T13:00:56.295271+0000","last_active":"2023-11-28T13:00:56.298210+0000","last_peered":"2023-11-28T13:00:56.298210+0000","last_clean":"2023-11-28T13:00:56.298210+0000","last_became_active":"2023-11-28T13:00:56.294659+0000","last_became_peered":"2023-11-28T13:00:56.294659+0000","last_unstale":"2023-11-28T13:00:56.298210+0000","last_undegraded":"2023-11-28T13:00:56.298210+0000","last_fullsized":"2023-11-28T13:00:56.298210+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:36:18.740266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301250+0000","last_change":"2023-11-28T13:00:56.294715+0000","last_active":"2023-11-28T13:00:56.301250+0000","last_peered":"2023-11-28T13:00:56.301250+0000","last_clean":"2023-11-28T13:00:56.301250+0000","last_became_active":"2023-11-28T13:00:56.293637+0000","last_became_peered":"2023-11-28T13:00:56.293637+0000","last_unstale":"2023-11-28T13:00:56.301250+0000","last_undegraded":"2023-11-28T13:00:56.301250+0000","last_fullsized":"2023-11-28T13:00:56.301250+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:44:32.911023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.300878+0000","last_change":"2023-11-28T13:00:56.296998+0000","last_active":"2023-11-28T13:00:56.300878+0000","last_peered":"2023-11-28T13:00:56.300878+0000","last_clean":"2023-11-28T13:00:56.300878+0000","last_became_active":"2023-11-28T13:00:56.296705+0000","last_became_peered":"2023-11-28T13:00:56.296705+0000","last_unstale":"2023-11-28T13:00:56.300878+0000","last_undegraded":"2023-11-28T13:00:56.300878+0000","last_fullsized":"2023-11-28T13:00:56.300878+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:50:37.941307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.297636+0000","last_change":"2023-11-28T13:00:56.288478+0000","last_active":"2023-11-28T13:00:56.297636+0000","last_peered":"2023-11-28T13:00:56.297636+0000","last_clean":"2023-11-28T13:00:56.297636+0000","last_became_active":"2023-11-28T13:00:56.288201+0000","last_became_peered":"2023-11-28T13:00:56.288201+0000","last_unstale":"2023-11-28T13:00:56.297636+0000","last_undegraded":"2023-11-28T13:00:56.297636+0000","last_fullsized":"2023-11-28T13:00:56.297636+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:32:21.431054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298107+0000","last_change":"2023-11-28T13:00:56.291756+0000","last_active":"2023-11-28T13:00:56.298107+0000","last_peered":"2023-11-28T13:00:56.298107+0000","last_clean":"2023-11-28T13:00:56.298107+0000","last_became_active":"2023-11-28T13:00:56.287654+0000","last_became_peered":"2023-11-28T13:00:56.287654+0000","last_unstale":"2023-11-28T13:00:56.298107+0000","last_undegraded":"2023-11-28T13:00:56.298107+0000","last_fullsized":"2023-11-28T13:00:56.298107+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:12.468194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.304306+0000","last_change":"2023-11-28T13:00:56.298915+0000","last_active":"2023-11-28T13:00:56.304306+0000","last_peered":"2023-11-28T13:00:56.304306+0000","last_clean":"2023-11-28T13:00:56.304306+0000","last_became_active":"2023-11-28T13:00:56.298583+0000","last_became_peered":"2023-11-28T13:00:56.298583+0000","last_unstale":"2023-11-28T13:00:56.304306+0000","last_undegraded":"2023-11-28T13:00:56.304306+0000","last_fullsized":"2023-11-28T13:00:56.304306+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:28:49.012453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298031+0000","last_change":"2023-11-28T13:00:56.293009+0000","last_active":"2023-11-28T13:00:56.298031+0000","last_peered":"2023-11-28T13:00:56.298031+0000","last_clean":"2023-11-28T13:00:56.298031+0000","last_became_active":"2023-11-28T13:00:56.292771+0000","last_became_peered":"2023-11-28T13:00:56.292771+0000","last_unstale":"2023-11-28T13:00:56.298031+0000","last_undegraded":"2023-11-28T13:00:56.298031+0000","last_fullsized":"2023-11-28T13:00:56.298031+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:18:03.399413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.303701+0000","last_change":"2023-11-28T13:00:56.301047+0000","last_active":"2023-11-28T13:00:56.303701+0000","last_peered":"2023-11-28T13:00:56.303701+0000","last_clean":"2023-11-28T13:00:56.303701+0000","last_became_active":"2023-11-28T13:00:56.300756+0000","last_became_peered":"2023-11-28T13:00:56.300756+0000","last_unstale":"2023-11-28T13:00:56.303701+0000","last_undegraded":"2023-11-28T13:00:56.303701+0000","last_fullsized":"2023-11-28T13:00:56.303701+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:59:22.039571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.303114+0000","last_change":"2023-11-28T13:00:56.296639+0000","last_active":"2023-11-28T13:00:56.303114+0000","last_peered":"2023-11-28T13:00:56.303114+0000","last_clean":"2023-11-28T13:00:56.303114+0000","last_became_active":"2023-11-28T13:00:56.295056+0000","last_became_peered":"2023-11-28T13:00:56.295056+0000","last_unstale":"2023-11-28T13:00:56.303114+0000","last_undegraded":"2023-11-28T13:00:56.303114+0000","last_fullsized":"2023-11-28T13:00:56.303114+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:39:11.338916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307476+0000","last_change":"2023-11-28T13:00:56.303529+0000","last_active":"2023-11-28T13:00:56.307476+0000","last_peered":"2023-11-28T13:00:56.307476+0000","last_clean":"2023-11-28T13:00:56.307476+0000","last_became_active":"2023-11-28T13:00:56.302836+0000","last_became_peered":"2023-11-28T13:00:56.302836+0000","last_unstale":"2023-11-28T13:00:56.307476+0000","last_undegraded":"2023-11-28T13:00:56.307476+0000","last_fullsized":"2023-11-28T13:00:56.307476+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:24.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.300864+0000","last_change":"2023-11-28T13:00:56.296498+0000","last_active":"2023-11-28T13:00:56.300864+0000","last_peered":"2023-11-28T13:00:56.300864+0000","last_clean":"2023-11-28T13:00:56.300864+0000","last_became_active":"2023-11-28T13:00:56.295880+0000","last_became_peered":"2023-11-28T13:00:56.295880+0000","last_unstale":"2023-11-28T13:00:56.300864+0000","last_undegraded":"2023-11-28T13:00:56.300864+0000","last_fullsized":"2023-11-28T13:00:56.300864+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:23:36.199654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298869+0000","last_change":"2023-11-28T13:00:56.296062+0000","last_active":"2023-11-28T13:00:56.298869+0000","last_peered":"2023-11-28T13:00:56.298869+0000","last_clean":"2023-11-28T13:00:56.298869+0000","last_became_active":"2023-11-28T13:00:56.295653+0000","last_became_peered":"2023-11-28T13:00:56.295653+0000","last_unstale":"2023-11-28T13:00:56.298869+0000","last_undegraded":"2023-11-28T13:00:56.298869+0000","last_fullsized":"2023-11-28T13:00:56.298869+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:32:28.994903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306248+0000","last_change":"2023-11-28T13:00:56.303165+0000","last_active":"2023-11-28T13:00:56.306248+0000","last_peered":"2023-11-28T13:00:56.306248+0000","last_clean":"2023-11-28T13:00:56.306248+0000","last_became_active":"2023-11-28T13:00:56.302931+0000","last_became_peered":"2023-11-28T13:00:56.302931+0000","last_unstale":"2023-11-28T13:00:56.306248+0000","last_undegraded":"2023-11-28T13:00:56.306248+0000","last_fullsized":"2023-11-28T13:00:56.306248+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:58:47.885007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301113+0000","last_change":"2023-11-28T13:00:56.298226+0000","last_active":"2023-11-28T13:00:56.301113+0000","last_peered":"2023-11-28T13:00:56.301113+0000","last_clean":"2023-11-28T13:00:56.301113+0000","last_became_active":"2023-11-28T13:00:56.297953+0000","last_became_peered":"2023-11-28T13:00:56.297953+0000","last_unstale":"2023-11-28T13:00:56.301113+0000","last_undegraded":"2023-11-28T13:00:56.301113+0000","last_fullsized":"2023-11-28T13:00:56.301113+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:06:18.378950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.302402+0000","last_change":"2023-11-28T13:00:56.294478+0000","last_active":"2023-11-28T13:00:56.302402+0000","last_peered":"2023-11-28T13:00:56.302402+0000","last_clean":"2023-11-28T13:00:56.302402+0000","last_became_active":"2023-11-28T13:00:56.294167+0000","last_became_peered":"2023-11-28T13:00:56.294167+0000","last_unstale":"2023-11-28T13:00:56.302402+0000","last_undegraded":"2023-11-28T13:00:56.302402+0000","last_fullsized":"2023-11-28T13:00:56.302402+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:43:05.421035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.288957+0000","last_change":"2023-11-28T13:00:56.283786+0000","last_active":"2023-11-28T13:00:56.288957+0000","last_peered":"2023-11-28T13:00:56.288957+0000","last_clean":"2023-11-28T13:00:56.288957+0000","last_became_active":"2023-11-28T13:00:56.283148+0000","last_became_peered":"2023-11-28T13:00:56.283148+0000","last_unstale":"2023-11-28T13:00:56.288957+0000","last_undegraded":"2023-11-28T13:00:56.288957+0000","last_fullsized":"2023-11-28T13:00:56.288957+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:52.856174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301594+0000","last_change":"2023-11-28T13:00:56.298740+0000","last_active":"2023-11-28T13:00:56.301594+0000","last_peered":"2023-11-28T13:00:56.301594+0000","last_clean":"2023-11-28T13:00:56.301594+0000","last_became_active":"2023-11-28T13:00:56.298446+0000","last_became_peered":"2023-11-28T13:00:56.298446+0000","last_unstale":"2023-11-28T13:00:56.301594+0000","last_undegraded":"2023-11-28T13:00:56.301594+0000","last_fullsized":"2023-11-28T13:00:56.301594+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:12.468194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.294053+0000","last_change":"2023-11-28T13:00:56.286891+0000","last_active":"2023-11-28T13:00:56.294053+0000","last_peered":"2023-11-28T13:00:56.294053+0000","last_clean":"2023-11-28T13:00:56.294053+0000","last_became_active":"2023-11-28T13:00:56.286615+0000","last_became_peered":"2023-11-28T13:00:56.286615+0000","last_unstale":"2023-11-28T13:00:56.294053+0000","last_undegraded":"2023-11-28T13:00:56.294053+0000","last_fullsized":"2023-11-28T13:00:56.294053+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:17:23.028402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298193+0000","last_change":"2023-11-28T13:00:56.294982+0000","last_active":"2023-11-28T13:00:56.298193+0000","last_peered":"2023-11-28T13:00:56.298193+0000","last_clean":"2023-11-28T13:00:56.298193+0000","last_became_active":"2023-11-28T13:00:56.294744+0000","last_became_peered":"2023-11-28T13:00:56.294744+0000","last_unstale":"2023-11-28T13:00:56.298193+0000","last_undegraded":"2023-11-28T13:00:56.298193+0000","last_fullsized":"2023-11-28T13:00:56.298193+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:32:21.431054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301687+0000","last_change":"2023-11-28T13:00:56.300082+0000","last_active":"2023-11-28T13:00:56.301687+0000","last_peered":"2023-11-28T13:00:56.301687+0000","last_clean":"2023-11-28T13:00:56.301687+0000","last_became_active":"2023-11-28T13:00:56.299839+0000","last_became_peered":"2023-11-28T13:00:56.299839+0000","last_unstale":"2023-11-28T13:00:56.301687+0000","last_undegraded":"2023-11-28T13:00:56.301687+0000","last_fullsized":"2023-11-28T13:00:56.301687+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:23:58.317830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.296430+0000","last_change":"2023-11-28T13:00:56.289747+0000","last_active":"2023-11-28T13:00:56.296430+0000","last_peered":"2023-11-28T13:00:56.296430+0000","last_clean":"2023-11-28T13:00:56.296430+0000","last_became_active":"2023-11-28T13:00:56.289519+0000","last_became_peered":"2023-11-28T13:00:56.289519+0000","last_unstale":"2023-11-28T13:00:56.296430+0000","last_undegraded":"2023-11-28T13:00:56.296430+0000","last_fullsized":"2023-11-28T13:00:56.296430+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:18:51.253650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306141+0000","last_change":"2023-11-28T13:00:56.303230+0000","last_active":"2023-11-28T13:00:56.306141+0000","last_peered":"2023-11-28T13:00:56.306141+0000","last_clean":"2023-11-28T13:00:56.306141+0000","last_became_active":"2023-11-28T13:00:56.302976+0000","last_became_peered":"2023-11-28T13:00:56.302976+0000","last_unstale":"2023-11-28T13:00:56.306141+0000","last_undegraded":"2023-11-28T13:00:56.306141+0000","last_fullsized":"2023-11-28T13:00:56.306141+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:35:22.914408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.304810+0000","last_change":"2023-11-28T13:00:56.299762+0000","last_active":"2023-11-28T13:00:56.304810+0000","last_peered":"2023-11-28T13:00:56.304810+0000","last_clean":"2023-11-28T13:00:56.304810+0000","last_became_active":"2023-11-28T13:00:56.299538+0000","last_became_peered":"2023-11-28T13:00:56.299538+0000","last_unstale":"2023-11-28T13:00:56.304810+0000","last_undegraded":"2023-11-28T13:00:56.304810+0000","last_fullsized":"2023-11-28T13:00:56.304810+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:32:21.431054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"54'4","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.304014+0000","last_change":"2023-11-28T13:00:55.899907+0000","last_active":"2023-11-28T13:00:56.304014+0000","last_peered":"2023-11-28T13:00:56.304014+0000","last_clean":"2023-11-28T13:00:56.304014+0000","last_became_active":"2023-11-28T13:00:55.899661+0000","last_became_peered":"2023-11-28T13:00:55.899661+0000","last_unstale":"2023-11-28T13:00:56.304014+0000","last_undegraded":"2023-11-28T13:00:56.304014+0000","last_fullsized":"2023-11-28T13:00:56.304014+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:23:35.192962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.299197+0000","last_change":"2023-11-28T13:00:56.295845+0000","last_active":"2023-11-28T13:00:56.299197+0000","last_peered":"2023-11-28T13:00:56.299197+0000","last_clean":"2023-11-28T13:00:56.299197+0000","last_became_active":"2023-11-28T13:00:56.295353+0000","last_became_peered":"2023-11-28T13:00:56.295353+0000","last_unstale":"2023-11-28T13:00:56.299197+0000","last_undegraded":"2023-11-28T13:00:56.299197+0000","last_fullsized":"2023-11-28T13:00:56.299197+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:55.996169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.302871+0000","last_change":"2023-11-28T13:00:55.903352+0000","last_active":"2023-11-28T13:00:56.302871+0000","last_peered":"2023-11-28T13:00:56.302871+0000","last_clean":"2023-11-28T13:00:56.302871+0000","last_became_active":"2023-11-28T13:00:55.903082+0000","last_became_peered":"2023-11-28T13:00:55.903082+0000","last_unstale":"2023-11-28T13:00:56.302871+0000","last_undegraded":"2023-11-28T13:00:56.302871+0000","last_fullsized":"2023-11-28T13:00:56.302871+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:14:28.102186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.303983+0000","last_change":"2023-11-28T13:00:56.300594+0000","last_active":"2023-11-28T13:00:56.303983+0000","last_peered":"2023-11-28T13:00:56.303983+0000","last_clean":"2023-11-28T13:00:56.303983+0000","last_became_active":"2023-11-28T13:00:56.300350+0000","last_became_peered":"2023-11-28T13:00:56.300350+0000","last_unstale":"2023-11-28T13:00:56.303983+0000","last_undegraded":"2023-11-28T13:00:56.303983+0000","last_fullsized":"2023-11-28T13:00:56.303983+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:23:19.401044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.300391+0000","last_change":"2023-11-28T13:00:55.898990+0000","last_active":"2023-11-28T13:00:56.300391+0000","last_peered":"2023-11-28T13:00:56.300391+0000","last_clean":"2023-11-28T13:00:56.300391+0000","last_became_active":"2023-11-28T13:00:55.898683+0000","last_became_peered":"2023-11-28T13:00:55.898683+0000","last_unstale":"2023-11-28T13:00:56.300391+0000","last_undegraded":"2023-11-28T13:00:56.300391+0000","last_fullsized":"2023-11-28T13:00:56.300391+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:51.849483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306176+0000","last_change":"2023-11-28T13:00:56.303501+0000","last_active":"2023-11-28T13:00:56.306176+0000","last_peered":"2023-11-28T13:00:56.306176+0000","last_clean":"2023-11-28T13:00:56.306176+0000","last_became_active":"2023-11-28T13:00:56.303098+0000","last_became_peered":"2023-11-28T13:00:56.303098+0000","last_unstale":"2023-11-28T13:00:56.306176+0000","last_undegraded":"2023-11-28T13:00:56.306176+0000","last_fullsized":"2023-11-28T13:00:56.306176+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:24.340085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.295384+0000","last_change":"2023-11-28T13:00:55.900218+0000","last_active":"2023-11-28T13:00:56.295384+0000","last_peered":"2023-11-28T13:00:56.295384+0000","last_clean":"2023-11-28T13:00:56.295384+0000","last_became_active":"2023-11-28T13:00:55.899924+0000","last_became_peered":"2023-11-28T13:00:55.899924+0000","last_unstale":"2023-11-28T13:00:56.295384+0000","last_undegraded":"2023-11-28T13:00:56.295384+0000","last_fullsized":"2023-11-28T13:00:56.295384+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:41:19.536972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298850+0000","last_change":"2023-11-28T13:00:56.295890+0000","last_active":"2023-11-28T13:00:56.298850+0000","last_peered":"2023-11-28T13:00:56.298850+0000","last_clean":"2023-11-28T13:00:56.298850+0000","last_became_active":"2023-11-28T13:00:56.295321+0000","last_became_peered":"2023-11-28T13:00:56.295321+0000","last_unstale":"2023-11-28T13:00:56.298850+0000","last_undegraded":"2023-11-28T13:00:56.298850+0000","last_fullsized":"2023-11-28T13:00:56.298850+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:12.468194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.295557+0000","last_change":"2023-11-28T13:00:55.895221+0000","last_active":"2023-11-28T13:00:56.295557+0000","last_peered":"2023-11-28T13:00:56.295557+0000","last_clean":"2023-11-28T13:00:56.295557+0000","last_became_active":"2023-11-28T13:00:55.894983+0000","last_became_peered":"2023-11-28T13:00:55.894983+0000","last_unstale":"2023-11-28T13:00:56.295557+0000","last_undegraded":"2023-11-28T13:00:56.295557+0000","last_fullsized":"2023-11-28T13:00:56.295557+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:44:31.904332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307712+0000","last_change":"2023-11-28T13:00:56.305738+0000","last_active":"2023-11-28T13:00:56.307712+0000","last_peered":"2023-11-28T13:00:56.307712+0000","last_clean":"2023-11-28T13:00:56.307712+0000","last_became_active":"2023-11-28T13:00:56.305034+0000","last_became_peered":"2023-11-28T13:00:56.305034+0000","last_unstale":"2023-11-28T13:00:56.307712+0000","last_undegraded":"2023-11-28T13:00:56.307712+0000","last_fullsized":"2023-11-28T13:00:56.307712+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:24.340085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298596+0000","last_change":"2023-11-28T13:00:55.897110+0000","last_active":"2023-11-28T13:00:56.298596+0000","last_peered":"2023-11-28T13:00:56.298596+0000","last_clean":"2023-11-28T13:00:56.298596+0000","last_became_active":"2023-11-28T13:00:55.896835+0000","last_became_peered":"2023-11-28T13:00:55.896835+0000","last_unstale":"2023-11-28T13:00:56.298596+0000","last_undegraded":"2023-11-28T13:00:56.298596+0000","last_fullsized":"2023-11-28T13:00:56.298596+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:24:52.300564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.297707+0000","last_change":"2023-11-28T13:00:56.294119+0000","last_active":"2023-11-28T13:00:56.297707+0000","last_peered":"2023-11-28T13:00:56.297707+0000","last_clean":"2023-11-28T13:00:56.297707+0000","last_became_active":"2023-11-28T13:00:56.293824+0000","last_became_peered":"2023-11-28T13:00:56.293824+0000","last_unstale":"2023-11-28T13:00:56.297707+0000","last_undegraded":"2023-11-28T13:00:56.297707+0000","last_fullsized":"2023-11-28T13:00:56.297707+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:09:20.885653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.276161+0000","last_change":"2023-11-28T13:00:55.896198+0000","last_active":"2023-11-28T13:00:56.276161+0000","last_peered":"2023-11-28T13:00:56.276161+0000","last_clean":"2023-11-28T13:00:56.276161+0000","last_became_active":"2023-11-28T13:00:55.895752+0000","last_became_peered":"2023-11-28T13:00:55.895752+0000","last_unstale":"2023-11-28T13:00:56.276161+0000","last_undegraded":"2023-11-28T13:00:56.276161+0000","last_fullsized":"2023-11-28T13:00:56.276161+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:50:36.934616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.297895+0000","last_change":"2023-11-28T13:00:56.293727+0000","last_active":"2023-11-28T13:00:56.297895+0000","last_peered":"2023-11-28T13:00:56.297895+0000","last_clean":"2023-11-28T13:00:56.297895+0000","last_became_active":"2023-11-28T13:00:56.293291+0000","last_became_peered":"2023-11-28T13:00:56.293291+0000","last_unstale":"2023-11-28T13:00:56.297895+0000","last_undegraded":"2023-11-28T13:00:56.297895+0000","last_fullsized":"2023-11-28T13:00:56.297895+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:35:22.914408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.281161+0000","last_change":"2023-11-28T13:00:55.903981+0000","last_active":"2023-11-28T13:00:56.281161+0000","last_peered":"2023-11-28T13:00:56.281161+0000","last_clean":"2023-11-28T13:00:56.281161+0000","last_became_active":"2023-11-28T13:00:55.903778+0000","last_became_peered":"2023-11-28T13:00:55.903778+0000","last_unstale":"2023-11-28T13:00:56.281161+0000","last_undegraded":"2023-11-28T13:00:56.281161+0000","last_fullsized":"2023-11-28T13:00:56.281161+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:59:21.032880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.297782+0000","last_change":"2023-11-28T13:00:56.288864+0000","last_active":"2023-11-28T13:00:56.297782+0000","last_peered":"2023-11-28T13:00:56.297782+0000","last_clean":"2023-11-28T13:00:56.297782+0000","last_became_active":"2023-11-28T13:00:56.288593+0000","last_became_peered":"2023-11-28T13:00:56.288593+0000","last_unstale":"2023-11-28T13:00:56.297782+0000","last_undegraded":"2023-11-28T13:00:56.297782+0000","last_fullsized":"2023-11-28T13:00:56.297782+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:24.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.290084+0000","last_change":"2023-11-28T13:00:55.897376+0000","last_active":"2023-11-28T13:00:56.290084+0000","last_peered":"2023-11-28T13:00:56.290084+0000","last_clean":"2023-11-28T13:00:56.290084+0000","last_became_active":"2023-11-28T13:00:55.897019+0000","last_became_peered":"2023-11-28T13:00:55.897019+0000","last_unstale":"2023-11-28T13:00:56.290084+0000","last_undegraded":"2023-11-28T13:00:56.290084+0000","last_fullsized":"2023-11-28T13:00:56.290084+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:28:25.712868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301654+0000","last_change":"2023-11-28T13:00:56.299867+0000","last_active":"2023-11-28T13:00:56.301654+0000","last_peered":"2023-11-28T13:00:56.301654+0000","last_clean":"2023-11-28T13:00:56.301654+0000","last_became_active":"2023-11-28T13:00:56.299508+0000","last_became_peered":"2023-11-28T13:00:56.299508+0000","last_unstale":"2023-11-28T13:00:56.301654+0000","last_undegraded":"2023-11-28T13:00:56.301654+0000","last_fullsized":"2023-11-28T13:00:56.301654+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:28:45.846972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.286197+0000","last_change":"2023-11-28T13:00:55.901780+0000","last_active":"2023-11-28T13:00:56.286197+0000","last_peered":"2023-11-28T13:00:56.286197+0000","last_clean":"2023-11-28T13:00:56.286197+0000","last_became_active":"2023-11-28T13:00:55.901535+0000","last_became_peered":"2023-11-28T13:00:55.901535+0000","last_unstale":"2023-11-28T13:00:56.286197+0000","last_undegraded":"2023-11-28T13:00:56.286197+0000","last_fullsized":"2023-11-28T13:00:56.286197+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:58:00.945059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.287734+0000","last_change":"2023-11-28T13:00:56.282982+0000","last_active":"2023-11-28T13:00:56.287734+0000","last_peered":"2023-11-28T13:00:56.287734+0000","last_clean":"2023-11-28T13:00:56.287734+0000","last_became_active":"2023-11-28T13:00:56.282709+0000","last_became_peered":"2023-11-28T13:00:56.282709+0000","last_unstale":"2023-11-28T13:00:56.287734+0000","last_undegraded":"2023-11-28T13:00:56.287734+0000","last_fullsized":"2023-11-28T13:00:56.287734+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:27:51.480157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"54'3","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301107+0000","last_change":"2023-11-28T13:00:55.900946+0000","last_active":"2023-11-28T13:00:56.301107+0000","last_peered":"2023-11-28T13:00:56.301107+0000","last_clean":"2023-11-28T13:00:56.301107+0000","last_became_active":"2023-11-28T13:00:55.900765+0000","last_became_peered":"2023-11-28T13:00:55.900765+0000","last_unstale":"2023-11-28T13:00:56.301107+0000","last_undegraded":"2023-11-28T13:00:56.301107+0000","last_fullsized":"2023-11-28T13:00:56.301107+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:36:17.733575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301689+0000","last_change":"2023-11-28T13:00:56.300059+0000","last_active":"2023-11-28T13:00:56.301689+0000","last_peered":"2023-11-28T13:00:56.301689+0000","last_clean":"2023-11-28T13:00:56.301689+0000","last_became_active":"2023-11-28T13:00:56.299451+0000","last_became_peered":"2023-11-28T13:00:56.299451+0000","last_unstale":"2023-11-28T13:00:56.301689+0000","last_undegraded":"2023-11-28T13:00:56.301689+0000","last_fullsized":"2023-11-28T13:00:56.301689+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:36:13.832348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"54'6","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307117+0000","last_change":"2023-11-28T13:00:55.893615+0000","last_active":"2023-11-28T13:00:56.307117+0000","last_peered":"2023-11-28T13:00:56.307117+0000","last_clean":"2023-11-28T13:00:56.307117+0000","last_became_active":"2023-11-28T13:00:55.893349+0000","last_became_peered":"2023-11-28T13:00:55.893349+0000","last_unstale":"2023-11-28T13:00:56.307117+0000","last_undegraded":"2023-11-28T13:00:56.307117+0000","last_fullsized":"2023-11-28T13:00:56.307117+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:14:35.121834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.288929+0000","last_change":"2023-11-28T13:00:56.283129+0000","last_active":"2023-11-28T13:00:56.288929+0000","last_peered":"2023-11-28T13:00:56.288929+0000","last_clean":"2023-11-28T13:00:56.288929+0000","last_became_active":"2023-11-28T13:00:56.282517+0000","last_became_peered":"2023-11-28T13:00:56.282517+0000","last_unstale":"2023-11-28T13:00:56.288929+0000","last_undegraded":"2023-11-28T13:00:56.288929+0000","last_fullsized":"2023-11-28T13:00:56.288929+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:28:45.846972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.305601+0000","last_change":"2023-11-28T13:00:55.903352+0000","last_active":"2023-11-28T13:00:56.305601+0000","last_peered":"2023-11-28T13:00:56.305601+0000","last_clean":"2023-11-28T13:00:56.305601+0000","last_became_active":"2023-11-28T13:00:55.903045+0000","last_became_peered":"2023-11-28T13:00:55.903045+0000","last_unstale":"2023-11-28T13:00:56.305601+0000","last_undegraded":"2023-11-28T13:00:56.305601+0000","last_fullsized":"2023-11-28T13:00:56.305601+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:24:52.300564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.300884+0000","last_change":"2023-11-28T13:00:56.296576+0000","last_active":"2023-11-28T13:00:56.300884+0000","last_peered":"2023-11-28T13:00:56.300884+0000","last_clean":"2023-11-28T13:00:56.300884+0000","last_became_active":"2023-11-28T13:00:56.296189+0000","last_became_peered":"2023-11-28T13:00:56.296189+0000","last_unstale":"2023-11-28T13:00:56.300884+0000","last_undegraded":"2023-11-28T13:00:56.300884+0000","last_fullsized":"2023-11-28T13:00:56.300884+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:03:44.012770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.278516+0000","last_change":"2023-11-28T13:00:55.890105+0000","last_active":"2023-11-28T13:00:56.278516+0000","last_peered":"2023-11-28T13:00:56.278516+0000","last_clean":"2023-11-28T13:00:56.278516+0000","last_became_active":"2023-11-28T13:00:55.889879+0000","last_became_peered":"2023-11-28T13:00:55.889879+0000","last_unstale":"2023-11-28T13:00:56.278516+0000","last_undegraded":"2023-11-28T13:00:56.278516+0000","last_fullsized":"2023-11-28T13:00:56.278516+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:54.989478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301349+0000","last_change":"2023-11-28T13:00:56.294879+0000","last_active":"2023-11-28T13:00:56.301349+0000","last_peered":"2023-11-28T13:00:56.301349+0000","last_clean":"2023-11-28T13:00:56.301349+0000","last_became_active":"2023-11-28T13:00:56.294470+0000","last_became_peered":"2023-11-28T13:00:56.294470+0000","last_unstale":"2023-11-28T13:00:56.301349+0000","last_undegraded":"2023-11-28T13:00:56.301349+0000","last_fullsized":"2023-11-28T13:00:56.301349+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:28:49.012453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.276259+0000","last_change":"2023-11-28T13:00:55.900211+0000","last_active":"2023-11-28T13:00:56.276259+0000","last_peered":"2023-11-28T13:00:56.276259+0000","last_clean":"2023-11-28T13:00:56.276259+0000","last_became_active":"2023-11-28T13:00:55.899604+0000","last_became_peered":"2023-11-28T13:00:55.899604+0000","last_unstale":"2023-11-28T13:00:56.276259+0000","last_undegraded":"2023-11-28T13:00:56.276259+0000","last_fullsized":"2023-11-28T13:00:56.276259+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:34:04.255852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306264+0000","last_change":"2023-11-28T13:00:56.303325+0000","last_active":"2023-11-28T13:00:56.306264+0000","last_peered":"2023-11-28T13:00:56.306264+0000","last_clean":"2023-11-28T13:00:56.306264+0000","last_became_active":"2023-11-28T13:00:56.302995+0000","last_became_peered":"2023-11-28T13:00:56.302995+0000","last_unstale":"2023-11-28T13:00:56.306264+0000","last_undegraded":"2023-11-28T13:00:56.306264+0000","last_fullsized":"2023-11-28T13:00:56.306264+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:52.856174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"54'8","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298557+0000","last_change":"2023-11-28T13:00:55.897412+0000","last_active":"2023-11-28T13:00:56.298557+0000","last_peered":"2023-11-28T13:00:56.298557+0000","last_clean":"2023-11-28T13:00:56.298557+0000","last_became_active":"2023-11-28T13:00:55.897122+0000","last_became_peered":"2023-11-28T13:00:55.897122+0000","last_unstale":"2023-11-28T13:00:56.298557+0000","last_undegraded":"2023-11-28T13:00:56.298557+0000","last_fullsized":"2023-11-28T13:00:56.298557+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:27:50.473466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.295509+0000","last_change":"2023-11-28T13:00:56.292966+0000","last_active":"2023-11-28T13:00:56.295509+0000","last_peered":"2023-11-28T13:00:56.295509+0000","last_clean":"2023-11-28T13:00:56.295509+0000","last_became_active":"2023-11-28T13:00:56.292666+0000","last_became_peered":"2023-11-28T13:00:56.292666+0000","last_unstale":"2023-11-28T13:00:56.295509+0000","last_undegraded":"2023-11-28T13:00:56.295509+0000","last_fullsized":"2023-11-28T13:00:56.295509+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:58:47.885007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.295699+0000","last_change":"2023-11-28T13:00:55.903819+0000","last_active":"2023-11-28T13:00:56.295699+0000","last_peered":"2023-11-28T13:00:56.295699+0000","last_clean":"2023-11-28T13:00:56.295699+0000","last_became_active":"2023-11-28T13:00:55.903549+0000","last_became_peered":"2023-11-28T13:00:55.903549+0000","last_unstale":"2023-11-28T13:00:56.295699+0000","last_undegraded":"2023-11-28T13:00:56.295699+0000","last_fullsized":"2023-11-28T13:00:56.295699+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:00:32.746429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301212+0000","last_change":"2023-11-28T13:00:56.295826+0000","last_active":"2023-11-28T13:00:56.301212+0000","last_peered":"2023-11-28T13:00:56.301212+0000","last_clean":"2023-11-28T13:00:56.301212+0000","last_became_active":"2023-11-28T13:00:56.295517+0000","last_became_peered":"2023-11-28T13:00:56.295517+0000","last_unstale":"2023-11-28T13:00:56.301212+0000","last_undegraded":"2023-11-28T13:00:56.301212+0000","last_fullsized":"2023-11-28T13:00:56.301212+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:59:22.039571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.293195+0000","last_change":"2023-11-28T13:00:55.897215+0000","last_active":"2023-11-28T13:00:56.293195+0000","last_peered":"2023-11-28T13:00:56.293195+0000","last_clean":"2023-11-28T13:00:56.293195+0000","last_became_active":"2023-11-28T13:00:55.896978+0000","last_became_peered":"2023-11-28T13:00:55.896978+0000","last_unstale":"2023-11-28T13:00:56.293195+0000","last_undegraded":"2023-11-28T13:00:56.293195+0000","last_fullsized":"2023-11-28T13:00:56.293195+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:14:28.102186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.287714+0000","last_change":"2023-11-28T13:00:56.282281+0000","last_active":"2023-11-28T13:00:56.287714+0000","last_peered":"2023-11-28T13:00:56.287714+0000","last_clean":"2023-11-28T13:00:56.287714+0000","last_became_active":"2023-11-28T13:00:56.281940+0000","last_became_peered":"2023-11-28T13:00:56.281940+0000","last_unstale":"2023-11-28T13:00:56.287714+0000","last_undegraded":"2023-11-28T13:00:56.287714+0000","last_fullsized":"2023-11-28T13:00:56.287714+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:53.232769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.278601+0000","last_change":"2023-11-28T13:00:55.888527+0000","last_active":"2023-11-28T13:00:56.278601+0000","last_peered":"2023-11-28T13:00:56.278601+0000","last_clean":"2023-11-28T13:00:56.278601+0000","last_became_active":"2023-11-28T13:00:55.888251+0000","last_became_peered":"2023-11-28T13:00:55.888251+0000","last_unstale":"2023-11-28T13:00:56.278601+0000","last_undegraded":"2023-11-28T13:00:56.278601+0000","last_fullsized":"2023-11-28T13:00:56.278601+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:13:05.532344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306157+0000","last_change":"2023-11-28T13:00:56.303346+0000","last_active":"2023-11-28T13:00:56.306157+0000","last_peered":"2023-11-28T13:00:56.306157+0000","last_clean":"2023-11-28T13:00:56.306157+0000","last_became_active":"2023-11-28T13:00:56.302770+0000","last_became_peered":"2023-11-28T13:00:56.302770+0000","last_unstale":"2023-11-28T13:00:56.306157+0000","last_undegraded":"2023-11-28T13:00:56.306157+0000","last_fullsized":"2023-11-28T13:00:56.306157+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:47:29.484502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.277048+0000","last_change":"2023-11-28T13:00:55.893693+0000","last_active":"2023-11-28T13:00:56.277048+0000","last_peered":"2023-11-28T13:00:56.277048+0000","last_clean":"2023-11-28T13:00:56.277048+0000","last_became_active":"2023-11-28T13:00:55.893432+0000","last_became_peered":"2023-11-28T13:00:55.893432+0000","last_unstale":"2023-11-28T13:00:56.277048+0000","last_undegraded":"2023-11-28T13:00:56.277048+0000","last_fullsized":"2023-11-28T13:00:56.277048+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:52:10.079823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307427+0000","last_change":"2023-11-28T13:00:56.303392+0000","last_active":"2023-11-28T13:00:56.307427+0000","last_peered":"2023-11-28T13:00:56.307427+0000","last_clean":"2023-11-28T13:00:56.307427+0000","last_became_active":"2023-11-28T13:00:56.303151+0000","last_became_peered":"2023-11-28T13:00:56.303151+0000","last_unstale":"2023-11-28T13:00:56.307427+0000","last_undegraded":"2023-11-28T13:00:56.307427+0000","last_fullsized":"2023-11-28T13:00:56.307427+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:18.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.278103+0000","last_change":"2023-11-28T13:00:55.893605+0000","last_active":"2023-11-28T13:00:56.278103+0000","last_peered":"2023-11-28T13:00:56.278103+0000","last_clean":"2023-11-28T13:00:56.278103+0000","last_became_active":"2023-11-28T13:00:55.893001+0000","last_became_peered":"2023-11-28T13:00:55.893001+0000","last_unstale":"2023-11-28T13:00:56.278103+0000","last_undegraded":"2023-11-28T13:00:56.278103+0000","last_fullsized":"2023-11-28T13:00:56.278103+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:39:10.332224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306286+0000","last_change":"2023-11-28T13:00:56.303351+0000","last_active":"2023-11-28T13:00:56.306286+0000","last_peered":"2023-11-28T13:00:56.306286+0000","last_clean":"2023-11-28T13:00:56.306286+0000","last_became_active":"2023-11-28T13:00:56.303049+0000","last_became_peered":"2023-11-28T13:00:56.303049+0000","last_unstale":"2023-11-28T13:00:56.306286+0000","last_undegraded":"2023-11-28T13:00:56.306286+0000","last_fullsized":"2023-11-28T13:00:56.306286+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:47:11.371831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.281308+0000","last_change":"2023-11-28T13:00:55.893633+0000","last_active":"2023-11-28T13:00:56.281308+0000","last_peered":"2023-11-28T13:00:56.281308+0000","last_clean":"2023-11-28T13:00:56.281308+0000","last_became_active":"2023-11-28T13:00:55.893360+0000","last_became_peered":"2023-11-28T13:00:55.893360+0000","last_unstale":"2023-11-28T13:00:56.281308+0000","last_undegraded":"2023-11-28T13:00:56.281308+0000","last_fullsized":"2023-11-28T13:00:56.281308+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:23:18.394353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.287253+0000","last_change":"2023-11-28T13:00:56.281619+0000","last_active":"2023-11-28T13:00:56.287253+0000","last_peered":"2023-11-28T13:00:56.287253+0000","last_clean":"2023-11-28T13:00:56.287253+0000","last_became_active":"2023-11-28T13:00:56.281365+0000","last_became_peered":"2023-11-28T13:00:56.281365+0000","last_unstale":"2023-11-28T13:00:56.287253+0000","last_undegraded":"2023-11-28T13:00:56.287253+0000","last_fullsized":"2023-11-28T13:00:56.287253+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:28:26.719559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.277957+0000","last_change":"2023-11-28T13:00:55.893618+0000","last_active":"2023-11-28T13:00:56.277957+0000","last_peered":"2023-11-28T13:00:56.277957+0000","last_clean":"2023-11-28T13:00:56.277957+0000","last_became_active":"2023-11-28T13:00:55.893287+0000","last_became_peered":"2023-11-28T13:00:55.893287+0000","last_unstale":"2023-11-28T13:00:56.277957+0000","last_undegraded":"2023-11-28T13:00:56.277957+0000","last_fullsized":"2023-11-28T13:00:56.277957+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:52:09.384071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.297874+0000","last_change":"2023-11-28T13:00:56.293551+0000","last_active":"2023-11-28T13:00:56.297874+0000","last_peered":"2023-11-28T13:00:56.297874+0000","last_clean":"2023-11-28T13:00:56.297874+0000","last_became_active":"2023-11-28T13:00:56.292953+0000","last_became_peered":"2023-11-28T13:00:56.292953+0000","last_unstale":"2023-11-28T13:00:56.297874+0000","last_undegraded":"2023-11-28T13:00:56.297874+0000","last_fullsized":"2023-11-28T13:00:56.297874+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:58:42.270054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.284790+0000","last_change":"2023-11-28T13:00:55.901793+0000","last_active":"2023-11-28T13:00:56.284790+0000","last_peered":"2023-11-28T13:00:56.284790+0000","last_clean":"2023-11-28T13:00:56.284790+0000","last_became_active":"2023-11-28T13:00:55.901511+0000","last_became_peered":"2023-11-28T13:00:55.901511+0000","last_unstale":"2023-11-28T13:00:56.284790+0000","last_undegraded":"2023-11-28T13:00:56.284790+0000","last_fullsized":"2023-11-28T13:00:56.284790+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:18:12.918193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.289469+0000","last_change":"2023-11-28T13:00:56.282854+0000","last_active":"2023-11-28T13:00:56.289469+0000","last_peered":"2023-11-28T13:00:56.289469+0000","last_clean":"2023-11-28T13:00:56.289469+0000","last_became_active":"2023-11-28T13:00:56.282485+0000","last_became_peered":"2023-11-28T13:00:56.282485+0000","last_unstale":"2023-11-28T13:00:56.289469+0000","last_undegraded":"2023-11-28T13:00:56.289469+0000","last_fullsized":"2023-11-28T13:00:56.289469+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:47:11.371831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.304163+0000","last_change":"2023-11-28T13:00:55.898981+0000","last_active":"2023-11-28T13:00:56.304163+0000","last_peered":"2023-11-28T13:00:56.304163+0000","last_clean":"2023-11-28T13:00:56.304163+0000","last_became_active":"2023-11-28T13:00:55.898727+0000","last_became_peered":"2023-11-28T13:00:55.898727+0000","last_unstale":"2023-11-28T13:00:56.304163+0000","last_undegraded":"2023-11-28T13:00:56.304163+0000","last_fullsized":"2023-11-28T13:00:56.304163+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:06.820051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":43,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.279371+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T13:00:56.279371+0000","last_peered":"2023-11-28T13:00:56.279371+0000","last_clean":"2023-11-28T13:00:56.279371+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:00:56.279371+0000","last_undegraded":"2023-11-28T13:00:56.279371+0000","last_fullsized":"2023-11-28T13:00:56.279371+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301607+0000","last_change":"2023-11-28T13:00:56.298759+0000","last_active":"2023-11-28T13:00:56.301607+0000","last_peered":"2023-11-28T13:00:56.301607+0000","last_clean":"2023-11-28T13:00:56.301607+0000","last_became_active":"2023-11-28T13:00:56.298469+0000","last_became_peered":"2023-11-28T13:00:56.298469+0000","last_unstale":"2023-11-28T13:00:56.301607+0000","last_undegraded":"2023-11-28T13:00:56.301607+0000","last_fullsized":"2023-11-28T13:00:56.301607+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:24.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.276120+0000","last_change":"2023-11-28T13:00:55.904336+0000","last_active":"2023-11-28T13:00:56.276120+0000","last_peered":"2023-11-28T13:00:56.276120+0000","last_clean":"2023-11-28T13:00:56.276120+0000","last_became_active":"2023-11-28T13:00:55.904145+0000","last_became_peered":"2023-11-28T13:00:55.904145+0000","last_unstale":"2023-11-28T13:00:56.276120+0000","last_undegraded":"2023-11-28T13:00:56.276120+0000","last_fullsized":"2023-11-28T13:00:56.276120+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:34:49.706762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.277876+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T13:00:56.277876+0000","last_peered":"2023-11-28T13:00:56.277876+0000","last_clean":"2023-11-28T13:00:56.277876+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:00:56.277876+0000","last_undegraded":"2023-11-28T13:00:56.277876+0000","last_fullsized":"2023-11-28T13:00:56.277876+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.288908+0000","last_change":"2023-11-28T13:00:56.283012+0000","last_active":"2023-11-28T13:00:56.288908+0000","last_peered":"2023-11-28T13:00:56.288908+0000","last_clean":"2023-11-28T13:00:56.288908+0000","last_became_active":"2023-11-28T13:00:56.282722+0000","last_became_peered":"2023-11-28T13:00:56.282722+0000","last_unstale":"2023-11-28T13:00:56.288908+0000","last_undegraded":"2023-11-28T13:00:56.288908+0000","last_fullsized":"2023-11-28T13:00:56.288908+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:36:13.832348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.305312+0000","last_change":"2023-11-28T13:00:55.902610+0000","last_active":"2023-11-28T13:00:56.305312+0000","last_peered":"2023-11-28T13:00:56.305312+0000","last_clean":"2023-11-28T13:00:56.305312+0000","last_became_active":"2023-11-28T13:00:55.902224+0000","last_became_peered":"2023-11-28T13:00:55.902224+0000","last_unstale":"2023-11-28T13:00:56.305312+0000","last_undegraded":"2023-11-28T13:00:56.305312+0000","last_fullsized":"2023-11-28T13:00:56.305312+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:46:36.832924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":39,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.280027+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T13:00:56.280027+0000","last_peered":"2023-11-28T13:00:56.280027+0000","last_clean":"2023-11-28T13:00:56.280027+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:00:56.280027+0000","last_undegraded":"2023-11-28T13:00:56.280027+0000","last_fullsized":"2023-11-28T13:00:56.280027+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307441+0000","last_change":"2023-11-28T13:00:56.303379+0000","last_active":"2023-11-28T13:00:56.307441+0000","last_peered":"2023-11-28T13:00:56.307441+0000","last_clean":"2023-11-28T13:00:56.307441+0000","last_became_active":"2023-11-28T13:00:56.302900+0000","last_became_peered":"2023-11-28T13:00:56.302900+0000","last_unstale":"2023-11-28T13:00:56.307441+0000","last_undegraded":"2023-11-28T13:00:56.307441+0000","last_fullsized":"2023-11-28T13:00:56.307441+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:08.749520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.277910+0000","last_change":"2023-11-28T13:00:55.896258+0000","last_active":"2023-11-28T13:00:56.277910+0000","last_peered":"2023-11-28T13:00:56.277910+0000","last_clean":"2023-11-28T13:00:56.277910+0000","last_became_active":"2023-11-28T13:00:55.895925+0000","last_became_peered":"2023-11-28T13:00:55.895925+0000","last_unstale":"2023-11-28T13:00:56.277910+0000","last_undegraded":"2023-11-28T13:00:56.277910+0000","last_fullsized":"2023-11-28T13:00:56.277910+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:46:39.890621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":40,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.276286+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T13:00:56.276286+0000","last_peered":"2023-11-28T13:00:56.276286+0000","last_clean":"2023-11-28T13:00:56.276286+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:00:56.276286+0000","last_undegraded":"2023-11-28T13:00:56.276286+0000","last_fullsized":"2023-11-28T13:00:56.276286+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":194,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-28T13:00:55.887106+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T13:00:55.887106+0000","last_peered":"2023-11-28T13:00:55.887106+0000","last_clean":"2023-11-28T13:00:55.887106+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:00:55.887106+0000","last_undegraded":"2023-11-28T13:00:55.887106+0000","last_fullsized":"2023-11-28T13:00:55.887106+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301503+0000","last_change":"2023-11-28T13:00:56.298014+0000","last_active":"2023-11-28T13:00:56.301503+0000","last_peered":"2023-11-28T13:00:56.301503+0000","last_clean":"2023-11-28T13:00:56.301503+0000","last_became_active":"2023-11-28T13:00:56.297776+0000","last_became_peered":"2023-11-28T13:00:56.297776+0000","last_unstale":"2023-11-28T13:00:56.301503+0000","last_undegraded":"2023-11-28T13:00:56.301503+0000","last_fullsized":"2023-11-28T13:00:56.301503+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:08.749520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.278906+0000","last_change":"2023-11-28T13:00:55.894723+0000","last_active":"2023-11-28T13:00:56.278906+0000","last_peered":"2023-11-28T13:00:56.278906+0000","last_clean":"2023-11-28T13:00:56.278906+0000","last_became_active":"2023-11-28T13:00:55.894507+0000","last_became_peered":"2023-11-28T13:00:55.894507+0000","last_unstale":"2023-11-28T13:00:56.278906+0000","last_undegraded":"2023-11-28T13:00:56.278906+0000","last_fullsized":"2023-11-28T13:00:56.278906+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:30:18.611882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-28T13:00:55.887195+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T13:00:55.887195+0000","last_peered":"2023-11-28T13:00:55.887195+0000","last_clean":"2023-11-28T13:00:55.887195+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:00:55.887195+0000","last_undegraded":"2023-11-28T13:00:55.887195+0000","last_fullsized":"2023-11-28T13:00:55.887195+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301561+0000","last_change":"2023-11-28T13:00:56.298856+0000","last_active":"2023-11-28T13:00:56.301561+0000","last_peered":"2023-11-28T13:00:56.301561+0000","last_clean":"2023-11-28T13:00:56.301561+0000","last_became_active":"2023-11-28T13:00:56.298533+0000","last_became_peered":"2023-11-28T13:00:56.298533+0000","last_unstale":"2023-11-28T13:00:56.301561+0000","last_undegraded":"2023-11-28T13:00:56.301561+0000","last_fullsized":"2023-11-28T13:00:56.301561+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:41:20.543663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.302589+0000","last_change":"2023-11-28T13:00:55.901799+0000","last_active":"2023-11-28T13:00:56.302589+0000","last_peered":"2023-11-28T13:00:56.302589+0000","last_clean":"2023-11-28T13:00:56.302589+0000","last_became_active":"2023-11-28T13:00:55.901524+0000","last_became_peered":"2023-11-28T13:00:55.901524+0000","last_unstale":"2023-11-28T13:00:56.302589+0000","last_undegraded":"2023-11-28T13:00:56.302589+0000","last_fullsized":"2023-11-28T13:00:56.302589+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:23:57.311139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.289811+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T13:00:56.289811+0000","last_peered":"2023-11-28T13:00:56.289811+0000","last_clean":"2023-11-28T13:00:56.289811+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:00:56.289811+0000","last_undegraded":"2023-11-28T13:00:56.289811+0000","last_fullsized":"2023-11-28T13:00:56.289811+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301665+0000","last_change":"2023-11-28T13:00:56.299872+0000","last_active":"2023-11-28T13:00:56.301665+0000","last_peered":"2023-11-28T13:00:56.301665+0000","last_clean":"2023-11-28T13:00:56.301665+0000","last_became_active":"2023-11-28T13:00:56.299171+0000","last_became_peered":"2023-11-28T13:00:56.299171+0000","last_unstale":"2023-11-28T13:00:56.301665+0000","last_undegraded":"2023-11-28T13:00:56.301665+0000","last_fullsized":"2023-11-28T13:00:56.301665+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:07.826742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.295916+0000","last_change":"2023-11-28T13:00:55.904411+0000","last_active":"2023-11-28T13:00:56.295916+0000","last_peered":"2023-11-28T13:00:56.295916+0000","last_clean":"2023-11-28T13:00:56.295916+0000","last_became_active":"2023-11-28T13:00:55.904130+0000","last_became_peered":"2023-11-28T13:00:55.904130+0000","last_unstale":"2023-11-28T13:00:56.295916+0000","last_undegraded":"2023-11-28T13:00:56.295916+0000","last_fullsized":"2023-11-28T13:00:56.295916+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:58:00.945059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-28T13:00:55.890209+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T13:00:55.890209+0000","last_peered":"2023-11-28T13:00:55.890209+0000","last_clean":"2023-11-28T13:00:55.890209+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:00:55.890209+0000","last_undegraded":"2023-11-28T13:00:55.890209+0000","last_fullsized":"2023-11-28T13:00:55.890209+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307494+0000","last_change":"2023-11-28T13:00:56.303702+0000","last_active":"2023-11-28T13:00:56.307494+0000","last_peered":"2023-11-28T13:00:56.307494+0000","last_clean":"2023-11-28T13:00:56.307494+0000","last_became_active":"2023-11-28T13:00:56.303257+0000","last_became_peered":"2023-11-28T13:00:56.303257+0000","last_unstale":"2023-11-28T13:00:56.307494+0000","last_undegraded":"2023-11-28T13:00:56.307494+0000","last_fullsized":"2023-11-28T13:00:56.307494+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:47:29.484502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.296946+0000","last_change":"2023-11-28T13:00:55.900175+0000","last_active":"2023-11-28T13:00:56.296946+0000","last_peered":"2023-11-28T13:00:56.296946+0000","last_clean":"2023-11-28T13:00:56.296946+0000","last_became_active":"2023-11-28T13:00:55.899889+0000","last_became_peered":"2023-11-28T13:00:55.899889+0000","last_unstale":"2023-11-28T13:00:56.296946+0000","last_undegraded":"2023-11-28T13:00:56.296946+0000","last_fullsized":"2023-11-28T13:00:56.296946+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:03:43.006079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":46,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-28T13:00:55.883351+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T13:00:55.883351+0000","last_peered":"2023-11-28T13:00:55.883351+0000","last_clean":"2023-11-28T13:00:55.883351+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:00:55.883351+0000","last_undegraded":"2023-11-28T13:00:55.883351+0000","last_fullsized":"2023-11-28T13:00:55.883351+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10312,"kb_used_data":1216,"kb_used_omap":0,"kb_used_meta":9088,"kb_avail":94361528,"statfs":{"total":96636764160,"available":96626204672,"internally_reserved":0,"allocated":1245184,"data_stored":887519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9306112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.64700000000000002,"15min":0.67000000000000004},"min":{"1min":0.34000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66400000000000003,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":0.504},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.65700000000000003,"15min":0.68500000000000005},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63300000000000001,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.52300000000000002}]},{"osd":1,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.41499999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.60799999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.501,"15min":0.50700000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64100000000000001,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.63500000000000001}]},{"osd":2,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.54000000000000004,"15min":0.55000000000000004},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.628,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53600000000000003,"15min":0.54100000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67300000000000004,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.55800000000000005}]},{"osd":3,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.67800000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56299999999999994,"15min":0.56599999999999995},"min":{"1min":0.39000000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66300000000000003,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.59999999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.33000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.68799999999999994,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.623,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.90900000000000003}]},{"osd":5,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.57199999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.73399999999999999}]},{"osd":6,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.46100000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.70099999999999996,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.80000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9060,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362780,"statfs":{"total":96636764160,"available":96627486720,"internally_reserved":0,"allocated":618496,"data_stored":282603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.69699999999999995,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.51500000000000001,"15min":0.51300000000000001},"min":{"1min":0.35999999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.436}]},{"osd":1,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.36299999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.41399999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77700000000000002,"5min":1.016,"15min":1.016},"last":0.629}]},{"osd":2,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.41399999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.503},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81599999999999995,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.59199999999999997}]},{"osd":3,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.44700000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.67500000000000004,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57699999999999996,"15min":0.57999999999999996},"min":{"1min":0.42599999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84899999999999998,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.64600000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.53200000000000003,"15min":0.54000000000000004},"min":{"1min":0.38200000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.63400000000000001,"5min":1,"15min":1},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.53900000000000003,"15min":0.54300000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.749,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.56399999999999995}]},{"osd":5,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68100000000000005,"5min":1.052,"15min":1.052},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.626,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89900000000000002,"5min":1.258,"15min":1.258},"last":0.66900000000000004}]},{"osd":7,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.42299999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.872,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.41799999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.95199999999999996,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.748}]}]},{"osd":1,"up_from":13,"seq":55834574878,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9060,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362780,"statfs":{"total":96636764160,"available":96627486720,"internally_reserved":0,"allocated":618496,"data_stored":282603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.23999999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.505,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.35799999999999998,"15min":0.35399999999999998},"min":{"1min":0.24399999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.54000000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.309}]},{"osd":2,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.33700000000000002,"15min":0.33300000000000002},"min":{"1min":0.29599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.48199999999999998,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27000000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.56999999999999995,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.28499999999999998}]},{"osd":3,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39100000000000001,"15min":0.38900000000000001},"min":{"1min":0.28499999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.53700000000000003,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39300000000000002,"15min":0.39400000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.58499999999999996,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.36899999999999999}]},{"osd":4,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57199999999999995,"15min":0.56599999999999995},"min":{"1min":0.39200000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81100000000000005,"5min":1.028,"15min":1.028},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.435,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84799999999999998,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.85899999999999999}]},{"osd":5,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59599999999999997,"15min":0.59099999999999997},"min":{"1min":0.46300000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.874,"5min":1.042,"15min":1.042},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.56799999999999995,"15min":0.55700000000000005},"min":{"1min":0.40999999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.79200000000000004}]},{"osd":6,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.40000000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.87,"5min":1.006,"15min":1.006},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60099999999999998,"15min":0.59799999999999998},"min":{"1min":0.42599999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85899999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.65400000000000003}]},{"osd":7,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57899999999999996,"15min":0.57299999999999995},"min":{"1min":0.42499999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90000000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.64400000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10292,"kb_used_data":1196,"kb_used_omap":0,"kb_used_meta":9088,"kb_avail":94361548,"statfs":{"total":96636764160,"available":96626225152,"internally_reserved":0,"allocated":1224704,"data_stored":879911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9306112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.32400000000000001,"15min":0.32200000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34200000000000003,"15min":0.34100000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.32000000000000001}]},{"osd":2,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.34999999999999998,"15min":0.34499999999999997},"min":{"1min":0.26100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36699999999999999,"15min":0.36399999999999999},"min":{"1min":0.27700000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.29099999999999998}]},{"osd":3,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.37,"15min":0.372},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.55100000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38900000000000001,"15min":0.38600000000000001},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.33200000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56699999999999995,"15min":0.56899999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.70999999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64300000000000002}]},{"osd":5,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.432,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.754,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.56200000000000006}]},{"osd":6,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.73399999999999999,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60899999999999999,"15min":0.61599999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72299999999999998,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.56699999999999995}]},{"osd":7,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.627,"15min":0.629},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.78600000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.621,"15min":0.628},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.71899999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.623}]}]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9008,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362832,"statfs":{"total":96636764160,"available":96627539968,"internally_reserved":0,"allocated":630784,"data_stored":289543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.32500000000000001,"15min":0.316},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.33700000000000002,"15min":0.33300000000000002},"min":{"1min":0.249,"5min":0.184,"15min":0.184},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.432}]},{"osd":1,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38100000000000001,"15min":0.38200000000000001},"min":{"1min":0.28599999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.35199999999999998},"min":{"1min":0.29599999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51300000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.47599999999999998}]},{"osd":3,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.33800000000000002},"min":{"1min":0.26600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45400000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.376},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.28999999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.60799999999999998,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.48999999999999999}]},{"osd":4,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.41399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79100000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.79100000000000004}]},{"osd":5,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.45700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.83199999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82399999999999995,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.78000000000000003}]},{"osd":6,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.57499999999999996,"15min":0.57099999999999995},"min":{"1min":0.45000000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77700000000000002,"5min":1.002,"15min":1.002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.85299999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.76900000000000002}]},{"osd":7,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.621,"15min":0.627},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.80900000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.77500000000000002,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.69699999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9200,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362640,"statfs":{"total":96636764160,"available":96627343360,"internally_reserved":0,"allocated":630784,"data_stored":289543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.31900000000000001},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.44700000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.441},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.32700000000000001,"15min":0.32600000000000001},"min":{"1min":0.22700000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.505,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.33000000000000002}]},{"osd":1,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34000000000000002,"15min":0.33800000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.47899999999999998,"5min":0.502,"15min":0.502},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.373,"15min":0.372},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.314}]},{"osd":2,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.47699999999999998,"15min":0.49099999999999999},"min":{"1min":0.28100000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.57399999999999995,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.51900000000000002,"15min":0.53500000000000003},"min":{"1min":0.30299999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58999999999999997,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":0.371}]},{"osd":4,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.91700000000000004,"15min":0.96699999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.88200000000000001,"5min":8.2889999999999997,"15min":8.2889999999999997},"last":0.60299999999999998}]},{"osd":5,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.62,"15min":0.62},"min":{"1min":0.439,"5min":0.432,"15min":0.432},"max":{"1min":0.82899999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66500000000000004,"15min":0.67500000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.83099999999999996,"5min":2.165,"15min":2.165},"last":0.78500000000000003}]},{"osd":6,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.628,"15min":0.627},"min":{"1min":0.46400000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.86599999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.61199999999999999}]},{"osd":7,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.79000000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.755},{"interface":"front","average":{"1min":0.623,"5min":0.91100000000000003,"15min":0.95899999999999996},"min":{"1min":0.45500000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81499999999999995,"5min":7.0899999999999999,"15min":7.0899999999999999},"last":0.64900000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9328,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362512,"statfs":{"total":96636764160,"available":96627212288,"internally_reserved":0,"allocated":630784,"data_stored":289543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83499999999999996,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.72199999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.54200000000000004}]},{"osd":1,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.38700000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41199999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70499999999999996,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.51800000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.55400000000000005,"15min":0.54700000000000004},"min":{"1min":0.441,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.876,"5min":1.069,"15min":1.069},"last":0.46000000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.51800000000000002,"15min":0.50900000000000001},"min":{"1min":0.39600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85599999999999998,"5min":0.879,"15min":0.879},"last":0.53000000000000003}]},{"osd":5,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.32000000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.72099999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.42799999999999999}]},{"osd":6,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69199999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58999999999999997,"15min":0.59099999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78800000000000003,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.48299999999999998}]},{"osd":7,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.40500000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.80100000000000005,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.44800000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8996,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362844,"statfs":{"total":96636764160,"available":96627552256,"internally_reserved":0,"allocated":618496,"data_stored":282603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.34599999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69899999999999995,"5min":1.819,"15min":1.819},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.52600000000000002,"15min":0.52100000000000002},"min":{"1min":0.40799999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73599999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.30399999999999999}]},{"osd":2,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.75800000000000001,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.63,"15min":0.64800000000000002},"min":{"1min":0.372,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.748,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.38200000000000001}]},{"osd":3,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72799999999999998,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.66300000000000003,"15min":0.68600000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71099999999999997,"5min":2.702,"15min":2.702},"last":0.40699999999999997}]},{"osd":4,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.60099999999999998,"15min":0.61899999999999999},"min":{"1min":0.30399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.66500000000000004,"5min":3.0739999999999998,"15min":3.0739999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76500000000000001,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.34399999999999997}]},{"osd":6,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.34499999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.70199999999999996,"15min":0.72699999999999998},"min":{"1min":0.39100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73099999999999998,"5min":3.7679999999999998,"15min":3.7679999999999998},"last":0.318}]},{"osd":7,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78400000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.44},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.65500000000000003,"15min":0.66600000000000004},"min":{"1min":0.41399999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.751,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:01:00.351 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:01:00.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.502+0000 7fc27901e700 1 Processor -- start 2023-11-28T13:01:00.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.502+0000 7fc27901e700 1 -- start start 2023-11-28T13:01:00.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.503+0000 7fc27901e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740cc550 0x7fc2740cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.503+0000 7fc27901e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc27415cd90 con 0x7fc2741604b0 2023-11-28T13:01:00.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.503+0000 7fc27901e700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2741613e0 con 0x7fc2740b0d70 2023-11-28T13:01:00.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.503+0000 7fc27901e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc274160950 con 0x7fc2740cc550 2023-11-28T13:01:00.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.503+0000 7fc27259c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc2741604b0 0x7fc2740ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37034/0 (socket says 172.21.15.111:37034) 2023-11-28T13:01:00.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.503+0000 7fc27259c700 1 -- 172.21.15.111:0/3734257282 learned_addr learned my addr 172.21.15.111:0/3734257282 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:00.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.503+0000 7fc27359e700 1 --2- 172.21.15.111:0/3734257282 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740cc550 0x7fc2740cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.503+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407411075 0 0) 0x7fc264001110 con 0x7fc2741604b0 2023-11-28T13:01:00.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.503+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc25c0035b0 con 0x7fc2741604b0 2023-11-28T13:01:00.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058715578 0 0) 0x7fc25c0035b0 con 0x7fc2741604b0 2023-11-28T13:01:00.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740cc550 msgr2=0x7fc2740cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:00.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 --2- 172.21.15.111:0/3734257282 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740cc550 0x7fc2740cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 >> v1:172.21.15.149:6789/0 conn(0x7fc2740b0d70 legacy=0x7fc2740b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc274160b40 con 0x7fc2741604b0 2023-11-28T13:01:00.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fc264001800 con 0x7fc2741604b0 2023-11-28T13:01:00.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 >> v1:172.21.15.111:6789/0 conn(0x7fc2741604b0 legacy=0x7fc2740ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 --2- 172.21.15.111:0/3734257282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc25c003a30 0x7fc25c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 --2- 172.21.15.111:0/3734257282 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740cc550 0x7fc25c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 --2- 172.21.15.111:0/3734257282 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc25c004d90 0x7fc25c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c00ba10 con 0x7fc25c003a30 2023-11-28T13:01:00.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c00bb50 con 0x7fc25c004d90 2023-11-28T13:01:00.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c00bc90 con 0x7fc2740cc550 2023-11-28T13:01:00.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc272d9d700 1 --2- 172.21.15.111:0/3734257282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc25c003a30 0x7fc25c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc27359e700 1 --2- 172.21.15.111:0/3734257282 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc25c004d90 0x7fc25c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc27259c700 1 --2- 172.21.15.111:0/3734257282 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740cc550 0x7fc25c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc272d9d700 1 -- 172.21.15.111:0/3734257282 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740cc550 msgr2=0x7fc25c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:00.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc272d9d700 1 --2- 172.21.15.111:0/3734257282 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740cc550 0x7fc25c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc272d9d700 1 -- 172.21.15.111:0/3734257282 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc25c004d90 msgr2=0x7fc25c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:00.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc272d9d700 1 --2- 172.21.15.111:0/3734257282 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc25c004d90 0x7fc25c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc272d9d700 1 -- 172.21.15.111:0/3734257282 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c0035b0 con 0x7fc25c003a30 2023-11-28T13:01:00.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc27359e700 1 --2- 172.21.15.111:0/3734257282 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc25c004d90 0x7fc25c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:00.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc272d9d700 1 --2- 172.21.15.111:0/3734257282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc25c003a30 0x7fc25c003e00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fc2680018e0 tx=0x7fc25c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bddfd64a6d5ee0fd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:00.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26800c660 con 0x7fc25c003a30 2023-11-28T13:01:00.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26800c7e0 con 0x7fc25c003a30 2023-11-28T13:01:00.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.504+0000 7fc271d9b700 1 -- 172.21.15.111:0/3734257282 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26800cb90 con 0x7fc25c003a30 2023-11-28T13:01:00.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.505+0000 7fc27901e700 1 -- 172.21.15.111:0/3734257282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc25c003a30 msgr2=0x7fc25c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.505+0000 7fc27901e700 1 --2- 172.21.15.111:0/3734257282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc25c003a30 0x7fc25c003e00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fc2680018e0 tx=0x7fc25c005880 comp rx=0 tx=0).stop 2023-11-28T13:01:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.505+0000 7fc27359e700 1 -- 172.21.15.111:0/3734257282 reap_dead start 2023-11-28T13:01:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.505+0000 7fc27901e700 1 -- 172.21.15.111:0/3734257282 shutdown_connections 2023-11-28T13:01:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.505+0000 7fc27901e700 1 -- 172.21.15.111:0/3734257282 >> 172.21.15.111:0/3734257282 conn(0x7fc274098ee0 msgr2=0x7fc27406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.505+0000 7fc27901e700 1 -- 172.21.15.111:0/3734257282 shutdown_connections 2023-11-28T13:01:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.505+0000 7fc27901e700 1 -- 172.21.15.111:0/3734257282 wait complete. 2023-11-28T13:01:00.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.507+0000 7fc27901e700 1 Processor -- start 2023-11-28T13:01:00.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.507+0000 7fc27901e700 1 -- start start 2023-11-28T13:01:00.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27901e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc25c003a30 0x7fc2740c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27901e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc27415ae50 0x7fc2740c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27901e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740c3b00 0x7fc2740c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27901e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2740c7630 con 0x7fc27415ae50 2023-11-28T13:01:00.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27901e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2740c7910 con 0x7fc25c003a30 2023-11-28T13:01:00.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27901e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2740c7bf0 con 0x7fc2740c3b00 2023-11-28T13:01:00.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc27415ae50 0x7fc2740c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc27415ae50 0x7fc2740c35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36268/0 (socket says 172.21.15.111:36268) 2023-11-28T13:01:00.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27259c700 1 -- 172.21.15.111:0/3034610245 learned_addr learned my addr 172.21.15.111:0/3034610245 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:00.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc272d9d700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc25c003a30 0x7fc2740c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27359e700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740c3b00 0x7fc2740c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27259c700 1 -- 172.21.15.111:0/3034610245 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740c3b00 msgr2=0x7fc2740c3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27259c700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740c3b00 0x7fc2740c3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27259c700 1 -- 172.21.15.111:0/3034610245 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc25c003a30 msgr2=0x7fc2740c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27259c700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc25c003a30 0x7fc2740c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27259c700 1 -- 172.21.15.111:0/3034610245 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc274161100 con 0x7fc27415ae50 2023-11-28T13:01:00.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27359e700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740c3b00 0x7fc2740c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:00.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.508+0000 7fc27259c700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc27415ae50 0x7fc2740c35c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc264001cf0 tx=0x7fc26400e430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:00.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.509+0000 7fc25b7fe700 1 -- 172.21.15.111:0/3034610245 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2640017e0 con 0x7fc27415ae50 2023-11-28T13:01:00.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.509+0000 7fc25b7fe700 1 -- 172.21.15.111:0/3034610245 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26400ea10 con 0x7fc27415ae50 2023-11-28T13:01:00.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.509+0000 7fc27901e700 1 -- 172.21.15.111:0/3034610245 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc274160e20 con 0x7fc27415ae50 2023-11-28T13:01:00.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.509+0000 7fc25b7fe700 1 -- 172.21.15.111:0/3034610245 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26400ed80 con 0x7fc27415ae50 2023-11-28T13:01:00.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.509+0000 7fc27901e700 1 -- 172.21.15.111:0/3034610245 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2740af340 con 0x7fc27415ae50 2023-11-28T13:01:00.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.510+0000 7fc27901e700 1 -- 172.21.15.111:0/3034610245 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2740c64e0 con 0x7fc27415ae50 2023-11-28T13:01:00.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.510+0000 7fc25b7fe700 1 -- 172.21.15.111:0/3034610245 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc26401d460 con 0x7fc27415ae50 2023-11-28T13:01:00.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.510+0000 7fc25b7fe700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc2600682f0 0x7fc26006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.510+0000 7fc25b7fe700 1 -- 172.21.15.111:0/3034610245 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc264085d90 con 0x7fc27415ae50 2023-11-28T13:01:00.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.510+0000 7fc272d9d700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc2600682f0 0x7fc26006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.511+0000 7fc272d9d700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc2600682f0 0x7fc26006a7a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc2680018e0 tx=0x7fc268009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:00.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.515+0000 7fc25b7fe700 1 -- 172.21.15.111:0/3034610245 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc264052450 con 0x7fc27415ae50 2023-11-28T13:01:00.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.691+0000 7fc27901e700 1 -- 172.21.15.111:0/3034610245 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc27415cd90 con 0x7fc2600682f0 2023-11-28T13:01:00.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.699+0000 7fc25b7fe700 1 -- 172.21.15.111:0/3034610245 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311595 (secure 0 0 0) 0x7fc27415cd90 con 0x7fc2600682f0 2023-11-28T13:01:00.704 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:00.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.702+0000 7fc27901e700 1 -- 172.21.15.111:0/3034610245 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc2600682f0 msgr2=0x7fc26006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.702+0000 7fc27901e700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc2600682f0 0x7fc26006a7a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc2680018e0 tx=0x7fc268009030 comp rx=0 tx=0).stop 2023-11-28T13:01:00.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.702+0000 7fc27901e700 1 -- 172.21.15.111:0/3034610245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc27415ae50 msgr2=0x7fc2740c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.702+0000 7fc27901e700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc27415ae50 0x7fc2740c35c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc264001cf0 tx=0x7fc26400e430 comp rx=0 tx=0).stop 2023-11-28T13:01:00.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.702+0000 7fc27901e700 1 -- 172.21.15.111:0/3034610245 shutdown_connections 2023-11-28T13:01:00.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.702+0000 7fc27901e700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2740c3b00 0x7fc2740c3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.702+0000 7fc27901e700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc2600682f0 0x7fc26006a7a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.702+0000 7fc27901e700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc25c003a30 0x7fc2740c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.702+0000 7fc27901e700 1 --2- 172.21.15.111:0/3034610245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc27415ae50 0x7fc2740c35c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.702+0000 7fc27901e700 1 -- 172.21.15.111:0/3034610245 >> 172.21.15.111:0/3034610245 conn(0x7fc274098ee0 msgr2=0x7fc27415bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:00.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.702+0000 7fc27901e700 1 -- 172.21.15.111:0/3034610245 shutdown_connections 2023-11-28T13:01:00.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.702+0000 7fc27901e700 1 -- 172.21.15.111:0/3034610245 wait complete. 2023-11-28T13:01:00.708 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:01:00.716 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2023-11-28T13:01:00.360986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":75256,"kb_used_data":6072,"kb_used_omap":0,"kb_used_meta":69120,"kb_avail":754899464,"statfs":{"total":773094113280,"available":773017051136,"internally_reserved":0,"allocated":6217728,"data_stored":3483868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":70778880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.489070"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.304647+0000","last_change":"2023-11-28T13:00:56.297180+0000","last_active":"2023-11-28T13:00:56.304647+0000","last_peered":"2023-11-28T13:00:56.304647+0000","last_clean":"2023-11-28T13:00:56.304647+0000","last_became_active":"2023-11-28T13:00:56.296928+0000","last_became_peered":"2023-11-28T13:00:56.296928+0000","last_unstale":"2023-11-28T13:00:56.304647+0000","last_undegraded":"2023-11-28T13:00:56.304647+0000","last_fullsized":"2023-11-28T13:00:56.304647+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:06:40.264221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.297885+0000","last_change":"2023-11-28T13:00:56.293874+0000","last_active":"2023-11-28T13:00:56.297885+0000","last_peered":"2023-11-28T13:00:56.297885+0000","last_clean":"2023-11-28T13:00:56.297885+0000","last_became_active":"2023-11-28T13:00:56.293612+0000","last_became_peered":"2023-11-28T13:00:56.293612+0000","last_unstale":"2023-11-28T13:00:56.297885+0000","last_undegraded":"2023-11-28T13:00:56.297885+0000","last_fullsized":"2023-11-28T13:00:56.297885+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:41:30.521782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301641+0000","last_change":"2023-11-28T13:00:56.299702+0000","last_active":"2023-11-28T13:00:56.301641+0000","last_peered":"2023-11-28T13:00:56.301641+0000","last_clean":"2023-11-28T13:00:56.301641+0000","last_became_active":"2023-11-28T13:00:56.298684+0000","last_became_peered":"2023-11-28T13:00:56.298684+0000","last_unstale":"2023-11-28T13:00:56.301641+0000","last_undegraded":"2023-11-28T13:00:56.301641+0000","last_fullsized":"2023-11-28T13:00:56.301641+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:36.003632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.292127+0000","last_change":"2023-11-28T13:00:56.284480+0000","last_active":"2023-11-28T13:00:56.292127+0000","last_peered":"2023-11-28T13:00:56.292127+0000","last_clean":"2023-11-28T13:00:56.292127+0000","last_became_active":"2023-11-28T13:00:56.284161+0000","last_became_peered":"2023-11-28T13:00:56.284161+0000","last_unstale":"2023-11-28T13:00:56.292127+0000","last_undegraded":"2023-11-28T13:00:56.292127+0000","last_fullsized":"2023-11-28T13:00:56.292127+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:36:36.003632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.287312+0000","last_change":"2023-11-28T13:00:56.284274+0000","last_active":"2023-11-28T13:00:56.287312+0000","last_peered":"2023-11-28T13:00:56.287312+0000","last_clean":"2023-11-28T13:00:56.287312+0000","last_became_active":"2023-11-28T13:00:56.284042+0000","last_became_peered":"2023-11-28T13:00:56.284042+0000","last_unstale":"2023-11-28T13:00:56.287312+0000","last_undegraded":"2023-11-28T13:00:56.287312+0000","last_fullsized":"2023-11-28T13:00:56.287312+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:06:57.178129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306439+0000","last_change":"2023-11-28T13:00:56.304342+0000","last_active":"2023-11-28T13:00:56.306439+0000","last_peered":"2023-11-28T13:00:56.306439+0000","last_clean":"2023-11-28T13:00:56.306439+0000","last_became_active":"2023-11-28T13:00:56.304070+0000","last_became_peered":"2023-11-28T13:00:56.304070+0000","last_unstale":"2023-11-28T13:00:56.306439+0000","last_undegraded":"2023-11-28T13:00:56.306439+0000","last_fullsized":"2023-11-28T13:00:56.306439+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:18.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.295655+0000","last_change":"2023-11-28T13:00:56.284160+0000","last_active":"2023-11-28T13:00:56.295655+0000","last_peered":"2023-11-28T13:00:56.295655+0000","last_clean":"2023-11-28T13:00:56.295655+0000","last_became_active":"2023-11-28T13:00:56.283893+0000","last_became_peered":"2023-11-28T13:00:56.283893+0000","last_unstale":"2023-11-28T13:00:56.295655+0000","last_undegraded":"2023-11-28T13:00:56.295655+0000","last_fullsized":"2023-11-28T13:00:56.295655+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:35:22.914408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307678+0000","last_change":"2023-11-28T13:00:56.303857+0000","last_active":"2023-11-28T13:00:56.307678+0000","last_peered":"2023-11-28T13:00:56.307678+0000","last_clean":"2023-11-28T13:00:56.307678+0000","last_became_active":"2023-11-28T13:00:56.303308+0000","last_became_peered":"2023-11-28T13:00:56.303308+0000","last_unstale":"2023-11-28T13:00:56.307678+0000","last_undegraded":"2023-11-28T13:00:56.307678+0000","last_fullsized":"2023-11-28T13:00:56.307678+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:42:06.976808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.303888+0000","last_change":"2023-11-28T13:00:56.300909+0000","last_active":"2023-11-28T13:00:56.303888+0000","last_peered":"2023-11-28T13:00:56.303888+0000","last_clean":"2023-11-28T13:00:56.303888+0000","last_became_active":"2023-11-28T13:00:56.300658+0000","last_became_peered":"2023-11-28T13:00:56.300658+0000","last_unstale":"2023-11-28T13:00:56.303888+0000","last_undegraded":"2023-11-28T13:00:56.303888+0000","last_fullsized":"2023-11-28T13:00:56.303888+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:18.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.287343+0000","last_change":"2023-11-28T13:00:56.282305+0000","last_active":"2023-11-28T13:00:56.287343+0000","last_peered":"2023-11-28T13:00:56.287343+0000","last_clean":"2023-11-28T13:00:56.287343+0000","last_became_active":"2023-11-28T13:00:56.282022+0000","last_became_peered":"2023-11-28T13:00:56.282022+0000","last_unstale":"2023-11-28T13:00:56.287343+0000","last_undegraded":"2023-11-28T13:00:56.287343+0000","last_fullsized":"2023-11-28T13:00:56.287343+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:09:20.885653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298210+0000","last_change":"2023-11-28T13:00:56.295271+0000","last_active":"2023-11-28T13:00:56.298210+0000","last_peered":"2023-11-28T13:00:56.298210+0000","last_clean":"2023-11-28T13:00:56.298210+0000","last_became_active":"2023-11-28T13:00:56.294659+0000","last_became_peered":"2023-11-28T13:00:56.294659+0000","last_unstale":"2023-11-28T13:00:56.298210+0000","last_undegraded":"2023-11-28T13:00:56.298210+0000","last_fullsized":"2023-11-28T13:00:56.298210+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:36:18.740266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301250+0000","last_change":"2023-11-28T13:00:56.294715+0000","last_active":"2023-11-28T13:00:56.301250+0000","last_peered":"2023-11-28T13:00:56.301250+0000","last_clean":"2023-11-28T13:00:56.301250+0000","last_became_active":"2023-11-28T13:00:56.293637+0000","last_became_peered":"2023-11-28T13:00:56.293637+0000","last_unstale":"2023-11-28T13:00:56.301250+0000","last_undegraded":"2023-11-28T13:00:56.301250+0000","last_fullsized":"2023-11-28T13:00:56.301250+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:44:32.911023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.300878+0000","last_change":"2023-11-28T13:00:56.296998+0000","last_active":"2023-11-28T13:00:56.300878+0000","last_peered":"2023-11-28T13:00:56.300878+0000","last_clean":"2023-11-28T13:00:56.300878+0000","last_became_active":"2023-11-28T13:00:56.296705+0000","last_became_peered":"2023-11-28T13:00:56.296705+0000","last_unstale":"2023-11-28T13:00:56.300878+0000","last_undegraded":"2023-11-28T13:00:56.300878+0000","last_fullsized":"2023-11-28T13:00:56.300878+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:50:37.941307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.297636+0000","last_change":"2023-11-28T13:00:56.288478+0000","last_active":"2023-11-28T13:00:56.297636+0000","last_peered":"2023-11-28T13:00:56.297636+0000","last_clean":"2023-11-28T13:00:56.297636+0000","last_became_active":"2023-11-28T13:00:56.288201+0000","last_became_peered":"2023-11-28T13:00:56.288201+0000","last_unstale":"2023-11-28T13:00:56.297636+0000","last_undegraded":"2023-11-28T13:00:56.297636+0000","last_fullsized":"2023-11-28T13:00:56.297636+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:32:21.431054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298107+0000","last_change":"2023-11-28T13:00:56.291756+0000","last_active":"2023-11-28T13:00:56.298107+0000","last_peered":"2023-11-28T13:00:56.298107+0000","last_clean":"2023-11-28T13:00:56.298107+0000","last_became_active":"2023-11-28T13:00:56.287654+0000","last_became_peered":"2023-11-28T13:00:56.287654+0000","last_unstale":"2023-11-28T13:00:56.298107+0000","last_undegraded":"2023-11-28T13:00:56.298107+0000","last_fullsized":"2023-11-28T13:00:56.298107+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:12.468194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.304306+0000","last_change":"2023-11-28T13:00:56.298915+0000","last_active":"2023-11-28T13:00:56.304306+0000","last_peered":"2023-11-28T13:00:56.304306+0000","last_clean":"2023-11-28T13:00:56.304306+0000","last_became_active":"2023-11-28T13:00:56.298583+0000","last_became_peered":"2023-11-28T13:00:56.298583+0000","last_unstale":"2023-11-28T13:00:56.304306+0000","last_undegraded":"2023-11-28T13:00:56.304306+0000","last_fullsized":"2023-11-28T13:00:56.304306+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:28:49.012453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298031+0000","last_change":"2023-11-28T13:00:56.293009+0000","last_active":"2023-11-28T13:00:56.298031+0000","last_peered":"2023-11-28T13:00:56.298031+0000","last_clean":"2023-11-28T13:00:56.298031+0000","last_became_active":"2023-11-28T13:00:56.292771+0000","last_became_peered":"2023-11-28T13:00:56.292771+0000","last_unstale":"2023-11-28T13:00:56.298031+0000","last_undegraded":"2023-11-28T13:00:56.298031+0000","last_fullsized":"2023-11-28T13:00:56.298031+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:18:03.399413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.303701+0000","last_change":"2023-11-28T13:00:56.301047+0000","last_active":"2023-11-28T13:00:56.303701+0000","last_peered":"2023-11-28T13:00:56.303701+0000","last_clean":"2023-11-28T13:00:56.303701+0000","last_became_active":"2023-11-28T13:00:56.300756+0000","last_became_peered":"2023-11-28T13:00:56.300756+0000","last_unstale":"2023-11-28T13:00:56.303701+0000","last_undegraded":"2023-11-28T13:00:56.303701+0000","last_fullsized":"2023-11-28T13:00:56.303701+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:59:22.039571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.303114+0000","last_change":"2023-11-28T13:00:56.296639+0000","last_active":"2023-11-28T13:00:56.303114+0000","last_peered":"2023-11-28T13:00:56.303114+0000","last_clean":"2023-11-28T13:00:56.303114+0000","last_became_active":"2023-11-28T13:00:56.295056+0000","last_became_peered":"2023-11-28T13:00:56.295056+0000","last_unstale":"2023-11-28T13:00:56.303114+0000","last_undegraded":"2023-11-28T13:00:56.303114+0000","last_fullsized":"2023-11-28T13:00:56.303114+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:39:11.338916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307476+0000","last_change":"2023-11-28T13:00:56.303529+0000","last_active":"2023-11-28T13:00:56.307476+0000","last_peered":"2023-11-28T13:00:56.307476+0000","last_clean":"2023-11-28T13:00:56.307476+0000","last_became_active":"2023-11-28T13:00:56.302836+0000","last_became_peered":"2023-11-28T13:00:56.302836+0000","last_unstale":"2023-11-28T13:00:56.307476+0000","last_undegraded":"2023-11-28T13:00:56.307476+0000","last_fullsized":"2023-11-28T13:00:56.307476+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:24.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.300864+0000","last_change":"2023-11-28T13:00:56.296498+0000","last_active":"2023-11-28T13:00:56.300864+0000","last_peered":"2023-11-28T13:00:56.300864+0000","last_clean":"2023-11-28T13:00:56.300864+0000","last_became_active":"2023-11-28T13:00:56.295880+0000","last_became_peered":"2023-11-28T13:00:56.295880+0000","last_unstale":"2023-11-28T13:00:56.300864+0000","last_undegraded":"2023-11-28T13:00:56.300864+0000","last_fullsized":"2023-11-28T13:00:56.300864+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:23:36.199654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298869+0000","last_change":"2023-11-28T13:00:56.296062+0000","last_active":"2023-11-28T13:00:56.298869+0000","last_peered":"2023-11-28T13:00:56.298869+0000","last_clean":"2023-11-28T13:00:56.298869+0000","last_became_active":"2023-11-28T13:00:56.295653+0000","last_became_peered":"2023-11-28T13:00:56.295653+0000","last_unstale":"2023-11-28T13:00:56.298869+0000","last_undegraded":"2023-11-28T13:00:56.298869+0000","last_fullsized":"2023-11-28T13:00:56.298869+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:32:28.994903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306248+0000","last_change":"2023-11-28T13:00:56.303165+0000","last_active":"2023-11-28T13:00:56.306248+0000","last_peered":"2023-11-28T13:00:56.306248+0000","last_clean":"2023-11-28T13:00:56.306248+0000","last_became_active":"2023-11-28T13:00:56.302931+0000","last_became_peered":"2023-11-28T13:00:56.302931+0000","last_unstale":"2023-11-28T13:00:56.306248+0000","last_undegraded":"2023-11-28T13:00:56.306248+0000","last_fullsized":"2023-11-28T13:00:56.306248+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:58:47.885007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301113+0000","last_change":"2023-11-28T13:00:56.298226+0000","last_active":"2023-11-28T13:00:56.301113+0000","last_peered":"2023-11-28T13:00:56.301113+0000","last_clean":"2023-11-28T13:00:56.301113+0000","last_became_active":"2023-11-28T13:00:56.297953+0000","last_became_peered":"2023-11-28T13:00:56.297953+0000","last_unstale":"2023-11-28T13:00:56.301113+0000","last_undegraded":"2023-11-28T13:00:56.301113+0000","last_fullsized":"2023-11-28T13:00:56.301113+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:06:18.378950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.302402+0000","last_change":"2023-11-28T13:00:56.294478+0000","last_active":"2023-11-28T13:00:56.302402+0000","last_peered":"2023-11-28T13:00:56.302402+0000","last_clean":"2023-11-28T13:00:56.302402+0000","last_became_active":"2023-11-28T13:00:56.294167+0000","last_became_peered":"2023-11-28T13:00:56.294167+0000","last_unstale":"2023-11-28T13:00:56.302402+0000","last_undegraded":"2023-11-28T13:00:56.302402+0000","last_fullsized":"2023-11-28T13:00:56.302402+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:43:05.421035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.288957+0000","last_change":"2023-11-28T13:00:56.283786+0000","last_active":"2023-11-28T13:00:56.288957+0000","last_peered":"2023-11-28T13:00:56.288957+0000","last_clean":"2023-11-28T13:00:56.288957+0000","last_became_active":"2023-11-28T13:00:56.283148+0000","last_became_peered":"2023-11-28T13:00:56.283148+0000","last_unstale":"2023-11-28T13:00:56.288957+0000","last_undegraded":"2023-11-28T13:00:56.288957+0000","last_fullsized":"2023-11-28T13:00:56.288957+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:52.856174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301594+0000","last_change":"2023-11-28T13:00:56.298740+0000","last_active":"2023-11-28T13:00:56.301594+0000","last_peered":"2023-11-28T13:00:56.301594+0000","last_clean":"2023-11-28T13:00:56.301594+0000","last_became_active":"2023-11-28T13:00:56.298446+0000","last_became_peered":"2023-11-28T13:00:56.298446+0000","last_unstale":"2023-11-28T13:00:56.301594+0000","last_undegraded":"2023-11-28T13:00:56.301594+0000","last_fullsized":"2023-11-28T13:00:56.301594+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:12.468194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.294053+0000","last_change":"2023-11-28T13:00:56.286891+0000","last_active":"2023-11-28T13:00:56.294053+0000","last_peered":"2023-11-28T13:00:56.294053+0000","last_clean":"2023-11-28T13:00:56.294053+0000","last_became_active":"2023-11-28T13:00:56.286615+0000","last_became_peered":"2023-11-28T13:00:56.286615+0000","last_unstale":"2023-11-28T13:00:56.294053+0000","last_undegraded":"2023-11-28T13:00:56.294053+0000","last_fullsized":"2023-11-28T13:00:56.294053+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:17:23.028402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298193+0000","last_change":"2023-11-28T13:00:56.294982+0000","last_active":"2023-11-28T13:00:56.298193+0000","last_peered":"2023-11-28T13:00:56.298193+0000","last_clean":"2023-11-28T13:00:56.298193+0000","last_became_active":"2023-11-28T13:00:56.294744+0000","last_became_peered":"2023-11-28T13:00:56.294744+0000","last_unstale":"2023-11-28T13:00:56.298193+0000","last_undegraded":"2023-11-28T13:00:56.298193+0000","last_fullsized":"2023-11-28T13:00:56.298193+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:32:21.431054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301687+0000","last_change":"2023-11-28T13:00:56.300082+0000","last_active":"2023-11-28T13:00:56.301687+0000","last_peered":"2023-11-28T13:00:56.301687+0000","last_clean":"2023-11-28T13:00:56.301687+0000","last_became_active":"2023-11-28T13:00:56.299839+0000","last_became_peered":"2023-11-28T13:00:56.299839+0000","last_unstale":"2023-11-28T13:00:56.301687+0000","last_undegraded":"2023-11-28T13:00:56.301687+0000","last_fullsized":"2023-11-28T13:00:56.301687+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:23:58.317830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.296430+0000","last_change":"2023-11-28T13:00:56.289747+0000","last_active":"2023-11-28T13:00:56.296430+0000","last_peered":"2023-11-28T13:00:56.296430+0000","last_clean":"2023-11-28T13:00:56.296430+0000","last_became_active":"2023-11-28T13:00:56.289519+0000","last_became_peered":"2023-11-28T13:00:56.289519+0000","last_unstale":"2023-11-28T13:00:56.296430+0000","last_undegraded":"2023-11-28T13:00:56.296430+0000","last_fullsized":"2023-11-28T13:00:56.296430+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:18:51.253650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306141+0000","last_change":"2023-11-28T13:00:56.303230+0000","last_active":"2023-11-28T13:00:56.306141+0000","last_peered":"2023-11-28T13:00:56.306141+0000","last_clean":"2023-11-28T13:00:56.306141+0000","last_became_active":"2023-11-28T13:00:56.302976+0000","last_became_peered":"2023-11-28T13:00:56.302976+0000","last_unstale":"2023-11-28T13:00:56.306141+0000","last_undegraded":"2023-11-28T13:00:56.306141+0000","last_fullsized":"2023-11-28T13:00:56.306141+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:35:22.914408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.304810+0000","last_change":"2023-11-28T13:00:56.299762+0000","last_active":"2023-11-28T13:00:56.304810+0000","last_peered":"2023-11-28T13:00:56.304810+0000","last_clean":"2023-11-28T13:00:56.304810+0000","last_became_active":"2023-11-28T13:00:56.299538+0000","last_became_peered":"2023-11-28T13:00:56.299538+0000","last_unstale":"2023-11-28T13:00:56.304810+0000","last_undegraded":"2023-11-28T13:00:56.304810+0000","last_fullsized":"2023-11-28T13:00:56.304810+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:32:21.431054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"54'4","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.304014+0000","last_change":"2023-11-28T13:00:55.899907+0000","last_active":"2023-11-28T13:00:56.304014+0000","last_peered":"2023-11-28T13:00:56.304014+0000","last_clean":"2023-11-28T13:00:56.304014+0000","last_became_active":"2023-11-28T13:00:55.899661+0000","last_became_peered":"2023-11-28T13:00:55.899661+0000","last_unstale":"2023-11-28T13:00:56.304014+0000","last_undegraded":"2023-11-28T13:00:56.304014+0000","last_fullsized":"2023-11-28T13:00:56.304014+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:23:35.192962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.299197+0000","last_change":"2023-11-28T13:00:56.295845+0000","last_active":"2023-11-28T13:00:56.299197+0000","last_peered":"2023-11-28T13:00:56.299197+0000","last_clean":"2023-11-28T13:00:56.299197+0000","last_became_active":"2023-11-28T13:00:56.295353+0000","last_became_peered":"2023-11-28T13:00:56.295353+0000","last_unstale":"2023-11-28T13:00:56.299197+0000","last_undegraded":"2023-11-28T13:00:56.299197+0000","last_fullsized":"2023-11-28T13:00:56.299197+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:55.996169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.302871+0000","last_change":"2023-11-28T13:00:55.903352+0000","last_active":"2023-11-28T13:00:56.302871+0000","last_peered":"2023-11-28T13:00:56.302871+0000","last_clean":"2023-11-28T13:00:56.302871+0000","last_became_active":"2023-11-28T13:00:55.903082+0000","last_became_peered":"2023-11-28T13:00:55.903082+0000","last_unstale":"2023-11-28T13:00:56.302871+0000","last_undegraded":"2023-11-28T13:00:56.302871+0000","last_fullsized":"2023-11-28T13:00:56.302871+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:14:28.102186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.303983+0000","last_change":"2023-11-28T13:00:56.300594+0000","last_active":"2023-11-28T13:00:56.303983+0000","last_peered":"2023-11-28T13:00:56.303983+0000","last_clean":"2023-11-28T13:00:56.303983+0000","last_became_active":"2023-11-28T13:00:56.300350+0000","last_became_peered":"2023-11-28T13:00:56.300350+0000","last_unstale":"2023-11-28T13:00:56.303983+0000","last_undegraded":"2023-11-28T13:00:56.303983+0000","last_fullsized":"2023-11-28T13:00:56.303983+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:23:19.401044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.300391+0000","last_change":"2023-11-28T13:00:55.898990+0000","last_active":"2023-11-28T13:00:56.300391+0000","last_peered":"2023-11-28T13:00:56.300391+0000","last_clean":"2023-11-28T13:00:56.300391+0000","last_became_active":"2023-11-28T13:00:55.898683+0000","last_became_peered":"2023-11-28T13:00:55.898683+0000","last_unstale":"2023-11-28T13:00:56.300391+0000","last_undegraded":"2023-11-28T13:00:56.300391+0000","last_fullsized":"2023-11-28T13:00:56.300391+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:51.849483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306176+0000","last_change":"2023-11-28T13:00:56.303501+0000","last_active":"2023-11-28T13:00:56.306176+0000","last_peered":"2023-11-28T13:00:56.306176+0000","last_clean":"2023-11-28T13:00:56.306176+0000","last_became_active":"2023-11-28T13:00:56.303098+0000","last_became_peered":"2023-11-28T13:00:56.303098+0000","last_unstale":"2023-11-28T13:00:56.306176+0000","last_undegraded":"2023-11-28T13:00:56.306176+0000","last_fullsized":"2023-11-28T13:00:56.306176+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:24.340085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.295384+0000","last_change":"2023-11-28T13:00:55.900218+0000","last_active":"2023-11-28T13:00:56.295384+0000","last_peered":"2023-11-28T13:00:56.295384+0000","last_clean":"2023-11-28T13:00:56.295384+0000","last_became_active":"2023-11-28T13:00:55.899924+0000","last_became_peered":"2023-11-28T13:00:55.899924+0000","last_unstale":"2023-11-28T13:00:56.295384+0000","last_undegraded":"2023-11-28T13:00:56.295384+0000","last_fullsized":"2023-11-28T13:00:56.295384+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:41:19.536972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298850+0000","last_change":"2023-11-28T13:00:56.295890+0000","last_active":"2023-11-28T13:00:56.298850+0000","last_peered":"2023-11-28T13:00:56.298850+0000","last_clean":"2023-11-28T13:00:56.298850+0000","last_became_active":"2023-11-28T13:00:56.295321+0000","last_became_peered":"2023-11-28T13:00:56.295321+0000","last_unstale":"2023-11-28T13:00:56.298850+0000","last_undegraded":"2023-11-28T13:00:56.298850+0000","last_fullsized":"2023-11-28T13:00:56.298850+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:12.468194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.295557+0000","last_change":"2023-11-28T13:00:55.895221+0000","last_active":"2023-11-28T13:00:56.295557+0000","last_peered":"2023-11-28T13:00:56.295557+0000","last_clean":"2023-11-28T13:00:56.295557+0000","last_became_active":"2023-11-28T13:00:55.894983+0000","last_became_peered":"2023-11-28T13:00:55.894983+0000","last_unstale":"2023-11-28T13:00:56.295557+0000","last_undegraded":"2023-11-28T13:00:56.295557+0000","last_fullsized":"2023-11-28T13:00:56.295557+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:44:31.904332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307712+0000","last_change":"2023-11-28T13:00:56.305738+0000","last_active":"2023-11-28T13:00:56.307712+0000","last_peered":"2023-11-28T13:00:56.307712+0000","last_clean":"2023-11-28T13:00:56.307712+0000","last_became_active":"2023-11-28T13:00:56.305034+0000","last_became_peered":"2023-11-28T13:00:56.305034+0000","last_unstale":"2023-11-28T13:00:56.307712+0000","last_undegraded":"2023-11-28T13:00:56.307712+0000","last_fullsized":"2023-11-28T13:00:56.307712+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:24.340085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298596+0000","last_change":"2023-11-28T13:00:55.897110+0000","last_active":"2023-11-28T13:00:56.298596+0000","last_peered":"2023-11-28T13:00:56.298596+0000","last_clean":"2023-11-28T13:00:56.298596+0000","last_became_active":"2023-11-28T13:00:55.896835+0000","last_became_peered":"2023-11-28T13:00:55.896835+0000","last_unstale":"2023-11-28T13:00:56.298596+0000","last_undegraded":"2023-11-28T13:00:56.298596+0000","last_fullsized":"2023-11-28T13:00:56.298596+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:24:52.300564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.297707+0000","last_change":"2023-11-28T13:00:56.294119+0000","last_active":"2023-11-28T13:00:56.297707+0000","last_peered":"2023-11-28T13:00:56.297707+0000","last_clean":"2023-11-28T13:00:56.297707+0000","last_became_active":"2023-11-28T13:00:56.293824+0000","last_became_peered":"2023-11-28T13:00:56.293824+0000","last_unstale":"2023-11-28T13:00:56.297707+0000","last_undegraded":"2023-11-28T13:00:56.297707+0000","last_fullsized":"2023-11-28T13:00:56.297707+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:09:20.885653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.276161+0000","last_change":"2023-11-28T13:00:55.896198+0000","last_active":"2023-11-28T13:00:56.276161+0000","last_peered":"2023-11-28T13:00:56.276161+0000","last_clean":"2023-11-28T13:00:56.276161+0000","last_became_active":"2023-11-28T13:00:55.895752+0000","last_became_peered":"2023-11-28T13:00:55.895752+0000","last_unstale":"2023-11-28T13:00:56.276161+0000","last_undegraded":"2023-11-28T13:00:56.276161+0000","last_fullsized":"2023-11-28T13:00:56.276161+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:50:36.934616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.297895+0000","last_change":"2023-11-28T13:00:56.293727+0000","last_active":"2023-11-28T13:00:56.297895+0000","last_peered":"2023-11-28T13:00:56.297895+0000","last_clean":"2023-11-28T13:00:56.297895+0000","last_became_active":"2023-11-28T13:00:56.293291+0000","last_became_peered":"2023-11-28T13:00:56.293291+0000","last_unstale":"2023-11-28T13:00:56.297895+0000","last_undegraded":"2023-11-28T13:00:56.297895+0000","last_fullsized":"2023-11-28T13:00:56.297895+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:35:22.914408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.281161+0000","last_change":"2023-11-28T13:00:55.903981+0000","last_active":"2023-11-28T13:00:56.281161+0000","last_peered":"2023-11-28T13:00:56.281161+0000","last_clean":"2023-11-28T13:00:56.281161+0000","last_became_active":"2023-11-28T13:00:55.903778+0000","last_became_peered":"2023-11-28T13:00:55.903778+0000","last_unstale":"2023-11-28T13:00:56.281161+0000","last_undegraded":"2023-11-28T13:00:56.281161+0000","last_fullsized":"2023-11-28T13:00:56.281161+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:59:21.032880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.297782+0000","last_change":"2023-11-28T13:00:56.288864+0000","last_active":"2023-11-28T13:00:56.297782+0000","last_peered":"2023-11-28T13:00:56.297782+0000","last_clean":"2023-11-28T13:00:56.297782+0000","last_became_active":"2023-11-28T13:00:56.288593+0000","last_became_peered":"2023-11-28T13:00:56.288593+0000","last_unstale":"2023-11-28T13:00:56.297782+0000","last_undegraded":"2023-11-28T13:00:56.297782+0000","last_fullsized":"2023-11-28T13:00:56.297782+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:24.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.290084+0000","last_change":"2023-11-28T13:00:55.897376+0000","last_active":"2023-11-28T13:00:56.290084+0000","last_peered":"2023-11-28T13:00:56.290084+0000","last_clean":"2023-11-28T13:00:56.290084+0000","last_became_active":"2023-11-28T13:00:55.897019+0000","last_became_peered":"2023-11-28T13:00:55.897019+0000","last_unstale":"2023-11-28T13:00:56.290084+0000","last_undegraded":"2023-11-28T13:00:56.290084+0000","last_fullsized":"2023-11-28T13:00:56.290084+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:28:25.712868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301654+0000","last_change":"2023-11-28T13:00:56.299867+0000","last_active":"2023-11-28T13:00:56.301654+0000","last_peered":"2023-11-28T13:00:56.301654+0000","last_clean":"2023-11-28T13:00:56.301654+0000","last_became_active":"2023-11-28T13:00:56.299508+0000","last_became_peered":"2023-11-28T13:00:56.299508+0000","last_unstale":"2023-11-28T13:00:56.301654+0000","last_undegraded":"2023-11-28T13:00:56.301654+0000","last_fullsized":"2023-11-28T13:00:56.301654+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:28:45.846972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.286197+0000","last_change":"2023-11-28T13:00:55.901780+0000","last_active":"2023-11-28T13:00:56.286197+0000","last_peered":"2023-11-28T13:00:56.286197+0000","last_clean":"2023-11-28T13:00:56.286197+0000","last_became_active":"2023-11-28T13:00:55.901535+0000","last_became_peered":"2023-11-28T13:00:55.901535+0000","last_unstale":"2023-11-28T13:00:56.286197+0000","last_undegraded":"2023-11-28T13:00:56.286197+0000","last_fullsized":"2023-11-28T13:00:56.286197+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:58:00.945059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.287734+0000","last_change":"2023-11-28T13:00:56.282982+0000","last_active":"2023-11-28T13:00:56.287734+0000","last_peered":"2023-11-28T13:00:56.287734+0000","last_clean":"2023-11-28T13:00:56.287734+0000","last_became_active":"2023-11-28T13:00:56.282709+0000","last_became_peered":"2023-11-28T13:00:56.282709+0000","last_unstale":"2023-11-28T13:00:56.287734+0000","last_undegraded":"2023-11-28T13:00:56.287734+0000","last_fullsized":"2023-11-28T13:00:56.287734+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:27:51.480157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"54'3","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301107+0000","last_change":"2023-11-28T13:00:55.900946+0000","last_active":"2023-11-28T13:00:56.301107+0000","last_peered":"2023-11-28T13:00:56.301107+0000","last_clean":"2023-11-28T13:00:56.301107+0000","last_became_active":"2023-11-28T13:00:55.900765+0000","last_became_peered":"2023-11-28T13:00:55.900765+0000","last_unstale":"2023-11-28T13:00:56.301107+0000","last_undegraded":"2023-11-28T13:00:56.301107+0000","last_fullsized":"2023-11-28T13:00:56.301107+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:36:17.733575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301689+0000","last_change":"2023-11-28T13:00:56.300059+0000","last_active":"2023-11-28T13:00:56.301689+0000","last_peered":"2023-11-28T13:00:56.301689+0000","last_clean":"2023-11-28T13:00:56.301689+0000","last_became_active":"2023-11-28T13:00:56.299451+0000","last_became_peered":"2023-11-28T13:00:56.299451+0000","last_unstale":"2023-11-28T13:00:56.301689+0000","last_undegraded":"2023-11-28T13:00:56.301689+0000","last_fullsized":"2023-11-28T13:00:56.301689+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:36:13.832348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"54'6","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307117+0000","last_change":"2023-11-28T13:00:55.893615+0000","last_active":"2023-11-28T13:00:56.307117+0000","last_peered":"2023-11-28T13:00:56.307117+0000","last_clean":"2023-11-28T13:00:56.307117+0000","last_became_active":"2023-11-28T13:00:55.893349+0000","last_became_peered":"2023-11-28T13:00:55.893349+0000","last_unstale":"2023-11-28T13:00:56.307117+0000","last_undegraded":"2023-11-28T13:00:56.307117+0000","last_fullsized":"2023-11-28T13:00:56.307117+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:14:35.121834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.288929+0000","last_change":"2023-11-28T13:00:56.283129+0000","last_active":"2023-11-28T13:00:56.288929+0000","last_peered":"2023-11-28T13:00:56.288929+0000","last_clean":"2023-11-28T13:00:56.288929+0000","last_became_active":"2023-11-28T13:00:56.282517+0000","last_became_peered":"2023-11-28T13:00:56.282517+0000","last_unstale":"2023-11-28T13:00:56.288929+0000","last_undegraded":"2023-11-28T13:00:56.288929+0000","last_fullsized":"2023-11-28T13:00:56.288929+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:28:45.846972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.305601+0000","last_change":"2023-11-28T13:00:55.903352+0000","last_active":"2023-11-28T13:00:56.305601+0000","last_peered":"2023-11-28T13:00:56.305601+0000","last_clean":"2023-11-28T13:00:56.305601+0000","last_became_active":"2023-11-28T13:00:55.903045+0000","last_became_peered":"2023-11-28T13:00:55.903045+0000","last_unstale":"2023-11-28T13:00:56.305601+0000","last_undegraded":"2023-11-28T13:00:56.305601+0000","last_fullsized":"2023-11-28T13:00:56.305601+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:24:52.300564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.300884+0000","last_change":"2023-11-28T13:00:56.296576+0000","last_active":"2023-11-28T13:00:56.300884+0000","last_peered":"2023-11-28T13:00:56.300884+0000","last_clean":"2023-11-28T13:00:56.300884+0000","last_became_active":"2023-11-28T13:00:56.296189+0000","last_became_peered":"2023-11-28T13:00:56.296189+0000","last_unstale":"2023-11-28T13:00:56.300884+0000","last_undegraded":"2023-11-28T13:00:56.300884+0000","last_fullsized":"2023-11-28T13:00:56.300884+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:03:44.012770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.278516+0000","last_change":"2023-11-28T13:00:55.890105+0000","last_active":"2023-11-28T13:00:56.278516+0000","last_peered":"2023-11-28T13:00:56.278516+0000","last_clean":"2023-11-28T13:00:56.278516+0000","last_became_active":"2023-11-28T13:00:55.889879+0000","last_became_peered":"2023-11-28T13:00:55.889879+0000","last_unstale":"2023-11-28T13:00:56.278516+0000","last_undegraded":"2023-11-28T13:00:56.278516+0000","last_fullsized":"2023-11-28T13:00:56.278516+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:38:54.989478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301349+0000","last_change":"2023-11-28T13:00:56.294879+0000","last_active":"2023-11-28T13:00:56.301349+0000","last_peered":"2023-11-28T13:00:56.301349+0000","last_clean":"2023-11-28T13:00:56.301349+0000","last_became_active":"2023-11-28T13:00:56.294470+0000","last_became_peered":"2023-11-28T13:00:56.294470+0000","last_unstale":"2023-11-28T13:00:56.301349+0000","last_undegraded":"2023-11-28T13:00:56.301349+0000","last_fullsized":"2023-11-28T13:00:56.301349+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:28:49.012453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.276259+0000","last_change":"2023-11-28T13:00:55.900211+0000","last_active":"2023-11-28T13:00:56.276259+0000","last_peered":"2023-11-28T13:00:56.276259+0000","last_clean":"2023-11-28T13:00:56.276259+0000","last_became_active":"2023-11-28T13:00:55.899604+0000","last_became_peered":"2023-11-28T13:00:55.899604+0000","last_unstale":"2023-11-28T13:00:56.276259+0000","last_undegraded":"2023-11-28T13:00:56.276259+0000","last_fullsized":"2023-11-28T13:00:56.276259+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:34:04.255852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306264+0000","last_change":"2023-11-28T13:00:56.303325+0000","last_active":"2023-11-28T13:00:56.306264+0000","last_peered":"2023-11-28T13:00:56.306264+0000","last_clean":"2023-11-28T13:00:56.306264+0000","last_became_active":"2023-11-28T13:00:56.302995+0000","last_became_peered":"2023-11-28T13:00:56.302995+0000","last_unstale":"2023-11-28T13:00:56.306264+0000","last_undegraded":"2023-11-28T13:00:56.306264+0000","last_fullsized":"2023-11-28T13:00:56.306264+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:43:52.856174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"54'8","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.298557+0000","last_change":"2023-11-28T13:00:55.897412+0000","last_active":"2023-11-28T13:00:56.298557+0000","last_peered":"2023-11-28T13:00:56.298557+0000","last_clean":"2023-11-28T13:00:56.298557+0000","last_became_active":"2023-11-28T13:00:55.897122+0000","last_became_peered":"2023-11-28T13:00:55.897122+0000","last_unstale":"2023-11-28T13:00:56.298557+0000","last_undegraded":"2023-11-28T13:00:56.298557+0000","last_fullsized":"2023-11-28T13:00:56.298557+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:27:50.473466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.295509+0000","last_change":"2023-11-28T13:00:56.292966+0000","last_active":"2023-11-28T13:00:56.295509+0000","last_peered":"2023-11-28T13:00:56.295509+0000","last_clean":"2023-11-28T13:00:56.295509+0000","last_became_active":"2023-11-28T13:00:56.292666+0000","last_became_peered":"2023-11-28T13:00:56.292666+0000","last_unstale":"2023-11-28T13:00:56.295509+0000","last_undegraded":"2023-11-28T13:00:56.295509+0000","last_fullsized":"2023-11-28T13:00:56.295509+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:58:47.885007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.295699+0000","last_change":"2023-11-28T13:00:55.903819+0000","last_active":"2023-11-28T13:00:56.295699+0000","last_peered":"2023-11-28T13:00:56.295699+0000","last_clean":"2023-11-28T13:00:56.295699+0000","last_became_active":"2023-11-28T13:00:55.903549+0000","last_became_peered":"2023-11-28T13:00:55.903549+0000","last_unstale":"2023-11-28T13:00:56.295699+0000","last_undegraded":"2023-11-28T13:00:56.295699+0000","last_fullsized":"2023-11-28T13:00:56.295699+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:00:32.746429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301212+0000","last_change":"2023-11-28T13:00:56.295826+0000","last_active":"2023-11-28T13:00:56.301212+0000","last_peered":"2023-11-28T13:00:56.301212+0000","last_clean":"2023-11-28T13:00:56.301212+0000","last_became_active":"2023-11-28T13:00:56.295517+0000","last_became_peered":"2023-11-28T13:00:56.295517+0000","last_unstale":"2023-11-28T13:00:56.301212+0000","last_undegraded":"2023-11-28T13:00:56.301212+0000","last_fullsized":"2023-11-28T13:00:56.301212+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:59:22.039571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.293195+0000","last_change":"2023-11-28T13:00:55.897215+0000","last_active":"2023-11-28T13:00:56.293195+0000","last_peered":"2023-11-28T13:00:56.293195+0000","last_clean":"2023-11-28T13:00:56.293195+0000","last_became_active":"2023-11-28T13:00:55.896978+0000","last_became_peered":"2023-11-28T13:00:55.896978+0000","last_unstale":"2023-11-28T13:00:56.293195+0000","last_undegraded":"2023-11-28T13:00:56.293195+0000","last_fullsized":"2023-11-28T13:00:56.293195+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:14:28.102186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.287714+0000","last_change":"2023-11-28T13:00:56.282281+0000","last_active":"2023-11-28T13:00:56.287714+0000","last_peered":"2023-11-28T13:00:56.287714+0000","last_clean":"2023-11-28T13:00:56.287714+0000","last_became_active":"2023-11-28T13:00:56.281940+0000","last_became_peered":"2023-11-28T13:00:56.281940+0000","last_unstale":"2023-11-28T13:00:56.287714+0000","last_undegraded":"2023-11-28T13:00:56.287714+0000","last_fullsized":"2023-11-28T13:00:56.287714+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:24:53.232769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.278601+0000","last_change":"2023-11-28T13:00:55.888527+0000","last_active":"2023-11-28T13:00:56.278601+0000","last_peered":"2023-11-28T13:00:56.278601+0000","last_clean":"2023-11-28T13:00:56.278601+0000","last_became_active":"2023-11-28T13:00:55.888251+0000","last_became_peered":"2023-11-28T13:00:55.888251+0000","last_unstale":"2023-11-28T13:00:56.278601+0000","last_undegraded":"2023-11-28T13:00:56.278601+0000","last_fullsized":"2023-11-28T13:00:56.278601+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:13:05.532344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306157+0000","last_change":"2023-11-28T13:00:56.303346+0000","last_active":"2023-11-28T13:00:56.306157+0000","last_peered":"2023-11-28T13:00:56.306157+0000","last_clean":"2023-11-28T13:00:56.306157+0000","last_became_active":"2023-11-28T13:00:56.302770+0000","last_became_peered":"2023-11-28T13:00:56.302770+0000","last_unstale":"2023-11-28T13:00:56.306157+0000","last_undegraded":"2023-11-28T13:00:56.306157+0000","last_fullsized":"2023-11-28T13:00:56.306157+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:47:29.484502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.277048+0000","last_change":"2023-11-28T13:00:55.893693+0000","last_active":"2023-11-28T13:00:56.277048+0000","last_peered":"2023-11-28T13:00:56.277048+0000","last_clean":"2023-11-28T13:00:56.277048+0000","last_became_active":"2023-11-28T13:00:55.893432+0000","last_became_peered":"2023-11-28T13:00:55.893432+0000","last_unstale":"2023-11-28T13:00:56.277048+0000","last_undegraded":"2023-11-28T13:00:56.277048+0000","last_fullsized":"2023-11-28T13:00:56.277048+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:52:10.079823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307427+0000","last_change":"2023-11-28T13:00:56.303392+0000","last_active":"2023-11-28T13:00:56.307427+0000","last_peered":"2023-11-28T13:00:56.307427+0000","last_clean":"2023-11-28T13:00:56.307427+0000","last_became_active":"2023-11-28T13:00:56.303151+0000","last_became_peered":"2023-11-28T13:00:56.303151+0000","last_unstale":"2023-11-28T13:00:56.307427+0000","last_undegraded":"2023-11-28T13:00:56.307427+0000","last_fullsized":"2023-11-28T13:00:56.307427+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:44:18.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.278103+0000","last_change":"2023-11-28T13:00:55.893605+0000","last_active":"2023-11-28T13:00:56.278103+0000","last_peered":"2023-11-28T13:00:56.278103+0000","last_clean":"2023-11-28T13:00:56.278103+0000","last_became_active":"2023-11-28T13:00:55.893001+0000","last_became_peered":"2023-11-28T13:00:55.893001+0000","last_unstale":"2023-11-28T13:00:56.278103+0000","last_undegraded":"2023-11-28T13:00:56.278103+0000","last_fullsized":"2023-11-28T13:00:56.278103+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:39:10.332224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.306286+0000","last_change":"2023-11-28T13:00:56.303351+0000","last_active":"2023-11-28T13:00:56.306286+0000","last_peered":"2023-11-28T13:00:56.306286+0000","last_clean":"2023-11-28T13:00:56.306286+0000","last_became_active":"2023-11-28T13:00:56.303049+0000","last_became_peered":"2023-11-28T13:00:56.303049+0000","last_unstale":"2023-11-28T13:00:56.306286+0000","last_undegraded":"2023-11-28T13:00:56.306286+0000","last_fullsized":"2023-11-28T13:00:56.306286+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:47:11.371831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.281308+0000","last_change":"2023-11-28T13:00:55.893633+0000","last_active":"2023-11-28T13:00:56.281308+0000","last_peered":"2023-11-28T13:00:56.281308+0000","last_clean":"2023-11-28T13:00:56.281308+0000","last_became_active":"2023-11-28T13:00:55.893360+0000","last_became_peered":"2023-11-28T13:00:55.893360+0000","last_unstale":"2023-11-28T13:00:56.281308+0000","last_undegraded":"2023-11-28T13:00:56.281308+0000","last_fullsized":"2023-11-28T13:00:56.281308+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:23:18.394353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.287253+0000","last_change":"2023-11-28T13:00:56.281619+0000","last_active":"2023-11-28T13:00:56.287253+0000","last_peered":"2023-11-28T13:00:56.287253+0000","last_clean":"2023-11-28T13:00:56.287253+0000","last_became_active":"2023-11-28T13:00:56.281365+0000","last_became_peered":"2023-11-28T13:00:56.281365+0000","last_unstale":"2023-11-28T13:00:56.287253+0000","last_undegraded":"2023-11-28T13:00:56.287253+0000","last_fullsized":"2023-11-28T13:00:56.287253+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:28:26.719559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.277957+0000","last_change":"2023-11-28T13:00:55.893618+0000","last_active":"2023-11-28T13:00:56.277957+0000","last_peered":"2023-11-28T13:00:56.277957+0000","last_clean":"2023-11-28T13:00:56.277957+0000","last_became_active":"2023-11-28T13:00:55.893287+0000","last_became_peered":"2023-11-28T13:00:55.893287+0000","last_unstale":"2023-11-28T13:00:56.277957+0000","last_undegraded":"2023-11-28T13:00:56.277957+0000","last_fullsized":"2023-11-28T13:00:56.277957+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:52:09.384071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.297874+0000","last_change":"2023-11-28T13:00:56.293551+0000","last_active":"2023-11-28T13:00:56.297874+0000","last_peered":"2023-11-28T13:00:56.297874+0000","last_clean":"2023-11-28T13:00:56.297874+0000","last_became_active":"2023-11-28T13:00:56.292953+0000","last_became_peered":"2023-11-28T13:00:56.292953+0000","last_unstale":"2023-11-28T13:00:56.297874+0000","last_undegraded":"2023-11-28T13:00:56.297874+0000","last_fullsized":"2023-11-28T13:00:56.297874+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:58:42.270054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.284790+0000","last_change":"2023-11-28T13:00:55.901793+0000","last_active":"2023-11-28T13:00:56.284790+0000","last_peered":"2023-11-28T13:00:56.284790+0000","last_clean":"2023-11-28T13:00:56.284790+0000","last_became_active":"2023-11-28T13:00:55.901511+0000","last_became_peered":"2023-11-28T13:00:55.901511+0000","last_unstale":"2023-11-28T13:00:56.284790+0000","last_undegraded":"2023-11-28T13:00:56.284790+0000","last_fullsized":"2023-11-28T13:00:56.284790+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:18:12.918193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.289469+0000","last_change":"2023-11-28T13:00:56.282854+0000","last_active":"2023-11-28T13:00:56.289469+0000","last_peered":"2023-11-28T13:00:56.289469+0000","last_clean":"2023-11-28T13:00:56.289469+0000","last_became_active":"2023-11-28T13:00:56.282485+0000","last_became_peered":"2023-11-28T13:00:56.282485+0000","last_unstale":"2023-11-28T13:00:56.289469+0000","last_undegraded":"2023-11-28T13:00:56.289469+0000","last_fullsized":"2023-11-28T13:00:56.289469+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:47:11.371831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.304163+0000","last_change":"2023-11-28T13:00:55.898981+0000","last_active":"2023-11-28T13:00:56.304163+0000","last_peered":"2023-11-28T13:00:56.304163+0000","last_clean":"2023-11-28T13:00:56.304163+0000","last_became_active":"2023-11-28T13:00:55.898727+0000","last_became_peered":"2023-11-28T13:00:55.898727+0000","last_unstale":"2023-11-28T13:00:56.304163+0000","last_undegraded":"2023-11-28T13:00:56.304163+0000","last_fullsized":"2023-11-28T13:00:56.304163+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:06.820051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":43,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.279371+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T13:00:56.279371+0000","last_peered":"2023-11-28T13:00:56.279371+0000","last_clean":"2023-11-28T13:00:56.279371+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:00:56.279371+0000","last_undegraded":"2023-11-28T13:00:56.279371+0000","last_fullsized":"2023-11-28T13:00:56.279371+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301607+0000","last_change":"2023-11-28T13:00:56.298759+0000","last_active":"2023-11-28T13:00:56.301607+0000","last_peered":"2023-11-28T13:00:56.301607+0000","last_clean":"2023-11-28T13:00:56.301607+0000","last_became_active":"2023-11-28T13:00:56.298469+0000","last_became_peered":"2023-11-28T13:00:56.298469+0000","last_unstale":"2023-11-28T13:00:56.301607+0000","last_undegraded":"2023-11-28T13:00:56.301607+0000","last_fullsized":"2023-11-28T13:00:56.301607+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:21:24.326680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.276120+0000","last_change":"2023-11-28T13:00:55.904336+0000","last_active":"2023-11-28T13:00:56.276120+0000","last_peered":"2023-11-28T13:00:56.276120+0000","last_clean":"2023-11-28T13:00:56.276120+0000","last_became_active":"2023-11-28T13:00:55.904145+0000","last_became_peered":"2023-11-28T13:00:55.904145+0000","last_unstale":"2023-11-28T13:00:56.276120+0000","last_undegraded":"2023-11-28T13:00:56.276120+0000","last_fullsized":"2023-11-28T13:00:56.276120+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:34:49.706762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.277876+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T13:00:56.277876+0000","last_peered":"2023-11-28T13:00:56.277876+0000","last_clean":"2023-11-28T13:00:56.277876+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:00:56.277876+0000","last_undegraded":"2023-11-28T13:00:56.277876+0000","last_fullsized":"2023-11-28T13:00:56.277876+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.288908+0000","last_change":"2023-11-28T13:00:56.283012+0000","last_active":"2023-11-28T13:00:56.288908+0000","last_peered":"2023-11-28T13:00:56.288908+0000","last_clean":"2023-11-28T13:00:56.288908+0000","last_became_active":"2023-11-28T13:00:56.282722+0000","last_became_peered":"2023-11-28T13:00:56.282722+0000","last_unstale":"2023-11-28T13:00:56.288908+0000","last_undegraded":"2023-11-28T13:00:56.288908+0000","last_fullsized":"2023-11-28T13:00:56.288908+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:36:13.832348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.305312+0000","last_change":"2023-11-28T13:00:55.902610+0000","last_active":"2023-11-28T13:00:56.305312+0000","last_peered":"2023-11-28T13:00:56.305312+0000","last_clean":"2023-11-28T13:00:56.305312+0000","last_became_active":"2023-11-28T13:00:55.902224+0000","last_became_peered":"2023-11-28T13:00:55.902224+0000","last_unstale":"2023-11-28T13:00:56.305312+0000","last_undegraded":"2023-11-28T13:00:56.305312+0000","last_fullsized":"2023-11-28T13:00:56.305312+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:46:36.832924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":39,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.280027+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T13:00:56.280027+0000","last_peered":"2023-11-28T13:00:56.280027+0000","last_clean":"2023-11-28T13:00:56.280027+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:00:56.280027+0000","last_undegraded":"2023-11-28T13:00:56.280027+0000","last_fullsized":"2023-11-28T13:00:56.280027+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307441+0000","last_change":"2023-11-28T13:00:56.303379+0000","last_active":"2023-11-28T13:00:56.307441+0000","last_peered":"2023-11-28T13:00:56.307441+0000","last_clean":"2023-11-28T13:00:56.307441+0000","last_became_active":"2023-11-28T13:00:56.302900+0000","last_became_peered":"2023-11-28T13:00:56.302900+0000","last_unstale":"2023-11-28T13:00:56.307441+0000","last_undegraded":"2023-11-28T13:00:56.307441+0000","last_fullsized":"2023-11-28T13:00:56.307441+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:08.749520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.277910+0000","last_change":"2023-11-28T13:00:55.896258+0000","last_active":"2023-11-28T13:00:56.277910+0000","last_peered":"2023-11-28T13:00:56.277910+0000","last_clean":"2023-11-28T13:00:56.277910+0000","last_became_active":"2023-11-28T13:00:55.895925+0000","last_became_peered":"2023-11-28T13:00:55.895925+0000","last_unstale":"2023-11-28T13:00:56.277910+0000","last_undegraded":"2023-11-28T13:00:56.277910+0000","last_fullsized":"2023-11-28T13:00:56.277910+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:46:39.890621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":40,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.276286+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T13:00:56.276286+0000","last_peered":"2023-11-28T13:00:56.276286+0000","last_clean":"2023-11-28T13:00:56.276286+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:00:56.276286+0000","last_undegraded":"2023-11-28T13:00:56.276286+0000","last_fullsized":"2023-11-28T13:00:56.276286+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":194,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-28T13:00:55.887106+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T13:00:55.887106+0000","last_peered":"2023-11-28T13:00:55.887106+0000","last_clean":"2023-11-28T13:00:55.887106+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:00:55.887106+0000","last_undegraded":"2023-11-28T13:00:55.887106+0000","last_fullsized":"2023-11-28T13:00:55.887106+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301503+0000","last_change":"2023-11-28T13:00:56.298014+0000","last_active":"2023-11-28T13:00:56.301503+0000","last_peered":"2023-11-28T13:00:56.301503+0000","last_clean":"2023-11-28T13:00:56.301503+0000","last_became_active":"2023-11-28T13:00:56.297776+0000","last_became_peered":"2023-11-28T13:00:56.297776+0000","last_unstale":"2023-11-28T13:00:56.301503+0000","last_undegraded":"2023-11-28T13:00:56.301503+0000","last_fullsized":"2023-11-28T13:00:56.301503+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:11:08.749520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.278906+0000","last_change":"2023-11-28T13:00:55.894723+0000","last_active":"2023-11-28T13:00:56.278906+0000","last_peered":"2023-11-28T13:00:56.278906+0000","last_clean":"2023-11-28T13:00:56.278906+0000","last_became_active":"2023-11-28T13:00:55.894507+0000","last_became_peered":"2023-11-28T13:00:55.894507+0000","last_unstale":"2023-11-28T13:00:56.278906+0000","last_undegraded":"2023-11-28T13:00:56.278906+0000","last_fullsized":"2023-11-28T13:00:56.278906+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:30:18.611882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-28T13:00:55.887195+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T13:00:55.887195+0000","last_peered":"2023-11-28T13:00:55.887195+0000","last_clean":"2023-11-28T13:00:55.887195+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:00:55.887195+0000","last_undegraded":"2023-11-28T13:00:55.887195+0000","last_fullsized":"2023-11-28T13:00:55.887195+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301561+0000","last_change":"2023-11-28T13:00:56.298856+0000","last_active":"2023-11-28T13:00:56.301561+0000","last_peered":"2023-11-28T13:00:56.301561+0000","last_clean":"2023-11-28T13:00:56.301561+0000","last_became_active":"2023-11-28T13:00:56.298533+0000","last_became_peered":"2023-11-28T13:00:56.298533+0000","last_unstale":"2023-11-28T13:00:56.301561+0000","last_undegraded":"2023-11-28T13:00:56.301561+0000","last_fullsized":"2023-11-28T13:00:56.301561+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:41:20.543663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.302589+0000","last_change":"2023-11-28T13:00:55.901799+0000","last_active":"2023-11-28T13:00:56.302589+0000","last_peered":"2023-11-28T13:00:56.302589+0000","last_clean":"2023-11-28T13:00:56.302589+0000","last_became_active":"2023-11-28T13:00:55.901524+0000","last_became_peered":"2023-11-28T13:00:55.901524+0000","last_unstale":"2023-11-28T13:00:56.302589+0000","last_undegraded":"2023-11-28T13:00:56.302589+0000","last_fullsized":"2023-11-28T13:00:56.302589+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:23:57.311139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.289811+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T13:00:56.289811+0000","last_peered":"2023-11-28T13:00:56.289811+0000","last_clean":"2023-11-28T13:00:56.289811+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:00:56.289811+0000","last_undegraded":"2023-11-28T13:00:56.289811+0000","last_fullsized":"2023-11-28T13:00:56.289811+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.301665+0000","last_change":"2023-11-28T13:00:56.299872+0000","last_active":"2023-11-28T13:00:56.301665+0000","last_peered":"2023-11-28T13:00:56.301665+0000","last_clean":"2023-11-28T13:00:56.301665+0000","last_became_active":"2023-11-28T13:00:56.299171+0000","last_became_peered":"2023-11-28T13:00:56.299171+0000","last_unstale":"2023-11-28T13:00:56.301665+0000","last_undegraded":"2023-11-28T13:00:56.301665+0000","last_fullsized":"2023-11-28T13:00:56.301665+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:07.826742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.295916+0000","last_change":"2023-11-28T13:00:55.904411+0000","last_active":"2023-11-28T13:00:56.295916+0000","last_peered":"2023-11-28T13:00:56.295916+0000","last_clean":"2023-11-28T13:00:56.295916+0000","last_became_active":"2023-11-28T13:00:55.904130+0000","last_became_peered":"2023-11-28T13:00:55.904130+0000","last_unstale":"2023-11-28T13:00:56.295916+0000","last_undegraded":"2023-11-28T13:00:56.295916+0000","last_fullsized":"2023-11-28T13:00:56.295916+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:58:00.945059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-28T13:00:55.890209+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T13:00:55.890209+0000","last_peered":"2023-11-28T13:00:55.890209+0000","last_clean":"2023-11-28T13:00:55.890209+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:00:55.890209+0000","last_undegraded":"2023-11-28T13:00:55.890209+0000","last_fullsized":"2023-11-28T13:00:55.890209+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.307494+0000","last_change":"2023-11-28T13:00:56.303702+0000","last_active":"2023-11-28T13:00:56.307494+0000","last_peered":"2023-11-28T13:00:56.307494+0000","last_clean":"2023-11-28T13:00:56.307494+0000","last_became_active":"2023-11-28T13:00:56.303257+0000","last_became_peered":"2023-11-28T13:00:56.303257+0000","last_unstale":"2023-11-28T13:00:56.307494+0000","last_undegraded":"2023-11-28T13:00:56.307494+0000","last_fullsized":"2023-11-28T13:00:56.307494+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:55.871276+0000","last_clean_scrub_stamp":"2023-11-28T13:00:55.871276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:47:29.484502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2023-11-28T13:00:56.296946+0000","last_change":"2023-11-28T13:00:55.900175+0000","last_active":"2023-11-28T13:00:56.296946+0000","last_peered":"2023-11-28T13:00:56.296946+0000","last_clean":"2023-11-28T13:00:56.296946+0000","last_became_active":"2023-11-28T13:00:55.899889+0000","last_became_peered":"2023-11-28T13:00:55.899889+0000","last_unstale":"2023-11-28T13:00:56.296946+0000","last_undegraded":"2023-11-28T13:00:56.296946+0000","last_fullsized":"2023-11-28T13:00:56.296946+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:00:54.864585+0000","last_clean_scrub_stamp":"2023-11-28T13:00:54.864585+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:03:43.006079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":46,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-11-28T13:00:55.883351+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T13:00:55.883351+0000","last_peered":"2023-11-28T13:00:55.883351+0000","last_clean":"2023-11-28T13:00:55.883351+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:00:55.883351+0000","last_undegraded":"2023-11-28T13:00:55.883351+0000","last_fullsized":"2023-11-28T13:00:55.883351+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10312,"kb_used_data":1216,"kb_used_omap":0,"kb_used_meta":9088,"kb_avail":94361528,"statfs":{"total":96636764160,"available":96626204672,"internally_reserved":0,"allocated":1245184,"data_stored":887519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9306112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.64700000000000002,"15min":0.67000000000000004},"min":{"1min":0.34000000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66400000000000003,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":0.504},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.65700000000000003,"15min":0.68500000000000005},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63300000000000001,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.52300000000000002}]},{"osd":1,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.41499999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.60799999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.501,"15min":0.50700000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64100000000000001,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.63500000000000001}]},{"osd":2,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.54000000000000004,"15min":0.55000000000000004},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.628,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53600000000000003,"15min":0.54100000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67300000000000004,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.55800000000000005}]},{"osd":3,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.67800000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56299999999999994,"15min":0.56599999999999995},"min":{"1min":0.39000000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66300000000000003,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.59999999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.33000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.68799999999999994,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53300000000000003,"15min":0.53600000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.623,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.90900000000000003}]},{"osd":5,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.57199999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.73399999999999999}]},{"osd":6,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.46100000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.70099999999999996,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.80000000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9060,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362780,"statfs":{"total":96636764160,"available":96627486720,"internally_reserved":0,"allocated":618496,"data_stored":282603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.69699999999999995,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.51500000000000001,"15min":0.51300000000000001},"min":{"1min":0.35999999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.436}]},{"osd":1,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.36299999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.41399999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77700000000000002,"5min":1.016,"15min":1.016},"last":0.629}]},{"osd":2,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.41399999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.503},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81599999999999995,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.59199999999999997}]},{"osd":3,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.44700000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.67500000000000004,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57699999999999996,"15min":0.57999999999999996},"min":{"1min":0.42599999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84899999999999998,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.64600000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.53200000000000003,"15min":0.54000000000000004},"min":{"1min":0.38200000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.63400000000000001,"5min":1,"15min":1},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.53900000000000003,"15min":0.54300000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.749,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.56399999999999995}]},{"osd":5,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68100000000000005,"5min":1.052,"15min":1.052},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.626,"15min":0.63200000000000001},"min":{"1min":0.39200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89900000000000002,"5min":1.258,"15min":1.258},"last":0.66900000000000004}]},{"osd":7,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.42299999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.872,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.41799999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.95199999999999996,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.748}]}]},{"osd":1,"up_from":13,"seq":55834574878,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9060,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362780,"statfs":{"total":96636764160,"available":96627486720,"internally_reserved":0,"allocated":618496,"data_stored":282603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.23999999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.505,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.35799999999999998,"15min":0.35399999999999998},"min":{"1min":0.24399999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.54000000000000004,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.309}]},{"osd":2,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.33700000000000002,"15min":0.33300000000000002},"min":{"1min":0.29599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.48199999999999998,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.36899999999999999,"15min":0.36699999999999999},"min":{"1min":0.27000000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.56999999999999995,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.28499999999999998}]},{"osd":3,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.39100000000000001,"15min":0.38900000000000001},"min":{"1min":0.28499999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.53700000000000003,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.39300000000000002,"15min":0.39400000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.58499999999999996,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.36899999999999999}]},{"osd":4,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57199999999999995,"15min":0.56599999999999995},"min":{"1min":0.39200000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81100000000000005,"5min":1.028,"15min":1.028},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.435,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84799999999999998,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.85899999999999999}]},{"osd":5,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59599999999999997,"15min":0.59099999999999997},"min":{"1min":0.46300000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.874,"5min":1.042,"15min":1.042},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.56799999999999995,"15min":0.55700000000000005},"min":{"1min":0.40999999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.79200000000000004}]},{"osd":6,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.40000000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.87,"5min":1.006,"15min":1.006},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60099999999999998,"15min":0.59799999999999998},"min":{"1min":0.42599999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85899999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.65400000000000003}]},{"osd":7,"last update":"Tue Nov 28 13:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57899999999999996,"15min":0.57299999999999995},"min":{"1min":0.42499999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90000000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.64400000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10292,"kb_used_data":1196,"kb_used_omap":0,"kb_used_meta":9088,"kb_avail":94361548,"statfs":{"total":96636764160,"available":96626225152,"internally_reserved":0,"allocated":1224704,"data_stored":879911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9306112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.32400000000000001,"15min":0.32200000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34200000000000003,"15min":0.34100000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.32000000000000001}]},{"osd":2,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.34999999999999998,"15min":0.34499999999999997},"min":{"1min":0.26100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36699999999999999,"15min":0.36399999999999999},"min":{"1min":0.27700000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.29099999999999998}]},{"osd":3,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.37,"15min":0.372},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.55100000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38900000000000001,"15min":0.38600000000000001},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.33200000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56699999999999995,"15min":0.56899999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.70999999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64300000000000002}]},{"osd":5,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.432,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.754,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.56200000000000006}]},{"osd":6,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.73399999999999999,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60899999999999999,"15min":0.61599999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72299999999999998,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.56699999999999995}]},{"osd":7,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.627,"15min":0.629},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.78600000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.621,"15min":0.628},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.71899999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.623}]}]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9008,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362832,"statfs":{"total":96636764160,"available":96627539968,"internally_reserved":0,"allocated":630784,"data_stored":289543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.32500000000000001,"15min":0.316},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.33700000000000002,"15min":0.33300000000000002},"min":{"1min":0.249,"5min":0.184,"15min":0.184},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.432}]},{"osd":1,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38100000000000001,"15min":0.38200000000000001},"min":{"1min":0.28599999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.35199999999999998},"min":{"1min":0.29599999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51300000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.47599999999999998}]},{"osd":3,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.33800000000000002},"min":{"1min":0.26600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45400000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.376},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.28999999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.60799999999999998,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.48999999999999999}]},{"osd":4,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.41399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79100000000000004,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.79100000000000004}]},{"osd":5,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.45700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.83199999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82399999999999995,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.78000000000000003}]},{"osd":6,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.57499999999999996,"15min":0.57099999999999995},"min":{"1min":0.45000000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77700000000000002,"5min":1.002,"15min":1.002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.85299999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.76900000000000002}]},{"osd":7,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.621,"15min":0.627},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.80900000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.77500000000000002,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.69699999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9200,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":8576,"kb_avail":94362640,"statfs":{"total":96636764160,"available":96627343360,"internally_reserved":0,"allocated":630784,"data_stored":289543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8781824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.31900000000000001},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.44700000000000001,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.441},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.32700000000000001,"15min":0.32600000000000001},"min":{"1min":0.22700000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.505,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.33000000000000002}]},{"osd":1,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34000000000000002,"15min":0.33800000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.47899999999999998,"5min":0.502,"15min":0.502},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.373,"15min":0.372},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.314}]},{"osd":2,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.47699999999999998,"15min":0.49099999999999999},"min":{"1min":0.28100000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.57399999999999995,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.51900000000000002,"15min":0.53500000000000003},"min":{"1min":0.30299999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58999999999999997,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":0.371}]},{"osd":4,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.91700000000000004,"15min":0.96699999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.88200000000000001,"5min":8.2889999999999997,"15min":8.2889999999999997},"last":0.60299999999999998}]},{"osd":5,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.62,"15min":0.62},"min":{"1min":0.439,"5min":0.432,"15min":0.432},"max":{"1min":0.82899999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.66500000000000004,"15min":0.67500000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.83099999999999996,"5min":2.165,"15min":2.165},"last":0.78500000000000003}]},{"osd":6,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.628,"15min":0.627},"min":{"1min":0.46400000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.86599999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.61199999999999999}]},{"osd":7,"last update":"Tue Nov 28 13:00:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.79000000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.755},{"interface":"front","average":{"1min":0.623,"5min":0.91100000000000003,"15min":0.95899999999999996},"min":{"1min":0.45500000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81499999999999995,"5min":7.0899999999999999,"15min":7.0899999999999999},"last":0.64900000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9328,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362512,"statfs":{"total":96636764160,"available":96627212288,"internally_reserved":0,"allocated":630784,"data_stored":289543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83499999999999996,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.72199999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.54200000000000004}]},{"osd":1,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.38700000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41199999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70499999999999996,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.51800000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.55400000000000005,"15min":0.54700000000000004},"min":{"1min":0.441,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.876,"5min":1.069,"15min":1.069},"last":0.46000000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.51800000000000002,"15min":0.50900000000000001},"min":{"1min":0.39600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58499999999999996,"15min":0.58299999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85599999999999998,"5min":0.879,"15min":0.879},"last":0.53000000000000003}]},{"osd":5,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51600000000000001,"15min":0.51800000000000002},"min":{"1min":0.32000000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.72099999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.42799999999999999}]},{"osd":6,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69199999999999995,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58999999999999997,"15min":0.59099999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78800000000000003,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.48299999999999998}]},{"osd":7,"last update":"Tue Nov 28 13:00:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.40500000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.80100000000000005,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.44800000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8996,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362844,"statfs":{"total":96636764160,"available":96627552256,"internally_reserved":0,"allocated":618496,"data_stored":282603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.34599999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69899999999999995,"5min":1.819,"15min":1.819},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.52600000000000002,"15min":0.52100000000000002},"min":{"1min":0.40799999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73599999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.30399999999999999}]},{"osd":2,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.75800000000000001,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.63,"15min":0.64800000000000002},"min":{"1min":0.372,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.748,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.38200000000000001}]},{"osd":3,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72799999999999998,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.66300000000000003,"15min":0.68600000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71099999999999997,"5min":2.702,"15min":2.702},"last":0.40699999999999997}]},{"osd":4,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.60099999999999998,"15min":0.61899999999999999},"min":{"1min":0.30399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.66500000000000004,"5min":3.0739999999999998,"15min":3.0739999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76500000000000001,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.34399999999999997}]},{"osd":6,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.34499999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.70199999999999996,"15min":0.72699999999999998},"min":{"1min":0.39100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73099999999999998,"5min":3.7679999999999998,"15min":3.7679999999999998},"last":0.318}]},{"osd":7,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78400000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.44},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.65500000000000003,"15min":0.66600000000000004},"min":{"1min":0.41399999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.751,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:01:00.722 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T13:01:00.722 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2023-11-28T13:01:00.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.882+0000 7f52c77da700 1 Processor -- start 2023-11-28T13:01:00.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.882+0000 7f52c77da700 1 -- start start 2023-11-28T13:01:00.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.883+0000 7f52c77da700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52c00cde60 0x7f52c015d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.883+0000 7f52c77da700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52c0160ca0 0x7f52c00b1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.883+0000 7f52c77da700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c0161100 con 0x7f52c006bff0 2023-11-28T13:01:00.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.883+0000 7f52c77da700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c00b1b40 con 0x7f52c0160ca0 2023-11-28T13:01:00.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.883+0000 7f52c77da700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c00b1d50 con 0x7f52c00cde60 2023-11-28T13:01:00.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.883+0000 7f52c5d77700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52c0160ca0 0x7f52c00b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.883+0000 7f52c5d77700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52c0160ca0 0x7f52c00b1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36280/0 (socket says 172.21.15.111:36280) 2023-11-28T13:01:00.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.883+0000 7f52c5d77700 1 -- 172.21.15.111:0/4000320259 learned_addr learned my addr 172.21.15.111:0/4000320259 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:00.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.884+0000 7f52c4d75700 1 --2- 172.21.15.111:0/4000320259 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52c00cde60 0x7f52c015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.884+0000 7f52c5d77700 1 -- 172.21.15.111:0/4000320259 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52c00cde60 msgr2=0x7f52c015d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:00.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.884+0000 7f52c5d77700 1 --2- 172.21.15.111:0/4000320259 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52c00cde60 0x7f52c015d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.884+0000 7f52c5d77700 1 -- 172.21.15.111:0/4000320259 >> v1:172.21.15.149:6789/0 conn(0x7f52c006bff0 legacy=0x7f52c00cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.884+0000 7f52c5d77700 1 -- 172.21.15.111:0/4000320259 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c015cd40 con 0x7f52c0160ca0 2023-11-28T13:01:00.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.884+0000 7f52c5d77700 1 --2- 172.21.15.111:0/4000320259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c0160ca0 0x7f52c00b1600 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f52b8009e50 tx=0x7f52b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6994469965381ab server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:00.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.884+0000 7f52b7fff700 1 -- 172.21.15.111:0/4000320259 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b800e070 con 0x7f52c0160ca0 2023-11-28T13:01:00.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.884+0000 7f52b7fff700 1 -- 172.21.15.111:0/4000320259 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b800b910 con 0x7f52c0160ca0 2023-11-28T13:01:00.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.884+0000 7f52b7fff700 1 -- 172.21.15.111:0/4000320259 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b800bcc0 con 0x7f52c0160ca0 2023-11-28T13:01:00.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.885+0000 7f52c77da700 1 -- 172.21.15.111:0/4000320259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c0160ca0 msgr2=0x7f52c00b1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.885+0000 7f52c77da700 1 --2- 172.21.15.111:0/4000320259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c0160ca0 0x7f52c00b1600 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f52b8009e50 tx=0x7f52b800b330 comp rx=0 tx=0).stop 2023-11-28T13:01:00.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.885+0000 7f52c77da700 1 -- 172.21.15.111:0/4000320259 shutdown_connections 2023-11-28T13:01:00.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.885+0000 7f52c77da700 1 --2- 172.21.15.111:0/4000320259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c0160ca0 0x7f52c00b1600 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.885+0000 7f52c77da700 1 --2- 172.21.15.111:0/4000320259 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52c00cde60 0x7f52c015d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.885+0000 7f52c77da700 1 -- 172.21.15.111:0/4000320259 >> 172.21.15.111:0/4000320259 conn(0x7f52c006e790 msgr2=0x7f52c00cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:00.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.885+0000 7f52c77da700 1 -- 172.21.15.111:0/4000320259 shutdown_connections 2023-11-28T13:01:00.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.885+0000 7f52c77da700 1 -- 172.21.15.111:0/4000320259 wait complete. 2023-11-28T13:01:00.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.886+0000 7f52c77da700 1 Processor -- start 2023-11-28T13:01:00.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.886+0000 7f52c77da700 1 -- start start 2023-11-28T13:01:00.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c77da700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c006bff0 0x7f52c00c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c77da700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52c00cde60 0x7f52c01f1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c77da700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52c0160ca0 0x7f52c00abc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c77da700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c00c60a0 con 0x7f52c006bff0 2023-11-28T13:01:00.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c77da700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c00c5110 con 0x7f52c0160ca0 2023-11-28T13:01:00.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c77da700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c00b2b70 con 0x7f52c00cde60 2023-11-28T13:01:00.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c5576700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c006bff0 0x7f52c00c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c5576700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c006bff0 0x7f52c00c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36282/0 (socket says 172.21.15.111:36282) 2023-11-28T13:01:00.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c5576700 1 -- 172.21.15.111:0/1923665415 learned_addr learned my addr 172.21.15.111:0/1923665415 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:00.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c5d77700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52c0160ca0 0x7f52c00abc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c4d75700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52c00cde60 0x7f52c01f1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c5576700 1 -- 172.21.15.111:0/1923665415 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52c00cde60 msgr2=0x7f52c01f1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c5576700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52c00cde60 0x7f52c01f1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c5576700 1 -- 172.21.15.111:0/1923665415 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52c0160ca0 msgr2=0x7f52c00abc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:00.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c5576700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52c0160ca0 0x7f52c00abc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:00.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c5576700 1 -- 172.21.15.111:0/1923665415 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b8009b30 con 0x7f52c006bff0 2023-11-28T13:01:00.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c5576700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c006bff0 0x7f52c00c5b60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f52b000c4e0 tx=0x7f52b000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:00.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52b5ffb700 1 -- 172.21.15.111:0/1923665415 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b00052d0 con 0x7f52c006bff0 2023-11-28T13:01:00.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52b5ffb700 1 -- 172.21.15.111:0/1923665415 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b0005430 con 0x7f52c006bff0 2023-11-28T13:01:00.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c77da700 1 -- 172.21.15.111:0/1923665415 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c015d300 con 0x7f52c006bff0 2023-11-28T13:01:00.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52b5ffb700 1 -- 172.21.15.111:0/1923665415 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b00038c0 con 0x7f52c006bff0 2023-11-28T13:01:00.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.887+0000 7f52c77da700 1 -- 172.21.15.111:0/1923665415 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c015d020 con 0x7f52c006bff0 2023-11-28T13:01:00.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.888+0000 7f52c77da700 1 -- 172.21.15.111:0/1923665415 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c00b2030 con 0x7f52c006bff0 2023-11-28T13:01:00.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.889+0000 7f52b5ffb700 1 -- 172.21.15.111:0/1923665415 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f52b0019030 con 0x7f52c006bff0 2023-11-28T13:01:00.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.889+0000 7f52b5ffb700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52a40682f0 0x7f52a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:00.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.889+0000 7f52b5ffb700 1 -- 172.21.15.111:0/1923665415 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f52b0085010 con 0x7f52c006bff0 2023-11-28T13:01:00.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.889+0000 7f52c4d75700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52a40682f0 0x7f52a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:00.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.889+0000 7f52c4d75700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52a40682f0 0x7f52a406a7a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f52c00b4230 tx=0x7f52bc009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:00.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:00.893+0000 7f52b5ffb700 1 -- 172.21.15.111:0/1923665415 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f52b0051730 con 0x7f52c006bff0 2023-11-28T13:01:01.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.149+0000 7f52c77da700 1 -- 172.21.15.111:0/1923665415 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7f52c0067170 con 0x7f52c006bff0 2023-11-28T13:01:01.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.153+0000 7f52b5ffb700 1 -- 172.21.15.111:0/1923665415 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 184+0+0 (secure 0 0 0) 0x7f52b00170e0 con 0x7f52c006bff0 2023-11-28T13:01:01.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.155+0000 7f52c77da700 1 -- 172.21.15.111:0/1923665415 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52a40682f0 msgr2=0x7f52a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:01.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.155+0000 7f52c77da700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52a40682f0 0x7f52a406a7a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f52c00b4230 tx=0x7f52bc009210 comp rx=0 tx=0).stop 2023-11-28T13:01:01.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.155+0000 7f52c77da700 1 -- 172.21.15.111:0/1923665415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c006bff0 msgr2=0x7f52c00c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:01.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.155+0000 7f52c77da700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c006bff0 0x7f52c00c5b60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f52b000c4e0 tx=0x7f52b000c7f0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.155+0000 7f52c77da700 1 -- 172.21.15.111:0/1923665415 shutdown_connections 2023-11-28T13:01:01.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.155+0000 7f52c77da700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52a40682f0 0x7f52a406a7a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.155+0000 7f52c77da700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52c0160ca0 0x7f52c00abc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.155+0000 7f52c77da700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52c006bff0 0x7f52c00c5b60 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.155+0000 7f52c77da700 1 --2- 172.21.15.111:0/1923665415 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52c00cde60 0x7f52c01f1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.155+0000 7f52c77da700 1 -- 172.21.15.111:0/1923665415 >> 172.21.15.111:0/1923665415 conn(0x7f52c006e790 msgr2=0x7f52c00cd730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:01.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.155+0000 7f52c77da700 1 -- 172.21.15.111:0/1923665415 shutdown_connections 2023-11-28T13:01:01.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.155+0000 7f52c77da700 1 -- 172.21.15.111:0/1923665415 wait complete. 2023-11-28T13:01:01.158 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-11-28T13:01:01.169 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:01:01.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.288+0000 7f3d04cb1700 1 Processor -- start 2023-11-28T13:01:01.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.288+0000 7f3d04cb1700 1 -- start start 2023-11-28T13:01:01.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3d04cb1700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f3d000cb240 0x7f3d001590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:01.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3d04cb1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3d0015d270 0x7f3d000cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:01.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3d04cb1700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d000cd540 con 0x7f3d000cdf10 2023-11-28T13:01:01.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3d04cb1700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d00159a20 con 0x7f3d0015d270 2023-11-28T13:01:01.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3d04cb1700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d00159d00 con 0x7f3d000cb240 2023-11-28T13:01:01.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfdd9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3d0015d270 0x7f3d000cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:01.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3d000cdf10 0x7f3d0006e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37072/0 (socket says 172.21.15.111:37072) 2023-11-28T13:01:01.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfdd9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3d0015d270 0x7f3d000cd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36292/0 (socket says 172.21.15.111:36292) 2023-11-28T13:01:01.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfed9d700 1 -- 172.21.15.111:0/4047443618 learned_addr learned my addr 172.21.15.111:0/4047443618 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:01.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfe59c700 1 --2- 172.21.15.111:0/4047443618 >> v2:172.21.15.149:3300/0 conn(0x7f3d000cb240 0x7f3d001590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:01.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfdd9b700 1 -- 172.21.15.111:0/4047443618 >> v2:172.21.15.149:3300/0 conn(0x7f3d000cb240 msgr2=0x7f3d001590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:01.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfdd9b700 1 --2- 172.21.15.111:0/4047443618 >> v2:172.21.15.149:3300/0 conn(0x7f3d000cb240 0x7f3d001590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfdd9b700 1 -- 172.21.15.111:0/4047443618 >> v1:172.21.15.111:6789/0 conn(0x7f3d000cdf10 legacy=0x7f3d0006e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:01.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfdd9b700 1 -- 172.21.15.111:0/4047443618 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d00159fe0 con 0x7f3d0015d270 2023-11-28T13:01:01.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfd59a700 1 -- 172.21.15.111:0/4047443618 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167046076 0 0) 0x7f3d000cd540 con 0x7f3d000cdf10 2023-11-28T13:01:01.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfdd9b700 1 --2- 172.21.15.111:0/4047443618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d0015d270 0x7f3d000cd9d0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3d00070d90 tx=0x7f3ce8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd07e10f4c755fc8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:01.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfd59a700 1 -- 172.21.15.111:0/4047443618 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8011830 con 0x7f3d0015d270 2023-11-28T13:01:01.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfd59a700 1 -- 172.21.15.111:0/4047443618 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce8011990 con 0x7f3d0015d270 2023-11-28T13:01:01.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.289+0000 7f3cfd59a700 1 -- 172.21.15.111:0/4047443618 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8011d20 con 0x7f3d0015d270 2023-11-28T13:01:01.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.290+0000 7f3d04cb1700 1 -- 172.21.15.111:0/4047443618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d0015d270 msgr2=0x7f3d000cd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:01.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.290+0000 7f3d04cb1700 1 --2- 172.21.15.111:0/4047443618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d0015d270 0x7f3d000cd9d0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3d00070d90 tx=0x7f3ce8009b20 comp rx=0 tx=0).stop 2023-11-28T13:01:01.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.290+0000 7f3d04cb1700 1 -- 172.21.15.111:0/4047443618 shutdown_connections 2023-11-28T13:01:01.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.290+0000 7f3d04cb1700 1 --2- 172.21.15.111:0/4047443618 >> v2:172.21.15.149:3300/0 conn(0x7f3d000cb240 0x7f3d001590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.290+0000 7f3d04cb1700 1 --2- 172.21.15.111:0/4047443618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d0015d270 0x7f3d000cd9d0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.290+0000 7f3d04cb1700 1 -- 172.21.15.111:0/4047443618 >> 172.21.15.111:0/4047443618 conn(0x7f3d000b0db0 msgr2=0x7f3d000b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:01.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.290+0000 7f3d04cb1700 1 -- 172.21.15.111:0/4047443618 shutdown_connections 2023-11-28T13:01:01.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.290+0000 7f3d04cb1700 1 -- 172.21.15.111:0/4047443618 wait complete. 2023-11-28T13:01:01.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.291+0000 7f3d04cb1700 1 Processor -- start 2023-11-28T13:01:01.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.291+0000 7f3d04cb1700 1 -- start start 2023-11-28T13:01:01.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.292+0000 7f3d04cb1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d000cb240 0x7f3d000ae570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:01.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.292+0000 7f3d04cb1700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d000cdf10 0x7f3d001f14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:01.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.292+0000 7f3d04cb1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d0015d270 0x7f3d001f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:01.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.292+0000 7f3d04cb1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d00152260 con 0x7f3d0015d270 2023-11-28T13:01:01.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.292+0000 7f3d04cb1700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d000a76d0 con 0x7f3d000cdf10 2023-11-28T13:01:01.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.292+0000 7f3d04cb1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d000aeab0 con 0x7f3d000cb240 2023-11-28T13:01:01.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.292+0000 7f3cfed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d0015d270 0x7f3d001f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:01.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.292+0000 7f3cfed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d0015d270 0x7f3d001f1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36300/0 (socket says 172.21.15.111:36300) 2023-11-28T13:01:01.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.292+0000 7f3cfed9d700 1 -- 172.21.15.111:0/48747235 learned_addr learned my addr 172.21.15.111:0/48747235 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:01.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.292+0000 7f3cfe59c700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d000cb240 0x7f3d000ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:01.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.292+0000 7f3cfdd9b700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d000cdf10 0x7f3d001f14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:01.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.293+0000 7f3cfed9d700 1 -- 172.21.15.111:0/48747235 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d000cb240 msgr2=0x7f3d000ae570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:01.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.293+0000 7f3cfed9d700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d000cb240 0x7f3d000ae570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.293+0000 7f3cfed9d700 1 -- 172.21.15.111:0/48747235 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d000cdf10 msgr2=0x7f3d001f14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:01.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.293+0000 7f3cfed9d700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d000cdf10 0x7f3d001f14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.293+0000 7f3cfed9d700 1 -- 172.21.15.111:0/48747235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce80094a0 con 0x7f3d0015d270 2023-11-28T13:01:01.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.293+0000 7f3cfed9d700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d0015d270 0x7f3d001f1a20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80011e0 tx=0x7f3cf800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:01.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.293+0000 7f3cf6ffd700 1 -- 172.21.15.111:0/48747235 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8004c80 con 0x7f3d0015d270 2023-11-28T13:01:01.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.294+0000 7f3cf6ffd700 1 -- 172.21.15.111:0/48747235 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf8004de0 con 0x7f3d0015d270 2023-11-28T13:01:01.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.294+0000 7f3d04cb1700 1 -- 172.21.15.111:0/48747235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d001595e0 con 0x7f3d0015d270 2023-11-28T13:01:01.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.294+0000 7f3cf6ffd700 1 -- 172.21.15.111:0/48747235 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf80072d0 con 0x7f3d0015d270 2023-11-28T13:01:01.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.294+0000 7f3d04cb1700 1 -- 172.21.15.111:0/48747235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d000cd1d0 con 0x7f3d0015d270 2023-11-28T13:01:01.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.295+0000 7f3d04cb1700 1 -- 172.21.15.111:0/48747235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d00159770 con 0x7f3d0015d270 2023-11-28T13:01:01.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.295+0000 7f3cf6ffd700 1 -- 172.21.15.111:0/48747235 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3cf801a020 con 0x7f3d0015d270 2023-11-28T13:01:01.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.295+0000 7f3cf6ffd700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3ce00682f0 0x7f3ce006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:01.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.295+0000 7f3cfe59c700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3ce00682f0 0x7f3ce006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:01.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.295+0000 7f3cf6ffd700 1 -- 172.21.15.111:0/48747235 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3cf8085890 con 0x7f3d0015d270 2023-11-28T13:01:01.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.296+0000 7f3cfe59c700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3ce00682f0 0x7f3ce006a7a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0003c10 tx=0x7f3cf00098d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:01.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.300+0000 7f3cf6ffd700 1 -- 172.21.15.111:0/48747235 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3cf8051fb0 con 0x7f3d0015d270 2023-11-28T13:01:01.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.530+0000 7f3d04cb1700 1 -- 172.21.15.111:0/48747235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d001f2350 con 0x7f3d0015d270 2023-11-28T13:01:01.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.531+0000 7f3cf6ffd700 1 -- 172.21.15.111:0/48747235 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f3cf80555d0 con 0x7f3d0015d270 2023-11-28T13:01:01.532 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:01.532 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2072621958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2072621958},{"type":"v1","addr":"172.21.15.149:6834","nonce":2072621958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3348427170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3348427170},{"type":"v1","addr":"172.21.15.149:6835","nonce":3348427170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1745443298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1745443298},{"type":"v1","addr":"172.21.15.111:6836","nonce":1745443298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:56.272312+0000","modified":"2023-11-28T13:00:59.284265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5814":{"gid":5814,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/4146274267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":4146274267},{"type":"v1","addr":"172.21.15.111:6837","nonce":4146274267}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T13:01:01.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.533+0000 7f3d04cb1700 1 -- 172.21.15.111:0/48747235 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3ce00682f0 msgr2=0x7f3ce006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:01.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.533+0000 7f3d04cb1700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3ce00682f0 0x7f3ce006a7a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0003c10 tx=0x7f3cf00098d0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.533+0000 7f3d04cb1700 1 -- 172.21.15.111:0/48747235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d0015d270 msgr2=0x7f3d001f1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:01.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.533+0000 7f3d04cb1700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d0015d270 0x7f3d001f1a20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80011e0 tx=0x7f3cf800c800 comp rx=0 tx=0).stop 2023-11-28T13:01:01.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.533+0000 7f3d04cb1700 1 -- 172.21.15.111:0/48747235 shutdown_connections 2023-11-28T13:01:01.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.533+0000 7f3d04cb1700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d0015d270 0x7f3d001f1a20 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.533+0000 7f3d04cb1700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d000cb240 0x7f3d000ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.533+0000 7f3d04cb1700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3ce00682f0 0x7f3ce006a7a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.533+0000 7f3d04cb1700 1 --2- 172.21.15.111:0/48747235 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d000cdf10 0x7f3d001f14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:01.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.533+0000 7f3d04cb1700 1 -- 172.21.15.111:0/48747235 >> 172.21.15.111:0/48747235 conn(0x7f3d000b0db0 msgr2=0x7f3d00071bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:01.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.533+0000 7f3d04cb1700 1 -- 172.21.15.111:0/48747235 shutdown_connections 2023-11-28T13:01:01.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:01.533+0000 7f3d04cb1700 1 -- 172.21.15.111:0/48747235 wait complete. 2023-11-28T13:01:01.536 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 25 2023-11-28T13:01:01.764 INFO:tasks.ceph.mds.c.smithi111.stderr:2023-11-28T13:01:01.762+0000 7fa2f7c7b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T13:01:01.948 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T13:00:56.272312+0000', 'modified': '2023-11-28T13:00:59.284265+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5814}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5814': {'gid': 5814, 'name': 'c', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6837/4146274267', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6835', 'nonce': 4146274267}, {'type': 'v1', 'addr': '172.21.15.111:6837', 'nonce': 4146274267}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T13:01:01.949 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T13:01:01.949 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T13:01:02.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.076+0000 7f1d4c1be700 1 Processor -- start 2023-11-28T13:01:02.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.077+0000 7f1d4c1be700 1 -- start start 2023-11-28T13:01:02.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.077+0000 7f1d4c1be700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1d4415fed0 0x7f1d440cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.077+0000 7f1d4c1be700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f1d440cf7d0 0x7f1d440b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.077+0000 7f1d4c1be700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1d440af5a0 0x7f1d440b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.077+0000 7f1d4c1be700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d4406ac80 con 0x7f1d440af5a0 2023-11-28T13:01:02.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.077+0000 7f1d4c1be700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d440cee00 con 0x7f1d440cf7d0 2023-11-28T13:01:02.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.077+0000 7f1d4c1be700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4406aed0 con 0x7f1d4415fed0 2023-11-28T13:01:02.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d4a75b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1d440af5a0 0x7f1d440b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d4a75b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1d440af5a0 0x7f1d440b09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36308/0 (socket says 172.21.15.111:36308) 2023-11-28T13:01:02.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d4a75b700 1 -- 172.21.15.111:0/439300399 learned_addr learned my addr 172.21.15.111:0/439300399 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:02.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/439300399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1d4415fed0 0x7f1d440cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d49759700 1 --2- 172.21.15.111:0/439300399 >> v2:172.21.15.149:3300/0 conn(0x7f1d440cf7d0 0x7f1d440b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d4a75b700 1 -- 172.21.15.111:0/439300399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1d4415fed0 msgr2=0x7f1d440cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:02.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/439300399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1d4415fed0 0x7f1d440cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d4a75b700 1 -- 172.21.15.111:0/439300399 >> v2:172.21.15.149:3300/0 conn(0x7f1d440cf7d0 msgr2=0x7f1d440b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:02.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/439300399 >> v2:172.21.15.149:3300/0 conn(0x7f1d440cf7d0 0x7f1d440b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d4a75b700 1 -- 172.21.15.111:0/439300399 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d440cea90 con 0x7f1d440af5a0 2023-11-28T13:01:02.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/439300399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1d4415fed0 0x7f1d440cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:02.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d49759700 1 --2- 172.21.15.111:0/439300399 >> v2:172.21.15.149:3300/0 conn(0x7f1d440cf7d0 0x7f1d440b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:02.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/439300399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d440af5a0 0x7f1d440b09f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1d440cd750 tx=0x7f1d3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=636cbf946379cb02 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:02.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d48f58700 1 -- 172.21.15.111:0/439300399 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c00e070 con 0x7f1d440af5a0 2023-11-28T13:01:02.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d48f58700 1 -- 172.21.15.111:0/439300399 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c00b9c0 con 0x7f1d440af5a0 2023-11-28T13:01:02.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.078+0000 7f1d48f58700 1 -- 172.21.15.111:0/439300399 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c00bd70 con 0x7f1d440af5a0 2023-11-28T13:01:02.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.079+0000 7f1d4c1be700 1 -- 172.21.15.111:0/439300399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d440af5a0 msgr2=0x7f1d440b09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.079+0000 7f1d4c1be700 1 --2- 172.21.15.111:0/439300399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d440af5a0 0x7f1d440b09f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1d440cd750 tx=0x7f1d3c00b330 comp rx=0 tx=0).stop 2023-11-28T13:01:02.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.079+0000 7f1d4c1be700 1 -- 172.21.15.111:0/439300399 shutdown_connections 2023-11-28T13:01:02.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.079+0000 7f1d4c1be700 1 --2- 172.21.15.111:0/439300399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d440af5a0 0x7f1d440b09f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.079+0000 7f1d4c1be700 1 --2- 172.21.15.111:0/439300399 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1d4415fed0 0x7f1d440cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.079+0000 7f1d4c1be700 1 --2- 172.21.15.111:0/439300399 >> v2:172.21.15.149:3300/0 conn(0x7f1d440cf7d0 0x7f1d440b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.079+0000 7f1d4c1be700 1 -- 172.21.15.111:0/439300399 >> 172.21.15.111:0/439300399 conn(0x7f1d4406bca0 msgr2=0x7f1d440cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:02.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.079+0000 7f1d4c1be700 1 -- 172.21.15.111:0/439300399 shutdown_connections 2023-11-28T13:01:02.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.079+0000 7f1d4c1be700 1 -- 172.21.15.111:0/439300399 wait complete. 2023-11-28T13:01:02.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.080+0000 7f1d4c1be700 1 Processor -- start 2023-11-28T13:01:02.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.080+0000 7f1d4c1be700 1 -- start start 2023-11-28T13:01:02.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4c1be700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1d440af5a0 0x7f1d441f59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4c1be700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1d440cf7d0 0x7f1d4415aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4c1be700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d4415fed0 0x7f1d440ad060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4c1be700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440acc40 con 0x7f1d4415fed0 2023-11-28T13:01:02.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4c1be700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440ac880 con 0x7f1d440cf7d0 2023-11-28T13:01:02.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4c1be700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d441f6240 con 0x7f1d440af5a0 2023-11-28T13:01:02.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4a75b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d4415fed0 0x7f1d440ad060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4a75b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d4415fed0 0x7f1d440ad060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36314/0 (socket says 172.21.15.111:36314) 2023-11-28T13:01:02.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4a75b700 1 -- 172.21.15.111:0/1511683462 learned_addr learned my addr 172.21.15.111:0/1511683462 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:02.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4a75b700 1 -- 172.21.15.111:0/1511683462 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1d440af5a0 msgr2=0x7f1d441f59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/1511683462 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1d440af5a0 0x7f1d441f59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4a75b700 1 -- 172.21.15.111:0/1511683462 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1d440cf7d0 msgr2=0x7f1d4415aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/1511683462 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1d440cf7d0 0x7f1d4415aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4a75b700 1 -- 172.21.15.111:0/1511683462 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c009b30 con 0x7f1d4415fed0 2023-11-28T13:01:02.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/1511683462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d4415fed0 0x7f1d440ad060 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c012000 tx=0x7f1d3c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:02.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d3a7fc700 1 -- 172.21.15.111:0/1511683462 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c0099b0 con 0x7f1d4415fed0 2023-11-28T13:01:02.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d3a7fc700 1 -- 172.21.15.111:0/1511683462 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c0092e0 con 0x7f1d4415fed0 2023-11-28T13:01:02.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d3a7fc700 1 -- 172.21.15.111:0/1511683462 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c01c640 con 0x7f1d4415fed0 2023-11-28T13:01:02.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4c1be700 1 -- 172.21.15.111:0/1511683462 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d440ce6b0 con 0x7f1d4415fed0 2023-11-28T13:01:02.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.081+0000 7f1d4c1be700 1 -- 172.21.15.111:0/1511683462 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d441602a0 con 0x7f1d4415fed0 2023-11-28T13:01:02.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.083+0000 7f1d3a7fc700 1 -- 172.21.15.111:0/1511683462 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1d3c022030 con 0x7f1d4415fed0 2023-11-28T13:01:02.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.083+0000 7f1d3a7fc700 1 --2- 172.21.15.111:0/1511683462 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1d280682f0 0x7f1d2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.083+0000 7f1d3a7fc700 1 -- 172.21.15.111:0/1511683462 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f1d3c087740 con 0x7f1d4415fed0 2023-11-28T13:01:02.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.083+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/1511683462 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1d280682f0 0x7f1d2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.084+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/1511683462 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1d280682f0 0x7f1d2806a7a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1d34006fd0 tx=0x7f1d34008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:02.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.085+0000 7f1d58347b80 1 Processor -- start 2023-11-28T13:01:02.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.085+0000 7f1d58347b80 1 -- start start 2023-11-28T13:01:02.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d58347b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b40810 0x55d4b3b16a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d58347b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b3e130 0x55d4b3b40be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d58347b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 0x55d4b3b5c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d58347b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b3a00bc0 con 0x55d4b3b41120 2023-11-28T13:01:02.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d58347b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b3a6f350 con 0x55d4b3b40810 2023-11-28T13:01:02.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d58347b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4b3a68880 con 0x55d4b3b3e130 2023-11-28T13:01:02.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d4a75b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 0x55d4b3b5c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d4a75b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 0x55d4b3b5c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36322/0 (socket says 172.21.15.111:36322) 2023-11-28T13:01:02.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d4a75b700 1 -- 172.21.15.111:0/807301290 learned_addr learned my addr 172.21.15.111:0/807301290 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:02.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d49759700 1 --2- 172.21.15.111:0/807301290 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b3e130 0x55d4b3b40be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/807301290 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b40810 0x55d4b3b16a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d4a75b700 1 -- 172.21.15.111:0/807301290 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b3e130 msgr2=0x55d4b3b40be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:02.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/807301290 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b3e130 0x55d4b3b40be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d4a75b700 1 -- 172.21.15.111:0/807301290 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b40810 msgr2=0x55d4b3b16a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:02.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/807301290 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b40810 0x55d4b3b16a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d4a75b700 1 -- 172.21.15.111:0/807301290 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4b38ea5e0 con 0x55d4b3b41120 2023-11-28T13:01:02.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d49759700 1 --2- 172.21.15.111:0/807301290 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b3e130 0x55d4b3b40be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:02.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/807301290 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b40810 0x55d4b3b16a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:02.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/807301290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 0x55d4b3b5c770 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c004aa0 tx=0x7f1d3c057750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb083bbfa6f80713 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:02.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d277fe700 1 -- 172.21.15.111:0/807301290 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c056fd0 con 0x55d4b3b41120 2023-11-28T13:01:02.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.086+0000 7f1d277fe700 1 -- 172.21.15.111:0/807301290 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c057130 con 0x55d4b3b41120 2023-11-28T13:01:02.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.087+0000 7f1d277fe700 1 -- 172.21.15.111:0/807301290 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c0572a0 con 0x55d4b3b41120 2023-11-28T13:01:02.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.087+0000 7f1d58347b80 1 -- 172.21.15.111:0/807301290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 msgr2=0x55d4b3b5c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.087+0000 7f1d58347b80 1 --2- 172.21.15.111:0/807301290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 0x55d4b3b5c770 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c004aa0 tx=0x7f1d3c057750 comp rx=0 tx=0).stop 2023-11-28T13:01:02.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.087+0000 7f1d58347b80 1 -- 172.21.15.111:0/807301290 shutdown_connections 2023-11-28T13:01:02.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.087+0000 7f1d58347b80 1 --2- 172.21.15.111:0/807301290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 0x55d4b3b5c770 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.087+0000 7f1d58347b80 1 --2- 172.21.15.111:0/807301290 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b40810 0x55d4b3b16a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.087+0000 7f1d58347b80 1 --2- 172.21.15.111:0/807301290 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b3e130 0x55d4b3b40be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.087+0000 7f1d58347b80 1 -- 172.21.15.111:0/807301290 >> 172.21.15.111:0/807301290 conn(0x55d4b3b13d60 msgr2=0x55d4b3b146f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.087+0000 7f1d58347b80 1 -- 172.21.15.111:0/807301290 shutdown_connections 2023-11-28T13:01:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.087+0000 7f1d58347b80 1 -- 172.21.15.111:0/807301290 wait complete. 2023-11-28T13:01:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.088+0000 7f1d58347b80 1 -- start start 2023-11-28T13:01:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.088+0000 7f1d58347b80 1 Processor -- start 2023-11-28T13:01:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d58347b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 0x55d4b3b5ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d58347b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 0x55d4b3b25a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d58347b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b41120 0x55d4b3b25fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d58347b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b38ec8d0 con 0x55d4b3b40810 2023-11-28T13:01:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d58347b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b3b26500 con 0x55d4b3b3e130 2023-11-28T13:01:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d58347b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4b3b53b50 con 0x55d4b3b41120 2023-11-28T13:01:02.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d58347b80 10 client.0 _ll_register_callbacks cb 0x55d4b390e240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:01:02.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d49759700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 0x55d4b3b25a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d49759700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 0x55d4b3b25a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36326/0 (socket says 172.21.15.111:36326) 2023-11-28T13:01:02.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d49759700 1 -- 172.21.15.111:0/651138121 learned_addr learned my addr 172.21.15.111:0/651138121 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:02.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/651138121 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b41120 0x55d4b3b25fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/651138121 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 0x55d4b3b5ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d49759700 1 -- 172.21.15.111:0/651138121 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b41120 msgr2=0x55d4b3b25fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:02.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d49759700 1 --2- 172.21.15.111:0/651138121 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b41120 0x55d4b3b25fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d49759700 1 -- 172.21.15.111:0/651138121 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 msgr2=0x55d4b3b5ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:02.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d49759700 1 --2- 172.21.15.111:0/651138121 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 0x55d4b3b5ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d49759700 1 -- 172.21.15.111:0/651138121 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d441602a0 con 0x55d4b3b40810 2023-11-28T13:01:02.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d49759700 1 --2- 172.21.15.111:0/651138121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 0x55d4b3b25a80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1d4000e4d0 tx=0x7f1d4000e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=844b19ca2bea25a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.089+0000 7f1d24ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:01:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d24ff9700 1 -- 172.21.15.111:0/651138121 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d40013450 con 0x55d4b3b40810 2023-11-28T13:01:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d24ff9700 1 -- 172.21.15.111:0/651138121 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f1d400135b0 con 0x55d4b3b40810 2023-11-28T13:01:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d58347b80 1 -- 172.21.15.111:0/651138121 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4b3a36140 con 0x55d4b3b40810 2023-11-28T13:01:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d24ff9700 1 -- 172.21.15.111:0/651138121 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d40013770 con 0x55d4b3b40810 2023-11-28T13:01:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d58347b80 10 client.5946 fetch_fsmap learned FSMap version 25 2023-11-28T13:01:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d58347b80 1 -- 172.21.15.111:0/651138121 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55d4b3896d90 con 0x55d4b3b40810 2023-11-28T13:01:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d24ff9700 1 -- 172.21.15.111:0/651138121 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d4b3a36140 con 0x55d4b3b40810 2023-11-28T13:01:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d24ff9700 1 -- 172.21.15.111:0/651138121 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 25) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f1d40011070 con 0x55d4b3b40810 2023-11-28T13:01:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d58347b80 10 client.5946 fetch_fsmap finished waiting for FSMap version 25 2023-11-28T13:01:02.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d58347b80 10 client.5946 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T13:01:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d58347b80 20 client.5946 populate_metadata read hostname 'smithi111' 2023-11-28T13:01:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d58347b80 1 --2- 172.21.15.111:0/651138121 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55d4b3b63870 0x55d4b3b783b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d58347b80 4 client.5946 mds_command: new command op to 5814 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-28T13:01:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d58347b80 1 -- 172.21.15.111:0/651138121 --> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d4b38ecf10 con 0x55d4b3b63870 2023-11-28T13:01:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/651138121 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55d4b3b63870 0x55d4b3b783b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d24ff9700 10 client.5946 ms_handle_connect on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.090+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/651138121 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55d4b3b63870 0x55d4b3b783b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=656dc4857d4ccf60 server_cookie=537821ba19fd49fb in_seq=0 out_seq=0 2023-11-28T13:01:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d24ff9700 1 -- 172.21.15.111:0/651138121 <== mds.0 v2:172.21.15.111:6835/4146274267 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55d4b38ecf10 con 0x55d4b3b63870 2023-11-28T13:01:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d24ff9700 10 client.5946 handle_command_reply: tid=0 2023-11-28T13:01:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d58347b80 1 client.5946 shutdown 2023-11-28T13:01:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d58347b80 1 -- 172.21.15.111:0/651138121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 msgr2=0x55d4b3b25a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d58347b80 1 --2- 172.21.15.111:0/651138121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 0x55d4b3b25a80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1d4000e4d0 tx=0x7f1d4000e7e0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d58347b80 1 -- 172.21.15.111:0/651138121 shutdown_connections 2023-11-28T13:01:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d58347b80 1 --2- 172.21.15.111:0/651138121 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55d4b3b63870 0x55d4b3b783b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d58347b80 1 --2- 172.21.15.111:0/651138121 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b41120 0x55d4b3b25fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d24ff9700 0 client.5946 ms_handle_reset on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d58347b80 1 --2- 172.21.15.111:0/651138121 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 0x55d4b3b5ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d58347b80 1 --2- 172.21.15.111:0/651138121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 0x55d4b3b25a80 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d58347b80 1 -- 172.21.15.111:0/651138121 >> 172.21.15.111:0/651138121 conn(0x55d4b3b13d60 msgr2=0x55d4b3b5dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d58347b80 1 -- 172.21.15.111:0/651138121 shutdown_connections 2023-11-28T13:01:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d58347b80 1 -- 172.21.15.111:0/651138121 wait complete. 2023-11-28T13:01:02.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.091+0000 7f1d58347b80 20 client.5946 trim_cache size 0 max 16384 2023-11-28T13:01:02.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.108+0000 7f1d58347b80 1 Processor -- start 2023-11-28T13:01:02.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.108+0000 7f1d58347b80 1 -- start start 2023-11-28T13:01:02.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d58347b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 0x55d4b3b3f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d58347b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 0x55d4b3b5dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d58347b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b41120 0x55d4b3b5e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d58347b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b3b36ea0 con 0x55d4b3b40810 2023-11-28T13:01:02.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d58347b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b3b10e20 con 0x55d4b3b3e130 2023-11-28T13:01:02.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d58347b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4b3b61de0 con 0x55d4b3b41120 2023-11-28T13:01:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d49759700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 0x55d4b3b5dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d49759700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 0x55d4b3b5dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36334/0 (socket says 172.21.15.111:36334) 2023-11-28T13:01:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d49759700 1 -- 172.21.15.111:0/396935369 learned_addr learned my addr 172.21.15.111:0/396935369 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/396935369 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 0x55d4b3b3f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/396935369 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b41120 0x55d4b3b5e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d49759700 1 -- 172.21.15.111:0/396935369 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b41120 msgr2=0x55d4b3b5e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d49759700 1 --2- 172.21.15.111:0/396935369 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b41120 0x55d4b3b5e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d49759700 1 -- 172.21.15.111:0/396935369 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 msgr2=0x55d4b3b3f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d49759700 1 --2- 172.21.15.111:0/396935369 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 0x55d4b3b3f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.109+0000 7f1d49759700 1 -- 172.21.15.111:0/396935369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4b3a2f140 con 0x55d4b3b40810 2023-11-28T13:01:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/396935369 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b41120 0x55d4b3b5e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d49759700 1 --2- 172.21.15.111:0/396935369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 0x55d4b3b5dfd0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1d40010010 tx=0x7f1d40014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06daacebe1e6110 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d27fff700 1 -- 172.21.15.111:0/396935369 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d40013da0 con 0x55d4b3b40810 2023-11-28T13:01:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d27fff700 1 -- 172.21.15.111:0/396935369 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d4002a070 con 0x55d4b3b40810 2023-11-28T13:01:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d27fff700 1 -- 172.21.15.111:0/396935369 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d40013690 con 0x55d4b3b40810 2023-11-28T13:01:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d58347b80 1 -- 172.21.15.111:0/396935369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 msgr2=0x55d4b3b5dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d58347b80 1 --2- 172.21.15.111:0/396935369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 0x55d4b3b5dfd0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1d40010010 tx=0x7f1d40014000 comp rx=0 tx=0).stop 2023-11-28T13:01:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d58347b80 1 -- 172.21.15.111:0/396935369 shutdown_connections 2023-11-28T13:01:02.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d58347b80 1 --2- 172.21.15.111:0/396935369 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b41120 0x55d4b3b5e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d58347b80 1 --2- 172.21.15.111:0/396935369 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 0x55d4b3b3f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d58347b80 1 --2- 172.21.15.111:0/396935369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b40810 0x55d4b3b5dfd0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d58347b80 1 -- 172.21.15.111:0/396935369 >> 172.21.15.111:0/396935369 conn(0x55d4b3b13d60 msgr2=0x55d4b3b38780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:02.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d58347b80 1 -- 172.21.15.111:0/396935369 shutdown_connections 2023-11-28T13:01:02.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.110+0000 7f1d58347b80 1 -- 172.21.15.111:0/396935369 wait complete. 2023-11-28T13:01:02.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.111+0000 7f1d58347b80 1 -- start start 2023-11-28T13:01:02.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.111+0000 7f1d58347b80 1 Processor -- start 2023-11-28T13:01:02.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d58347b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 0x55d4b3bbabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d58347b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b40810 0x55d4b3b439e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d58347b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 0x55d4b3b5ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d58347b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b3b54910 con 0x55d4b3b41120 2023-11-28T13:01:02.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d58347b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b3b783b0 con 0x55d4b3b3e130 2023-11-28T13:01:02.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d58347b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4b3b556a0 con 0x55d4b3b40810 2023-11-28T13:01:02.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d58347b80 10 client.0 _ll_register_callbacks cb 0x55d4b390e240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:01:02.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d4a75b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 0x55d4b3b5ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d4a75b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 0x55d4b3b5ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36342/0 (socket says 172.21.15.111:36342) 2023-11-28T13:01:02.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d4a75b700 1 -- 172.21.15.111:0/3896163997 learned_addr learned my addr 172.21.15.111:0/3896163997 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:02.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d49759700 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b40810 0x55d4b3b439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 0x55d4b3bbabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d4a75b700 1 -- 172.21.15.111:0/3896163997 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b40810 msgr2=0x55d4b3b439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:02.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b40810 0x55d4b3b439e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d4a75b700 1 -- 172.21.15.111:0/3896163997 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 msgr2=0x55d4b3bbabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 0x55d4b3bbabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d4a75b700 1 -- 172.21.15.111:0/3896163997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d4b3896d90 con 0x55d4b3b41120 2023-11-28T13:01:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d49759700 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b40810 0x55d4b3b439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 0x55d4b3bbabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d4a75b700 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 0x55d4b3b5ed40 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c007130 tx=0x7f1d3c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e893413f14f7d9e9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.112+0000 7f1d24ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:01:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d24ff9700 1 -- 172.21.15.111:0/3896163997 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c084070 con 0x55d4b3b41120 2023-11-28T13:01:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d24ff9700 1 -- 172.21.15.111:0/3896163997 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f1d3c0571e0 con 0x55d4b3b41120 2023-11-28T13:01:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d58347b80 1 -- 172.21.15.111:0/3896163997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4b3a26160 con 0x55d4b3b41120 2023-11-28T13:01:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d24ff9700 1 -- 172.21.15.111:0/3896163997 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d3c085ad0 con 0x55d4b3b41120 2023-11-28T13:01:02.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d58347b80 10 client.5952 fetch_fsmap learned FSMap version 25 2023-11-28T13:01:02.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d58347b80 1 -- 172.21.15.111:0/3896163997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55d4b38bd200 con 0x55d4b3b41120 2023-11-28T13:01:02.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d24ff9700 1 -- 172.21.15.111:0/3896163997 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d4b3a26160 con 0x55d4b3b41120 2023-11-28T13:01:02.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d24ff9700 1 -- 172.21.15.111:0/3896163997 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 25) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f1d3c067050 con 0x55d4b3b41120 2023-11-28T13:01:02.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d58347b80 10 client.5952 fetch_fsmap finished waiting for FSMap version 25 2023-11-28T13:01:02.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d58347b80 10 client.5952 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T13:01:02.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d58347b80 20 client.5952 populate_metadata read hostname 'smithi111' 2023-11-28T13:01:02.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d58347b80 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55d4b3b3e9f0 0x55d4b3b3edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d58347b80 4 client.5952 mds_command: new command op to 5814 tid=0[{"prefix": "status"}] 2023-11-28T13:01:02.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d58347b80 1 -- 172.21.15.111:0/3896163997 --> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d4b3a10170 con 0x55d4b3b3e9f0 2023-11-28T13:01:02.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55d4b3b3e9f0 0x55d4b3b3edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d24ff9700 10 client.5952 ms_handle_connect on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.113+0000 7f1d49f5a700 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55d4b3b3e9f0 0x55d4b3b3edc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3f2d68ee8004a2c server_cookie=13f4c820072a8f5f in_seq=0 out_seq=0 2023-11-28T13:01:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d24ff9700 1 -- 172.21.15.111:0/3896163997 <== mds.0 v2:172.21.15.111:6835/4146274267 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+316 (crc 0 0 0) 0x55d4b3a10170 con 0x55d4b3b3e9f0 2023-11-28T13:01:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d24ff9700 10 client.5952 handle_command_reply: tid=0 2023-11-28T13:01:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d58347b80 1 client.5952 shutdown 2023-11-28T13:01:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d58347b80 1 -- 172.21.15.111:0/3896163997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 msgr2=0x55d4b3b5ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d58347b80 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 0x55d4b3b5ed40 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c007130 tx=0x7f1d3c00be30 comp rx=0 tx=0).stop 2023-11-28T13:01:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d58347b80 1 -- 172.21.15.111:0/3896163997 shutdown_connections 2023-11-28T13:01:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d58347b80 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55d4b3b3e9f0 0x55d4b3b3edc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d58347b80 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d4b3b41120 0x55d4b3b5ed40 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d24ff9700 0 client.5952 ms_handle_reset on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d58347b80 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d4b3b3e130 0x55d4b3bbabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d58347b80 1 --2- 172.21.15.111:0/3896163997 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d4b3b40810 0x55d4b3b439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d58347b80 1 -- 172.21.15.111:0/3896163997 >> 172.21.15.111:0/3896163997 conn(0x55d4b3b13d60 msgr2=0x55d4b3b5e410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:02.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d58347b80 1 -- 172.21.15.111:0/3896163997 shutdown_connections 2023-11-28T13:01:02.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d58347b80 1 -- 172.21.15.111:0/3896163997 wait complete. 2023-11-28T13:01:02.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d58347b80 20 client.5952 trim_cache size 0 max 16384 2023-11-28T13:01:02.125 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:01:02.125 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "a1b2d00c-aeec-4b5d-93e3-41820c4189ba", 2023-11-28T13:01:02.125 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-11-28T13:01:02.125 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 5814, 2023-11-28T13:01:02.126 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-11-28T13:01:02.126 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-11-28T13:01:02.126 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-11-28T13:01:02.126 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 5.841170601, 2023-11-28T13:01:02.126 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 25, 2023-11-28T13:01:02.126 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 55, 2023-11-28T13:01:02.126 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 55, 2023-11-28T13:01:02.126 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 11.2988348 2023-11-28T13:01:02.126 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:01:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d4c1be700 1 -- 172.21.15.111:0/1511683462 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1d280682f0 msgr2=0x7f1d2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d4c1be700 1 --2- 172.21.15.111:0/1511683462 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1d280682f0 0x7f1d2806a7a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1d34006fd0 tx=0x7f1d34008040 comp rx=0 tx=0).stop 2023-11-28T13:01:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d4c1be700 1 -- 172.21.15.111:0/1511683462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d4415fed0 msgr2=0x7f1d440ad060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d4c1be700 1 --2- 172.21.15.111:0/1511683462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d4415fed0 0x7f1d440ad060 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c012000 tx=0x7f1d3c004ad0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d4c1be700 1 -- 172.21.15.111:0/1511683462 shutdown_connections 2023-11-28T13:01:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d4c1be700 1 --2- 172.21.15.111:0/1511683462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d4415fed0 0x7f1d440ad060 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d4c1be700 1 --2- 172.21.15.111:0/1511683462 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1d440af5a0 0x7f1d441f59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d4c1be700 1 --2- 172.21.15.111:0/1511683462 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1d280682f0 0x7f1d2806a7a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d4c1be700 1 --2- 172.21.15.111:0/1511683462 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1d440cf7d0 0x7f1d4415aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.114+0000 7f1d4c1be700 1 -- 172.21.15.111:0/1511683462 >> 172.21.15.111:0/1511683462 conn(0x7f1d4406bca0 msgr2=0x7f1d4406e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:02.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.115+0000 7f1d4c1be700 1 -- 172.21.15.111:0/1511683462 shutdown_connections 2023-11-28T13:01:02.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.115+0000 7f1d4c1be700 1 -- 172.21.15.111:0/1511683462 wait complete. 2023-11-28T13:01:02.129 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T13:01:02.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.247+0000 7f01d735d700 1 Processor -- start 2023-11-28T13:01:02.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.247+0000 7f01d735d700 1 -- start start 2023-11-28T13:01:02.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01d735d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01d015cc50 0x7f01d00712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01d735d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d00717f0 con 0x7f01d00cceb0 2023-11-28T13:01:02.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01d735d700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d006fb90 con 0x7f01d0072300 2023-11-28T13:01:02.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01d735d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d00726d0 con 0x7f01d015cc50 2023-11-28T13:01:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01d58fa700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f01d00cceb0 0x7f01d006a610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37128/0 (socket says 172.21.15.111:37128) 2023-11-28T13:01:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01d58fa700 1 -- 172.21.15.111:0/1292282306 learned_addr learned my addr 172.21.15.111:0/1292282306 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01d48f8700 1 --2- 172.21.15.111:0/1292282306 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01d015cc50 0x7f01d00712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114409875 0 0) 0x7f01d00717f0 con 0x7f01d00cceb0 2023-11-28T13:01:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01bc0035b0 con 0x7f01d00cceb0 2023-11-28T13:01:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2928802491 0 0) 0x7f01bc0035b0 con 0x7f01d00cceb0 2023-11-28T13:01:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01d015cc50 msgr2=0x7f01d00712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 --2- 172.21.15.111:0/1292282306 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01d015cc50 0x7f01d00712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 >> v1:172.21.15.149:6789/0 conn(0x7f01d0072300 legacy=0x7f01d006b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d015d0f0 con 0x7f01d00cceb0 2023-11-28T13:01:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f01c80019f0 con 0x7f01d00cceb0 2023-11-28T13:01:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 >> v1:172.21.15.111:6789/0 conn(0x7f01d00cceb0 legacy=0x7f01d006a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 --2- 172.21.15.111:0/1292282306 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01d015cc50 0x7f01bc003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 --2- 172.21.15.111:0/1292282306 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01bc003f70 0x7f01bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 --2- 172.21.15.111:0/1292282306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01bc004d90 0x7f01bc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d00726d0 con 0x7f01bc004d90 2023-11-28T13:01:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc00baf0 con 0x7f01bc003f70 2023-11-28T13:01:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc00bcb0 con 0x7f01d015cc50 2023-11-28T13:01:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01d58fa700 1 --2- 172.21.15.111:0/1292282306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01bc004d90 0x7f01bc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.248+0000 7f01d50f9700 1 --2- 172.21.15.111:0/1292282306 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01d015cc50 0x7f01bc003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.249+0000 7f01d48f8700 1 --2- 172.21.15.111:0/1292282306 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01bc003f70 0x7f01bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.249+0000 7f01d58fa700 1 -- 172.21.15.111:0/1292282306 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01d015cc50 msgr2=0x7f01bc003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.249+0000 7f01d58fa700 1 --2- 172.21.15.111:0/1292282306 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01d015cc50 0x7f01bc003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.249+0000 7f01d58fa700 1 -- 172.21.15.111:0/1292282306 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01bc003f70 msgr2=0x7f01bc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.249+0000 7f01d58fa700 1 --2- 172.21.15.111:0/1292282306 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01bc003f70 0x7f01bc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.249+0000 7f01d58fa700 1 -- 172.21.15.111:0/1292282306 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc0035b0 con 0x7f01bc004d90 2023-11-28T13:01:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.249+0000 7f01d58fa700 1 --2- 172.21.15.111:0/1292282306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01bc004d90 0x7f01bc005200 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f01c8002230 tx=0x7f01bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d75c012c56bfb0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.249+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c8012600 con 0x7f01bc004d90 2023-11-28T13:01:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.249+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c8012780 con 0x7f01bc004d90 2023-11-28T13:01:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.249+0000 7f01c3fff700 1 -- 172.21.15.111:0/1292282306 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c8012b30 con 0x7f01bc004d90 2023-11-28T13:01:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.249+0000 7f01d735d700 1 -- 172.21.15.111:0/1292282306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01bc004d90 msgr2=0x7f01bc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.249+0000 7f01d735d700 1 --2- 172.21.15.111:0/1292282306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01bc004d90 0x7f01bc005200 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f01c8002230 tx=0x7f01bc005880 comp rx=0 tx=0).stop 2023-11-28T13:01:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.249+0000 7f01d58fa700 1 -- 172.21.15.111:0/1292282306 reap_dead start 2023-11-28T13:01:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.250+0000 7f01d735d700 1 -- 172.21.15.111:0/1292282306 shutdown_connections 2023-11-28T13:01:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.250+0000 7f01d735d700 1 -- 172.21.15.111:0/1292282306 >> 172.21.15.111:0/1292282306 conn(0x7f01d00cf2e0 msgr2=0x7f01d00cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.250+0000 7f01d735d700 1 -- 172.21.15.111:0/1292282306 shutdown_connections 2023-11-28T13:01:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.250+0000 7f01d735d700 1 -- 172.21.15.111:0/1292282306 wait complete. 2023-11-28T13:01:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.251+0000 7f01d735d700 1 Processor -- start 2023-11-28T13:01:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.251+0000 7f01d735d700 1 -- start start 2023-11-28T13:01:02.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.251+0000 7f01d735d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01bc004d90 0x7f01d00aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.251+0000 7f01d735d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01d015dd00 0x7f01d01519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.251+0000 7f01d735d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01d0160270 0x7f01d01606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.251+0000 7f01d735d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0160c30 con 0x7f01bc004d90 2023-11-28T13:01:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.251+0000 7f01d735d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d00b1d50 con 0x7f01d0160270 2023-11-28T13:01:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.251+0000 7f01d735d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d015e470 con 0x7f01d015dd00 2023-11-28T13:01:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d50f9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01bc004d90 0x7f01d00aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d50f9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01bc004d90 0x7f01d00aed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36358/0 (socket says 172.21.15.111:36358) 2023-11-28T13:01:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d50f9700 1 -- 172.21.15.111:0/1365836759 learned_addr learned my addr 172.21.15.111:0/1365836759 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d48f8700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01d015dd00 0x7f01d01519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d58fa700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01d0160270 0x7f01d01606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d50f9700 1 -- 172.21.15.111:0/1365836759 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01d015dd00 msgr2=0x7f01d01519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d50f9700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01d015dd00 0x7f01d01519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d50f9700 1 -- 172.21.15.111:0/1365836759 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01d0160270 msgr2=0x7f01d01606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d50f9700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01d0160270 0x7f01d01606f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d50f9700 1 -- 172.21.15.111:0/1365836759 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d015d6b0 con 0x7f01bc004d90 2023-11-28T13:01:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d58fa700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01d0160270 0x7f01d01606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d50f9700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01bc004d90 0x7f01d00aed40 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00f860 tx=0x7f01cc00fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01c1ffb700 1 -- 172.21.15.111:0/1365836759 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc0050d0 con 0x7f01bc004d90 2023-11-28T13:01:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01c1ffb700 1 -- 172.21.15.111:0/1365836759 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01cc005250 con 0x7f01bc004d90 2023-11-28T13:01:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d735d700 1 -- 172.21.15.111:0/1365836759 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d015d3d0 con 0x7f01bc004d90 2023-11-28T13:01:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01c1ffb700 1 -- 172.21.15.111:0/1365836759 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc003680 con 0x7f01bc004d90 2023-11-28T13:01:02.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.252+0000 7f01d735d700 1 -- 172.21.15.111:0/1365836759 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d015d990 con 0x7f01bc004d90 2023-11-28T13:01:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.253+0000 7f01d735d700 1 -- 172.21.15.111:0/1365836759 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d00b3530 con 0x7f01bc004d90 2023-11-28T13:01:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.254+0000 7f01c1ffb700 1 -- 172.21.15.111:0/1365836759 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f01cc0037e0 con 0x7f01bc004d90 2023-11-28T13:01:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.254+0000 7f01c1ffb700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01b40682f0 0x7f01b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.254+0000 7f01c1ffb700 1 -- 172.21.15.111:0/1365836759 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f01cc085d80 con 0x7f01bc004d90 2023-11-28T13:01:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.254+0000 7f01d48f8700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01b40682f0 0x7f01b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.254+0000 7f01d48f8700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01b40682f0 0x7f01b406a7a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f01d01571f0 tx=0x7f01c4009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:02.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.258+0000 7f01c1ffb700 1 -- 172.21.15.111:0/1365836759 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f01cc0524a0 con 0x7f01bc004d90 2023-11-28T13:01:02.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.485+0000 7f01d735d700 1 -- 172.21.15.111:0/1365836759 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f01d0067170 con 0x7f01bc004d90 2023-11-28T13:01:02.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.485+0000 7f01c1ffb700 1 -- 172.21.15.111:0/1365836759 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7f01cc087070 con 0x7f01bc004d90 2023-11-28T13:01:02.487 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T13:01:02.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.487+0000 7f01d735d700 1 -- 172.21.15.111:0/1365836759 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01b40682f0 msgr2=0x7f01b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.487+0000 7f01d735d700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01b40682f0 0x7f01b406a7a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f01d01571f0 tx=0x7f01c4009210 comp rx=0 tx=0).stop 2023-11-28T13:01:02.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.487+0000 7f01d735d700 1 -- 172.21.15.111:0/1365836759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01bc004d90 msgr2=0x7f01d00aed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:02.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.487+0000 7f01d735d700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01bc004d90 0x7f01d00aed40 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00f860 tx=0x7f01cc00fc20 comp rx=0 tx=0).stop 2023-11-28T13:01:02.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.487+0000 7f01d735d700 1 -- 172.21.15.111:0/1365836759 shutdown_connections 2023-11-28T13:01:02.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.487+0000 7f01d735d700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f01b40682f0 0x7f01b406a7a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.487+0000 7f01d735d700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f01d0160270 0x7f01d01606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.487+0000 7f01d735d700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01bc004d90 0x7f01d00aed40 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.487+0000 7f01d735d700 1 --2- 172.21.15.111:0/1365836759 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f01d015dd00 0x7f01d01519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:02.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.487+0000 7f01d735d700 1 -- 172.21.15.111:0/1365836759 >> 172.21.15.111:0/1365836759 conn(0x7f01d00cf2e0 msgr2=0x7f01d00c85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:02.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.487+0000 7f01d735d700 1 -- 172.21.15.111:0/1365836759 shutdown_connections 2023-11-28T13:01:02.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:02.487+0000 7f01d735d700 1 -- 172.21.15.111:0/1365836759 wait complete. 2023-11-28T13:01:02.500 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T13:01:02.500 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T13:01:02.501 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-11-28T13:01:02.501 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T13:01:02.501 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T13:01:02.501 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T13:01:02.501 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T13:01:02.501 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T13:01:02.501 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T13:01:02.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:01:02.501 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-11-28T13:01:02.518 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T13:01:02.519 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T13:01:02.519 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T13:01:02.519 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:01:02.519 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-11-28T13:01:02.519 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:01:02.519 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T13:01:02.519 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-11-28T13:01:02.519 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T13:01:02.519 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-11-28T13:01:02.519 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T13:01:02.519 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-11-28T13:01:02.519 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T13:01:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-11-28T13:01:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T13:01:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 42263sec preferred_lft 42263sec 2023-11-28T13:01:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::9195:85a0:8e3c:2da4/64 scope link noprefixroute 2023-11-28T13:01:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:01:02.520 INFO:teuthology.orchestra.run.smithi111.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T13:01:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 96:9a:a2:c9:67:8f brd ff:ff:ff:ff:ff:ff 2023-11-28T13:01:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T13:01:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:01:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::f893:44ff:fe17:3abf/64 scope link 2023-11-28T13:01:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:01:02.520 INFO:teuthology.orchestra.run.smithi111.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T13:01:02.521 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 96:9a:a2:c9:67:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T13:01:02.521 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::949a:a2ff:fec9:678f/64 scope link 2023-11-28T13:01:02.521 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:01:02.521 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T13:01:02.521 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ee:e6:45:35:2f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T13:01:02.521 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ece6:45ff:fe35:2fc2/64 scope link 2023-11-28T13:01:02.521 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:01:02.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:01:02.521 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T13:01:02.584 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-11-28T13:01:02.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:01:02.585 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-11-28T13:01:02.643 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-11-28T13:01:02.643 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T13:01:02.644 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T13:01:02.644 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T13:01:02.644 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T13:01:02.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:01:02.644 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T13:01:02.644 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T13:01:02.644 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T13:01:02.644 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T13:01:02.644 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T13:01:02.645 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T13:01:02.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:01:02.748 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-11-28T13:01:02.765 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T13:01:02.765 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T13:01:02.766 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T13:01:02.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:01:02.766 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-11-28T13:01:02.833 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T13:01:02.833 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T13:01:02.890 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T13:01:02.890 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T13:01:02.891 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T13:01:02.947 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T13:01:02.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:01:02.948 DEBUG:teuthology.orchestra.run.smithi111:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T13:01:03.033 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T13:01:03.033 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T13:01:03.033 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T13:01:03.042 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T13:01:03.042 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T13:01:03.042 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T13:01:03.042 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T13:01:03.043 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T13:01:03.116 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T13:01:03.116 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T13:01:03.116 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T13:01:03.117 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T13:01:03.117 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T13:01:03.117 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790" 2023-11-28T13:01:03.117 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790:/ 2023-11-28T13:01:03.117 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=a1b2d00c-aeec-4b5d-93e3-41820c4189ba" will pass to kernel 2023-11-28T13:01:03.118 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T13:01:03.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:01:03.119 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T13:01:03.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:01:03.145 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T13:01:03.229 INFO:teuthology.orchestra.run.smithi111.stdout:client5219 2023-11-28T13:01:03.230 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:01:03.230 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:01:03.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:01:03.288 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_a) 2023-11-28T13:01:03.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:01:03.350 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parent_a/alpha) 2023-11-28T13:01:03.408 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-28T13:01:03.409 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-11-28T13:01:03.409 DEBUG:teuthology.orchestra.run.smithi111:> import stat 2023-11-28T13:01:03.409 DEBUG:teuthology.orchestra.run.smithi111:> 2023-11-28T13:01:03.409 DEBUG:teuthology.orchestra.run.smithi111:> print(os.stat("/home/ubuntu/cephtest/mnt.0/parent_a/alpha").st_ino) 2023-11-28T13:01:03.409 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-11-28T13:01:03.513 INFO:teuthology.orchestra.run.smithi111.stdout:1099511627777 2023-11-28T13:01:03.703 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:01:03.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.862+0000 7f3c44003700 1 Processor -- start 2023-11-28T13:01:03.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.862+0000 7f3c44003700 1 -- start start 2023-11-28T13:01:03.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c44003700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3c3c0b0440 0x7f3c3c0b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:03.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c44003700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3c3c0ca2f0 0x7f3c3c0ca770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:03.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c44003700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c3c0cc750 con 0x7f3c3c15c6c0 2023-11-28T13:01:03.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c44003700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c3c0b0260 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c44003700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c0b0df0 con 0x7f3c3c0b0440 2023-11-28T13:01:03.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c425a0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3c3c0ca2f0 0x7f3c3c0ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:03.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c41d9f700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3c3c15c6c0 0x7f3c3c0afb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37148/0 (socket says 172.21.15.111:37148) 2023-11-28T13:01:03.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c41d9f700 1 -- 172.21.15.111:0/1892589096 learned_addr learned my addr 172.21.15.111:0/1892589096 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:03.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c4159e700 1 --2- 172.21.15.111:0/1892589096 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3c3c0b0440 0x7f3c3c0b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:03.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c425a0700 1 -- 172.21.15.111:0/1892589096 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3c3c0b0440 msgr2=0x7f3c3c0b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:03.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c425a0700 1 --2- 172.21.15.111:0/1892589096 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3c3c0b0440 0x7f3c3c0b08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:03.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c425a0700 1 -- 172.21.15.111:0/1892589096 >> v1:172.21.15.111:6789/0 conn(0x7f3c3c15c6c0 legacy=0x7f3c3c0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:03.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c425a0700 1 -- 172.21.15.111:0/1892589096 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3c0cca30 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c425a0700 1 --2- 172.21.15.111:0/1892589096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3c0ca2f0 0x7f3c3c0ca770 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009d90 tx=0x7f3c3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d205b267a3796 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:03.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c40d9d700 1 -- 172.21.15.111:0/1892589096 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3400e070 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c40d9d700 1 -- 172.21.15.111:0/1892589096 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3400bbd0 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.863+0000 7f3c40d9d700 1 -- 172.21.15.111:0/1892589096 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c34012430 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.864+0000 7f3c44003700 1 -- 172.21.15.111:0/1892589096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3c0ca2f0 msgr2=0x7f3c3c0ca770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:03.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.864+0000 7f3c44003700 1 --2- 172.21.15.111:0/1892589096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3c0ca2f0 0x7f3c3c0ca770 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009d90 tx=0x7f3c3400b330 comp rx=0 tx=0).stop 2023-11-28T13:01:03.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.864+0000 7f3c44003700 1 -- 172.21.15.111:0/1892589096 shutdown_connections 2023-11-28T13:01:03.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.864+0000 7f3c44003700 1 --2- 172.21.15.111:0/1892589096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3c0ca2f0 0x7f3c3c0ca770 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:03.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.864+0000 7f3c44003700 1 --2- 172.21.15.111:0/1892589096 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3c3c0b0440 0x7f3c3c0b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:03.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.864+0000 7f3c44003700 1 -- 172.21.15.111:0/1892589096 >> 172.21.15.111:0/1892589096 conn(0x7f3c3c098ee0 msgr2=0x7f3c3c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:03.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.864+0000 7f3c44003700 1 -- 172.21.15.111:0/1892589096 shutdown_connections 2023-11-28T13:01:03.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.864+0000 7f3c44003700 1 -- 172.21.15.111:0/1892589096 wait complete. 2023-11-28T13:01:03.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.865+0000 7f3c44003700 1 Processor -- start 2023-11-28T13:01:03.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.865+0000 7f3c44003700 1 -- start start 2023-11-28T13:01:03.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c44003700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c3c0b0440 0x7f3c3c1f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:03.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c44003700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3c0ca2f0 0x7f3c3c0c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:03.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c44003700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3c3c15c6c0 0x7f3c3c0c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:03.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c44003700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c1587c0 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c44003700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c0b2ae0 con 0x7f3c3c0b0440 2023-11-28T13:01:03.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c44003700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3c0b2dc0 con 0x7f3c3c15c6c0 2023-11-28T13:01:03.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c4159e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3c0ca2f0 0x7f3c3c0c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:03.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c4159e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3c0ca2f0 0x7f3c3c0c83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36376/0 (socket says 172.21.15.111:36376) 2023-11-28T13:01:03.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c4159e700 1 -- 172.21.15.111:0/358157869 learned_addr learned my addr 172.21.15.111:0/358157869 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:03.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c41d9f700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c3c0b0440 0x7f3c3c1f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:03.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c425a0700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3c3c15c6c0 0x7f3c3c0c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:03.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c4159e700 1 -- 172.21.15.111:0/358157869 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3c3c15c6c0 msgr2=0x7f3c3c0c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:03.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c4159e700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3c3c15c6c0 0x7f3c3c0c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:03.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c4159e700 1 -- 172.21.15.111:0/358157869 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c3c0b0440 msgr2=0x7f3c3c1f30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:03.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c4159e700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c3c0b0440 0x7f3c3c1f30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:03.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.866+0000 7f3c4159e700 1 -- 172.21.15.111:0/358157869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c34009a70 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.867+0000 7f3c4159e700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3c0ca2f0 0x7f3c3c0c83e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00d800 tx=0x7f3c2c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:03.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.867+0000 7f3c327fc700 1 -- 172.21.15.111:0/358157869 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c0125f0 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.867+0000 7f3c327fc700 1 -- 172.21.15.111:0/358157869 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2c012750 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.867+0000 7f3c44003700 1 -- 172.21.15.111:0/358157869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c3c0ccff0 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.867+0000 7f3c327fc700 1 -- 172.21.15.111:0/358157869 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c012b00 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.867+0000 7f3c44003700 1 -- 172.21.15.111:0/358157869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c3c0ccd10 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.868+0000 7f3c44003700 1 -- 172.21.15.111:0/358157869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3c0cd2d0 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.868+0000 7f3c327fc700 1 -- 172.21.15.111:0/358157869 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3c2c012c60 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.868+0000 7f3c327fc700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3c200682f0 0x7f3c2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:03.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.869+0000 7f3c327fc700 1 -- 172.21.15.111:0/358157869 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3c2c086250 con 0x7f3c3c0ca2f0 2023-11-28T13:01:03.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.869+0000 7f3c41d9f700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3c200682f0 0x7f3c2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:03.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.869+0000 7f3c41d9f700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3c200682f0 0x7f3c2006a7a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3c38005910 tx=0x7f3c38009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:03.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:03.873+0000 7f3c327fc700 1 -- 172.21.15.111:0/358157869 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3c2c0528f0 con 0x7f3c3c0ca2f0 2023-11-28T13:01:04.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.100+0000 7f3c44003700 1 -- 172.21.15.111:0/358157869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c3c0cdf80 con 0x7f3c3c0ca2f0 2023-11-28T13:01:04.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.101+0000 7f3c327fc700 1 -- 172.21.15.111:0/358157869 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f3c2c055f10 con 0x7f3c3c0ca2f0 2023-11-28T13:01:04.102 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:04.102 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2072621958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2072621958},{"type":"v1","addr":"172.21.15.149:6834","nonce":2072621958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3348427170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3348427170},{"type":"v1","addr":"172.21.15.149:6835","nonce":3348427170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1745443298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1745443298},{"type":"v1","addr":"172.21.15.111:6836","nonce":1745443298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:56.272312+0000","modified":"2023-11-28T13:00:59.284265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5814":{"gid":5814,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/4146274267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":4146274267},{"type":"v1","addr":"172.21.15.111:6837","nonce":4146274267}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T13:01:04.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.102+0000 7f3c44003700 1 -- 172.21.15.111:0/358157869 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3c200682f0 msgr2=0x7f3c2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.102+0000 7f3c44003700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3c200682f0 0x7f3c2006a7a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3c38005910 tx=0x7f3c38009040 comp rx=0 tx=0).stop 2023-11-28T13:01:04.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.102+0000 7f3c44003700 1 -- 172.21.15.111:0/358157869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3c0ca2f0 msgr2=0x7f3c3c0c83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.102+0000 7f3c44003700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3c0ca2f0 0x7f3c3c0c83e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00d800 tx=0x7f3c2c00db10 comp rx=0 tx=0).stop 2023-11-28T13:01:04.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.103+0000 7f3c44003700 1 -- 172.21.15.111:0/358157869 shutdown_connections 2023-11-28T13:01:04.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.103+0000 7f3c44003700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3c3c15c6c0 0x7f3c3c0c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.103+0000 7f3c44003700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3c200682f0 0x7f3c2006a7a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.103+0000 7f3c44003700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3c3c0b0440 0x7f3c3c1f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.103+0000 7f3c44003700 1 --2- 172.21.15.111:0/358157869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3c0ca2f0 0x7f3c3c0c83e0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.103+0000 7f3c44003700 1 -- 172.21.15.111:0/358157869 >> 172.21.15.111:0/358157869 conn(0x7f3c3c098ee0 msgr2=0x7f3c3c0cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.103+0000 7f3c44003700 1 -- 172.21.15.111:0/358157869 shutdown_connections 2023-11-28T13:01:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.103+0000 7f3c44003700 1 -- 172.21.15.111:0/358157869 wait complete. 2023-11-28T13:01:04.105 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 25 2023-11-28T13:01:04.116 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2023-11-28T13:01:04.263 INFO:teuthology.orchestra.run.smithi111.stdout:{"message":"","return_code":0} 2023-11-28T13:01:04.263 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-11-28T13:01:04.263 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2023-11-28T13:01:04.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.307+0000 7f4f7e5d4080 1 Processor -- start 2023-11-28T13:01:04.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.307+0000 7f4f7e5d4080 1 -- start start 2023-11-28T13:01:04.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f7e5d4080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f0a8a4e040 0x55f0a8a506b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f7e5d4080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f0a8a4e410 0x55f0a8a4ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f7e5d4080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f0a8a6e9f0 0x55f0a8a4f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f7e5d4080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0a89cc000 con 0x55f0a8a4e410 2023-11-28T13:01:04.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f7e5d4080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0a895efe0 con 0x55f0a8a4e040 2023-11-28T13:01:04.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f7e5d4080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f0a8a536d0 con 0x55f0a8a6e9f0 2023-11-28T13:01:04.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f73ed6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f0a8a4e410 0x55f0a8a4ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f73ed6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f0a8a4e410 0x55f0a8a4ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36390/0 (socket says 172.21.15.111:36390) 2023-11-28T13:01:04.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f73ed6700 1 -- 172.21.15.111:0/676761487 learned_addr learned my addr 172.21.15.111:0/676761487 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f746d7700 1 --2- 172.21.15.111:0/676761487 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f0a8a4e040 0x55f0a8a506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f74ed8700 1 --2- 172.21.15.111:0/676761487 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f0a8a6e9f0 0x55f0a8a4f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f73ed6700 1 -- 172.21.15.111:0/676761487 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f0a8a6e9f0 msgr2=0x55f0a8a4f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f73ed6700 1 --2- 172.21.15.111:0/676761487 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f0a8a6e9f0 0x55f0a8a4f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f73ed6700 1 -- 172.21.15.111:0/676761487 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f0a8a4e040 msgr2=0x55f0a8a506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f73ed6700 1 --2- 172.21.15.111:0/676761487 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f0a8a4e040 0x55f0a8a506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f73ed6700 1 -- 172.21.15.111:0/676761487 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f68005d20 con 0x55f0a8a4e410 2023-11-28T13:01:04.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f746d7700 1 --2- 172.21.15.111:0/676761487 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f0a8a4e040 0x55f0a8a506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:04.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.308+0000 7f4f73ed6700 1 --2- 172.21.15.111:0/676761487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f0a8a4e410 0x55f0a8a4ed10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4f68000c00 tx=0x7f4f6800d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18384f99d12dfe5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.310+0000 7f4f72ed4700 1 -- 172.21.15.111:0/676761487 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6801a040 con 0x55f0a8a4e410 2023-11-28T13:01:04.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.310+0000 7f4f72ed4700 1 -- 172.21.15.111:0/676761487 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f68013a80 con 0x55f0a8a4e410 2023-11-28T13:01:04.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.310+0000 7f4f72ed4700 1 -- 172.21.15.111:0/676761487 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68013e30 con 0x55f0a8a4e410 2023-11-28T13:01:04.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.310+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/676761487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f0a8a4e410 msgr2=0x55f0a8a4ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.310+0000 7f4f7e5d4080 1 --2- 172.21.15.111:0/676761487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f0a8a4e410 0x55f0a8a4ed10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4f68000c00 tx=0x7f4f6800d000 comp rx=0 tx=0).stop 2023-11-28T13:01:04.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.310+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/676761487 shutdown_connections 2023-11-28T13:01:04.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.310+0000 7f4f7e5d4080 1 --2- 172.21.15.111:0/676761487 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f0a8a6e9f0 0x55f0a8a4f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.310+0000 7f4f7e5d4080 1 --2- 172.21.15.111:0/676761487 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f0a8a4e040 0x55f0a8a506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.310+0000 7f4f7e5d4080 1 --2- 172.21.15.111:0/676761487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f0a8a4e410 0x55f0a8a4ed10 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.310+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/676761487 >> 172.21.15.111:0/676761487 conn(0x55f0a8a6c5a0 msgr2=0x55f0a8b858c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.310+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/676761487 shutdown_connections 2023-11-28T13:01:04.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.310+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/676761487 wait complete. 2023-11-28T13:01:04.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.311+0000 7f4f7e5d4080 1 Processor -- start 2023-11-28T13:01:04.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.311+0000 7f4f7e5d4080 1 -- start start 2023-11-28T13:01:04.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f7e5d4080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f0a8a4e040 0x55f0a8a22ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f7e5d4080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f0a8a4e410 0x55f0a8b8faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f7e5d4080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f0a8a6e9f0 0x55f0a8a64940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f7e5d4080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0a8a644b0 con 0x55f0a8a4e410 2023-11-28T13:01:04.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f7e5d4080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0a8b86530 con 0x55f0a8a6e9f0 2023-11-28T13:01:04.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f7e5d4080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f0a8b7e460 con 0x55f0a8a4e040 2023-11-28T13:01:04.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f73ed6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f0a8a4e410 0x55f0a8b8faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f73ed6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f0a8a4e410 0x55f0a8b8faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36396/0 (socket says 172.21.15.111:36396) 2023-11-28T13:01:04.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f73ed6700 1 -- 172.21.15.111:0/2474127699 learned_addr learned my addr 172.21.15.111:0/2474127699 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f74ed8700 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f0a8a6e9f0 0x55f0a8a64940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f746d7700 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f0a8a4e040 0x55f0a8a22ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f73ed6700 1 -- 172.21.15.111:0/2474127699 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f0a8a4e040 msgr2=0x55f0a8a22ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f73ed6700 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f0a8a4e040 0x55f0a8a22ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f73ed6700 1 -- 172.21.15.111:0/2474127699 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f0a8a6e9f0 msgr2=0x55f0a8a64940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f73ed6700 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f0a8a6e9f0 0x55f0a8a64940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f73ed6700 1 -- 172.21.15.111:0/2474127699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0a8977120 con 0x55f0a8a4e410 2023-11-28T13:01:04.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f73ed6700 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f0a8a4e410 0x55f0a8b8faa0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800c010 tx=0x7f4f68002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f71ed2700 1 -- 172.21.15.111:0/2474127699 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68008d00 con 0x55f0a8a4e410 2023-11-28T13:01:04.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f71ed2700 1 -- 172.21.15.111:0/2474127699 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f68008e80 con 0x55f0a8a4e410 2023-11-28T13:01:04.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f71ed2700 1 -- 172.21.15.111:0/2474127699 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68031650 con 0x55f0a8a4e410 2023-11-28T13:01:04.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.312+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/2474127699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f0a896a830 con 0x55f0a8a4e410 2023-11-28T13:01:04.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.313+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/2474127699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f0a8977360 con 0x55f0a8a4e410 2023-11-28T13:01:04.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.314+0000 7f4f71ed2700 1 -- 172.21.15.111:0/2474127699 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4f680273f0 con 0x55f0a8a4e410 2023-11-28T13:01:04.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.314+0000 7f4f71ed2700 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4f60066c10 0x7f4f600690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.314+0000 7f4f71ed2700 1 -- 172.21.15.111:0/2474127699 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4f680882f0 con 0x55f0a8a4e410 2023-11-28T13:01:04.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.314+0000 7f4f746d7700 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4f60066c10 0x7f4f600690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.314+0000 7f4f7e5d4080 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55f0a8a6b1d0 0x55f0a8b7db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.314+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/2474127699 --> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] -- osd_op(unknown.0.0:1 9.7 9:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55) v8 -- 0x55f0a8a532c0 con 0x55f0a8a6b1d0 2023-11-28T13:01:04.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.314+0000 7f4f746d7700 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4f60066c10 0x7f4f600690c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x55f0a8b92f70 tx=0x7f4f64003890 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.314+0000 7f4f74ed8700 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55f0a8a6b1d0 0x55f0a8b7db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.315+0000 7f4f74ed8700 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55f0a8a6b1d0 0x55f0a8b7db80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.315+0000 7f4f74ed8700 1 -- 172.21.15.111:0/2474127699 <== osd.5 v2:172.21.15.149:6808/103795 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x7f4f6c004560 con 0x55f0a8a6b1d0 2023-11-28T13:01:04.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.315+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/2474127699 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55f0a8a6b1d0 msgr2=0x55f0a8b7db80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.315+0000 7f4f7e5d4080 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55f0a8a6b1d0 0x55f0a8b7db80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.315+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/2474127699 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4f60066c10 msgr2=0x7f4f600690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.315+0000 7f4f7e5d4080 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4f60066c10 0x7f4f600690c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x55f0a8b92f70 tx=0x7f4f64003890 comp rx=0 tx=0).stop 2023-11-28T13:01:04.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.315+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/2474127699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f0a8a4e410 msgr2=0x55f0a8b8faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.315+0000 7f4f7e5d4080 1 --2- 172.21.15.111:0/2474127699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f0a8a4e410 0x55f0a8b8faa0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800c010 tx=0x7f4f68002fb0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.315+0000 7f4f74ed8700 1 -- 172.21.15.111:0/2474127699 reap_dead start 2023-11-28T13:01:04.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.315+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/2474127699 shutdown_connections 2023-11-28T13:01:04.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.315+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/2474127699 >> 172.21.15.111:0/2474127699 conn(0x55f0a8a6c5a0 msgr2=0x55f0a8a506b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.315+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/2474127699 shutdown_connections 2023-11-28T13:01:04.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.315+0000 7f4f7e5d4080 1 -- 172.21.15.111:0/2474127699 wait complete. 2023-11-28T13:01:04.325 INFO:teuthology.orchestra.run.smithi111.stdout:Yalphaparent_a 2023-11-28T13:01:04.326 DEBUG:teuthology.orchestra.run.smithi111:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2023-11-28T13:01:04.429 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:01:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 1099511627777, 2023-11-28T13:01:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "ancestors": [ 2023-11-28T13:01:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "dirino": 1099511627776, 2023-11-28T13:01:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "dname": "alpha", 2023-11-28T13:01:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "version": 2 2023-11-28T13:01:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "dirino": 1, 2023-11-28T13:01:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "dname": "parent_a", 2023-11-28T13:01:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "version": 4 2023-11-28T13:01:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-11-28T13:01:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 9, 2023-11-28T13:01:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: "old_pools": [] 2023-11-28T13:01:04.431 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:01:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:04.432 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'oh i'"'"'m sorry did i overwrite your xattr?' 2023-11-28T13:01:04.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.475+0000 7f35348a1080 1 Processor -- start 2023-11-28T13:01:04.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.475+0000 7f35348a1080 1 -- start start 2023-11-28T13:01:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f35348a1080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56018978e200 0x5601897b6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f35348a1080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56018978e7c0 0x5601898f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f35348a1080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5601897bca30 0x5601898f8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f35348a1080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601896c8fe0 con 0x56018978e200 2023-11-28T13:01:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f35348a1080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601897d2f30 con 0x56018978e7c0 2023-11-28T13:01:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f35348a1080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x56018985fcd0 con 0x5601897bca30 2023-11-28T13:01:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f352a9a4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56018978e200 0x5601897b6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f352a9a4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56018978e200 0x5601897b6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36410/0 (socket says 172.21.15.111:36410) 2023-11-28T13:01:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f352a9a4700 1 -- 172.21.15.111:0/3816152687 learned_addr learned my addr 172.21.15.111:0/3816152687 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f352b1a5700 1 --2- 172.21.15.111:0/3816152687 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5601897bca30 0x5601898f8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f352a1a3700 1 --2- 172.21.15.111:0/3816152687 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56018978e7c0 0x5601898f7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f352a9a4700 1 -- 172.21.15.111:0/3816152687 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5601897bca30 msgr2=0x5601898f8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f352a9a4700 1 --2- 172.21.15.111:0/3816152687 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5601897bca30 0x5601898f8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f352a9a4700 1 -- 172.21.15.111:0/3816152687 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56018978e7c0 msgr2=0x5601898f7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f352a9a4700 1 --2- 172.21.15.111:0/3816152687 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56018978e7c0 0x5601898f7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f352a9a4700 1 -- 172.21.15.111:0/3816152687 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352400a100 con 0x56018978e200 2023-11-28T13:01:04.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f352a9a4700 1 --2- 172.21.15.111:0/3816152687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56018978e200 0x5601897b6700 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f352000b240 tx=0x7f352000b550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd0ce59c3f6d3ec server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f35291a1700 1 -- 172.21.15.111:0/3816152687 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520014070 con 0x56018978e200 2023-11-28T13:01:04.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f35291a1700 1 -- 172.21.15.111:0/3816152687 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352000f3f0 con 0x56018978e200 2023-11-28T13:01:04.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.476+0000 7f35291a1700 1 -- 172.21.15.111:0/3816152687 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352000f760 con 0x56018978e200 2023-11-28T13:01:04.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.477+0000 7f35348a1080 1 -- 172.21.15.111:0/3816152687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56018978e200 msgr2=0x5601897b6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.477+0000 7f35348a1080 1 --2- 172.21.15.111:0/3816152687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56018978e200 0x5601897b6700 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f352000b240 tx=0x7f352000b550 comp rx=0 tx=0).stop 2023-11-28T13:01:04.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.477+0000 7f35348a1080 1 -- 172.21.15.111:0/3816152687 shutdown_connections 2023-11-28T13:01:04.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.477+0000 7f35348a1080 1 --2- 172.21.15.111:0/3816152687 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5601897bca30 0x5601898f8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.477+0000 7f35348a1080 1 --2- 172.21.15.111:0/3816152687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56018978e200 0x5601897b6700 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.477+0000 7f35348a1080 1 --2- 172.21.15.111:0/3816152687 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56018978e7c0 0x5601898f7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.477+0000 7f35348a1080 1 -- 172.21.15.111:0/3816152687 >> 172.21.15.111:0/3816152687 conn(0x5601897d61a0 msgr2=0x56018978d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.477+0000 7f35348a1080 1 -- 172.21.15.111:0/3816152687 shutdown_connections 2023-11-28T13:01:04.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.477+0000 7f35348a1080 1 -- 172.21.15.111:0/3816152687 wait complete. 2023-11-28T13:01:04.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.478+0000 7f35348a1080 1 Processor -- start 2023-11-28T13:01:04.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.478+0000 7f35348a1080 1 -- start start 2023-11-28T13:01:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f35348a1080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56018978e200 0x5601897cd940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f35348a1080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56018978e7c0 0x56018985e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f35348a1080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601897bca30 0x5601897b74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f35348a1080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601897cdf30 con 0x5601897bca30 2023-11-28T13:01:04.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f35348a1080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601897ce210 con 0x56018978e7c0 2023-11-28T13:01:04.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f35348a1080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5601897ce4f0 con 0x56018978e200 2023-11-28T13:01:04.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f352b1a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601897bca30 0x5601897b74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f352b1a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601897bca30 0x5601897b74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36420/0 (socket says 172.21.15.111:36420) 2023-11-28T13:01:04.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f352b1a5700 1 -- 172.21.15.111:0/1205241302 learned_addr learned my addr 172.21.15.111:0/1205241302 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f352a9a4700 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56018978e200 0x5601897cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f352a1a3700 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56018978e7c0 0x56018985e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f352b1a5700 1 -- 172.21.15.111:0/1205241302 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56018978e200 msgr2=0x5601897cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f352b1a5700 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56018978e200 0x5601897cd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f352b1a5700 1 -- 172.21.15.111:0/1205241302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56018978e7c0 msgr2=0x56018985e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f352b1a5700 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56018978e7c0 0x56018985e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f352b1a5700 1 -- 172.21.15.111:0/1205241302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3520001b40 con 0x5601897bca30 2023-11-28T13:01:04.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f352a1a3700 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56018978e7c0 0x56018985e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:04.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f352a9a4700 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56018978e200 0x5601897cd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:04.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f352b1a5700 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601897bca30 0x5601897b74e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x56018977ba60 tx=0x7f3524003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f3513fff700 1 -- 172.21.15.111:0/1205241302 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352401b860 con 0x5601897bca30 2023-11-28T13:01:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f3513fff700 1 -- 172.21.15.111:0/1205241302 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352401b9c0 con 0x5601897bca30 2023-11-28T13:01:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f35348a1080 1 -- 172.21.15.111:0/1205241302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5601896e1120 con 0x5601897bca30 2023-11-28T13:01:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f3513fff700 1 -- 172.21.15.111:0/1205241302 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352401bd50 con 0x5601897bca30 2023-11-28T13:01:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.479+0000 7f35348a1080 1 -- 172.21.15.111:0/1205241302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601896d4830 con 0x5601897bca30 2023-11-28T13:01:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.481+0000 7f3513fff700 1 -- 172.21.15.111:0/1205241302 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f352402c070 con 0x5601897bca30 2023-11-28T13:01:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.481+0000 7f3513fff700 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3518066c10 0x7f35180690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.481+0000 7f3513fff700 1 -- 172.21.15.111:0/1205241302 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3524085fb0 con 0x5601897bca30 2023-11-28T13:01:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.481+0000 7f352a9a4700 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3518066c10 0x7f35180690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.481+0000 7f35348a1080 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x5601897b7a20 0x560189908340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.481+0000 7f35348a1080 1 -- 172.21.15.111:0/1205241302 --> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] -- osd_op(unknown.0.0:1 9.7 9:e2e07930:::10000000001.00000000:head [setxattr parent (40) in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e55) v8 -- 0x560189908880 con 0x5601897b7a20 2023-11-28T13:01:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.481+0000 7f352a9a4700 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3518066c10 0x7f35180690c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f352000b240 tx=0x7f3520008000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.481+0000 7f352a1a3700 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x5601897b7a20 0x560189908340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.482+0000 7f352a1a3700 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x5601897b7a20 0x560189908340 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.483+0000 7f352a1a3700 1 -- 172.21.15.111:0/1205241302 <== osd.5 v2:172.21.15.149:6808/103795 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (40)] v55'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f351c009810 con 0x5601897b7a20 2023-11-28T13:01:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.483+0000 7f35348a1080 1 -- 172.21.15.111:0/1205241302 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x5601897b7a20 msgr2=0x560189908340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.483+0000 7f35348a1080 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x5601897b7a20 0x560189908340 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.483+0000 7f35348a1080 1 -- 172.21.15.111:0/1205241302 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3518066c10 msgr2=0x7f35180690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.483+0000 7f35348a1080 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3518066c10 0x7f35180690c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f352000b240 tx=0x7f3520008000 comp rx=0 tx=0).stop 2023-11-28T13:01:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.483+0000 7f35348a1080 1 -- 172.21.15.111:0/1205241302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601897bca30 msgr2=0x5601897b74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.483+0000 7f35348a1080 1 --2- 172.21.15.111:0/1205241302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601897bca30 0x5601897b74e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x56018977ba60 tx=0x7f3524003330 comp rx=0 tx=0).stop 2023-11-28T13:01:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.483+0000 7f352b1a5700 1 -- 172.21.15.111:0/1205241302 reap_dead start 2023-11-28T13:01:04.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.483+0000 7f35348a1080 1 -- 172.21.15.111:0/1205241302 shutdown_connections 2023-11-28T13:01:04.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.483+0000 7f35348a1080 1 -- 172.21.15.111:0/1205241302 >> 172.21.15.111:0/1205241302 conn(0x5601897d61a0 msgr2=0x5601897b6700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.484+0000 7f35348a1080 1 -- 172.21.15.111:0/1205241302 shutdown_connections 2023-11-28T13:01:04.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.484+0000 7f35348a1080 1 -- 172.21.15.111:0/1205241302 wait complete. 2023-11-28T13:01:04.491 DEBUG:teuthology.orchestra.run.smithi111:> sudo daemon-helper kill ceph --cluster ceph -w 2023-11-28T13:01:04.493 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start / repair,recursive 2023-11-28T13:01:04.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.617+0000 7f8d5ac09700 1 Processor -- start 2023-11-28T13:01:04.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.617+0000 7f8d5ac09700 1 -- start start 2023-11-28T13:01:04.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.618+0000 7f8d5ac09700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f8d5415cc50 0x7f8d540712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.618+0000 7f8d5ac09700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8d540cceb0 0x7f8d540cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.618+0000 7f8d5ac09700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d540717f0 con 0x7f8d54072300 2023-11-28T13:01:04.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.618+0000 7f8d5ac09700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d540726d0 con 0x7f8d540cceb0 2023-11-28T13:01:04.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.618+0000 7f8d5ac09700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d540cd810 con 0x7f8d5415cc50 2023-11-28T13:01:04.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.618+0000 7f8d591a6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8d540cceb0 0x7f8d540cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.618+0000 7f8d591a6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8d540cceb0 0x7f8d540cd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36430/0 (socket says 172.21.15.111:36430) 2023-11-28T13:01:04.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.618+0000 7f8d591a6700 1 -- 172.21.15.111:0/1038076335 learned_addr learned my addr 172.21.15.111:0/1038076335 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.618+0000 7f8d53fff700 1 --2- 172.21.15.111:0/1038076335 >> v2:172.21.15.149:3300/0 conn(0x7f8d5415cc50 0x7f8d540712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.618+0000 7f8d591a6700 1 -- 172.21.15.111:0/1038076335 >> v2:172.21.15.149:3300/0 conn(0x7f8d5415cc50 msgr2=0x7f8d540712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.618+0000 7f8d591a6700 1 --2- 172.21.15.111:0/1038076335 >> v2:172.21.15.149:3300/0 conn(0x7f8d5415cc50 0x7f8d540712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.618+0000 7f8d591a6700 1 -- 172.21.15.111:0/1038076335 >> v1:172.21.15.149:6789/0 conn(0x7f8d54072300 legacy=0x7f8d5406b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.618+0000 7f8d591a6700 1 -- 172.21.15.111:0/1038076335 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5406fb90 con 0x7f8d540cceb0 2023-11-28T13:01:04.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.619+0000 7f8d591a6700 1 --2- 172.21.15.111:0/1038076335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d540cceb0 0x7f8d540cd2d0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8d540aed10 tx=0x7f8d4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a507e0ec3c627eb0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.619+0000 7f8d537fe700 1 -- 172.21.15.111:0/1038076335 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c00e070 con 0x7f8d540cceb0 2023-11-28T13:01:04.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.619+0000 7f8d537fe700 1 -- 172.21.15.111:0/1038076335 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4c00b910 con 0x7f8d540cceb0 2023-11-28T13:01:04.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.619+0000 7f8d537fe700 1 -- 172.21.15.111:0/1038076335 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c00bcc0 con 0x7f8d540cceb0 2023-11-28T13:01:04.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.620+0000 7f8d5ac09700 1 -- 172.21.15.111:0/1038076335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d540cceb0 msgr2=0x7f8d540cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.620+0000 7f8d5ac09700 1 --2- 172.21.15.111:0/1038076335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d540cceb0 0x7f8d540cd2d0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8d540aed10 tx=0x7f8d4c00b330 comp rx=0 tx=0).stop 2023-11-28T13:01:04.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.620+0000 7f8d5ac09700 1 -- 172.21.15.111:0/1038076335 shutdown_connections 2023-11-28T13:01:04.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.620+0000 7f8d5ac09700 1 --2- 172.21.15.111:0/1038076335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d540cceb0 0x7f8d540cd2d0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.620+0000 7f8d5ac09700 1 --2- 172.21.15.111:0/1038076335 >> v2:172.21.15.149:3300/0 conn(0x7f8d5415cc50 0x7f8d540712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.620+0000 7f8d5ac09700 1 -- 172.21.15.111:0/1038076335 >> 172.21.15.111:0/1038076335 conn(0x7f8d540cf2e0 msgr2=0x7f8d540cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.620+0000 7f8d5ac09700 1 -- 172.21.15.111:0/1038076335 shutdown_connections 2023-11-28T13:01:04.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.620+0000 7f8d5ac09700 1 -- 172.21.15.111:0/1038076335 wait complete. 2023-11-28T13:01:04.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.621+0000 7f8d5ac09700 1 Processor -- start 2023-11-28T13:01:04.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.621+0000 7f8d5ac09700 1 -- start start 2023-11-28T13:01:04.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.622+0000 7f8d5ac09700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d54072300 0x7f8d541f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.622+0000 7f8d5ac09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d540cceb0 0x7f8d541f4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.622+0000 7f8d5ac09700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d5415cc50 0x7f8d540b1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.622+0000 7f8d5ac09700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d540af6d0 con 0x7f8d540cceb0 2023-11-28T13:01:04.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.622+0000 7f8d5ac09700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d540aede0 con 0x7f8d54072300 2023-11-28T13:01:04.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.622+0000 7f8d5ac09700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d541f3800 con 0x7f8d5415cc50 2023-11-28T13:01:04.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.622+0000 7f8d53fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d540cceb0 0x7f8d541f4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.622+0000 7f8d53fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d540cceb0 0x7f8d541f4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36434/0 (socket says 172.21.15.111:36434) 2023-11-28T13:01:04.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.622+0000 7f8d53fff700 1 -- 172.21.15.111:0/3343768027 learned_addr learned my addr 172.21.15.111:0/3343768027 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.623+0000 7f8d589a5700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d54072300 0x7f8d541f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.623+0000 7f8d591a6700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d5415cc50 0x7f8d540b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.623+0000 7f8d53fff700 1 -- 172.21.15.111:0/3343768027 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d5415cc50 msgr2=0x7f8d540b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.623+0000 7f8d53fff700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d5415cc50 0x7f8d540b1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.623+0000 7f8d53fff700 1 -- 172.21.15.111:0/3343768027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d54072300 msgr2=0x7f8d541f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.623+0000 7f8d53fff700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d54072300 0x7f8d541f32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.623+0000 7f8d53fff700 1 -- 172.21.15.111:0/3343768027 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c009b30 con 0x7f8d540cceb0 2023-11-28T13:01:04.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.623+0000 7f8d53fff700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d540cceb0 0x7f8d541f4eb0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f8d4000e840 tx=0x7f8d4000eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.623+0000 7f8d517fa700 1 -- 172.21.15.111:0/3343768027 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d40013680 con 0x7f8d540cceb0 2023-11-28T13:01:04.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.623+0000 7f8d517fa700 1 -- 172.21.15.111:0/3343768027 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d400137e0 con 0x7f8d540cceb0 2023-11-28T13:01:04.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.623+0000 7f8d5ac09700 1 -- 172.21.15.111:0/3343768027 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5415d3d0 con 0x7f8d540cceb0 2023-11-28T13:01:04.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.623+0000 7f8d517fa700 1 -- 172.21.15.111:0/3343768027 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d40013b90 con 0x7f8d540cceb0 2023-11-28T13:01:04.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.623+0000 7f8d5ac09700 1 -- 172.21.15.111:0/3343768027 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5415d0f0 con 0x7f8d540cceb0 2023-11-28T13:01:04.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.625+0000 7f8d517fa700 1 -- 172.21.15.111:0/3343768027 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8d40019030 con 0x7f8d540cceb0 2023-11-28T13:01:04.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.625+0000 7f8d517fa700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8d380682f0 0x7f8d3806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.625+0000 7f8d517fa700 1 -- 172.21.15.111:0/3343768027 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8d40085bd0 con 0x7f8d540cceb0 2023-11-28T13:01:04.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.625+0000 7f8d589a5700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8d380682f0 0x7f8d3806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.626+0000 7f8d589a5700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8d380682f0 0x7f8d3806a7a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8d540c6f00 tx=0x7f8d48009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.627+0000 7f8d66d92b80 1 Processor -- start 2023-11-28T13:01:04.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.627+0000 7f8d66d92b80 1 -- start start 2023-11-28T13:01:04.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d66d92b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 0x5650b775caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d66d92b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b7784200 0x5650b7786cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d66d92b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77a2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d66d92b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b7646c90 con 0x5650b77871f0 2023-11-28T13:01:04.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d66d92b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b76b5420 con 0x5650b7784200 2023-11-28T13:01:04.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d66d92b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5650b76ae950 con 0x5650b77868e0 2023-11-28T13:01:04.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d591a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d591a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77a2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36444/0 (socket says 172.21.15.111:36444) 2023-11-28T13:01:04.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d591a6700 1 -- 172.21.15.111:0/732016294 learned_addr learned my addr 172.21.15.111:0/732016294 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d53fff700 1 --2- 172.21.15.111:0/732016294 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b7784200 0x5650b7786cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d589a5700 1 --2- 172.21.15.111:0/732016294 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 0x5650b775caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d591a6700 1 -- 172.21.15.111:0/732016294 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 msgr2=0x5650b775caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d591a6700 1 --2- 172.21.15.111:0/732016294 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 0x5650b775caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d591a6700 1 -- 172.21.15.111:0/732016294 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b7784200 msgr2=0x5650b7786cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d591a6700 1 --2- 172.21.15.111:0/732016294 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b7784200 0x5650b7786cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d591a6700 1 -- 172.21.15.111:0/732016294 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650b75306b0 con 0x5650b77871f0 2023-11-28T13:01:04.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d53fff700 1 --2- 172.21.15.111:0/732016294 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b7784200 0x5650b7786cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:04.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d591a6700 1 --2- 172.21.15.111:0/732016294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77a2840 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c012000 tx=0x7f8d4c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d76ab241de23a76 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.628+0000 7f8d3e7fc700 1 -- 172.21.15.111:0/732016294 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c00bc80 con 0x5650b77871f0 2023-11-28T13:01:04.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.629+0000 7f8d3e7fc700 1 -- 172.21.15.111:0/732016294 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4c003a20 con 0x5650b77871f0 2023-11-28T13:01:04.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.629+0000 7f8d3e7fc700 1 -- 172.21.15.111:0/732016294 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c003dd0 con 0x5650b77871f0 2023-11-28T13:01:04.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.629+0000 7f8d66d92b80 1 -- 172.21.15.111:0/732016294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 msgr2=0x5650b77a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.629+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/732016294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77a2840 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c012000 tx=0x7f8d4c010000 comp rx=0 tx=0).stop 2023-11-28T13:01:04.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.629+0000 7f8d66d92b80 1 -- 172.21.15.111:0/732016294 shutdown_connections 2023-11-28T13:01:04.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.629+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/732016294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77a2840 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.629+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/732016294 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 0x5650b775caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.629+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/732016294 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b7784200 0x5650b7786cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.629+0000 7f8d66d92b80 1 -- 172.21.15.111:0/732016294 >> 172.21.15.111:0/732016294 conn(0x5650b7759e30 msgr2=0x5650b775a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.629+0000 7f8d66d92b80 1 -- 172.21.15.111:0/732016294 shutdown_connections 2023-11-28T13:01:04.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.629+0000 7f8d66d92b80 1 -- 172.21.15.111:0/732016294 wait complete. 2023-11-28T13:01:04.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.630+0000 7f8d66d92b80 1 -- start start 2023-11-28T13:01:04.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.631+0000 7f8d66d92b80 1 Processor -- start 2023-11-28T13:01:04.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d66d92b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b7784200 0x5650b778a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d66d92b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 0x5650b7785580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d66d92b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b77871f0 0x5650b7785ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d66d92b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b75329a0 con 0x5650b7784200 2023-11-28T13:01:04.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d66d92b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b7798d50 con 0x5650b77871f0 2023-11-28T13:01:04.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d66d92b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5650b7779bb0 con 0x5650b77868e0 2023-11-28T13:01:04.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d66d92b80 10 client.0 _ll_register_callbacks cb 0x5650b7554310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:01:04.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d589a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b7784200 0x5650b778a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d589a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b7784200 0x5650b778a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36446/0 (socket says 172.21.15.111:36446) 2023-11-28T13:01:04.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d589a5700 1 -- 172.21.15.111:0/2509707283 learned_addr learned my addr 172.21.15.111:0/2509707283 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d591a6700 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b77871f0 0x5650b7785ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d53fff700 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 0x5650b7785580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d589a5700 1 -- 172.21.15.111:0/2509707283 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 msgr2=0x5650b7785580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d589a5700 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 0x5650b7785580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d589a5700 1 -- 172.21.15.111:0/2509707283 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b77871f0 msgr2=0x5650b7785ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d589a5700 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b77871f0 0x5650b7785ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d589a5700 1 -- 172.21.15.111:0/2509707283 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4c00be70 con 0x5650b7784200 2023-11-28T13:01:04.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d591a6700 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b77871f0 0x5650b7785ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:04.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d589a5700 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b7784200 0x5650b778a600 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8d480016b0 tx=0x7f8d48001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01edecfb4f35853 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.632+0000 7f8d337fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:01:04.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d337fe700 1 -- 172.21.15.111:0/2509707283 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d48014c50 con 0x5650b7784200 2023-11-28T13:01:04.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d337fe700 1 -- 172.21.15.111:0/2509707283 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8d4801c040 con 0x5650b7784200 2023-11-28T13:01:04.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2509707283 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5650b767c210 con 0x5650b7784200 2023-11-28T13:01:04.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d337fe700 1 -- 172.21.15.111:0/2509707283 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d48014db0 con 0x5650b7784200 2023-11-28T13:01:04.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d66d92b80 10 client.6000 fetch_fsmap learned FSMap version 25 2023-11-28T13:01:04.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2509707283 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5650b74dce60 con 0x5650b7784200 2023-11-28T13:01:04.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d337fe700 1 -- 172.21.15.111:0/2509707283 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5650b767c210 con 0x5650b7784200 2023-11-28T13:01:04.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d337fe700 1 -- 172.21.15.111:0/2509707283 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 25) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f8d48029350 con 0x5650b7784200 2023-11-28T13:01:04.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d66d92b80 10 client.6000 fetch_fsmap finished waiting for FSMap version 25 2023-11-28T13:01:04.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d66d92b80 10 client.6000 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2023-11-28T13:01:04.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d66d92b80 20 client.6000 populate_metadata read hostname 'smithi111' 2023-11-28T13:01:04.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x5650b7798650 0x5650b7775b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d66d92b80 4 client.6000 mds_command: new command op to 5814 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-28T13:01:04.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2509707283 --> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5650b7532fe0 con 0x5650b7798650 2023-11-28T13:01:04.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.633+0000 7f8d53fff700 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x5650b7798650 0x5650b7775b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d337fe700 10 client.6000 ms_handle_connect on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:04.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d53fff700 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x5650b7798650 0x5650b7775b20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5740d29f6485e0b4 server_cookie=fabf1dbe477ae7e8 in_seq=0 out_seq=0 2023-11-28T13:01:04.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d337fe700 1 -- 172.21.15.111:0/2509707283 <== mds.0 v2:172.21.15.111:6835/4146274267 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5650b7532fe0 con 0x5650b7798650 2023-11-28T13:01:04.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d337fe700 10 client.6000 handle_command_reply: tid=0 2023-11-28T13:01:04.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d66d92b80 1 client.6000 shutdown 2023-11-28T13:01:04.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2509707283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b7784200 msgr2=0x5650b778a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b7784200 0x5650b778a600 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8d480016b0 tx=0x7f8d48001120 comp rx=0 tx=0).stop 2023-11-28T13:01:04.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2509707283 shutdown_connections 2023-11-28T13:01:04.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x5650b7798650 0x5650b7775b20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b77871f0 0x5650b7785ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d337fe700 0 client.6000 ms_handle_reset on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:04.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b7784200 0x5650b778a600 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/2509707283 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 0x5650b7785580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2509707283 >> 172.21.15.111:0/2509707283 conn(0x5650b7759e30 msgr2=0x5650b775ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2509707283 shutdown_connections 2023-11-28T13:01:04.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.634+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2509707283 wait complete. 2023-11-28T13:01:04.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.635+0000 7f8d66d92b80 20 client.6000 trim_cache size 0 max 16384 2023-11-28T13:01:04.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.643+0000 7f027e5a4700 1 Processor -- start 2023-11-28T13:01:04.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.643+0000 7f027e5a4700 1 -- start start 2023-11-28T13:01:04.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.643+0000 7f027e5a4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f02780cdde0 0x7f02780b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f027e5a4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f027815d7a0 0x7f027815cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f027e5a4700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02780b35a0 con 0x7f027815d090 2023-11-28T13:01:04.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f027e5a4700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027806b820 con 0x7f02780cdde0 2023-11-28T13:01:04.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f027e5a4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027815d510 con 0x7f027815d7a0 2023-11-28T13:01:04.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f0277fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f02780cdde0 0x7f02780b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f0277fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f02780cdde0 0x7f02780b26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36454/0 (socket says 172.21.15.111:36454) 2023-11-28T13:01:04.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f0277fff700 1 -- 172.21.15.111:0/928486753 learned_addr learned my addr 172.21.15.111:0/928486753 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f02777fe700 1 --2- 172.21.15.111:0/928486753 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f027815d7a0 0x7f027815cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f0277fff700 1 -- 172.21.15.111:0/928486753 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f027815d7a0 msgr2=0x7f027815cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f0277fff700 1 --2- 172.21.15.111:0/928486753 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f027815d7a0 0x7f027815cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f0277fff700 1 -- 172.21.15.111:0/928486753 >> v1:172.21.15.149:6789/0 conn(0x7f027815d090 legacy=0x7f0278160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f0277fff700 1 -- 172.21.15.111:0/928486753 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02780b2fe0 con 0x7f02780cdde0 2023-11-28T13:01:04.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f0277fff700 1 --2- 172.21.15.111:0/928486753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780cdde0 0x7f02780b26e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f02680098f0 tx=0x7f0268009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385c1f376e9b29a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f0276ffd700 1 -- 172.21.15.111:0/928486753 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026800b8a0 con 0x7f02780cdde0 2023-11-28T13:01:04.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f0276ffd700 1 -- 172.21.15.111:0/928486753 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026800ba00 con 0x7f02780cdde0 2023-11-28T13:01:04.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.644+0000 7f0276ffd700 1 -- 172.21.15.111:0/928486753 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026800bd90 con 0x7f02780cdde0 2023-11-28T13:01:04.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.645+0000 7f027e5a4700 1 -- 172.21.15.111:0/928486753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780cdde0 msgr2=0x7f02780b26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.645+0000 7f027e5a4700 1 --2- 172.21.15.111:0/928486753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780cdde0 0x7f02780b26e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f02680098f0 tx=0x7f0268009c00 comp rx=0 tx=0).stop 2023-11-28T13:01:04.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.645+0000 7f027e5a4700 1 -- 172.21.15.111:0/928486753 shutdown_connections 2023-11-28T13:01:04.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.645+0000 7f027e5a4700 1 --2- 172.21.15.111:0/928486753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02780cdde0 0x7f02780b26e0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.645+0000 7f027e5a4700 1 --2- 172.21.15.111:0/928486753 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f027815d7a0 0x7f027815cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.645+0000 7f027e5a4700 1 -- 172.21.15.111:0/928486753 >> 172.21.15.111:0/928486753 conn(0x7f027806ebf0 msgr2=0x7f027806efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.645+0000 7f027e5a4700 1 -- 172.21.15.111:0/928486753 shutdown_connections 2023-11-28T13:01:04.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.645+0000 7f027e5a4700 1 -- 172.21.15.111:0/928486753 wait complete. 2023-11-28T13:01:04.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.647+0000 7f027e5a4700 1 Processor -- start 2023-11-28T13:01:04.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.647+0000 7f027e5a4700 1 -- start start 2023-11-28T13:01:04.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027e5a4700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f02780cdde0 0x7f0278158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027e5a4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f027815d090 0x7f02780c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027e5a4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f027815d7a0 0x7f0278158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027e5a4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02780b4e70 con 0x7f027815d7a0 2023-11-28T13:01:04.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027e5a4700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02780c7510 con 0x7f02780cdde0 2023-11-28T13:01:04.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027e5a4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02780c5340 con 0x7f027815d090 2023-11-28T13:01:04.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027cb41700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f027815d7a0 0x7f0278158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027cb41700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f027815d7a0 0x7f0278158590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36464/0 (socket says 172.21.15.111:36464) 2023-11-28T13:01:04.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027cb41700 1 -- 172.21.15.111:0/856373349 learned_addr learned my addr 172.21.15.111:0/856373349 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f0277fff700 1 --2- 172.21.15.111:0/856373349 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f02780cdde0 0x7f0278158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f02777fe700 1 --2- 172.21.15.111:0/856373349 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f027815d090 0x7f02780c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027cb41700 1 -- 172.21.15.111:0/856373349 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f027815d090 msgr2=0x7f02780c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027cb41700 1 --2- 172.21.15.111:0/856373349 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f027815d090 0x7f02780c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027cb41700 1 -- 172.21.15.111:0/856373349 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f02780cdde0 msgr2=0x7f0278158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027cb41700 1 --2- 172.21.15.111:0/856373349 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f02780cdde0 0x7f0278158da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027cb41700 1 -- 172.21.15.111:0/856373349 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0268009550 con 0x7f027815d7a0 2023-11-28T13:01:04.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027cb41700 1 --2- 172.21.15.111:0/856373349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f027815d7a0 0x7f0278158590 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f027000cc50 tx=0x7f027000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f0274ff9700 1 -- 172.21.15.111:0/856373349 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0270003c00 con 0x7f027815d7a0 2023-11-28T13:01:04.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f0274ff9700 1 -- 172.21.15.111:0/856373349 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0270003d60 con 0x7f027815d7a0 2023-11-28T13:01:04.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027e5a4700 1 -- 172.21.15.111:0/856373349 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027806b4b0 con 0x7f027815d7a0 2023-11-28T13:01:04.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f0274ff9700 1 -- 172.21.15.111:0/856373349 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027000aab0 con 0x7f027815d7a0 2023-11-28T13:01:04.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.648+0000 7f027e5a4700 1 -- 172.21.15.111:0/856373349 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02780b32c0 con 0x7f027815d7a0 2023-11-28T13:01:04.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.649+0000 7f027e5a4700 1 -- 172.21.15.111:0/856373349 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f02780b2c20 con 0x7f027815d7a0 2023-11-28T13:01:04.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.650+0000 7f0274ff9700 1 -- 172.21.15.111:0/856373349 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0270010050 con 0x7f027815d7a0 2023-11-28T13:01:04.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.650+0000 7f0274ff9700 1 --2- 172.21.15.111:0/856373349 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f02640682f0 0x7f026406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.650+0000 7f0274ff9700 1 -- 172.21.15.111:0/856373349 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0270086670 con 0x7f027815d7a0 2023-11-28T13:01:04.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.650+0000 7f0277fff700 1 --2- 172.21.15.111:0/856373349 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f02640682f0 0x7f026406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.650+0000 7f0274ff9700 1 -- 172.21.15.111:0/856373349 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+502 (secure 0 0 0) 0x7f0270086af0 con 0x7f027815d7a0 2023-11-28T13:01:04.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.651+0000 7f027e5a4700 1 -- 172.21.15.111:0/856373349 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7f027804e620 con 0x7f027815d7a0 2023-11-28T13:01:04.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.651+0000 7f0277fff700 1 --2- 172.21.15.111:0/856373349 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f02640682f0 0x7f026406a7a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0268009c40 tx=0x7f0268008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.654+0000 7f8d66d92b80 1 Processor -- start 2023-11-28T13:01:04.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.654+0000 7f8d66d92b80 1 -- start start 2023-11-28T13:01:04.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d66d92b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b7784200 0x5650b77a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d66d92b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b77868e0 0x5650b77a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d66d92b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77a5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d66d92b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b7785ae0 con 0x5650b77871f0 2023-11-28T13:01:04.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d66d92b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b779ab60 con 0x5650b77868e0 2023-11-28T13:01:04.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d66d92b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5650b7786090 con 0x5650b7784200 2023-11-28T13:01:04.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d591a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d591a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77a5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36472/0 (socket says 172.21.15.111:36472) 2023-11-28T13:01:04.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d591a6700 1 -- 172.21.15.111:0/3597519997 learned_addr learned my addr 172.21.15.111:0/3597519997 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d53fff700 1 --2- 172.21.15.111:0/3597519997 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b77868e0 0x5650b77a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d589a5700 1 --2- 172.21.15.111:0/3597519997 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b7784200 0x5650b77a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d591a6700 1 -- 172.21.15.111:0/3597519997 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b7784200 msgr2=0x5650b77a3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d591a6700 1 --2- 172.21.15.111:0/3597519997 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b7784200 0x5650b77a3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d591a6700 1 -- 172.21.15.111:0/3597519997 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b77868e0 msgr2=0x5650b77a4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d591a6700 1 --2- 172.21.15.111:0/3597519997 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b77868e0 0x5650b77a4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d591a6700 1 -- 172.21.15.111:0/3597519997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650b7675210 con 0x5650b77871f0 2023-11-28T13:01:04.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d53fff700 1 --2- 172.21.15.111:0/3597519997 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b77868e0 0x5650b77a4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:04.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d591a6700 1 --2- 172.21.15.111:0/3597519997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77a5500 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x5650b77581a0 tx=0x7f8d4c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4e2d1123ac1e05 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d3effd700 1 -- 172.21.15.111:0/3597519997 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c00bc30 con 0x5650b77871f0 2023-11-28T13:01:04.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d3effd700 1 -- 172.21.15.111:0/3597519997 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4c003a20 con 0x5650b77871f0 2023-11-28T13:01:04.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.655+0000 7f8d3effd700 1 -- 172.21.15.111:0/3597519997 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c003dd0 con 0x5650b77871f0 2023-11-28T13:01:04.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.656+0000 7f8d66d92b80 1 -- 172.21.15.111:0/3597519997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 msgr2=0x5650b77a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.656+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/3597519997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77a5500 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x5650b77581a0 tx=0x7f8d4c011000 comp rx=0 tx=0).stop 2023-11-28T13:01:04.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.656+0000 7f8d66d92b80 1 -- 172.21.15.111:0/3597519997 shutdown_connections 2023-11-28T13:01:04.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.656+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/3597519997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77a5500 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.656+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/3597519997 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b7784200 0x5650b77a3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.656+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/3597519997 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b77868e0 0x5650b77a4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.656+0000 7f8d66d92b80 1 -- 172.21.15.111:0/3597519997 >> 172.21.15.111:0/3597519997 conn(0x5650b7759e30 msgr2=0x5650b7789d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.656+0000 7f8d66d92b80 1 -- 172.21.15.111:0/3597519997 shutdown_connections 2023-11-28T13:01:04.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.656+0000 7f8d66d92b80 1 -- 172.21.15.111:0/3597519997 wait complete. 2023-11-28T13:01:04.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.657+0000 7f8d66d92b80 1 -- start start 2023-11-28T13:01:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.657+0000 7f8d66d92b80 1 Processor -- start 2023-11-28T13:01:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d66d92b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b7784200 0x5650b778a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d66d92b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 0x5650b7755920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d66d92b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77eca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d66d92b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b77bad10 con 0x5650b77871f0 2023-11-28T13:01:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d66d92b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b77a4850 con 0x5650b7784200 2023-11-28T13:01:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d66d92b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5650b7755e60 con 0x5650b77868e0 2023-11-28T13:01:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d66d92b80 10 client.0 _ll_register_callbacks cb 0x5650b7554310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:01:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d591a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77eca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d591a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77eca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36478/0 (socket says 172.21.15.111:36478) 2023-11-28T13:01:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d591a6700 1 -- 172.21.15.111:0/2511361882 learned_addr learned my addr 172.21.15.111:0/2511361882 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d589a5700 1 --2- 172.21.15.111:0/2511361882 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b7784200 0x5650b778a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d53fff700 1 --2- 172.21.15.111:0/2511361882 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 0x5650b7755920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d591a6700 1 -- 172.21.15.111:0/2511361882 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 msgr2=0x5650b7755920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d591a6700 1 --2- 172.21.15.111:0/2511361882 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 0x5650b7755920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d591a6700 1 -- 172.21.15.111:0/2511361882 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b7784200 msgr2=0x5650b778a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d591a6700 1 --2- 172.21.15.111:0/2511361882 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b7784200 0x5650b778a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d591a6700 1 -- 172.21.15.111:0/2511361882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650b75306b0 con 0x5650b77871f0 2023-11-28T13:01:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d591a6700 1 --2- 172.21.15.111:0/2511361882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77eca10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00b370 tx=0x7f8d4c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63281b53af954e97 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d337fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:01:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.658+0000 7f8d337fe700 1 -- 172.21.15.111:0/2511361882 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c01cc70 con 0x5650b77871f0 2023-11-28T13:01:04.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d337fe700 1 -- 172.21.15.111:0/2511361882 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8d4c01cdd0 con 0x5650b77871f0 2023-11-28T13:01:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2511361882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5650b766c230 con 0x5650b77871f0 2023-11-28T13:01:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d337fe700 1 -- 172.21.15.111:0/2511361882 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d4c037430 con 0x5650b77871f0 2023-11-28T13:01:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d66d92b80 10 client.6012 fetch_fsmap learned FSMap version 25 2023-11-28T13:01:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2511361882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5650b75032d0 con 0x5650b77871f0 2023-11-28T13:01:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d337fe700 1 -- 172.21.15.111:0/2511361882 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5650b766c230 con 0x5650b77871f0 2023-11-28T13:01:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d337fe700 1 -- 172.21.15.111:0/2511361882 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 25) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f8d4c022030 con 0x5650b77871f0 2023-11-28T13:01:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d66d92b80 10 client.6012 fetch_fsmap finished waiting for FSMap version 25 2023-11-28T13:01:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d66d92b80 10 client.6012 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2023-11-28T13:01:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d66d92b80 20 client.6012 populate_metadata read hostname 'smithi111' 2023-11-28T13:01:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/2511361882 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x5650b77eff20 0x5650b77ecf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d66d92b80 4 client.6012 mds_command: new command op to 5814 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2023-11-28T13:01:04.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2511361882 --> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) v1 -- 0x5650b7656240 con 0x5650b77eff20 2023-11-28T13:01:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d589a5700 1 --2- 172.21.15.111:0/2511361882 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x5650b77eff20 0x5650b77ecf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d337fe700 10 client.6012 ms_handle_connect on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.659+0000 7f8d589a5700 1 --2- 172.21.15.111:0/2511361882 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x5650b77eff20 0x5650b77ecf50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c37e806e3e65e46 server_cookie=b89311499c5e6643 in_seq=0 out_seq=0 2023-11-28T13:01:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.660+0000 7f8d337fe700 1 -- 172.21.15.111:0/2511361882 <== mds.0 v2:172.21.15.111:6835/4146274267 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5650b7656240 con 0x5650b77eff20 2023-11-28T13:01:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.660+0000 7f8d337fe700 10 client.6012 handle_command_reply: tid=0 2023-11-28T13:01:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.660+0000 7f8d66d92b80 1 client.6012 shutdown 2023-11-28T13:01:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.660+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2511361882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 msgr2=0x5650b77eca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.660+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/2511361882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77eca10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00b370 tx=0x7f8d4c004b40 comp rx=0 tx=0).stop 2023-11-28T13:01:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.660+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2511361882 shutdown_connections 2023-11-28T13:01:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.660+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/2511361882 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x5650b77eff20 0x5650b77ecf50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.660+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/2511361882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650b77871f0 0x5650b77eca10 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.660+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/2511361882 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5650b7784200 0x5650b778a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.660+0000 7f8d66d92b80 1 --2- 172.21.15.111:0/2511361882 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5650b77868e0 0x5650b7755920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.660+0000 7f8d337fe700 0 client.6012 ms_handle_reset on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:04.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.660+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2511361882 >> 172.21.15.111:0/2511361882 conn(0x5650b7759e30 msgr2=0x5650b77a4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.668 INFO:tasks.ceph.mds.c.smithi111.stderr:2023-11-28T13:01:04.660+0000 7fa2f4474700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5219=0-4194304@1} caps={5219=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5219 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x55abdfdffb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-28T13:01:04.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2511361882 shutdown_connections 2023-11-28T13:01:04.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d66d92b80 1 -- 172.21.15.111:0/2511361882 wait complete. 2023-11-28T13:01:04.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d66d92b80 20 client.6012 trim_cache size 0 max 16384 2023-11-28T13:01:04.669 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:01:04.669 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-11-28T13:01:04.669 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "88e043de-10f3-46f3-b9d8-60f2596d62dd", 2023-11-28T13:01:04.669 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-11-28T13:01:04.669 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:01:04.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d5ac09700 1 -- 172.21.15.111:0/3343768027 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8d380682f0 msgr2=0x7f8d3806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d5ac09700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8d380682f0 0x7f8d3806a7a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8d540c6f00 tx=0x7f8d48009040 comp rx=0 tx=0).stop 2023-11-28T13:01:04.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d5ac09700 1 -- 172.21.15.111:0/3343768027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d540cceb0 msgr2=0x7f8d541f4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d5ac09700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d540cceb0 0x7f8d541f4eb0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f8d4000e840 tx=0x7f8d4000eb50 comp rx=0 tx=0).stop 2023-11-28T13:01:04.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d5ac09700 1 -- 172.21.15.111:0/3343768027 shutdown_connections 2023-11-28T13:01:04.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d5ac09700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8d5415cc50 0x7f8d540b1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d5ac09700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8d380682f0 0x7f8d3806a7a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d5ac09700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8d54072300 0x7f8d541f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d5ac09700 1 --2- 172.21.15.111:0/3343768027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8d540cceb0 0x7f8d541f4eb0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d5ac09700 1 -- 172.21.15.111:0/3343768027 >> 172.21.15.111:0/3343768027 conn(0x7f8d540cf2e0 msgr2=0x7f8d540c9910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d5ac09700 1 -- 172.21.15.111:0/3343768027 shutdown_connections 2023-11-28T13:01:04.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.661+0000 7f8d5ac09700 1 -- 172.21.15.111:0/3343768027 wait complete. 2023-11-28T13:01:04.674 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub status 2023-11-28T13:01:04.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.794+0000 7fe630976700 1 Processor -- start 2023-11-28T13:01:04.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.794+0000 7fe630976700 1 -- start start 2023-11-28T13:01:04.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.795+0000 7fe630976700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe62c0b0d70 0x7fe62c1604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.795+0000 7fe630976700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe62c0cc550 0x7fe62c0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.795+0000 7fe630976700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe62c15cd90 con 0x7fe62c0b1140 2023-11-28T13:01:04.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.795+0000 7fe630976700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62c0cedd0 con 0x7fe62c0cc550 2023-11-28T13:01:04.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.795+0000 7fe630976700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62c0cced0 con 0x7fe62c0b0d70 2023-11-28T13:01:04.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.796+0000 7fe62ad9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe62c0cc550 0x7fe62c0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.796+0000 7fe62ad9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe62c0cc550 0x7fe62c0cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36486/0 (socket says 172.21.15.111:36486) 2023-11-28T13:01:04.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.796+0000 7fe62ad9d700 1 -- 172.21.15.111:0/3000400095 learned_addr learned my addr 172.21.15.111:0/3000400095 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.796+0000 7fe62a59c700 1 --2- 172.21.15.111:0/3000400095 >> v2:172.21.15.149:3300/0 conn(0x7fe62c0b0d70 0x7fe62c1604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.796+0000 7fe62ad9d700 1 -- 172.21.15.111:0/3000400095 >> v2:172.21.15.149:3300/0 conn(0x7fe62c0b0d70 msgr2=0x7fe62c1604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.796+0000 7fe62ad9d700 1 --2- 172.21.15.111:0/3000400095 >> v2:172.21.15.149:3300/0 conn(0x7fe62c0b0d70 0x7fe62c1604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.796+0000 7fe62ad9d700 1 -- 172.21.15.111:0/3000400095 >> v1:172.21.15.149:6789/0 conn(0x7fe62c0b1140 legacy=0x7fe62c0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.796+0000 7fe62ad9d700 1 -- 172.21.15.111:0/3000400095 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c1613e0 con 0x7fe62c0cc550 2023-11-28T13:01:04.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.796+0000 7fe62a59c700 1 --2- 172.21.15.111:0/3000400095 >> v2:172.21.15.149:3300/0 conn(0x7fe62c0b0d70 0x7fe62c1604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.796+0000 7fe62ad9d700 1 --2- 172.21.15.111:0/3000400095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe62c0cc550 0x7fe62c0cc990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe62c071960 tx=0x7fe62400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7973ab014bf2f7c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.796+0000 7fe62959a700 1 -- 172.21.15.111:0/3000400095 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62400e070 con 0x7fe62c0cc550 2023-11-28T13:01:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.796+0000 7fe62959a700 1 -- 172.21.15.111:0/3000400095 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe624003a20 con 0x7fe62c0cc550 2023-11-28T13:01:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.796+0000 7fe62959a700 1 -- 172.21.15.111:0/3000400095 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624003dd0 con 0x7fe62c0cc550 2023-11-28T13:01:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.797+0000 7fe630976700 1 -- 172.21.15.111:0/3000400095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe62c0cc550 msgr2=0x7fe62c0cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.797+0000 7fe630976700 1 --2- 172.21.15.111:0/3000400095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe62c0cc550 0x7fe62c0cc990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe62c071960 tx=0x7fe62400b330 comp rx=0 tx=0).stop 2023-11-28T13:01:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.797+0000 7fe630976700 1 -- 172.21.15.111:0/3000400095 shutdown_connections 2023-11-28T13:01:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.797+0000 7fe630976700 1 --2- 172.21.15.111:0/3000400095 >> v2:172.21.15.149:3300/0 conn(0x7fe62c0b0d70 0x7fe62c1604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.797+0000 7fe630976700 1 --2- 172.21.15.111:0/3000400095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe62c0cc550 0x7fe62c0cc990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.797+0000 7fe630976700 1 -- 172.21.15.111:0/3000400095 >> 172.21.15.111:0/3000400095 conn(0x7fe62c098ee0 msgr2=0x7fe62c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.797+0000 7fe630976700 1 -- 172.21.15.111:0/3000400095 shutdown_connections 2023-11-28T13:01:04.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.797+0000 7fe630976700 1 -- 172.21.15.111:0/3000400095 wait complete. 2023-11-28T13:01:04.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.798+0000 7fe630976700 1 Processor -- start 2023-11-28T13:01:04.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.798+0000 7fe630976700 1 -- start start 2023-11-28T13:01:04.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe630976700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe62c0b0d70 0x7fe62c0c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe630976700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe62c0b1140 0x7fe62c0c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe630976700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe62c0cc550 0x7fe62c0c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe630976700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c0c61f0 con 0x7fe62c0b0d70 2023-11-28T13:01:04.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe630976700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c0c55e0 con 0x7fe62c0b1140 2023-11-28T13:01:04.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe630976700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c0c58c0 con 0x7fe62c0cc550 2023-11-28T13:01:04.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe62a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe62c0b0d70 0x7fe62c0c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe62a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe62c0b0d70 0x7fe62c0c6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36488/0 (socket says 172.21.15.111:36488) 2023-11-28T13:01:04.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe62a59c700 1 -- 172.21.15.111:0/493206860 learned_addr learned my addr 172.21.15.111:0/493206860 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe629d9b700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe62c0b1140 0x7fe62c0c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe62ad9d700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe62c0cc550 0x7fe62c0c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe62a59c700 1 -- 172.21.15.111:0/493206860 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe62c0cc550 msgr2=0x7fe62c0c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe62a59c700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe62c0cc550 0x7fe62c0c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe62a59c700 1 -- 172.21.15.111:0/493206860 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe62c0b1140 msgr2=0x7fe62c0c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe62a59c700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe62c0b1140 0x7fe62c0c4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe62a59c700 1 -- 172.21.15.111:0/493206860 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe624009b30 con 0x7fe62c0b0d70 2023-11-28T13:01:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe62a59c700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe62c0b0d70 0x7fe62c0c6df0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe62000c7f0 tx=0x7fe62000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe61affd700 1 -- 172.21.15.111:0/493206860 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620005c20 con 0x7fe62c0b0d70 2023-11-28T13:01:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.799+0000 7fe61affd700 1 -- 172.21.15.111:0/493206860 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe620005d80 con 0x7fe62c0b0d70 2023-11-28T13:01:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.800+0000 7fe630976700 1 -- 172.21.15.111:0/493206860 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe62c160e20 con 0x7fe62c0b0d70 2023-11-28T13:01:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.800+0000 7fe61affd700 1 -- 172.21.15.111:0/493206860 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620011820 con 0x7fe62c0b0d70 2023-11-28T13:01:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.800+0000 7fe630976700 1 -- 172.21.15.111:0/493206860 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe62c160b40 con 0x7fe62c0b0d70 2023-11-28T13:01:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.801+0000 7fe61affd700 1 -- 172.21.15.111:0/493206860 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe620020070 con 0x7fe62c0b0d70 2023-11-28T13:01:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.801+0000 7fe61affd700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe60c0682f0 0x7fe60c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.801+0000 7fe61affd700 1 -- 172.21.15.111:0/493206860 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe620085eb0 con 0x7fe62c0b0d70 2023-11-28T13:01:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.801+0000 7fe629d9b700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe60c0682f0 0x7fe60c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.802+0000 7fe629d9b700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe60c0682f0 0x7fe60c06a7a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe62c0cd530 tx=0x7fe61c007ec0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.803+0000 7fe63caffb80 1 Processor -- start 2023-11-28T13:01:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.803+0000 7fe63caffb80 1 -- start start 2023-11-28T13:01:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.804+0000 7fe63caffb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbe860 0x56551ed94a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.804+0000 7fe63caffb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbc180 0x56551edbec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.804+0000 7fe63caffb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbf170 0x56551edda7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.804+0000 7fe63caffb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56551ec7ec10 con 0x56551edbc180 2023-11-28T13:01:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.804+0000 7fe63caffb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x56551eced3a0 con 0x56551edbf170 2023-11-28T13:01:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.804+0000 7fe63caffb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x56551ece68d0 con 0x56551edbe860 2023-11-28T13:01:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.804+0000 7fe629d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbc180 0x56551edbec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.804+0000 7fe629d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbc180 0x56551edbec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36498/0 (socket says 172.21.15.111:36498) 2023-11-28T13:01:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.804+0000 7fe629d9b700 1 -- 172.21.15.111:0/4225221524 learned_addr learned my addr 172.21.15.111:0/4225221524 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.804+0000 7fe62ad9d700 1 --2- 172.21.15.111:0/4225221524 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbf170 0x56551edda7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.804+0000 7fe62a59c700 1 --2- 172.21.15.111:0/4225221524 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbe860 0x56551ed94a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.805+0000 7fe629d9b700 1 -- 172.21.15.111:0/4225221524 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbe860 msgr2=0x56551ed94a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.805+0000 7fe629d9b700 1 --2- 172.21.15.111:0/4225221524 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbe860 0x56551ed94a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.805+0000 7fe629d9b700 1 -- 172.21.15.111:0/4225221524 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbf170 msgr2=0x56551edda7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.805+0000 7fe629d9b700 1 --2- 172.21.15.111:0/4225221524 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbf170 0x56551edda7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.805+0000 7fe629d9b700 1 -- 172.21.15.111:0/4225221524 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56551eb68630 con 0x56551edbc180 2023-11-28T13:01:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.805+0000 7fe62ad9d700 1 --2- 172.21.15.111:0/4225221524 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbf170 0x56551edda7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:04.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.805+0000 7fe62a59c700 1 --2- 172.21.15.111:0/4225221524 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbe860 0x56551ed94a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:04.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.805+0000 7fe629d9b700 1 --2- 172.21.15.111:0/4225221524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbc180 0x56551edbec30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe61c003e30 tx=0x7fe61c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ca372e5ab3262f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.805+0000 7fe60bfff700 1 -- 172.21.15.111:0/4225221524 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c00fd30 con 0x56551edbc180 2023-11-28T13:01:04.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.805+0000 7fe60bfff700 1 -- 172.21.15.111:0/4225221524 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c00fe90 con 0x56551edbc180 2023-11-28T13:01:04.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.805+0000 7fe60bfff700 1 -- 172.21.15.111:0/4225221524 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c004270 con 0x56551edbc180 2023-11-28T13:01:04.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.805+0000 7fe63caffb80 1 -- 172.21.15.111:0/4225221524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbc180 msgr2=0x56551edbec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.805+0000 7fe63caffb80 1 --2- 172.21.15.111:0/4225221524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbc180 0x56551edbec30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe61c003e30 tx=0x7fe61c012000 comp rx=0 tx=0).stop 2023-11-28T13:01:04.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.806+0000 7fe63caffb80 1 -- 172.21.15.111:0/4225221524 shutdown_connections 2023-11-28T13:01:04.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.806+0000 7fe63caffb80 1 --2- 172.21.15.111:0/4225221524 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbf170 0x56551edda7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.806+0000 7fe63caffb80 1 --2- 172.21.15.111:0/4225221524 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbe860 0x56551ed94a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.806+0000 7fe63caffb80 1 --2- 172.21.15.111:0/4225221524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbc180 0x56551edbec30 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.806+0000 7fe63caffb80 1 -- 172.21.15.111:0/4225221524 >> 172.21.15.111:0/4225221524 conn(0x56551ed91db0 msgr2=0x56551ed92740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.806+0000 7fe63caffb80 1 -- 172.21.15.111:0/4225221524 shutdown_connections 2023-11-28T13:01:04.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.806+0000 7fe63caffb80 1 -- 172.21.15.111:0/4225221524 wait complete. 2023-11-28T13:01:04.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.806+0000 7fe63caffb80 1 -- start start 2023-11-28T13:01:04.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.806+0000 7fe63caffb80 1 Processor -- start 2023-11-28T13:01:04.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.807+0000 7fe63caffb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbc180 0x56551edb55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.807+0000 7fe63caffb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbe860 0x56551eddeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.807+0000 7fe63caffb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 0x56551eddcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.807+0000 7fe63caffb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56551eb6a920 con 0x56551edbf170 2023-11-28T13:01:04.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.807+0000 7fe63caffb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x56551edd2a60 con 0x56551edbe860 2023-11-28T13:01:04.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.807+0000 7fe63caffb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x56551ede0860 con 0x56551edbc180 2023-11-28T13:01:04.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.807+0000 7fe63caffb80 10 client.0 _ll_register_callbacks cb 0x56551eb8c290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:01:04.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.807+0000 7fe62ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 0x56551eddcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.807+0000 7fe62ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 0x56551eddcb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36506/0 (socket says 172.21.15.111:36506) 2023-11-28T13:01:04.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.807+0000 7fe62ad9d700 1 -- 172.21.15.111:0/1927847961 learned_addr learned my addr 172.21.15.111:0/1927847961 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe62a59c700 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbc180 0x56551edb55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe629d9b700 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbe860 0x56551eddeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe62ad9d700 1 -- 172.21.15.111:0/1927847961 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbc180 msgr2=0x56551edb55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe62ad9d700 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbc180 0x56551edb55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe62ad9d700 1 -- 172.21.15.111:0/1927847961 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbe860 msgr2=0x56551eddeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe62ad9d700 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbe860 0x56551eddeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe62ad9d700 1 -- 172.21.15.111:0/1927847961 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe61c003b10 con 0x56551edbf170 2023-11-28T13:01:04.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe629d9b700 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbe860 0x56551eddeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:04.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe62ad9d700 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 0x56551eddcb80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe624009fd0 tx=0x7fe624004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef23a27b5c75f1a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe6097fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:01:04.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe6097fa700 1 -- 172.21.15.111:0/1927847961 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62400be10 con 0x56551edbf170 2023-11-28T13:01:04.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe6097fa700 1 -- 172.21.15.111:0/1927847961 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe624004240 con 0x56551edbf170 2023-11-28T13:01:04.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe63caffb80 1 -- 172.21.15.111:0/1927847961 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56551ecb4190 con 0x56551edbf170 2023-11-28T13:01:04.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe6097fa700 1 -- 172.21.15.111:0/1927847961 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe624004400 con 0x56551edbf170 2023-11-28T13:01:04.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe63caffb80 10 client.6024 fetch_fsmap learned FSMap version 25 2023-11-28T13:01:04.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe63caffb80 1 -- 172.21.15.111:0/1927847961 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x56551eb14de0 con 0x56551edbf170 2023-11-28T13:01:04.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe6097fa700 1 -- 172.21.15.111:0/1927847961 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56551ecb4190 con 0x56551edbf170 2023-11-28T13:01:04.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe6097fa700 1 -- 172.21.15.111:0/1927847961 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 25) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fe624004560 con 0x56551edbf170 2023-11-28T13:01:04.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe63caffb80 10 client.6024 fetch_fsmap finished waiting for FSMap version 25 2023-11-28T13:01:04.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe63caffb80 10 client.6024 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2023-11-28T13:01:04.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe63caffb80 20 client.6024 populate_metadata read hostname 'smithi111' 2023-11-28T13:01:04.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe63caffb80 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x56551edd1ad0 0x56551edd1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe63caffb80 4 client.6024 mds_command: new command op to 5814 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-28T13:01:04.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe63caffb80 1 -- 172.21.15.111:0/1927847961 --> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56551eb6af60 con 0x56551edd1ad0 2023-11-28T13:01:04.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.808+0000 7fe62a59c700 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x56551edd1ad0 0x56551edd1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe6097fa700 10 client.6024 ms_handle_connect on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:04.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe62a59c700 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x56551edd1ad0 0x56551edd1ed0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da93073feb394931 server_cookie=243881f662d71b8f in_seq=0 out_seq=0 2023-11-28T13:01:04.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe6097fa700 1 -- 172.21.15.111:0/1927847961 <== mds.0 v2:172.21.15.111:6835/4146274267 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56551eb6af60 con 0x56551edd1ad0 2023-11-28T13:01:04.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe6097fa700 10 client.6024 handle_command_reply: tid=0 2023-11-28T13:01:04.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe63caffb80 1 client.6024 shutdown 2023-11-28T13:01:04.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe63caffb80 1 -- 172.21.15.111:0/1927847961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 msgr2=0x56551eddcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe63caffb80 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 0x56551eddcb80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe624009fd0 tx=0x7fe624004b40 comp rx=0 tx=0).stop 2023-11-28T13:01:04.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe63caffb80 1 -- 172.21.15.111:0/1927847961 shutdown_connections 2023-11-28T13:01:04.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe63caffb80 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x56551edd1ad0 0x56551edd1ed0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe63caffb80 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 0x56551eddcb80 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe63caffb80 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbc180 0x56551edb55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe63caffb80 1 --2- 172.21.15.111:0/1927847961 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbe860 0x56551eddeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe6097fa700 0 client.6024 ms_handle_reset on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:04.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.809+0000 7fe63caffb80 1 -- 172.21.15.111:0/1927847961 >> 172.21.15.111:0/1927847961 conn(0x56551ed91db0 msgr2=0x56551edbded0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.810+0000 7fe63caffb80 1 -- 172.21.15.111:0/1927847961 shutdown_connections 2023-11-28T13:01:04.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.810+0000 7fe63caffb80 1 -- 172.21.15.111:0/1927847961 wait complete. 2023-11-28T13:01:04.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.810+0000 7fe63caffb80 20 client.6024 trim_cache size 0 max 16384 2023-11-28T13:01:04.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.826+0000 7fe63caffb80 1 Processor -- start 2023-11-28T13:01:04.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.826+0000 7fe63caffb80 1 -- start start 2023-11-28T13:01:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe63caffb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbc180 0x56551edb55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe63caffb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbe860 0x56551ee34d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe63caffb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbf170 0x56551ee35260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe63caffb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56551ed96240 con 0x56551edbe860 2023-11-28T13:01:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe63caffb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x56551ed91b40 con 0x56551edbc180 2023-11-28T13:01:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe63caffb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x56551ed89a40 con 0x56551edbf170 2023-11-28T13:01:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe629d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbe860 0x56551ee34d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe629d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbe860 0x56551ee34d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36512/0 (socket says 172.21.15.111:36512) 2023-11-28T13:01:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe629d9b700 1 -- 172.21.15.111:0/3108755070 learned_addr learned my addr 172.21.15.111:0/3108755070 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe629d9b700 1 -- 172.21.15.111:0/3108755070 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbf170 msgr2=0x56551ee35260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe629d9b700 1 --2- 172.21.15.111:0/3108755070 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbf170 0x56551ee35260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe629d9b700 1 -- 172.21.15.111:0/3108755070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbc180 msgr2=0x56551edb55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe629d9b700 1 --2- 172.21.15.111:0/3108755070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbc180 0x56551edb55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe629d9b700 1 -- 172.21.15.111:0/3108755070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56551ecad190 con 0x56551edbe860 2023-11-28T13:01:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe629d9b700 1 --2- 172.21.15.111:0/3108755070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbe860 0x56551ee34d20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe61c008010 tx=0x7fe61c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d64100dd3247fe8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe618ff9700 1 -- 172.21.15.111:0/3108755070 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c014650 con 0x56551edbe860 2023-11-28T13:01:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe618ff9700 1 -- 172.21.15.111:0/3108755070 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c0147b0 con 0x56551edbe860 2023-11-28T13:01:04.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.827+0000 7fe618ff9700 1 -- 172.21.15.111:0/3108755070 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c014b20 con 0x56551edbe860 2023-11-28T13:01:04.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.828+0000 7fe63caffb80 1 -- 172.21.15.111:0/3108755070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbe860 msgr2=0x56551ee34d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.828+0000 7fe63caffb80 1 --2- 172.21.15.111:0/3108755070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbe860 0x56551ee34d20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe61c008010 tx=0x7fe61c014000 comp rx=0 tx=0).stop 2023-11-28T13:01:04.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.828+0000 7fe63caffb80 1 -- 172.21.15.111:0/3108755070 shutdown_connections 2023-11-28T13:01:04.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.828+0000 7fe63caffb80 1 --2- 172.21.15.111:0/3108755070 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbf170 0x56551ee35260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.828+0000 7fe63caffb80 1 --2- 172.21.15.111:0/3108755070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbc180 0x56551edb55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.828+0000 7fe63caffb80 1 --2- 172.21.15.111:0/3108755070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbe860 0x56551ee34d20 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.828+0000 7fe63caffb80 1 -- 172.21.15.111:0/3108755070 >> 172.21.15.111:0/3108755070 conn(0x56551ed91db0 msgr2=0x56551ed92840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.828+0000 7fe63caffb80 1 -- 172.21.15.111:0/3108755070 shutdown_connections 2023-11-28T13:01:04.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.828+0000 7fe63caffb80 1 -- 172.21.15.111:0/3108755070 wait complete. 2023-11-28T13:01:04.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.828+0000 7fe63caffb80 1 -- start start 2023-11-28T13:01:04.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.828+0000 7fe63caffb80 1 Processor -- start 2023-11-28T13:01:04.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.829+0000 7fe63caffb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbc180 0x56551ee343b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.829+0000 7fe63caffb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbe860 0x56551ee37c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.829+0000 7fe63caffb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 0x56551ee38180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.829+0000 7fe63caffb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56551ee36980 con 0x56551edbf170 2023-11-28T13:01:04.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.829+0000 7fe63caffb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x56551edf23b0 con 0x56551edbc180 2023-11-28T13:01:04.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.829+0000 7fe63caffb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x56551ed881c0 con 0x56551edbe860 2023-11-28T13:01:04.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.829+0000 7fe63caffb80 10 client.0 _ll_register_callbacks cb 0x56551eb8c290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:01:04.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe62ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 0x56551ee38180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe62ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 0x56551ee38180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36520/0 (socket says 172.21.15.111:36520) 2023-11-28T13:01:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe62ad9d700 1 -- 172.21.15.111:0/2679953297 learned_addr learned my addr 172.21.15.111:0/2679953297 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe62a59c700 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbc180 0x56551ee343b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe629d9b700 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbe860 0x56551ee37c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe62ad9d700 1 -- 172.21.15.111:0/2679953297 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbe860 msgr2=0x56551ee37c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe62ad9d700 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbe860 0x56551ee37c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe62ad9d700 1 -- 172.21.15.111:0/2679953297 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbc180 msgr2=0x56551ee343b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe62ad9d700 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbc180 0x56551ee343b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe62ad9d700 1 -- 172.21.15.111:0/2679953297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56551eb68630 con 0x56551edbf170 2023-11-28T13:01:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe62a59c700 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbc180 0x56551ee343b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe62ad9d700 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 0x56551ee38180 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fe624003fb0 tx=0x7fe624004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6b738e9d3411c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe6097fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:01:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe6097fa700 1 -- 172.21.15.111:0/2679953297 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624034070 con 0x56551edbf170 2023-11-28T13:01:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe6097fa700 1 -- 172.21.15.111:0/2679953297 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe624004610 con 0x56551edbf170 2023-11-28T13:01:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe63caffb80 1 -- 172.21.15.111:0/2679953297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56551eca41b0 con 0x56551edbf170 2023-11-28T13:01:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe6097fa700 1 -- 172.21.15.111:0/2679953297 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe62400e070 con 0x56551edbf170 2023-11-28T13:01:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe63caffb80 10 client.6030 fetch_fsmap learned FSMap version 25 2023-11-28T13:01:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe63caffb80 1 -- 172.21.15.111:0/2679953297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x56551eb3b250 con 0x56551edbf170 2023-11-28T13:01:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe6097fa700 1 -- 172.21.15.111:0/2679953297 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56551eca41b0 con 0x56551edbf170 2023-11-28T13:01:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe6097fa700 1 -- 172.21.15.111:0/2679953297 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 25) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fe624037020 con 0x56551edbf170 2023-11-28T13:01:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe63caffb80 10 client.6030 fetch_fsmap finished waiting for FSMap version 25 2023-11-28T13:01:04.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe63caffb80 10 client.6030 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2023-11-28T13:01:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.830+0000 7fe63caffb80 20 client.6030 populate_metadata read hostname 'smithi111' 2023-11-28T13:01:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x56551edb31a0 0x56551edd2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 4 client.6030 mds_command: new command op to 5814 tid=0[{"prefix": "scrub status"}] 2023-11-28T13:01:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 1 -- 172.21.15.111:0/2679953297 --> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56551ec8e1c0 con 0x56551edb31a0 2023-11-28T13:01:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe62a59c700 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x56551edb31a0 0x56551edd2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe6097fa700 10 client.6030 ms_handle_connect on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe62a59c700 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x56551edb31a0 0x56551edd2ba0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46fe704c3904a963 server_cookie=192ca707a061bbca in_seq=0 out_seq=0 2023-11-28T13:01:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe6097fa700 1 -- 172.21.15.111:0/2679953297 <== mds.0 v2:172.21.15.111:6835/4146274267 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0x56551ec8e1c0 con 0x56551edb31a0 2023-11-28T13:01:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe6097fa700 10 client.6030 handle_command_reply: tid=0 2023-11-28T13:01:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 1 client.6030 shutdown 2023-11-28T13:01:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 1 -- 172.21.15.111:0/2679953297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 msgr2=0x56551ee38180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 0x56551ee38180 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fe624003fb0 tx=0x7fe624004ad0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 1 -- 172.21.15.111:0/2679953297 shutdown_connections 2023-11-28T13:01:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x56551edb31a0 0x56551edd2ba0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56551edbf170 0x56551ee38180 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56551edbc180 0x56551ee343b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 1 --2- 172.21.15.111:0/2679953297 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56551edbe860 0x56551ee37c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe6097fa700 0 client.6030 ms_handle_reset on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 1 -- 172.21.15.111:0/2679953297 >> 172.21.15.111:0/2679953297 conn(0x56551ed91db0 msgr2=0x56551edb55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 1 -- 172.21.15.111:0/2679953297 shutdown_connections 2023-11-28T13:01:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.831+0000 7fe63caffb80 1 -- 172.21.15.111:0/2679953297 wait complete. 2023-11-28T13:01:04.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.832+0000 7fe63caffb80 20 client.6030 trim_cache size 0 max 16384 2023-11-28T13:01:04.840 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:01:04.841 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-28T13:01:04.841 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-11-28T13:01:04.841 INFO:teuthology.orchestra.run.smithi111.stdout: "88e043de-10f3-46f3-b9d8-60f2596d62dd": { 2023-11-28T13:01:04.841 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-11-28T13:01:04.841 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "88e043de-10f3-46f3-b9d8-60f2596d62dd", 2023-11-28T13:01:04.841 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,repair" 2023-11-28T13:01:04.841 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:04.841 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:04.841 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:01:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.832+0000 7fe630976700 1 -- 172.21.15.111:0/493206860 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe60c0682f0 msgr2=0x7fe60c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.832+0000 7fe630976700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe60c0682f0 0x7fe60c06a7a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe62c0cd530 tx=0x7fe61c007ec0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.832+0000 7fe630976700 1 -- 172.21.15.111:0/493206860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe62c0b0d70 msgr2=0x7fe62c0c6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.832+0000 7fe630976700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe62c0b0d70 0x7fe62c0c6df0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe62000c7f0 tx=0x7fe62000cb00 comp rx=0 tx=0).stop 2023-11-28T13:01:04.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.832+0000 7fe630976700 1 -- 172.21.15.111:0/493206860 shutdown_connections 2023-11-28T13:01:04.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.832+0000 7fe630976700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe62c0cc550 0x7fe62c0c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.832+0000 7fe630976700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe62c0b0d70 0x7fe62c0c6df0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.832+0000 7fe630976700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe60c0682f0 0x7fe60c06a7a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.832+0000 7fe630976700 1 --2- 172.21.15.111:0/493206860 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe62c0b1140 0x7fe62c0c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:04.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.832+0000 7fe630976700 1 -- 172.21.15.111:0/493206860 >> 172.21.15.111:0/493206860 conn(0x7fe62c098ee0 msgr2=0x7fe62c072cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:04.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.832+0000 7fe630976700 1 -- 172.21.15.111:0/493206860 shutdown_connections 2023-11-28T13:01:04.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:04.832+0000 7fe630976700 1 -- 172.21.15.111:0/493206860 wait complete. 2023-11-28T13:01:04.845 INFO:tasks.cephfs.filesystem:scrub status for tag:88e043de-10f3-46f3-b9d8-60f2596d62dd - {'path': '/', 'tag': '88e043de-10f3-46f3-b9d8-60f2596d62dd', 'options': 'recursive,repair'} 2023-11-28T13:01:05.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:05.622+0000 7f0274ff9700 1 -- 172.21.15.111:0/856373349 <== mon.0 v2:172.21.15.111:3300/0 7 ==== log(6 entries from seq 1 at 2023-11-28T13:01:04.661360+0000) v1 ==== 1078+0+0 (secure 0 0 0) 0x7f0270086ad0 con 0x7f027815d7a0 2023-11-28T13:01:05.625 INFO:teuthology.orchestra.run.smithi111.stdout: cluster: 2023-11-28T13:01:05.625 INFO:teuthology.orchestra.run.smithi111.stdout: id: a1b2d00c-aeec-4b5d-93e3-41820c4189ba 2023-11-28T13:01:05.625 INFO:teuthology.orchestra.run.smithi111.stdout: health: HEALTH_OK 2023-11-28T13:01:05.625 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:05.625 INFO:teuthology.orchestra.run.smithi111.stdout: services: 2023-11-28T13:01:05.625 INFO:teuthology.orchestra.run.smithi111.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2023-11-28T13:01:05.625 INFO:teuthology.orchestra.run.smithi111.stdout: mgr: y(active, since 2m), standbys: x 2023-11-28T13:01:05.625 INFO:teuthology.orchestra.run.smithi111.stdout: mds: 1/1 daemons up, 3 standby 2023-11-28T13:01:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2023-11-28T13:01:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: data: 2023-11-28T13:01:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: volumes: 1/1 healthy 2023-11-28T13:01:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: pools: 4 pools, 105 pgs 2023-11-28T13:01:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: objects: 24 objects, 579 KiB 2023-11-28T13:01:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: usage: 74 MiB used, 720 GiB / 720 GiB avail 2023-11-28T13:01:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: pgs: 105 active+clean 2023-11-28T13:01:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: io: 2023-11-28T13:01:05.627 INFO:teuthology.orchestra.run.smithi111.stdout: client: 2.2 KiB/s wr, 0 op/s rd, 6 op/s wr 2023-11-28T13:01:05.627 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:05.627 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:05.627 INFO:teuthology.orchestra.run.smithi111.stdout:2023-11-28T13:01:04.661360+0000 mds.c [INF] scrub queued for path: / 2023-11-28T13:01:05.627 INFO:teuthology.orchestra.run.smithi111.stdout:2023-11-28T13:01:04.661364+0000 mds.c [INF] scrub summary: idle+waiting paths [/] 2023-11-28T13:01:05.627 INFO:teuthology.orchestra.run.smithi111.stdout:2023-11-28T13:01:04.661368+0000 mds.c [INF] scrub summary: active paths [/] 2023-11-28T13:01:05.627 INFO:teuthology.orchestra.run.smithi111.stdout:2023-11-28T13:01:04.662172+0000 mds.c [WRN] bad backtrace on inode 0x10000000001(/parent_a/alpha), rewriting it 2023-11-28T13:01:05.627 INFO:teuthology.orchestra.run.smithi111.stdout:2023-11-28T13:01:04.662183+0000 mds.c [INF] Scrub repaired inode 0x10000000001 (/parent_a/alpha) 2023-11-28T13:01:05.627 INFO:teuthology.orchestra.run.smithi111.stdout:2023-11-28T13:01:04.662628+0000 mds.c [INF] scrub summary: idle+waiting paths [/] 2023-11-28T13:01:07.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:07.625+0000 7f0274ff9700 1 -- 172.21.15.111:0/856373349 <== mon.0 v2:172.21.15.111:3300/0 8 ==== log(1 entries from seq 7 at 2023-11-28T13:01:05.818612+0000) v1 ==== 189+0+0 (secure 0 0 0) 0x7f027005eae0 con 0x7f027815d7a0 2023-11-28T13:01:07.626 INFO:teuthology.orchestra.run.smithi111.stdout:2023-11-28T13:01:05.818612+0000 mds.c [INF] scrub summary: idle 2023-11-28T13:01:28.815 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:01:28.818 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:01:34.846 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub status 2023-11-28T13:01:34.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.979+0000 7f2f78764700 1 Processor -- start 2023-11-28T13:01:34.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.979+0000 7f2f78764700 1 -- start start 2023-11-28T13:01:34.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.980+0000 7f2f78764700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2f7006b380 0x7f2f7006b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:34.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.980+0000 7f2f78764700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f2f700b5790 0x7f2f700b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:34.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.980+0000 7f2f78764700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f700b5300 con 0x7f2f700b20f0 2023-11-28T13:01:34.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.980+0000 7f2f78764700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7006bd30 con 0x7f2f7006b380 2023-11-28T13:01:34.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.980+0000 7f2f78764700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f700cfbb0 con 0x7f2f700b5790 2023-11-28T13:01:34.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.980+0000 7f2f75cff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2f7006b380 0x7f2f7006b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:34.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.980+0000 7f2f76500700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2f700b20f0 0x7f2f700b3b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37302/0 (socket says 172.21.15.111:37302) 2023-11-28T13:01:34.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.980+0000 7f2f76500700 1 -- 172.21.15.111:0/1987321867 learned_addr learned my addr 172.21.15.111:0/1987321867 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:34.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.980+0000 7f2f76d01700 1 --2- 172.21.15.111:0/1987321867 >> v2:172.21.15.149:3300/0 conn(0x7f2f700b5790 0x7f2f700b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:34.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.980+0000 7f2f75cff700 1 -- 172.21.15.111:0/1987321867 >> v2:172.21.15.149:3300/0 conn(0x7f2f700b5790 msgr2=0x7f2f700b5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:34.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.980+0000 7f2f75cff700 1 --2- 172.21.15.111:0/1987321867 >> v2:172.21.15.149:3300/0 conn(0x7f2f700b5790 0x7f2f700b5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:34.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f75cff700 1 -- 172.21.15.111:0/1987321867 >> v1:172.21.15.111:6789/0 conn(0x7f2f700b20f0 legacy=0x7f2f700b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:34.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f75cff700 1 -- 172.21.15.111:0/1987321867 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f700b46f0 con 0x7f2f7006b380 2023-11-28T13:01:34.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f754fe700 1 -- 172.21.15.111:0/1987321867 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290972323 0 0) 0x7f2f700b5300 con 0x7f2f700b20f0 2023-11-28T13:01:34.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f75cff700 1 --2- 172.21.15.111:0/1987321867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f7006b380 0x7f2f7006b7f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f2f7006d540 tx=0x7f2f6c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceba0f14f786937 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:34.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f754fe700 1 -- 172.21.15.111:0/1987321867 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c011830 con 0x7f2f7006b380 2023-11-28T13:01:34.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f754fe700 1 -- 172.21.15.111:0/1987321867 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6c011990 con 0x7f2f7006b380 2023-11-28T13:01:34.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f754fe700 1 -- 172.21.15.111:0/1987321867 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c011d20 con 0x7f2f7006b380 2023-11-28T13:01:34.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f78764700 1 -- 172.21.15.111:0/1987321867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f7006b380 msgr2=0x7f2f7006b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:34.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f78764700 1 --2- 172.21.15.111:0/1987321867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f7006b380 0x7f2f7006b7f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f2f7006d540 tx=0x7f2f6c009b20 comp rx=0 tx=0).stop 2023-11-28T13:01:34.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f78764700 1 -- 172.21.15.111:0/1987321867 shutdown_connections 2023-11-28T13:01:34.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f78764700 1 --2- 172.21.15.111:0/1987321867 >> v2:172.21.15.149:3300/0 conn(0x7f2f700b5790 0x7f2f700b5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:34.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f78764700 1 --2- 172.21.15.111:0/1987321867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f7006b380 0x7f2f7006b7f0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:34.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f78764700 1 -- 172.21.15.111:0/1987321867 >> 172.21.15.111:0/1987321867 conn(0x7f2f7009a040 msgr2=0x7f2f700d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:34.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.981+0000 7f2f78764700 1 -- 172.21.15.111:0/1987321867 shutdown_connections 2023-11-28T13:01:34.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.982+0000 7f2f78764700 1 -- 172.21.15.111:0/1987321867 wait complete. 2023-11-28T13:01:34.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.983+0000 7f2f78764700 1 Processor -- start 2023-11-28T13:01:34.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.983+0000 7f2f78764700 1 -- start start 2023-11-28T13:01:34.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.983+0000 7f2f78764700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f7006b380 0x7f2f701531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:34.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.983+0000 7f2f78764700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f700b20f0 0x7f2f7015b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:34.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.983+0000 7f2f78764700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f700b5790 0x7f2f701511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:34.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f78764700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f701f1ed0 con 0x7f2f700b20f0 2023-11-28T13:01:34.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f78764700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7015c3e0 con 0x7f2f700b5790 2023-11-28T13:01:34.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f78764700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f701f2190 con 0x7f2f7006b380 2023-11-28T13:01:34.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f75cff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f700b20f0 0x7f2f7015b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:34.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f75cff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f700b20f0 0x7f2f7015b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36532/0 (socket says 172.21.15.111:36532) 2023-11-28T13:01:34.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f75cff700 1 -- 172.21.15.111:0/233129393 learned_addr learned my addr 172.21.15.111:0/233129393 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:34.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f76d01700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f700b5790 0x7f2f701511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:34.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f76500700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f7006b380 0x7f2f701531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f75cff700 1 -- 172.21.15.111:0/233129393 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f7006b380 msgr2=0x7f2f701531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f75cff700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f7006b380 0x7f2f701531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f75cff700 1 -- 172.21.15.111:0/233129393 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f700b5790 msgr2=0x7f2f701511f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f75cff700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f700b5790 0x7f2f701511f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f75cff700 1 -- 172.21.15.111:0/233129393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c0094a0 con 0x7f2f700b20f0 2023-11-28T13:01:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f75cff700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f700b20f0 0x7f2f7015b970 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009b60 tx=0x7f2f6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f66ffd700 1 -- 172.21.15.111:0/233129393 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c0038a0 con 0x7f2f700b20f0 2023-11-28T13:01:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f66ffd700 1 -- 172.21.15.111:0/233129393 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6c01e070 con 0x7f2f700b20f0 2023-11-28T13:01:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f78764700 1 -- 172.21.15.111:0/233129393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f700b4cb0 con 0x7f2f700b20f0 2023-11-28T13:01:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f66ffd700 1 -- 172.21.15.111:0/233129393 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c003c00 con 0x7f2f700b20f0 2023-11-28T13:01:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.984+0000 7f2f78764700 1 -- 172.21.15.111:0/233129393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f700b49d0 con 0x7f2f700b20f0 2023-11-28T13:01:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.986+0000 7f2f66ffd700 1 -- 172.21.15.111:0/233129393 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f6c00f050 con 0x7f2f700b20f0 2023-11-28T13:01:34.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.986+0000 7f2f66ffd700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f540682f0 0x7f2f5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:34.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.986+0000 7f2f66ffd700 1 -- 172.21.15.111:0/233129393 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2f6c086f80 con 0x7f2f700b20f0 2023-11-28T13:01:34.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.986+0000 7f2f76500700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f540682f0 0x7f2f5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:34.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.986+0000 7f2f76500700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f540682f0 0x7f2f5406a7a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2f60000f60 tx=0x7f2f60007be0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:34.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.988+0000 7f2f848edb80 1 Processor -- start 2023-11-28T13:01:34.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.988+0000 7f2f848edb80 1 -- start start 2023-11-28T13:01:34.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f848edb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff41a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:34.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f848edb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 0x55a06ff6bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:34.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f848edb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff6c170 0x55a06ff877c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:34.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f848edb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06fe2bc10 con 0x55a06ff69180 2023-11-28T13:01:34.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f848edb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06fe9a3a0 con 0x55a06ff6c170 2023-11-28T13:01:34.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f848edb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a06fe938d0 con 0x55a06ff6b860 2023-11-28T13:01:34.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f75cff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 0x55a06ff6bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:34.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f75cff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 0x55a06ff6bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36540/0 (socket says 172.21.15.111:36540) 2023-11-28T13:01:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f75cff700 1 -- 172.21.15.111:0/2532768617 learned_addr learned my addr 172.21.15.111:0/2532768617 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f76d01700 1 --2- 172.21.15.111:0/2532768617 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff6c170 0x55a06ff877c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f76500700 1 --2- 172.21.15.111:0/2532768617 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff41a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f75cff700 1 -- 172.21.15.111:0/2532768617 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 msgr2=0x55a06ff41a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f75cff700 1 --2- 172.21.15.111:0/2532768617 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff41a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f75cff700 1 -- 172.21.15.111:0/2532768617 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff6c170 msgr2=0x55a06ff877c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f75cff700 1 --2- 172.21.15.111:0/2532768617 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff6c170 0x55a06ff877c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f75cff700 1 -- 172.21.15.111:0/2532768617 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a06fd15630 con 0x55a06ff69180 2023-11-28T13:01:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f76d01700 1 --2- 172.21.15.111:0/2532768617 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff6c170 0x55a06ff877c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.989+0000 7f2f75cff700 1 --2- 172.21.15.111:0/2532768617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 0x55a06ff6bc30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c085410 tx=0x7f2f6c056bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d38eadef2cb09a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.990+0000 7f2f53fff700 1 -- 172.21.15.111:0/2532768617 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c055610 con 0x55a06ff69180 2023-11-28T13:01:34.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.990+0000 7f2f53fff700 1 -- 172.21.15.111:0/2532768617 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6c003d60 con 0x55a06ff69180 2023-11-28T13:01:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.990+0000 7f2f53fff700 1 -- 172.21.15.111:0/2532768617 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c056960 con 0x55a06ff69180 2023-11-28T13:01:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.990+0000 7f2f848edb80 1 -- 172.21.15.111:0/2532768617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 msgr2=0x55a06ff6bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.990+0000 7f2f848edb80 1 --2- 172.21.15.111:0/2532768617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 0x55a06ff6bc30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c085410 tx=0x7f2f6c056bd0 comp rx=0 tx=0).stop 2023-11-28T13:01:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.990+0000 7f2f848edb80 1 -- 172.21.15.111:0/2532768617 shutdown_connections 2023-11-28T13:01:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.990+0000 7f2f848edb80 1 --2- 172.21.15.111:0/2532768617 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff6c170 0x55a06ff877c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.990+0000 7f2f848edb80 1 --2- 172.21.15.111:0/2532768617 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff41a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.990+0000 7f2f848edb80 1 --2- 172.21.15.111:0/2532768617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 0x55a06ff6bc30 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.990+0000 7f2f848edb80 1 -- 172.21.15.111:0/2532768617 >> 172.21.15.111:0/2532768617 conn(0x55a06ff3edb0 msgr2=0x55a06ff3f740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.990+0000 7f2f848edb80 1 -- 172.21.15.111:0/2532768617 shutdown_connections 2023-11-28T13:01:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.990+0000 7f2f848edb80 1 -- 172.21.15.111:0/2532768617 wait complete. 2023-11-28T13:01:34.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.991+0000 7f2f848edb80 1 -- start start 2023-11-28T13:01:34.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.991+0000 7f2f848edb80 1 Processor -- start 2023-11-28T13:01:34.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f848edb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff69180 0x55a06ff80110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:34.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f848edb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff5ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:34.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f848edb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 0x55a06ff6a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:34.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f848edb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06fd17920 con 0x55a06ff6c170 2023-11-28T13:01:35.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f848edb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06ff69630 con 0x55a06ff69180 2023-11-28T13:01:35.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f848edb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a06ff6d200 con 0x55a06ff6b860 2023-11-28T13:01:35.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f848edb80 10 client.0 _ll_register_callbacks cb 0x55a06fd39290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:01:35.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f76d01700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 0x55a06ff6a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f76d01700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 0x55a06ff6a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36548/0 (socket says 172.21.15.111:36548) 2023-11-28T13:01:35.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f76d01700 1 -- 172.21.15.111:0/760122969 learned_addr learned my addr 172.21.15.111:0/760122969 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:35.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f75cff700 1 --2- 172.21.15.111:0/760122969 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff5ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f76500700 1 --2- 172.21.15.111:0/760122969 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff69180 0x55a06ff80110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f76d01700 1 -- 172.21.15.111:0/760122969 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 msgr2=0x55a06ff5ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:35.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f76d01700 1 --2- 172.21.15.111:0/760122969 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff5ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f76d01700 1 -- 172.21.15.111:0/760122969 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff69180 msgr2=0x55a06ff80110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:35.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f76d01700 1 --2- 172.21.15.111:0/760122969 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff69180 0x55a06ff80110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f76d01700 1 -- 172.21.15.111:0/760122969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f700b49d0 con 0x55a06ff6c170 2023-11-28T13:01:35.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f76d01700 1 --2- 172.21.15.111:0/760122969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 0x55a06ff6a600 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2f68007a00 tx=0x7f2f68007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a24952806f154d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:35.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.992+0000 7f2f517fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:01:35.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f517fa700 1 -- 172.21.15.111:0/760122969 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68007e40 con 0x55a06ff6c170 2023-11-28T13:01:35.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f517fa700 1 -- 172.21.15.111:0/760122969 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2f680118d0 con 0x55a06ff6c170 2023-11-28T13:01:35.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f848edb80 1 -- 172.21.15.111:0/760122969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a06fe61190 con 0x55a06ff6c170 2023-11-28T13:01:35.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f517fa700 1 -- 172.21.15.111:0/760122969 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f68011a90 con 0x55a06ff6c170 2023-11-28T13:01:35.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f848edb80 10 client.6045 fetch_fsmap learned FSMap version 25 2023-11-28T13:01:35.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f848edb80 1 -- 172.21.15.111:0/760122969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55a06fcc1de0 con 0x55a06ff6c170 2023-11-28T13:01:35.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f517fa700 1 -- 172.21.15.111:0/760122969 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a06fe61190 con 0x55a06ff6c170 2023-11-28T13:01:35.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f517fa700 1 -- 172.21.15.111:0/760122969 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 25) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f2f68014070 con 0x55a06ff6c170 2023-11-28T13:01:35.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f848edb80 10 client.6045 fetch_fsmap finished waiting for FSMap version 25 2023-11-28T13:01:35.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f848edb80 10 client.6045 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2023-11-28T13:01:35.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f848edb80 20 client.6045 populate_metadata read hostname 'smithi111' 2023-11-28T13:01:35.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f848edb80 1 --2- 172.21.15.111:0/760122969 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55a06ff5da30 0x55a06ff5de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f848edb80 4 client.6045 mds_command: new command op to 5814 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-28T13:01:35.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f848edb80 1 -- 172.21.15.111:0/760122969 --> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a06fd17f60 con 0x55a06ff5da30 2023-11-28T13:01:35.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.993+0000 7f2f76500700 1 --2- 172.21.15.111:0/760122969 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55a06ff5da30 0x55a06ff5de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f517fa700 10 client.6045 ms_handle_connect on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:35.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f76500700 1 --2- 172.21.15.111:0/760122969 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55a06ff5da30 0x55a06ff5de00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e416e90fa7ff2aa server_cookie=3c1de5bb2d341baa in_seq=0 out_seq=0 2023-11-28T13:01:35.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f517fa700 1 -- 172.21.15.111:0/760122969 <== mds.0 v2:172.21.15.111:6835/4146274267 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a06fd17f60 con 0x55a06ff5da30 2023-11-28T13:01:35.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f517fa700 10 client.6045 handle_command_reply: tid=0 2023-11-28T13:01:35.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f848edb80 1 client.6045 shutdown 2023-11-28T13:01:35.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f848edb80 1 -- 172.21.15.111:0/760122969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 msgr2=0x55a06ff6a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f848edb80 1 --2- 172.21.15.111:0/760122969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 0x55a06ff6a600 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2f68007a00 tx=0x7f2f68007d10 comp rx=0 tx=0).stop 2023-11-28T13:01:35.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f848edb80 1 -- 172.21.15.111:0/760122969 shutdown_connections 2023-11-28T13:01:35.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f848edb80 1 --2- 172.21.15.111:0/760122969 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55a06ff5da30 0x55a06ff5de00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f848edb80 1 --2- 172.21.15.111:0/760122969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 0x55a06ff6a600 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f848edb80 1 --2- 172.21.15.111:0/760122969 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff69180 0x55a06ff80110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f848edb80 1 --2- 172.21.15.111:0/760122969 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff5ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f517fa700 0 client.6045 ms_handle_reset on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:35.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f848edb80 1 -- 172.21.15.111:0/760122969 >> 172.21.15.111:0/760122969 conn(0x55a06ff3edb0 msgr2=0x55a06ff419e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:35.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f848edb80 1 -- 172.21.15.111:0/760122969 shutdown_connections 2023-11-28T13:01:35.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f848edb80 1 -- 172.21.15.111:0/760122969 wait complete. 2023-11-28T13:01:35.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:34.994+0000 7f2f848edb80 20 client.6045 trim_cache size 0 max 16384 2023-11-28T13:01:35.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.011+0000 7f2f848edb80 1 Processor -- start 2023-11-28T13:01:35.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.011+0000 7f2f848edb80 1 -- start start 2023-11-28T13:01:35.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f848edb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 0x55a06ff5c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f848edb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff6e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f848edb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff6c170 0x55a06ff6e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f848edb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06ff3b980 con 0x55a06ff69180 2023-11-28T13:01:35.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f848edb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06ff59fa0 con 0x55a06ff6c170 2023-11-28T13:01:35.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f848edb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a06ffa89f0 con 0x55a06ff6b860 2023-11-28T13:01:35.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f76500700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 0x55a06ff5c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f76500700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 0x55a06ff5c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36552/0 (socket says 172.21.15.111:36552) 2023-11-28T13:01:35.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f76500700 1 -- 172.21.15.111:0/2836000722 learned_addr learned my addr 172.21.15.111:0/2836000722 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:35.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f76500700 1 -- 172.21.15.111:0/2836000722 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 msgr2=0x55a06ff6e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:35.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f76500700 1 --2- 172.21.15.111:0/2836000722 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff6e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f76500700 1 -- 172.21.15.111:0/2836000722 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff6c170 msgr2=0x55a06ff6e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:35.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f76500700 1 --2- 172.21.15.111:0/2836000722 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff6c170 0x55a06ff6e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f76500700 1 -- 172.21.15.111:0/2836000722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a06fe5a190 con 0x55a06ff69180 2023-11-28T13:01:35.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f76500700 1 --2- 172.21.15.111:0/2836000722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 0x55a06ff5c380 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2f600011e0 tx=0x7f2f60012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc692d38888e6e68 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:35.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f64ff9700 1 -- 172.21.15.111:0/2836000722 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60015680 con 0x55a06ff69180 2023-11-28T13:01:35.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f64ff9700 1 -- 172.21.15.111:0/2836000722 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f60015db0 con 0x55a06ff69180 2023-11-28T13:01:35.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.012+0000 7f2f64ff9700 1 -- 172.21.15.111:0/2836000722 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f600150d0 con 0x55a06ff69180 2023-11-28T13:01:35.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.013+0000 7f2f848edb80 1 -- 172.21.15.111:0/2836000722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 msgr2=0x55a06ff5c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.013+0000 7f2f848edb80 1 --2- 172.21.15.111:0/2836000722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 0x55a06ff5c380 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2f600011e0 tx=0x7f2f60012000 comp rx=0 tx=0).stop 2023-11-28T13:01:35.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.013+0000 7f2f848edb80 1 -- 172.21.15.111:0/2836000722 shutdown_connections 2023-11-28T13:01:35.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.013+0000 7f2f848edb80 1 --2- 172.21.15.111:0/2836000722 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff6c170 0x55a06ff6e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.013+0000 7f2f848edb80 1 --2- 172.21.15.111:0/2836000722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff69180 0x55a06ff5c380 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.013+0000 7f2f848edb80 1 --2- 172.21.15.111:0/2836000722 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff6e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.013+0000 7f2f848edb80 1 -- 172.21.15.111:0/2836000722 >> 172.21.15.111:0/2836000722 conn(0x55a06ff3edb0 msgr2=0x55a06ff40280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:35.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.013+0000 7f2f848edb80 1 -- 172.21.15.111:0/2836000722 shutdown_connections 2023-11-28T13:01:35.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.013+0000 7f2f848edb80 1 -- 172.21.15.111:0/2836000722 wait complete. 2023-11-28T13:01:35.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.013+0000 7f2f848edb80 1 -- start start 2023-11-28T13:01:35.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.013+0000 7f2f848edb80 1 Processor -- start 2023-11-28T13:01:35.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.014+0000 7f2f848edb80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff69180 0x55a06ffd5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.014+0000 7f2f848edb80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff6e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.014+0000 7f2f848edb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 0x55a06ff6eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.014+0000 7f2f848edb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06ff5c380 con 0x55a06ff6c170 2023-11-28T13:01:35.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.014+0000 7f2f848edb80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06ff5c740 con 0x55a06ff69180 2023-11-28T13:01:35.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.014+0000 7f2f848edb80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a06ffec080 con 0x55a06ff6b860 2023-11-28T13:01:35.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.014+0000 7f2f848edb80 10 client.0 _ll_register_callbacks cb 0x7f2f68009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:01:35.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.014+0000 7f2f76d01700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 0x55a06ff6eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.014+0000 7f2f76d01700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 0x55a06ff6eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36562/0 (socket says 172.21.15.111:36562) 2023-11-28T13:01:35.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.014+0000 7f2f76d01700 1 -- 172.21.15.111:0/3728254173 learned_addr learned my addr 172.21.15.111:0/3728254173 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:35.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f75cff700 1 --2- 172.21.15.111:0/3728254173 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff6e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f76500700 1 --2- 172.21.15.111:0/3728254173 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff69180 0x55a06ffd5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f76d01700 1 -- 172.21.15.111:0/3728254173 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 msgr2=0x55a06ff6e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:35.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f76d01700 1 --2- 172.21.15.111:0/3728254173 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff6e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f76d01700 1 -- 172.21.15.111:0/3728254173 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff69180 msgr2=0x55a06ffd5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:35.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f76d01700 1 --2- 172.21.15.111:0/3728254173 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff69180 0x55a06ffd5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f76d01700 1 -- 172.21.15.111:0/3728254173 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6001d040 con 0x55a06ff6c170 2023-11-28T13:01:35.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f76d01700 1 --2- 172.21.15.111:0/3728254173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 0x55a06ff6eac0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2f68007050 tx=0x7f2f68004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbdd039986bdf879 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:35.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f517fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:01:35.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f517fa700 1 -- 172.21.15.111:0/3728254173 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68011ae0 con 0x55a06ff6c170 2023-11-28T13:01:35.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f517fa700 1 -- 172.21.15.111:0/3728254173 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2f68011c90 con 0x55a06ff6c170 2023-11-28T13:01:35.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f848edb80 1 -- 172.21.15.111:0/3728254173 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a06fe511b0 con 0x55a06ff6c170 2023-11-28T13:01:35.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f517fa700 1 -- 172.21.15.111:0/3728254173 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f680203f0 con 0x55a06ff6c170 2023-11-28T13:01:35.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f848edb80 10 client.6051 fetch_fsmap learned FSMap version 25 2023-11-28T13:01:35.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f848edb80 1 -- 172.21.15.111:0/3728254173 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x55a06fce8250 con 0x55a06ff6c170 2023-11-28T13:01:35.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f517fa700 1 -- 172.21.15.111:0/3728254173 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a06fe511b0 con 0x55a06ff6c170 2023-11-28T13:01:35.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f517fa700 1 -- 172.21.15.111:0/3728254173 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 25) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f2f6802f020 con 0x55a06ff6c170 2023-11-28T13:01:35.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f848edb80 10 client.6051 fetch_fsmap finished waiting for FSMap version 25 2023-11-28T13:01:35.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f848edb80 10 client.6051 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2023-11-28T13:01:35.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f848edb80 20 client.6051 populate_metadata read hostname 'smithi111' 2023-11-28T13:01:35.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f848edb80 1 --2- 172.21.15.111:0/3728254173 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55a06ff68bc0 0x55a06ff5a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f848edb80 4 client.6051 mds_command: new command op to 5814 tid=0[{"prefix": "scrub status"}] 2023-11-28T13:01:35.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.015+0000 7f2f848edb80 1 -- 172.21.15.111:0/3728254173 --> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a06fe3b1c0 con 0x55a06ff68bc0 2023-11-28T13:01:35.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f76500700 1 --2- 172.21.15.111:0/3728254173 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55a06ff68bc0 0x55a06ff5a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f517fa700 10 client.6051 ms_handle_connect on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:35.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f76500700 1 --2- 172.21.15.111:0/3728254173 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55a06ff68bc0 0x55a06ff5a520 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d0e550e1bad1c47 server_cookie=118424e6af14f1bf in_seq=0 out_seq=0 2023-11-28T13:01:35.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f517fa700 1 -- 172.21.15.111:0/3728254173 <== mds.0 v2:172.21.15.111:6835/4146274267 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a06fe3b1c0 con 0x55a06ff68bc0 2023-11-28T13:01:35.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f517fa700 10 client.6051 handle_command_reply: tid=0 2023-11-28T13:01:35.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f848edb80 1 client.6051 shutdown 2023-11-28T13:01:35.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f848edb80 1 -- 172.21.15.111:0/3728254173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 msgr2=0x55a06ff6eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f848edb80 1 --2- 172.21.15.111:0/3728254173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 0x55a06ff6eac0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2f68007050 tx=0x7f2f68004b40 comp rx=0 tx=0).stop 2023-11-28T13:01:35.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f848edb80 1 -- 172.21.15.111:0/3728254173 shutdown_connections 2023-11-28T13:01:35.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f848edb80 1 --2- 172.21.15.111:0/3728254173 >> [v2:172.21.15.111:6835/4146274267,v1:172.21.15.111:6837/4146274267] conn(0x55a06ff68bc0 0x55a06ff5a520 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f848edb80 1 --2- 172.21.15.111:0/3728254173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a06ff6c170 0x55a06ff6eac0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f848edb80 1 --2- 172.21.15.111:0/3728254173 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55a06ff69180 0x55a06ffd5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f517fa700 0 client.6051 ms_handle_reset on v2:172.21.15.111:6835/4146274267 2023-11-28T13:01:35.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f848edb80 1 --2- 172.21.15.111:0/3728254173 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55a06ff6b860 0x55a06ff6e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f848edb80 1 -- 172.21.15.111:0/3728254173 >> 172.21.15.111:0/3728254173 conn(0x55a06ff3edb0 msgr2=0x55a06ff3c460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:35.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f848edb80 1 -- 172.21.15.111:0/3728254173 shutdown_connections 2023-11-28T13:01:35.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f848edb80 1 -- 172.21.15.111:0/3728254173 wait complete. 2023-11-28T13:01:35.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.016+0000 7f2f848edb80 20 client.6051 trim_cache size 0 max 16384 2023-11-28T13:01:35.026 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:01:35.026 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-11-28T13:01:35.026 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-11-28T13:01:35.027 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:01:35.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.017+0000 7f2f78764700 1 -- 172.21.15.111:0/233129393 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f540682f0 msgr2=0x7f2f5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.017+0000 7f2f78764700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f540682f0 0x7f2f5406a7a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2f60000f60 tx=0x7f2f60007be0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.017+0000 7f2f78764700 1 -- 172.21.15.111:0/233129393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f700b20f0 msgr2=0x7f2f7015b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.017+0000 7f2f78764700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f700b20f0 0x7f2f7015b970 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009b60 tx=0x7f2f6c004750 comp rx=0 tx=0).stop 2023-11-28T13:01:35.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.017+0000 7f2f78764700 1 -- 172.21.15.111:0/233129393 shutdown_connections 2023-11-28T13:01:35.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.017+0000 7f2f78764700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f540682f0 0x7f2f5406a7a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.017+0000 7f2f78764700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f700b5790 0x7f2f701511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.017+0000 7f2f78764700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f7006b380 0x7f2f701531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.017+0000 7f2f78764700 1 --2- 172.21.15.111:0/233129393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f700b20f0 0x7f2f7015b970 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.017+0000 7f2f78764700 1 -- 172.21.15.111:0/233129393 >> 172.21.15.111:0/233129393 conn(0x7f2f7009a040 msgr2=0x7f2f7006ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:35.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.017+0000 7f2f78764700 1 -- 172.21.15.111:0/233129393 shutdown_connections 2023-11-28T13:01:35.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.017+0000 7f2f78764700 1 -- 172.21.15.111:0/233129393 wait complete. 2023-11-28T13:01:35.030 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-28T13:01:35.131 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:01:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.249+0000 7fbee9a98700 1 Processor -- start 2023-11-28T13:01:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.249+0000 7fbee9a98700 1 -- start start 2023-11-28T13:01:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.250+0000 7fbee9a98700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fbee4072300 0x7fbee40712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.250+0000 7fbee9a98700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee40717f0 con 0x7fbee415cc50 2023-11-28T13:01:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.250+0000 7fbee9a98700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee406fb90 con 0x7fbee40cceb0 2023-11-28T13:01:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.250+0000 7fbee9a98700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbee40726d0 con 0x7fbee4072300 2023-11-28T13:01:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.250+0000 7fbee2ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbee415cc50 0x7fbee406b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37346/0 (socket says 172.21.15.111:37346) 2023-11-28T13:01:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.250+0000 7fbee2ffd700 1 -- 172.21.15.111:0/1650263910 learned_addr learned my addr 172.21.15.111:0/1650263910 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.250+0000 7fbee37fe700 1 --2- 172.21.15.111:0/1650263910 >> v2:172.21.15.149:3300/0 conn(0x7fbee4072300 0x7fbee40712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.250+0000 7fbee27fc700 1 -- 172.21.15.111:0/1650263910 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347396653 0 0) 0x7fbee40717f0 con 0x7fbee415cc50 2023-11-28T13:01:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.250+0000 7fbee27fc700 1 -- 172.21.15.111:0/1650263910 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbecc0035b0 con 0x7fbee415cc50 2023-11-28T13:01:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.250+0000 7fbee37fe700 1 -- 172.21.15.111:0/1650263910 >> v1:172.21.15.149:6789/0 conn(0x7fbee40cceb0 legacy=0x7fbee40cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.250+0000 7fbee37fe700 1 -- 172.21.15.111:0/1650263910 >> v1:172.21.15.111:6789/0 conn(0x7fbee415cc50 legacy=0x7fbee406b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.250+0000 7fbee27fc700 1 -- 172.21.15.111:0/1650263910 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3488541585 0 0) 0x7fbecc0035b0 con 0x7fbee415cc50 2023-11-28T13:01:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.250+0000 7fbee37fe700 1 -- 172.21.15.111:0/1650263910 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee415d0f0 con 0x7fbee4072300 2023-11-28T13:01:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.251+0000 7fbee37fe700 1 --2- 172.21.15.111:0/1650263910 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee4072300 0x7fbee40712b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009840 tx=0x7fbed8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a83373fcf33cb7db server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.251+0000 7fbee27fc700 1 -- 172.21.15.111:0/1650263910 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed4001670 con 0x7fbee4072300 2023-11-28T13:01:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.251+0000 7fbee27fc700 1 -- 172.21.15.111:0/1650263910 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed8004030 con 0x7fbee4072300 2023-11-28T13:01:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.251+0000 7fbee27fc700 1 -- 172.21.15.111:0/1650263910 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed80043c0 con 0x7fbee4072300 2023-11-28T13:01:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.251+0000 7fbee9a98700 1 -- 172.21.15.111:0/1650263910 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee4072300 msgr2=0x7fbee40712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.251+0000 7fbee9a98700 1 --2- 172.21.15.111:0/1650263910 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee4072300 0x7fbee40712b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009840 tx=0x7fbed8009b50 comp rx=0 tx=0).stop 2023-11-28T13:01:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.251+0000 7fbee9a98700 1 -- 172.21.15.111:0/1650263910 shutdown_connections 2023-11-28T13:01:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.251+0000 7fbee9a98700 1 --2- 172.21.15.111:0/1650263910 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee4072300 0x7fbee40712b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.251+0000 7fbee9a98700 1 -- 172.21.15.111:0/1650263910 >> 172.21.15.111:0/1650263910 conn(0x7fbee40cf2e0 msgr2=0x7fbee40cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.251+0000 7fbee9a98700 1 -- 172.21.15.111:0/1650263910 shutdown_connections 2023-11-28T13:01:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.251+0000 7fbee9a98700 1 -- 172.21.15.111:0/1650263910 wait complete. 2023-11-28T13:01:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.253+0000 7fbee9a98700 1 Processor -- start 2023-11-28T13:01:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.253+0000 7fbee9a98700 1 -- start start 2023-11-28T13:01:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.253+0000 7fbee9a98700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee4072300 0x7fbee4152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.253+0000 7fbee9a98700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee40cceb0 0x7fbee40b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.253+0000 7fbee9a98700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee415cc50 0x7fbee40add30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.253+0000 7fbee9a98700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee40b28c0 con 0x7fbee415cc50 2023-11-28T13:01:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee9a98700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee40aed40 con 0x7fbee40cceb0 2023-11-28T13:01:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee9a98700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee40aef90 con 0x7fbee4072300 2023-11-28T13:01:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee415cc50 0x7fbee40add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee415cc50 0x7fbee40add30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36576/0 (socket says 172.21.15.111:36576) 2023-11-28T13:01:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee3fff700 1 -- 172.21.15.111:0/8659280 learned_addr learned my addr 172.21.15.111:0/8659280 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee37fe700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee4072300 0x7fbee4152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee2ffd700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee40cceb0 0x7fbee40b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee3fff700 1 -- 172.21.15.111:0/8659280 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee4072300 msgr2=0x7fbee4152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee3fff700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee4072300 0x7fbee4152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee3fff700 1 -- 172.21.15.111:0/8659280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee40cceb0 msgr2=0x7fbee40b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee3fff700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee40cceb0 0x7fbee40b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee3fff700 1 -- 172.21.15.111:0/8659280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed80094a0 con 0x7fbee415cc50 2023-11-28T13:01:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee2ffd700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee40cceb0 0x7fbee40b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee3fff700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee415cc50 0x7fbee40add30 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00cc50 tx=0x7fbedc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbecbfff700 1 -- 172.21.15.111:0/8659280 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc003c00 con 0x7fbee415cc50 2023-11-28T13:01:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbecbfff700 1 -- 172.21.15.111:0/8659280 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbedc003d60 con 0x7fbee415cc50 2023-11-28T13:01:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee9a98700 1 -- 172.21.15.111:0/8659280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee415d6b0 con 0x7fbee415cc50 2023-11-28T13:01:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbecbfff700 1 -- 172.21.15.111:0/8659280 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc00ab60 con 0x7fbee415cc50 2023-11-28T13:01:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.254+0000 7fbee9a98700 1 -- 172.21.15.111:0/8659280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee415d3d0 con 0x7fbee415cc50 2023-11-28T13:01:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.255+0000 7fbee9a98700 1 -- 172.21.15.111:0/8659280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee415d990 con 0x7fbee415cc50 2023-11-28T13:01:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.255+0000 7fbecbfff700 1 -- 172.21.15.111:0/8659280 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbedc010050 con 0x7fbee415cc50 2023-11-28T13:01:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.256+0000 7fbecbfff700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbed00682f0 0x7fbed006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.256+0000 7fbecbfff700 1 -- 172.21.15.111:0/8659280 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fbedc085d30 con 0x7fbee415cc50 2023-11-28T13:01:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.256+0000 7fbee37fe700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbed00682f0 0x7fbed006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.256+0000 7fbee37fe700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbed00682f0 0x7fbed006a7a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbed80090b0 tx=0x7fbed8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:35.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.260+0000 7fbecbfff700 1 -- 172.21.15.111:0/8659280 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbedc0523d0 con 0x7fbee415cc50 2023-11-28T13:01:35.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.487+0000 7fbee9a98700 1 -- 172.21.15.111:0/8659280 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbee40cf150 con 0x7fbee415cc50 2023-11-28T13:01:35.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.488+0000 7fbecbfff700 1 -- 172.21.15.111:0/8659280 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4660 (secure 0 0 0) 0x7fbedc0559f0 con 0x7fbee415cc50 2023-11-28T13:01:35.490 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:35.490 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2072621958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2072621958},{"type":"v1","addr":"172.21.15.149:6834","nonce":2072621958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3348427170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3348427170},{"type":"v1","addr":"172.21.15.149:6835","nonce":3348427170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1745443298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1745443298},{"type":"v1","addr":"172.21.15.111:6836","nonce":1745443298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:56.272312+0000","modified":"2023-11-28T13:00:59.284265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5814":{"gid":5814,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/4146274267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":4146274267},{"type":"v1","addr":"172.21.15.111:6837","nonce":4146274267}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T13:01:35.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.490+0000 7fbee9a98700 1 -- 172.21.15.111:0/8659280 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbed00682f0 msgr2=0x7fbed006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.490+0000 7fbee9a98700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbed00682f0 0x7fbed006a7a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbed80090b0 tx=0x7fbed8008030 comp rx=0 tx=0).stop 2023-11-28T13:01:35.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.490+0000 7fbee9a98700 1 -- 172.21.15.111:0/8659280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee415cc50 msgr2=0x7fbee40add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.490+0000 7fbee9a98700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee415cc50 0x7fbee40add30 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00cc50 tx=0x7fbedc00cf60 comp rx=0 tx=0).stop 2023-11-28T13:01:35.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.490+0000 7fbee9a98700 1 -- 172.21.15.111:0/8659280 shutdown_connections 2023-11-28T13:01:35.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.490+0000 7fbee9a98700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbee415cc50 0x7fbee40add30 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.490+0000 7fbee9a98700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbee4072300 0x7fbee4152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.490+0000 7fbee9a98700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbed00682f0 0x7fbed006a7a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.490+0000 7fbee9a98700 1 --2- 172.21.15.111:0/8659280 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbee40cceb0 0x7fbee40b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.490+0000 7fbee9a98700 1 -- 172.21.15.111:0/8659280 >> 172.21.15.111:0/8659280 conn(0x7fbee40cf2e0 msgr2=0x7fbee415a850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:35.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.490+0000 7fbee9a98700 1 -- 172.21.15.111:0/8659280 shutdown_connections 2023-11-28T13:01:35.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.490+0000 7fbee9a98700 1 -- 172.21.15.111:0/8659280 wait complete. 2023-11-28T13:01:35.492 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 25 2023-11-28T13:01:35.503 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2023-11-28T13:01:35.647 INFO:teuthology.orchestra.run.smithi111.stdout:{"message":"","return_code":0} 2023-11-28T13:01:35.647 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-11-28T13:01:35.648 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2023-11-28T13:01:35.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.690+0000 7f5728214080 1 Processor -- start 2023-11-28T13:01:35.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.690+0000 7f5728214080 1 -- start start 2023-11-28T13:01:35.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f5728214080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55acc548fe60 0x55acc5517d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f5728214080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55acc5518890 0x55acc551e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f5728214080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55acc5490230 0x55acc5485120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f5728214080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc53ed000 con 0x55acc5518890 2023-11-28T13:01:35.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f5728214080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc537ffe0 con 0x55acc548fe60 2023-11-28T13:01:35.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f5728214080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acc551ecf0 con 0x55acc5490230 2023-11-28T13:01:35.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571db16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55acc5518890 0x55acc551e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571db16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55acc5518890 0x55acc551e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36594/0 (socket says 172.21.15.111:36594) 2023-11-28T13:01:35.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571db16700 1 -- 172.21.15.111:0/1797237120 learned_addr learned my addr 172.21.15.111:0/1797237120 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:35.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571e317700 1 --2- 172.21.15.111:0/1797237120 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55acc548fe60 0x55acc5517d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571db16700 1 -- 172.21.15.111:0/1797237120 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55acc5490230 msgr2=0x55acc5485120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:35.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571db16700 1 --2- 172.21.15.111:0/1797237120 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55acc5490230 0x55acc5485120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571db16700 1 -- 172.21.15.111:0/1797237120 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55acc548fe60 msgr2=0x55acc5517d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:35.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571db16700 1 --2- 172.21.15.111:0/1797237120 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55acc548fe60 0x55acc5517d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571db16700 1 -- 172.21.15.111:0/1797237120 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acc5398360 con 0x55acc5518890 2023-11-28T13:01:35.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571e317700 1 --2- 172.21.15.111:0/1797237120 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55acc548fe60 0x55acc5517d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:35.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571db16700 1 --2- 172.21.15.111:0/1797237120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55acc5518890 0x55acc551e7b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f57140059d0 tx=0x7f5714005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488e17f3af55d7e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:35.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571cb14700 1 -- 172.21.15.111:0/1797237120 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5714019070 con 0x55acc5518890 2023-11-28T13:01:35.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571cb14700 1 -- 172.21.15.111:0/1797237120 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5714010040 con 0x55acc5518890 2023-11-28T13:01:35.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.691+0000 7f571cb14700 1 -- 172.21.15.111:0/1797237120 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571400f050 con 0x55acc5518890 2023-11-28T13:01:35.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.692+0000 7f5728214080 1 -- 172.21.15.111:0/1797237120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55acc5518890 msgr2=0x55acc551e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.692+0000 7f5728214080 1 --2- 172.21.15.111:0/1797237120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55acc5518890 0x55acc551e7b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f57140059d0 tx=0x7f5714005a00 comp rx=0 tx=0).stop 2023-11-28T13:01:35.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.692+0000 7f5728214080 1 -- 172.21.15.111:0/1797237120 shutdown_connections 2023-11-28T13:01:35.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.692+0000 7f5728214080 1 --2- 172.21.15.111:0/1797237120 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55acc5490230 0x55acc5485120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.692+0000 7f5728214080 1 --2- 172.21.15.111:0/1797237120 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55acc548fe60 0x55acc5517d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.692+0000 7f5728214080 1 --2- 172.21.15.111:0/1797237120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55acc5518890 0x55acc551e7b0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.692+0000 7f5728214080 1 -- 172.21.15.111:0/1797237120 >> 172.21.15.111:0/1797237120 conn(0x55acc548d5a0 msgr2=0x55acc5520310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:35.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.692+0000 7f5728214080 1 -- 172.21.15.111:0/1797237120 shutdown_connections 2023-11-28T13:01:35.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.692+0000 7f5728214080 1 -- 172.21.15.111:0/1797237120 wait complete. 2023-11-28T13:01:35.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.692+0000 7f5728214080 1 Processor -- start 2023-11-28T13:01:35.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.693+0000 7f5728214080 1 -- start start 2023-11-28T13:01:35.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.693+0000 7f5728214080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55acc548fe60 0x55acc55b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.693+0000 7f5728214080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55acc5490230 0x55acc5443fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.693+0000 7f5728214080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55acc54444e0 0x55acc550f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.693+0000 7f5728214080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc55b4190 con 0x55acc54444e0 2023-11-28T13:01:35.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.693+0000 7f5728214080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acc5520170 con 0x55acc548fe60 2023-11-28T13:01:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.693+0000 7f5728214080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55acc551b3b0 con 0x55acc5490230 2023-11-28T13:01:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.693+0000 7f571eb18700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55acc54444e0 0x55acc550f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.693+0000 7f571eb18700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55acc54444e0 0x55acc550f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36602/0 (socket says 172.21.15.111:36602) 2023-11-28T13:01:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.693+0000 7f571eb18700 1 -- 172.21.15.111:0/3976404486 learned_addr learned my addr 172.21.15.111:0/3976404486 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.693+0000 7f571e317700 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55acc548fe60 0x55acc55b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.693+0000 7f571db16700 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55acc5490230 0x55acc5443fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.694+0000 7f571eb18700 1 -- 172.21.15.111:0/3976404486 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55acc5490230 msgr2=0x55acc5443fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.694+0000 7f571eb18700 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55acc5490230 0x55acc5443fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.694+0000 7f571eb18700 1 -- 172.21.15.111:0/3976404486 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55acc548fe60 msgr2=0x55acc55b3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.694+0000 7f571eb18700 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55acc548fe60 0x55acc55b3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.694+0000 7f571eb18700 1 -- 172.21.15.111:0/3976404486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5714009410 con 0x55acc54444e0 2023-11-28T13:01:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.694+0000 7f571eb18700 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55acc54444e0 0x55acc550f370 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5718010000 tx=0x7f5718001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:35.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.694+0000 7f570f7fe700 1 -- 172.21.15.111:0/3976404486 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5718020070 con 0x55acc54444e0 2023-11-28T13:01:35.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.694+0000 7f570f7fe700 1 -- 172.21.15.111:0/3976404486 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571800be50 con 0x55acc54444e0 2023-11-28T13:01:35.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.694+0000 7f570f7fe700 1 -- 172.21.15.111:0/3976404486 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5718003510 con 0x55acc54444e0 2023-11-28T13:01:35.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.694+0000 7f5728214080 1 -- 172.21.15.111:0/3976404486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55acc545d840 con 0x55acc54444e0 2023-11-28T13:01:35.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.694+0000 7f5728214080 1 -- 172.21.15.111:0/3976404486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55acc538b830 con 0x55acc54444e0 2023-11-28T13:01:35.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.695+0000 7f570f7fe700 1 -- 172.21.15.111:0/3976404486 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f571800f070 con 0x55acc54444e0 2023-11-28T13:01:35.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.696+0000 7f570f7fe700 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5700066c10 0x7f57000690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.696+0000 7f570f7fe700 1 -- 172.21.15.111:0/3976404486 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f5718084d30 con 0x55acc54444e0 2023-11-28T13:01:35.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.696+0000 7f571e317700 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5700066c10 0x7f57000690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.696+0000 7f5728214080 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55acc5446770 0x55acc55145b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:35.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.696+0000 7f5728214080 1 -- 172.21.15.111:0/3976404486 --> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] -- osd_op(unknown.0.0:1 9.7 9:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55) v8 -- 0x55acc5513500 con 0x55acc5446770 2023-11-28T13:01:35.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.696+0000 7f571e317700 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5700066c10 0x7f57000690c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x55acc5513d70 tx=0x7f57100029b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:35.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.696+0000 7f571db16700 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55acc5446770 0x55acc55145b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:35.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.696+0000 7f571db16700 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55acc5446770 0x55acc55145b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:35.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.697+0000 7f571db16700 1 -- 172.21.15.111:0/3976404486 <== osd.5 v2:172.21.15.149:6808/103795 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x7f571401b070 con 0x55acc5446770 2023-11-28T13:01:35.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.697+0000 7f5728214080 1 -- 172.21.15.111:0/3976404486 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55acc5446770 msgr2=0x55acc55145b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.697+0000 7f5728214080 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x55acc5446770 0x55acc55145b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.697+0000 7f5728214080 1 -- 172.21.15.111:0/3976404486 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5700066c10 msgr2=0x7f57000690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.697+0000 7f5728214080 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f5700066c10 0x7f57000690c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x55acc5513d70 tx=0x7f57100029b0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.697+0000 7f5728214080 1 -- 172.21.15.111:0/3976404486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55acc54444e0 msgr2=0x55acc550f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:35.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.697+0000 7f5728214080 1 --2- 172.21.15.111:0/3976404486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55acc54444e0 0x55acc550f370 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5718010000 tx=0x7f5718001ae0 comp rx=0 tx=0).stop 2023-11-28T13:01:35.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.697+0000 7f571eb18700 1 -- 172.21.15.111:0/3976404486 reap_dead start 2023-11-28T13:01:35.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.697+0000 7f5728214080 1 -- 172.21.15.111:0/3976404486 shutdown_connections 2023-11-28T13:01:35.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.697+0000 7f5728214080 1 -- 172.21.15.111:0/3976404486 >> 172.21.15.111:0/3976404486 conn(0x55acc548d5a0 msgr2=0x55acc5520310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:35.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.697+0000 7f5728214080 1 -- 172.21.15.111:0/3976404486 shutdown_connections 2023-11-28T13:01:35.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:35.697+0000 7f5728214080 1 -- 172.21.15.111:0/3976404486 wait complete. 2023-11-28T13:01:35.707 INFO:teuthology.orchestra.run.smithi111.stdout:Yalphaparent_a 2023-11-28T13:01:35.707 DEBUG:teuthology.orchestra.run.smithi111:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2023-11-28T13:01:35.801 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:01:35.801 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 1099511627777, 2023-11-28T13:01:35.801 INFO:teuthology.orchestra.run.smithi111.stdout: "ancestors": [ 2023-11-28T13:01:35.801 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:35.801 INFO:teuthology.orchestra.run.smithi111.stdout: "dirino": 1099511627776, 2023-11-28T13:01:35.801 INFO:teuthology.orchestra.run.smithi111.stdout: "dname": "alpha", 2023-11-28T13:01:35.802 INFO:teuthology.orchestra.run.smithi111.stdout: "version": 2 2023-11-28T13:01:35.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:35.802 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:35.802 INFO:teuthology.orchestra.run.smithi111.stdout: "dirino": 1, 2023-11-28T13:01:35.802 INFO:teuthology.orchestra.run.smithi111.stdout: "dname": "parent_a", 2023-11-28T13:01:35.802 INFO:teuthology.orchestra.run.smithi111.stdout: "version": 4 2023-11-28T13:01:35.802 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:35.802 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-11-28T13:01:35.803 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 9, 2023-11-28T13:01:35.803 INFO:teuthology.orchestra.run.smithi111.stdout: "old_pools": [] 2023-11-28T13:01:35.803 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:01:35.803 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:35.804 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T13:01:35.804 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T13:01:35.886 DEBUG:teuthology.orchestra.run.smithi149:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T13:01:35.936 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T13:01:35.937 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:01:35.937 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:01:35.954 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:01:35.954 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:01:36.012 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:01:36.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.169+0000 7f0b8fcbb700 1 Processor -- start 2023-11-28T13:01:36.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.170+0000 7f0b8fcbb700 1 -- start start 2023-11-28T13:01:36.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.170+0000 7f0b8fcbb700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f0b881604b0 0x7f0b880ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.170+0000 7f0b8fcbb700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b880cc550 0x7f0b880cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.170+0000 7f0b8fcbb700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8815cd90 con 0x7f0b880b0d70 2023-11-28T13:01:36.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.170+0000 7f0b8fcbb700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b88160950 con 0x7f0b881604b0 2023-11-28T13:01:36.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.170+0000 7f0b8fcbb700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880ccf10 con 0x7f0b880cc550 2023-11-28T13:01:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.171+0000 7f0b8d256700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f0b881604b0 0x7f0b880ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.171+0000 7f0b8e258700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b880cc550 0x7f0b880cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.171+0000 7f0b8da57700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f0b880b0d70 0x7f0b880b1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:45928/0 (socket says 172.21.15.111:45928) 2023-11-28T13:01:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.171+0000 7f0b8da57700 1 -- 172.21.15.111:0/556140972 learned_addr learned my addr 172.21.15.111:0/556140972 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.171+0000 7f0b8d256700 1 -- 172.21.15.111:0/556140972 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b880cc550 msgr2=0x7f0b880cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.171+0000 7f0b8d256700 1 --2- 172.21.15.111:0/556140972 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b880cc550 0x7f0b880cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.171+0000 7f0b8d256700 1 -- 172.21.15.111:0/556140972 >> v1:172.21.15.149:6789/0 conn(0x7f0b880b0d70 legacy=0x7f0b880b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.171+0000 7f0b8d256700 1 -- 172.21.15.111:0/556140972 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b881613e0 con 0x7f0b881604b0 2023-11-28T13:01:36.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.172+0000 7f0b8d256700 1 --2- 172.21.15.111:0/556140972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b881604b0 0x7f0b880ce6c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009840 tx=0x7f0b84009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53d0954df6b36f1e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.172+0000 7f0b8ca55700 1 -- 172.21.15.111:0/556140972 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b84011830 con 0x7f0b881604b0 2023-11-28T13:01:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.172+0000 7f0b8ca55700 1 -- 172.21.15.111:0/556140972 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b84011990 con 0x7f0b881604b0 2023-11-28T13:01:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.172+0000 7f0b8ca55700 1 -- 172.21.15.111:0/556140972 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b84011d20 con 0x7f0b881604b0 2023-11-28T13:01:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.172+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/556140972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b881604b0 msgr2=0x7f0b880ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.172+0000 7f0b8fcbb700 1 --2- 172.21.15.111:0/556140972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b881604b0 0x7f0b880ce6c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009840 tx=0x7f0b84009b50 comp rx=0 tx=0).stop 2023-11-28T13:01:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.172+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/556140972 shutdown_connections 2023-11-28T13:01:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.172+0000 7f0b8fcbb700 1 --2- 172.21.15.111:0/556140972 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b880cc550 0x7f0b880cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.172+0000 7f0b8fcbb700 1 --2- 172.21.15.111:0/556140972 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b881604b0 0x7f0b880ce6c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.172+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/556140972 >> 172.21.15.111:0/556140972 conn(0x7f0b88098ee0 msgr2=0x7f0b8806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.172+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/556140972 shutdown_connections 2023-11-28T13:01:36.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.172+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/556140972 wait complete. 2023-11-28T13:01:36.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.174+0000 7f0b8fcbb700 1 Processor -- start 2023-11-28T13:01:36.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.174+0000 7f0b8fcbb700 1 -- start start 2023-11-28T13:01:36.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.174+0000 7f0b8fcbb700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b880b0d70 0x7f0b881f1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.174+0000 7f0b8fcbb700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b880cc550 0x7f0b880ce680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.174+0000 7f0b8fcbb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b881604b0 0x7f0b880cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.174+0000 7f0b8fcbb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b881f2460 con 0x7f0b881604b0 2023-11-28T13:01:36.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.174+0000 7f0b8fcbb700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b881f2740 con 0x7f0b880b0d70 2023-11-28T13:01:36.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.174+0000 7f0b8fcbb700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b881f2a20 con 0x7f0b880cc550 2023-11-28T13:01:36.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.174+0000 7f0b8e258700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b881604b0 0x7f0b880cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.174+0000 7f0b8e258700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b881604b0 0x7f0b880cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36618/0 (socket says 172.21.15.111:36618) 2023-11-28T13:01:36.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.174+0000 7f0b8e258700 1 -- 172.21.15.111:0/144102574 learned_addr learned my addr 172.21.15.111:0/144102574 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:36.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b8da57700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b880b0d70 0x7f0b881f1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b8d256700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b880cc550 0x7f0b880ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b8e258700 1 -- 172.21.15.111:0/144102574 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b880cc550 msgr2=0x7f0b880ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b8e258700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b880cc550 0x7f0b880ce680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b8e258700 1 -- 172.21.15.111:0/144102574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b880b0d70 msgr2=0x7f0b881f1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b8e258700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b880b0d70 0x7f0b881f1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b8e258700 1 -- 172.21.15.111:0/144102574 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b840094a0 con 0x7f0b881604b0 2023-11-28T13:01:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b8d256700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b880cc550 0x7f0b880ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b8da57700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b880b0d70 0x7f0b881f1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b8e258700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b881604b0 0x7f0b880cebc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0b8000efc0 tx=0x7f0b8000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b7e7fc700 1 -- 172.21.15.111:0/144102574 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8000cd40 con 0x7f0b881604b0 2023-11-28T13:01:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b7e7fc700 1 -- 172.21.15.111:0/144102574 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8000f040 con 0x7f0b881604b0 2023-11-28T13:01:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/144102574 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b88160e20 con 0x7f0b881604b0 2023-11-28T13:01:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b7e7fc700 1 -- 172.21.15.111:0/144102574 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b800146c0 con 0x7f0b881604b0 2023-11-28T13:01:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.175+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/144102574 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b88160b40 con 0x7f0b881604b0 2023-11-28T13:01:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.176+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/144102574 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b88161100 con 0x7f0b881604b0 2023-11-28T13:01:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.176+0000 7f0b7e7fc700 1 -- 172.21.15.111:0/144102574 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0b80014820 con 0x7f0b881604b0 2023-11-28T13:01:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.177+0000 7f0b7e7fc700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0b6c0682f0 0x7f0b6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.177+0000 7f0b7e7fc700 1 -- 172.21.15.111:0/144102574 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f0b80086b40 con 0x7f0b881604b0 2023-11-28T13:01:36.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.177+0000 7f0b8da57700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0b6c0682f0 0x7f0b6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.177+0000 7f0b8da57700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0b6c0682f0 0x7f0b6c06a7a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0b78005960 tx=0x7f0b78009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:36.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.181+0000 7f0b7e7fc700 1 -- 172.21.15.111:0/144102574 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0b80053260 con 0x7f0b881604b0 2023-11-28T13:01:36.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.370+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/144102574 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b88067170 con 0x7f0b881604b0 2023-11-28T13:01:36.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.371+0000 7f0b7e7fc700 1 -- 172.21.15.111:0/144102574 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+18228 (secure 0 0 0) 0x7f0b80018090 con 0x7f0b881604b0 2023-11-28T13:01:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:36.373 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":55,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:57.274026+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:53.965825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:55.242768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:01:36.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.373+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/144102574 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0b6c0682f0 msgr2=0x7f0b6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.373+0000 7f0b8fcbb700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0b6c0682f0 0x7f0b6c06a7a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0b78005960 tx=0x7f0b78009040 comp rx=0 tx=0).stop 2023-11-28T13:01:36.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.373+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/144102574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b881604b0 msgr2=0x7f0b880cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.373+0000 7f0b8fcbb700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b881604b0 0x7f0b880cebc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0b8000efc0 tx=0x7f0b8000c3a0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.373+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/144102574 shutdown_connections 2023-11-28T13:01:36.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.373+0000 7f0b8fcbb700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b881604b0 0x7f0b880cebc0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.373+0000 7f0b8fcbb700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0b6c0682f0 0x7f0b6c06a7a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.373+0000 7f0b8fcbb700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0b880b0d70 0x7f0b881f1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.373+0000 7f0b8fcbb700 1 --2- 172.21.15.111:0/144102574 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f0b880cc550 0x7f0b880ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.373+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/144102574 >> 172.21.15.111:0/144102574 conn(0x7f0b88098ee0 msgr2=0x7f0b8815c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:36.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.373+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/144102574 shutdown_connections 2023-11-28T13:01:36.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.374+0000 7f0b8fcbb700 1 -- 172.21.15.111:0/144102574 wait complete. 2023-11-28T13:01:36.387 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T13:00:53.965825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T13:00:55.242768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T13:01:36.387 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T13:01:36.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.508+0000 7efbf81a7700 1 Processor -- start 2023-11-28T13:01:36.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.509+0000 7efbf81a7700 1 -- start start 2023-11-28T13:01:36.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.509+0000 7efbf81a7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efbf00cfc90 0x7efbf00d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.509+0000 7efbf81a7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efbf00cdef0 0x7efbf00ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.509+0000 7efbf81a7700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf00d05a0 con 0x7efbf00cc5a0 2023-11-28T13:01:36.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.509+0000 7efbf81a7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf00ca260 con 0x7efbf00cdef0 2023-11-28T13:01:36.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.509+0000 7efbf81a7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf00ccaa0 con 0x7efbf00cfc90 2023-11-28T13:01:36.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.510+0000 7efbf6744700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efbf00cdef0 0x7efbf00ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.510+0000 7efbf5742700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efbf00cc5a0 0x7efbf00cd7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37402/0 (socket says 172.21.15.111:37402) 2023-11-28T13:01:36.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.510+0000 7efbf5742700 1 -- 172.21.15.111:0/4201230539 learned_addr learned my addr 172.21.15.111:0/4201230539 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:36.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.510+0000 7efbf5f43700 1 --2- 172.21.15.111:0/4201230539 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efbf00cfc90 0x7efbf00d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.510+0000 7efbf6744700 1 -- 172.21.15.111:0/4201230539 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efbf00cfc90 msgr2=0x7efbf00d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:36.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.510+0000 7efbf6744700 1 --2- 172.21.15.111:0/4201230539 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efbf00cfc90 0x7efbf00d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.510+0000 7efbf6744700 1 -- 172.21.15.111:0/4201230539 >> v1:172.21.15.111:6789/0 conn(0x7efbf00cc5a0 legacy=0x7efbf00cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.510+0000 7efbf6744700 1 -- 172.21.15.111:0/4201230539 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf00ccd80 con 0x7efbf00cdef0 2023-11-28T13:01:36.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.510+0000 7efbf4f41700 1 -- 172.21.15.111:0/4201230539 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151957797 0 0) 0x7efbe4001110 con 0x7efbf00cc5a0 2023-11-28T13:01:36.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.510+0000 7efbf6744700 1 --2- 172.21.15.111:0/4201230539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf00cdef0 0x7efbf00ca6f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7efbe8009840 tx=0x7efbe8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5485e48af2a88e7a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:36.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.510+0000 7efbf4f41700 1 -- 172.21.15.111:0/4201230539 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe8011830 con 0x7efbf00cdef0 2023-11-28T13:01:36.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.510+0000 7efbf4f41700 1 -- 172.21.15.111:0/4201230539 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe8011990 con 0x7efbf00cdef0 2023-11-28T13:01:36.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.510+0000 7efbf4f41700 1 -- 172.21.15.111:0/4201230539 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe8011d20 con 0x7efbf00cdef0 2023-11-28T13:01:36.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.511+0000 7efbf81a7700 1 -- 172.21.15.111:0/4201230539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf00cdef0 msgr2=0x7efbf00ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.511+0000 7efbf81a7700 1 --2- 172.21.15.111:0/4201230539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf00cdef0 0x7efbf00ca6f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7efbe8009840 tx=0x7efbe8009b50 comp rx=0 tx=0).stop 2023-11-28T13:01:36.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.511+0000 7efbf81a7700 1 -- 172.21.15.111:0/4201230539 shutdown_connections 2023-11-28T13:01:36.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.511+0000 7efbf81a7700 1 --2- 172.21.15.111:0/4201230539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf00cdef0 0x7efbf00ca6f0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.511+0000 7efbf81a7700 1 --2- 172.21.15.111:0/4201230539 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efbf00cfc90 0x7efbf00d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.511+0000 7efbf81a7700 1 -- 172.21.15.111:0/4201230539 >> 172.21.15.111:0/4201230539 conn(0x7efbf0098ee0 msgr2=0x7efbf006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:36.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.511+0000 7efbf81a7700 1 -- 172.21.15.111:0/4201230539 shutdown_connections 2023-11-28T13:01:36.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.511+0000 7efbf81a7700 1 -- 172.21.15.111:0/4201230539 wait complete. 2023-11-28T13:01:36.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.512+0000 7efbf81a7700 1 Processor -- start 2023-11-28T13:01:36.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.512+0000 7efbf81a7700 1 -- start start 2023-11-28T13:01:36.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.513+0000 7efbf81a7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf00cc5a0 0x7efbf01496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.513+0000 7efbf81a7700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efbf00cdef0 0x7efbf00c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.513+0000 7efbf81a7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efbf00cfc90 0x7efbf00c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.513+0000 7efbf81a7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf00b3610 con 0x7efbf00cc5a0 2023-11-28T13:01:36.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.513+0000 7efbf81a7700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf00c30b0 con 0x7efbf00cdef0 2023-11-28T13:01:36.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.513+0000 7efbf81a7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf00c4cc0 con 0x7efbf00cfc90 2023-11-28T13:01:36.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.513+0000 7efbf5f43700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf00cc5a0 0x7efbf01496e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.513+0000 7efbf5f43700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf00cc5a0 0x7efbf01496e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36628/0 (socket says 172.21.15.111:36628) 2023-11-28T13:01:36.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.513+0000 7efbf5f43700 1 -- 172.21.15.111:0/569723125 learned_addr learned my addr 172.21.15.111:0/569723125 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:36.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.514+0000 7efbf6744700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efbf00cfc90 0x7efbf00c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.514+0000 7efbf5742700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efbf00cdef0 0x7efbf00c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.514+0000 7efbf5f43700 1 -- 172.21.15.111:0/569723125 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efbf00cfc90 msgr2=0x7efbf00c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.514+0000 7efbf5f43700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efbf00cfc90 0x7efbf00c61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.514+0000 7efbf5f43700 1 -- 172.21.15.111:0/569723125 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efbf00cdef0 msgr2=0x7efbf00c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.514+0000 7efbf5f43700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efbf00cdef0 0x7efbf00c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.514+0000 7efbf5f43700 1 -- 172.21.15.111:0/569723125 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe80094a0 con 0x7efbf00cc5a0 2023-11-28T13:01:36.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.514+0000 7efbf5f43700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf00cc5a0 0x7efbf01496e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7efbe0007ef0 tx=0x7efbe000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:36.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.514+0000 7efbee7fc700 1 -- 172.21.15.111:0/569723125 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe0014420 con 0x7efbf00cc5a0 2023-11-28T13:01:36.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.514+0000 7efbee7fc700 1 -- 172.21.15.111:0/569723125 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe0014580 con 0x7efbf00cc5a0 2023-11-28T13:01:36.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.514+0000 7efbf81a7700 1 -- 172.21.15.111:0/569723125 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf00cd340 con 0x7efbf00cc5a0 2023-11-28T13:01:36.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.514+0000 7efbee7fc700 1 -- 172.21.15.111:0/569723125 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe0014930 con 0x7efbf00cc5a0 2023-11-28T13:01:36.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.514+0000 7efbf81a7700 1 -- 172.21.15.111:0/569723125 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf00cd060 con 0x7efbf00cc5a0 2023-11-28T13:01:36.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.515+0000 7efbf81a7700 1 -- 172.21.15.111:0/569723125 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf0143280 con 0x7efbf00cc5a0 2023-11-28T13:01:36.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.515+0000 7efbee7fc700 1 -- 172.21.15.111:0/569723125 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efbe0012050 con 0x7efbf00cc5a0 2023-11-28T13:01:36.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.516+0000 7efbee7fc700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efbdc0682f0 0x7efbdc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.516+0000 7efbee7fc700 1 -- 172.21.15.111:0/569723125 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7efbe0086480 con 0x7efbf00cc5a0 2023-11-28T13:01:36.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.516+0000 7efbf5742700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efbdc0682f0 0x7efbdc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.516+0000 7efbf5742700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efbdc0682f0 0x7efbdc06a7a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7efbf00ae550 tx=0x7efbe4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:36.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.520+0000 7efbee7fc700 1 -- 172.21.15.111:0/569723125 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efbe0052ba0 con 0x7efbf00cc5a0 2023-11-28T13:01:36.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.703+0000 7efbf81a7700 1 -- 172.21.15.111:0/569723125 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efbf0067170 con 0x7efbf00cc5a0 2023-11-28T13:01:36.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.703+0000 7efbee7fc700 1 -- 172.21.15.111:0/569723125 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7efbe00561c0 con 0x7efbf00cc5a0 2023-11-28T13:01:36.705 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T13:01:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.705+0000 7efbf81a7700 1 -- 172.21.15.111:0/569723125 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efbdc0682f0 msgr2=0x7efbdc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.705+0000 7efbf81a7700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efbdc0682f0 0x7efbdc06a7a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7efbf00ae550 tx=0x7efbe4009040 comp rx=0 tx=0).stop 2023-11-28T13:01:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.705+0000 7efbf81a7700 1 -- 172.21.15.111:0/569723125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf00cc5a0 msgr2=0x7efbf01496e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.705+0000 7efbf81a7700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf00cc5a0 0x7efbf01496e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7efbe0007ef0 tx=0x7efbe000e3c0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.705+0000 7efbf81a7700 1 -- 172.21.15.111:0/569723125 shutdown_connections 2023-11-28T13:01:36.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.705+0000 7efbf81a7700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efbf00cfc90 0x7efbf00c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.705+0000 7efbf81a7700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf00cc5a0 0x7efbf01496e0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.705+0000 7efbf81a7700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efbdc0682f0 0x7efbdc06a7a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.705+0000 7efbf81a7700 1 --2- 172.21.15.111:0/569723125 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efbf00cdef0 0x7efbf00c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.705+0000 7efbf81a7700 1 -- 172.21.15.111:0/569723125 >> 172.21.15.111:0/569723125 conn(0x7efbf0098ee0 msgr2=0x7efbf006ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:36.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.706+0000 7efbf81a7700 1 -- 172.21.15.111:0/569723125 shutdown_connections 2023-11-28T13:01:36.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.706+0000 7efbf81a7700 1 -- 172.21.15.111:0/569723125 wait complete. 2023-11-28T13:01:36.719 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T13:01:36.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.840+0000 7fbb6a268700 1 Processor -- start 2023-11-28T13:01:36.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.840+0000 7fbb6a268700 1 -- start start 2023-11-28T13:01:36.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.841+0000 7fbb6a268700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fbb640cf010 0x7fbb640cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.841+0000 7fbb6a268700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb640cc7b0 con 0x7fbb640b2220 2023-11-28T13:01:36.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.841+0000 7fbb6a268700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb640b25f0 con 0x7fbb640cb380 2023-11-28T13:01:36.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.841+0000 7fbb6a268700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6408ae90 con 0x7fbb640cf010 2023-11-28T13:01:36.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.841+0000 7fbb63fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbb640b2220 0x7fbb640ce900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37414/0 (socket says 172.21.15.111:37414) 2023-11-28T13:01:36.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.841+0000 7fbb63fff700 1 -- 172.21.15.111:0/4271373850 learned_addr learned my addr 172.21.15.111:0/4271373850 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:36.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.841+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523432306 0 0) 0x7fbb640cc7b0 con 0x7fbb640b2220 2023-11-28T13:01:36.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb637fe700 1 --2- 172.21.15.111:0/4271373850 >> v2:172.21.15.149:3300/0 conn(0x7fbb640cf010 0x7fbb640cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb4c0035b0 con 0x7fbb640b2220 2023-11-28T13:01:36.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 267765255 0 0) 0x7fbb4c0035b0 con 0x7fbb640b2220 2023-11-28T13:01:36.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 >> v2:172.21.15.149:3300/0 conn(0x7fbb640cf010 msgr2=0x7fbb640cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:36.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 --2- 172.21.15.111:0/4271373850 >> v2:172.21.15.149:3300/0 conn(0x7fbb640cf010 0x7fbb640cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 >> v1:172.21.15.149:6789/0 conn(0x7fbb640cb380 legacy=0x7fbb640b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb640cca10 con 0x7fbb640b2220 2023-11-28T13:01:36.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fbb58001670 con 0x7fbb640b2220 2023-11-28T13:01:36.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 >> v1:172.21.15.111:6789/0 conn(0x7fbb640b2220 legacy=0x7fbb640ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 --2- 172.21.15.111:0/4271373850 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb4c003a00 0x7fbb4c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 --2- 172.21.15.111:0/4271373850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb4c004850 0x7fbb4c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 --2- 172.21.15.111:0/4271373850 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb4c005200 0x7fbb4c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6408ae90 con 0x7fbb4c004850 2023-11-28T13:01:36.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c00c080 con 0x7fbb4c005200 2023-11-28T13:01:36.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c00c240 con 0x7fbb4c003a00 2023-11-28T13:01:36.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb637fe700 1 --2- 172.21.15.111:0/4271373850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb4c004850 0x7fbb4c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb68805700 1 --2- 172.21.15.111:0/4271373850 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb4c005200 0x7fbb4c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb63fff700 1 --2- 172.21.15.111:0/4271373850 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb4c003a00 0x7fbb4c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb637fe700 1 -- 172.21.15.111:0/4271373850 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb4c003a00 msgr2=0x7fbb4c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:36.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb637fe700 1 --2- 172.21.15.111:0/4271373850 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb4c003a00 0x7fbb4c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb637fe700 1 -- 172.21.15.111:0/4271373850 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb4c005200 msgr2=0x7fbb4c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:36.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb637fe700 1 --2- 172.21.15.111:0/4271373850 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb4c005200 0x7fbb4c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb637fe700 1 -- 172.21.15.111:0/4271373850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c0035b0 con 0x7fbb4c004850 2023-11-28T13:01:36.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb68805700 1 --2- 172.21.15.111:0/4271373850 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb4c005200 0x7fbb4c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:36.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb68805700 1 -- 172.21.15.111:0/4271373850 reap_dead start 2023-11-28T13:01:36.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb637fe700 1 --2- 172.21.15.111:0/4271373850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb4c004850 0x7fbb4c004cc0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fbb54009b60 tx=0x7fbb4c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a7e23b4f4bcb09e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:36.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5400e6c0 con 0x7fbb4c004850 2023-11-28T13:01:36.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.842+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5400e820 con 0x7fbb4c004850 2023-11-28T13:01:36.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.843+0000 7fbb62ffd700 1 -- 172.21.15.111:0/4271373850 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5400eb90 con 0x7fbb4c004850 2023-11-28T13:01:36.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.843+0000 7fbb6a268700 1 -- 172.21.15.111:0/4271373850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb4c004850 msgr2=0x7fbb4c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.843+0000 7fbb6a268700 1 --2- 172.21.15.111:0/4271373850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb4c004850 0x7fbb4c004cc0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fbb54009b60 tx=0x7fbb4c005880 comp rx=0 tx=0).stop 2023-11-28T13:01:36.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.843+0000 7fbb6a268700 1 -- 172.21.15.111:0/4271373850 shutdown_connections 2023-11-28T13:01:36.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.843+0000 7fbb6a268700 1 --2- 172.21.15.111:0/4271373850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb4c004850 0x7fbb4c004cc0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.843+0000 7fbb6a268700 1 -- 172.21.15.111:0/4271373850 >> 172.21.15.111:0/4271373850 conn(0x7fbb6409a040 msgr2=0x7fbb640d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:36.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.843+0000 7fbb6a268700 1 -- 172.21.15.111:0/4271373850 shutdown_connections 2023-11-28T13:01:36.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.843+0000 7fbb6a268700 1 -- 172.21.15.111:0/4271373850 wait complete. 2023-11-28T13:01:36.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.845+0000 7fbb6a268700 1 Processor -- start 2023-11-28T13:01:36.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.845+0000 7fbb6a268700 1 -- start start 2023-11-28T13:01:36.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb6a268700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb4c004850 0x7fbb640b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb6a268700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb641f4160 0x7fbb640b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb6a268700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb641e4630 0x7fbb641e4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb6a268700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb641e4fe0 con 0x7fbb641e4630 2023-11-28T13:01:36.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb6a268700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6414ba90 con 0x7fbb641f4160 2023-11-28T13:01:36.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb6a268700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640b6b80 con 0x7fbb4c004850 2023-11-28T13:01:36.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb68805700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb641e4630 0x7fbb641e4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb68805700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb641e4630 0x7fbb641e4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36652/0 (socket says 172.21.15.111:36652) 2023-11-28T13:01:36.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb68805700 1 -- 172.21.15.111:0/214419704 learned_addr learned my addr 172.21.15.111:0/214419704 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:36.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb63fff700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb4c004850 0x7fbb640b5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb637fe700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb641f4160 0x7fbb640b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb68805700 1 -- 172.21.15.111:0/214419704 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb4c004850 msgr2=0x7fbb640b5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb68805700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb4c004850 0x7fbb640b5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb68805700 1 -- 172.21.15.111:0/214419704 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb641f4160 msgr2=0x7fbb640b6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:36.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb68805700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb641f4160 0x7fbb640b6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:36.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb68805700 1 -- 172.21.15.111:0/214419704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb640ccfd0 con 0x7fbb641e4630 2023-11-28T13:01:36.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb63fff700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb4c004850 0x7fbb640b5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:36.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb68805700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb641e4630 0x7fbb641e4aa0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fbb6408a1b0 tx=0x7fbb5c00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:36.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb60ff9700 1 -- 172.21.15.111:0/214419704 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c003c10 con 0x7fbb641e4630 2023-11-28T13:01:36.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb60ff9700 1 -- 172.21.15.111:0/214419704 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c003d70 con 0x7fbb641e4630 2023-11-28T13:01:36.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.846+0000 7fbb6a268700 1 -- 172.21.15.111:0/214419704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb640cccf0 con 0x7fbb641e4630 2023-11-28T13:01:36.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.847+0000 7fbb60ff9700 1 -- 172.21.15.111:0/214419704 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c005310 con 0x7fbb641e4630 2023-11-28T13:01:36.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.847+0000 7fbb6a268700 1 -- 172.21.15.111:0/214419704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb640cd2b0 con 0x7fbb641e4630 2023-11-28T13:01:36.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.847+0000 7fbb6a268700 1 -- 172.21.15.111:0/214419704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb64067170 con 0x7fbb641e4630 2023-11-28T13:01:36.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.848+0000 7fbb60ff9700 1 -- 172.21.15.111:0/214419704 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbb5c005470 con 0x7fbb641e4630 2023-11-28T13:01:36.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.848+0000 7fbb60ff9700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbb500682f0 0x7fbb5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:36.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.848+0000 7fbb60ff9700 1 -- 172.21.15.111:0/214419704 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fbb5c085df0 con 0x7fbb641e4630 2023-11-28T13:01:36.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.848+0000 7fbb63fff700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbb500682f0 0x7fbb5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:36.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.849+0000 7fbb63fff700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbb500682f0 0x7fbb5006a7a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbb58001050 tx=0x7fbb58009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:36.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:36.852+0000 7fbb60ff9700 1 -- 172.21.15.111:0/214419704 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbb5c052510 con 0x7fbb641e4630 2023-11-28T13:01:37.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.035+0000 7fbb6a268700 1 -- 172.21.15.111:0/214419704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fbb64150730 con 0x7fbb641e4630 2023-11-28T13:01:37.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.036+0000 7fbb60ff9700 1 -- 172.21.15.111:0/214419704 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v55) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbb5c055b30 con 0x7fbb641e4630 2023-11-28T13:01:37.037 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T13:01:37.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.037+0000 7fbb6a268700 1 -- 172.21.15.111:0/214419704 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbb500682f0 msgr2=0x7fbb5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.037+0000 7fbb6a268700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbb500682f0 0x7fbb5006a7a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbb58001050 tx=0x7fbb58009040 comp rx=0 tx=0).stop 2023-11-28T13:01:37.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.037+0000 7fbb6a268700 1 -- 172.21.15.111:0/214419704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb641e4630 msgr2=0x7fbb641e4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.037+0000 7fbb6a268700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb641e4630 0x7fbb641e4aa0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fbb6408a1b0 tx=0x7fbb5c00ed50 comp rx=0 tx=0).stop 2023-11-28T13:01:37.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.037+0000 7fbb6a268700 1 -- 172.21.15.111:0/214419704 shutdown_connections 2023-11-28T13:01:37.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.037+0000 7fbb6a268700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb641e4630 0x7fbb641e4aa0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.037+0000 7fbb6a268700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbb4c004850 0x7fbb640b5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.038+0000 7fbb6a268700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbb500682f0 0x7fbb5006a7a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.038+0000 7fbb6a268700 1 --2- 172.21.15.111:0/214419704 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbb641f4160 0x7fbb640b6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.038+0000 7fbb6a268700 1 -- 172.21.15.111:0/214419704 >> 172.21.15.111:0/214419704 conn(0x7fbb6409a040 msgr2=0x7fbb64070030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.038+0000 7fbb6a268700 1 -- 172.21.15.111:0/214419704 shutdown_connections 2023-11-28T13:01:37.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.038+0000 7fbb6a268700 1 -- 172.21.15.111:0/214419704 wait complete. 2023-11-28T13:01:37.051 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T13:01:37.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.172+0000 7f8819dfb700 1 Processor -- start 2023-11-28T13:01:37.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.172+0000 7f8819dfb700 1 -- start start 2023-11-28T13:01:37.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f8819dfb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f88140cb240 0x7f88141590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f8819dfb700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f881415d270 0x7f88140cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f8819dfb700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88140cd540 con 0x7f88140cdf10 2023-11-28T13:01:37.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f8819dfb700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8814159a20 con 0x7f88140cb240 2023-11-28T13:01:37.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f8819dfb700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8814159d00 con 0x7f881415d270 2023-11-28T13:01:37.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f88137fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f88140cb240 0x7f88141590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f8813fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f88140cdf10 0x7f881406e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37438/0 (socket says 172.21.15.111:37438) 2023-11-28T13:01:37.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f8813fff700 1 -- 172.21.15.111:0/3183420813 learned_addr learned my addr 172.21.15.111:0/3183420813 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:37.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f8812ffd700 1 --2- 172.21.15.111:0/3183420813 >> v2:172.21.15.149:3300/0 conn(0x7f881415d270 0x7f88140cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f88137fe700 1 -- 172.21.15.111:0/3183420813 >> v2:172.21.15.149:3300/0 conn(0x7f881415d270 msgr2=0x7f88140cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:37.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f88137fe700 1 --2- 172.21.15.111:0/3183420813 >> v2:172.21.15.149:3300/0 conn(0x7f881415d270 0x7f88140cd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f88137fe700 1 -- 172.21.15.111:0/3183420813 >> v1:172.21.15.111:6789/0 conn(0x7f88140cdf10 legacy=0x7f881406e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f88137fe700 1 -- 172.21.15.111:0/3183420813 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8814159fe0 con 0x7f88140cb240 2023-11-28T13:01:37.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f8812ffd700 1 --2- 172.21.15.111:0/3183420813 >> v2:172.21.15.149:3300/0 conn(0x7f881415d270 0x7f88140cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f88137fe700 1 --2- 172.21.15.111:0/3183420813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88140cb240 0x7f88141590a0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f88140b51f0 tx=0x7f8808009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a693c92dc737e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.173+0000 7f88127fc700 1 -- 172.21.15.111:0/3183420813 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808011830 con 0x7f88140cb240 2023-11-28T13:01:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.174+0000 7f88127fc700 1 -- 172.21.15.111:0/3183420813 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8808011990 con 0x7f88140cb240 2023-11-28T13:01:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.174+0000 7f88127fc700 1 -- 172.21.15.111:0/3183420813 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808011d20 con 0x7f88140cb240 2023-11-28T13:01:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.174+0000 7f8819dfb700 1 -- 172.21.15.111:0/3183420813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88140cb240 msgr2=0x7f88141590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.174+0000 7f8819dfb700 1 --2- 172.21.15.111:0/3183420813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88140cb240 0x7f88141590a0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f88140b51f0 tx=0x7f8808009b20 comp rx=0 tx=0).stop 2023-11-28T13:01:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.174+0000 7f8819dfb700 1 -- 172.21.15.111:0/3183420813 shutdown_connections 2023-11-28T13:01:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.174+0000 7f8819dfb700 1 --2- 172.21.15.111:0/3183420813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88140cb240 0x7f88141590a0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.174+0000 7f8819dfb700 1 --2- 172.21.15.111:0/3183420813 >> v2:172.21.15.149:3300/0 conn(0x7f881415d270 0x7f88140cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.174+0000 7f8819dfb700 1 -- 172.21.15.111:0/3183420813 >> 172.21.15.111:0/3183420813 conn(0x7f88140b0db0 msgr2=0x7f88140b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.174+0000 7f8819dfb700 1 -- 172.21.15.111:0/3183420813 shutdown_connections 2023-11-28T13:01:37.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.174+0000 7f8819dfb700 1 -- 172.21.15.111:0/3183420813 wait complete. 2023-11-28T13:01:37.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.175+0000 7f8819dfb700 1 Processor -- start 2023-11-28T13:01:37.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.176+0000 7f8819dfb700 1 -- start start 2023-11-28T13:01:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.176+0000 7f8819dfb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88140cb240 0x7f8814149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.176+0000 7f8819dfb700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88140cdf10 0x7f88141486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.176+0000 7f8819dfb700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f881415d270 0x7f88140c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.176+0000 7f8819dfb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8814148ea0 con 0x7f88140cb240 2023-11-28T13:01:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.176+0000 7f8819dfb700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881414a3b0 con 0x7f881415d270 2023-11-28T13:01:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.176+0000 7f8819dfb700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88141490b0 con 0x7f88140cdf10 2023-11-28T13:01:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.176+0000 7f88137fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88140cb240 0x7f8814149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.176+0000 7f88137fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88140cb240 0x7f8814149e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36662/0 (socket says 172.21.15.111:36662) 2023-11-28T13:01:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.176+0000 7f88137fe700 1 -- 172.21.15.111:0/1766141357 learned_addr learned my addr 172.21.15.111:0/1766141357 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.177+0000 7f8813fff700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f881415d270 0x7f88140c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.177+0000 7f8812ffd700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88140cdf10 0x7f88141486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.177+0000 7f88137fe700 1 -- 172.21.15.111:0/1766141357 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88140cdf10 msgr2=0x7f88141486f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.177+0000 7f88137fe700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88140cdf10 0x7f88141486f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.177+0000 7f88137fe700 1 -- 172.21.15.111:0/1766141357 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f881415d270 msgr2=0x7f88140c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.177+0000 7f88137fe700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f881415d270 0x7f88140c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.177+0000 7f88137fe700 1 -- 172.21.15.111:0/1766141357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88080094a0 con 0x7f88140cb240 2023-11-28T13:01:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.177+0000 7f88137fe700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88140cb240 0x7f8814149e70 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f8808000c00 tx=0x7f8808004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.177+0000 7f87fbfff700 1 -- 172.21.15.111:0/1766141357 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88080036a0 con 0x7f88140cb240 2023-11-28T13:01:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.177+0000 7f87fbfff700 1 -- 172.21.15.111:0/1766141357 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880801f070 con 0x7f88140cb240 2023-11-28T13:01:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.177+0000 7f8819dfb700 1 -- 172.21.15.111:0/1766141357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88141595e0 con 0x7f88140cb240 2023-11-28T13:01:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.177+0000 7f87fbfff700 1 -- 172.21.15.111:0/1766141357 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808003a60 con 0x7f88140cb240 2023-11-28T13:01:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.177+0000 7f8819dfb700 1 -- 172.21.15.111:0/1766141357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88140cd1d0 con 0x7f88140cb240 2023-11-28T13:01:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.178+0000 7f8819dfb700 1 -- 172.21.15.111:0/1766141357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8814159770 con 0x7f88140cb240 2023-11-28T13:01:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.178+0000 7f87fbfff700 1 -- 172.21.15.111:0/1766141357 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f880800f050 con 0x7f88140cb240 2023-11-28T13:01:37.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.179+0000 7f87fbfff700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f88000682f0 0x7f880006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.179+0000 7f87fbfff700 1 -- 172.21.15.111:0/1766141357 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8808021070 con 0x7f88140cb240 2023-11-28T13:01:37.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.179+0000 7f8812ffd700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f88000682f0 0x7f880006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.179+0000 7f8812ffd700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f88000682f0 0x7f880006a7a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f88140ae610 tx=0x7f8804008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:37.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.183+0000 7f87fbfff700 1 -- 172.21.15.111:0/1766141357 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8808053ed0 con 0x7f88140cb240 2023-11-28T13:01:37.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.364+0000 7f8819dfb700 1 -- 172.21.15.111:0/1766141357 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8814067170 con 0x7f88140cb240 2023-11-28T13:01:37.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.364+0000 7f87fbfff700 1 -- 172.21.15.111:0/1766141357 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8808015020 con 0x7f88140cb240 2023-11-28T13:01:37.366 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T13:01:37.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.367+0000 7f8819dfb700 1 -- 172.21.15.111:0/1766141357 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f88000682f0 msgr2=0x7f880006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.367+0000 7f8819dfb700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f88000682f0 0x7f880006a7a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f88140ae610 tx=0x7f8804008040 comp rx=0 tx=0).stop 2023-11-28T13:01:37.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.367+0000 7f8819dfb700 1 -- 172.21.15.111:0/1766141357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88140cb240 msgr2=0x7f8814149e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.367+0000 7f8819dfb700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88140cb240 0x7f8814149e70 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f8808000c00 tx=0x7f8808004750 comp rx=0 tx=0).stop 2023-11-28T13:01:37.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.367+0000 7f8819dfb700 1 -- 172.21.15.111:0/1766141357 shutdown_connections 2023-11-28T13:01:37.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.367+0000 7f8819dfb700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f88000682f0 0x7f880006a7a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.367+0000 7f8819dfb700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f881415d270 0x7f88140c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.367+0000 7f8819dfb700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88140cb240 0x7f8814149e70 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.367+0000 7f8819dfb700 1 --2- 172.21.15.111:0/1766141357 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88140cdf10 0x7f88141486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.367+0000 7f8819dfb700 1 -- 172.21.15.111:0/1766141357 >> 172.21.15.111:0/1766141357 conn(0x7f88140b0db0 msgr2=0x7f8814071bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:37.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.367+0000 7f8819dfb700 1 -- 172.21.15.111:0/1766141357 shutdown_connections 2023-11-28T13:01:37.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.367+0000 7f8819dfb700 1 -- 172.21.15.111:0/1766141357 wait complete. 2023-11-28T13:01:37.380 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T13:01:37.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.502+0000 7f879156c700 1 Processor -- start 2023-11-28T13:01:37.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.502+0000 7f879156c700 1 -- start start 2023-11-28T13:01:37.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.502+0000 7f879156c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f878c15cbf0 0x7f878c0b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.502+0000 7f879156c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f878c0b40a0 0x7f878c06b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.502+0000 7f879156c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f878c0cffd0 0x7f878c06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.502+0000 7f879156c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878c0cfdb0 con 0x7f878c0b40a0 2023-11-28T13:01:37.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.502+0000 7f879156c700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878c0cc740 con 0x7f878c0cffd0 2023-11-28T13:01:37.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.502+0000 7f879156c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c0b35e0 con 0x7f878c15cbf0 2023-11-28T13:01:37.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f878a7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f878c0b40a0 0x7f878c06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f878a7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f878c0b40a0 0x7f878c06b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36670/0 (socket says 172.21.15.111:36670) 2023-11-28T13:01:37.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f878a7fc700 1 -- 172.21.15.111:0/4192224681 learned_addr learned my addr 172.21.15.111:0/4192224681 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:37.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f878b7fe700 1 --2- 172.21.15.111:0/4192224681 >> v2:172.21.15.149:3300/0 conn(0x7f878c0cffd0 0x7f878c06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f878affd700 1 --2- 172.21.15.111:0/4192224681 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f878c15cbf0 0x7f878c0b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f878a7fc700 1 -- 172.21.15.111:0/4192224681 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f878c15cbf0 msgr2=0x7f878c0b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:37.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f878a7fc700 1 --2- 172.21.15.111:0/4192224681 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f878c15cbf0 0x7f878c0b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f878a7fc700 1 -- 172.21.15.111:0/4192224681 >> v2:172.21.15.149:3300/0 conn(0x7f878c0cffd0 msgr2=0x7f878c06b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:37.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f878a7fc700 1 --2- 172.21.15.111:0/4192224681 >> v2:172.21.15.149:3300/0 conn(0x7f878c0cffd0 0x7f878c06b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f878a7fc700 1 -- 172.21.15.111:0/4192224681 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c0cbe10 con 0x7f878c0b40a0 2023-11-28T13:01:37.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f878affd700 1 --2- 172.21.15.111:0/4192224681 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f878c15cbf0 0x7f878c0b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:37.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f878a7fc700 1 --2- 172.21.15.111:0/4192224681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f878c0b40a0 0x7f878c06b380 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f878c09a6e0 tx=0x7f877c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da0f3bfed431c23 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:37.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f8789ffb700 1 -- 172.21.15.111:0/4192224681 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c011830 con 0x7f878c0b40a0 2023-11-28T13:01:37.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f8789ffb700 1 -- 172.21.15.111:0/4192224681 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877c011990 con 0x7f878c0b40a0 2023-11-28T13:01:37.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.503+0000 7f8789ffb700 1 -- 172.21.15.111:0/4192224681 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c011d20 con 0x7f878c0b40a0 2023-11-28T13:01:37.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.504+0000 7f879156c700 1 -- 172.21.15.111:0/4192224681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f878c0b40a0 msgr2=0x7f878c06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.504+0000 7f879156c700 1 --2- 172.21.15.111:0/4192224681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f878c0b40a0 0x7f878c06b380 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f878c09a6e0 tx=0x7f877c009bd0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.504+0000 7f879156c700 1 -- 172.21.15.111:0/4192224681 shutdown_connections 2023-11-28T13:01:37.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.504+0000 7f879156c700 1 --2- 172.21.15.111:0/4192224681 >> v2:172.21.15.149:3300/0 conn(0x7f878c0cffd0 0x7f878c06b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.504+0000 7f879156c700 1 --2- 172.21.15.111:0/4192224681 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f878c15cbf0 0x7f878c0b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.504+0000 7f879156c700 1 --2- 172.21.15.111:0/4192224681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f878c0b40a0 0x7f878c06b380 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.504+0000 7f879156c700 1 -- 172.21.15.111:0/4192224681 >> 172.21.15.111:0/4192224681 conn(0x7f878c09a040 msgr2=0x7f878c0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:37.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.504+0000 7f879156c700 1 -- 172.21.15.111:0/4192224681 shutdown_connections 2023-11-28T13:01:37.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.504+0000 7f879156c700 1 -- 172.21.15.111:0/4192224681 wait complete. 2023-11-28T13:01:37.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.505+0000 7f879156c700 1 Processor -- start 2023-11-28T13:01:37.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.505+0000 7f879156c700 1 -- start start 2023-11-28T13:01:37.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f879156c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f878c0b40a0 0x7f878c1efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f879156c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f878c0cffd0 0x7f878c15cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f879156c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f878c15cbf0 0x7f878c15a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f879156c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c1f00a0 con 0x7f878c0b40a0 2023-11-28T13:01:37.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f879156c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c15d930 con 0x7f878c0cffd0 2023-11-28T13:01:37.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f879156c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c15dc10 con 0x7f878c15cbf0 2023-11-28T13:01:37.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f878affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f878c0b40a0 0x7f878c1efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f878affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f878c0b40a0 0x7f878c1efb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36676/0 (socket says 172.21.15.111:36676) 2023-11-28T13:01:37.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f878affd700 1 -- 172.21.15.111:0/2811444073 learned_addr learned my addr 172.21.15.111:0/2811444073 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:37.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f878a7fc700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f878c0cffd0 0x7f878c15cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f878b7fe700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f878c15cbf0 0x7f878c15a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f878affd700 1 -- 172.21.15.111:0/2811444073 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f878c15cbf0 msgr2=0x7f878c15a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f878affd700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f878c15cbf0 0x7f878c15a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f878affd700 1 -- 172.21.15.111:0/2811444073 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f878c0cffd0 msgr2=0x7f878c15cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f878affd700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f878c0cffd0 0x7f878c15cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f878affd700 1 -- 172.21.15.111:0/2811444073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877c009550 con 0x7f878c0b40a0 2023-11-28T13:01:37.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.506+0000 7f878a7fc700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f878c0cffd0 0x7f878c15cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:37.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.507+0000 7f878affd700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f878c0b40a0 0x7f878c1efb60 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f878000e810 tx=0x7f878000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:37.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.507+0000 7f87737fe700 1 -- 172.21.15.111:0/2811444073 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878000c020 con 0x7f878c0b40a0 2023-11-28T13:01:37.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.507+0000 7f87737fe700 1 -- 172.21.15.111:0/2811444073 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8780004250 con 0x7f878c0b40a0 2023-11-28T13:01:37.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.507+0000 7f879156c700 1 -- 172.21.15.111:0/2811444073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878c0cc3d0 con 0x7f878c0b40a0 2023-11-28T13:01:37.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.507+0000 7f87737fe700 1 -- 172.21.15.111:0/2811444073 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87800036a0 con 0x7f878c0b40a0 2023-11-28T13:01:37.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.507+0000 7f879156c700 1 -- 172.21.15.111:0/2811444073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f878c0cc0f0 con 0x7f878c0b40a0 2023-11-28T13:01:37.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.508+0000 7f879156c700 1 -- 172.21.15.111:0/2811444073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f878c0d06f0 con 0x7f878c0b40a0 2023-11-28T13:01:37.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.508+0000 7f87737fe700 1 -- 172.21.15.111:0/2811444073 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8780003800 con 0x7f878c0b40a0 2023-11-28T13:01:37.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.508+0000 7f87737fe700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87780682f0 0x7f877806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.509+0000 7f87737fe700 1 -- 172.21.15.111:0/2811444073 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8780085c40 con 0x7f878c0b40a0 2023-11-28T13:01:37.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.509+0000 7f878a7fc700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87780682f0 0x7f877806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.509+0000 7f878a7fc700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87780682f0 0x7f877806a7a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f878c0703f0 tx=0x7f877c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:37.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.513+0000 7f87737fe700 1 -- 172.21.15.111:0/2811444073 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8780052270 con 0x7f878c0b40a0 2023-11-28T13:01:37.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.694+0000 7f879156c700 1 -- 172.21.15.111:0/2811444073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f878c067880 con 0x7f878c0b40a0 2023-11-28T13:01:37.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.694+0000 7f87737fe700 1 -- 172.21.15.111:0/2811444073 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7f878000f040 con 0x7f878c0b40a0 2023-11-28T13:01:37.696 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T13:01:37.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.696+0000 7f879156c700 1 -- 172.21.15.111:0/2811444073 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87780682f0 msgr2=0x7f877806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.696+0000 7f879156c700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87780682f0 0x7f877806a7a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f878c0703f0 tx=0x7f877c008030 comp rx=0 tx=0).stop 2023-11-28T13:01:37.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.696+0000 7f879156c700 1 -- 172.21.15.111:0/2811444073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f878c0b40a0 msgr2=0x7f878c1efb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.696+0000 7f879156c700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f878c0b40a0 0x7f878c1efb60 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f878000e810 tx=0x7f878000ebd0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.697+0000 7f879156c700 1 -- 172.21.15.111:0/2811444073 shutdown_connections 2023-11-28T13:01:37.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.697+0000 7f879156c700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f878c15cbf0 0x7f878c15a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.697+0000 7f879156c700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f878c0b40a0 0x7f878c1efb60 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.697+0000 7f879156c700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87780682f0 0x7f877806a7a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.697+0000 7f879156c700 1 --2- 172.21.15.111:0/2811444073 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f878c0cffd0 0x7f878c15cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.697+0000 7f879156c700 1 -- 172.21.15.111:0/2811444073 >> 172.21.15.111:0/2811444073 conn(0x7f878c09a040 msgr2=0x7f878c0d09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:37.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.697+0000 7f879156c700 1 -- 172.21.15.111:0/2811444073 shutdown_connections 2023-11-28T13:01:37.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.697+0000 7f879156c700 1 -- 172.21.15.111:0/2811444073 wait complete. 2023-11-28T13:01:37.710 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T13:01:37.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.829+0000 7f3a1aaaa700 1 Processor -- start 2023-11-28T13:01:37.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.829+0000 7f3a1aaaa700 1 -- start start 2023-11-28T13:01:37.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.830+0000 7f3a1aaaa700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a140b0d70 0x7f3a141604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.830+0000 7f3a1aaaa700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3a140b1140 0x7f3a140ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.830+0000 7f3a1aaaa700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1415cd90 con 0x7f3a140b15b0 2023-11-28T13:01:37.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.830+0000 7f3a1aaaa700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a141613e0 con 0x7f3a140b1140 2023-11-28T13:01:37.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.830+0000 7f3a1aaaa700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a14160b40 con 0x7f3a140b0d70 2023-11-28T13:01:37.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.830+0000 7f3a13fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3a140b1140 0x7f3a140ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.830+0000 7f3a13fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3a140b1140 0x7f3a140ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36684/0 (socket says 172.21.15.111:36684) 2023-11-28T13:01:37.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.830+0000 7f3a13fff700 1 -- 172.21.15.111:0/3892509184 learned_addr learned my addr 172.21.15.111:0/3892509184 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.830+0000 7f3a18846700 1 --2- 172.21.15.111:0/3892509184 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a140b0d70 0x7f3a141604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.830+0000 7f3a13fff700 1 -- 172.21.15.111:0/3892509184 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a140b0d70 msgr2=0x7f3a141604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.830+0000 7f3a13fff700 1 --2- 172.21.15.111:0/3892509184 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a140b0d70 0x7f3a141604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.830+0000 7f3a13fff700 1 -- 172.21.15.111:0/3892509184 >> v1:172.21.15.149:6789/0 conn(0x7f3a140b15b0 legacy=0x7f3a140cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a13fff700 1 -- 172.21.15.111:0/3892509184 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a14160e20 con 0x7f3a140b1140 2023-11-28T13:01:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a18846700 1 --2- 172.21.15.111:0/3892509184 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a140b0d70 0x7f3a141604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a13fff700 1 --2- 172.21.15.111:0/3892509184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a140b1140 0x7f3a140ce6c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3a000098f0 tx=0x7f3a00009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b29738cb054a17 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a137fe700 1 -- 172.21.15.111:0/3892509184 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0000b8a0 con 0x7f3a140b1140 2023-11-28T13:01:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a137fe700 1 -- 172.21.15.111:0/3892509184 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0000ba00 con 0x7f3a140b1140 2023-11-28T13:01:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a137fe700 1 -- 172.21.15.111:0/3892509184 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0000bd90 con 0x7f3a140b1140 2023-11-28T13:01:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/3892509184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a140b1140 msgr2=0x7f3a140ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a1aaaa700 1 --2- 172.21.15.111:0/3892509184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a140b1140 0x7f3a140ce6c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3a000098f0 tx=0x7f3a00009c00 comp rx=0 tx=0).stop 2023-11-28T13:01:37.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/3892509184 shutdown_connections 2023-11-28T13:01:37.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a1aaaa700 1 --2- 172.21.15.111:0/3892509184 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a140b0d70 0x7f3a141604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a1aaaa700 1 --2- 172.21.15.111:0/3892509184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a140b1140 0x7f3a140ce6c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/3892509184 >> 172.21.15.111:0/3892509184 conn(0x7f3a14098ee0 msgr2=0x7f3a1406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:37.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/3892509184 shutdown_connections 2023-11-28T13:01:37.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.831+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/3892509184 wait complete. 2023-11-28T13:01:37.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.833+0000 7f3a1aaaa700 1 Processor -- start 2023-11-28T13:01:37.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.833+0000 7f3a1aaaa700 1 -- start start 2023-11-28T13:01:37.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.833+0000 7f3a1aaaa700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a140b0d70 0x7f3a14159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.833+0000 7f3a1aaaa700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a140b1140 0x7f3a140ad980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.833+0000 7f3a1aaaa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a140b15b0 0x7f3a14153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a1aaaa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a140b2aa0 con 0x7f3a140b15b0 2023-11-28T13:01:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a1aaaa700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1415f320 con 0x7f3a140b0d70 2023-11-28T13:01:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a1aaaa700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1415ea80 con 0x7f3a140b1140 2023-11-28T13:01:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a19047700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a140b15b0 0x7f3a14153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a19047700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a140b15b0 0x7f3a14153690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36694/0 (socket says 172.21.15.111:36694) 2023-11-28T13:01:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a19047700 1 -- 172.21.15.111:0/2789665417 learned_addr learned my addr 172.21.15.111:0/2789665417 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a18846700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a140b0d70 0x7f3a14159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a13fff700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a140b1140 0x7f3a140ad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a19047700 1 -- 172.21.15.111:0/2789665417 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a140b1140 msgr2=0x7f3a140ad980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a19047700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a140b1140 0x7f3a140ad980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a19047700 1 -- 172.21.15.111:0/2789665417 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a140b0d70 msgr2=0x7f3a14159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a19047700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a140b0d70 0x7f3a14159f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a19047700 1 -- 172.21.15.111:0/2789665417 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a00009550 con 0x7f3a140b15b0 2023-11-28T13:01:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a19047700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a140b15b0 0x7f3a14153690 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00caf0 tx=0x7f3a0c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a117fa700 1 -- 172.21.15.111:0/2789665417 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c003c00 con 0x7f3a140b15b0 2023-11-28T13:01:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/2789665417 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a140af340 con 0x7f3a140b15b0 2023-11-28T13:01:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/2789665417 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a14161100 con 0x7f3a140b15b0 2023-11-28T13:01:37.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a117fa700 1 -- 172.21.15.111:0/2789665417 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0c001bd0 con 0x7f3a140b15b0 2023-11-28T13:01:37.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.834+0000 7f3a117fa700 1 -- 172.21.15.111:0/2789665417 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c00a8d0 con 0x7f3a140b15b0 2023-11-28T13:01:37.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.835+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/2789665417 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a140b2c10 con 0x7f3a140b15b0 2023-11-28T13:01:37.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.835+0000 7f3a117fa700 1 -- 172.21.15.111:0/2789665417 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3a0c010050 con 0x7f3a140b15b0 2023-11-28T13:01:37.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.836+0000 7f3a117fa700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f39f80682f0 0x7f39f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:37.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.836+0000 7f3a117fa700 1 -- 172.21.15.111:0/2789665417 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f3a0c085b60 con 0x7f3a140b15b0 2023-11-28T13:01:37.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.836+0000 7f3a18846700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f39f80682f0 0x7f39f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:37.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.836+0000 7f3a18846700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f39f80682f0 0x7f39f806a7a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3a140ae4e0 tx=0x7f3a08008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:37.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:37.840+0000 7f3a117fa700 1 -- 172.21.15.111:0/2789665417 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3a0c052200 con 0x7f3a140b15b0 2023-11-28T13:01:38.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.038+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/2789665417 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3a14067170 con 0x7f3a140b15b0 2023-11-28T13:01:38.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.038+0000 7f3a117fa700 1 -- 172.21.15.111:0/2789665417 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3a0c055820 con 0x7f3a140b15b0 2023-11-28T13:01:38.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.040+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/2789665417 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f39f80682f0 msgr2=0x7f39f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.040+0000 7f3a1aaaa700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f39f80682f0 0x7f39f806a7a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3a140ae4e0 tx=0x7f3a08008040 comp rx=0 tx=0).stop 2023-11-28T13:01:38.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.040+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/2789665417 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a140b15b0 msgr2=0x7f3a14153690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.040+0000 7f3a1aaaa700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a140b15b0 0x7f3a14153690 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00caf0 tx=0x7f3a0c00ceb0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.040+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/2789665417 shutdown_connections 2023-11-28T13:01:38.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.040+0000 7f3a1aaaa700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a140b15b0 0x7f3a14153690 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.040+0000 7f3a1aaaa700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f39f80682f0 0x7f39f806a7a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.040+0000 7f3a1aaaa700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a140b0d70 0x7f3a14159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.040+0000 7f3a1aaaa700 1 --2- 172.21.15.111:0/2789665417 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a140b1140 0x7f3a140ad980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.040+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/2789665417 >> 172.21.15.111:0/2789665417 conn(0x7f3a14098ee0 msgr2=0x7f3a1415b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:38.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.040+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/2789665417 shutdown_connections 2023-11-28T13:01:38.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.040+0000 7f3a1aaaa700 1 -- 172.21.15.111:0/2789665417 wait complete. 2023-11-28T13:01:38.043 INFO:teuthology.orchestra.run.smithi111.stderr:listed 0 entries 2023-11-28T13:01:38.054 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T13:01:38.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:01:38.054 DEBUG:teuthology.orchestra.run.smithi111:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T13:01:38.113 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T13:01:38.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:01:38.114 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T13:01:38.151 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T13:01:38.151 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:01:38.170 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8026740k,nr_inodes=2006685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16040 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=1609048k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-28T13:01:38.170 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:01:38.170 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:01:38.210 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284268k,nr_inodes=4071067,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18839 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-28T13:01:38.211 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T13:01:38.211 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:01:38.229 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8026740k,nr_inodes=2006685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16040 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=1609048k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-28T13:01:38.229 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:01:38.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.369+0000 7fbd6734e700 1 Processor -- start 2023-11-28T13:01:38.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.369+0000 7fbd6734e700 1 -- start start 2023-11-28T13:01:38.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.370+0000 7fbd6734e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbd600b0d70 0x7fbd601604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:38.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.370+0000 7fbd6734e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6015cd90 con 0x7fbd600b1140 2023-11-28T13:01:38.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.370+0000 7fbd6734e700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd601613e0 con 0x7fbd600cc550 2023-11-28T13:01:38.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.370+0000 7fbd6734e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd600cedd0 con 0x7fbd600b0d70 2023-11-28T13:01:38.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.370+0000 7fbd650ea700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbd600b0d70 0x7fbd601604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:38.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.370+0000 7fbd648e9700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbd600b1140 0x7fbd600ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37480/0 (socket says 172.21.15.111:37480) 2023-11-28T13:01:38.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.370+0000 7fbd648e9700 1 -- 172.21.15.111:0/340409226 learned_addr learned my addr 172.21.15.111:0/340409226 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:38.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.370+0000 7fbd650ea700 1 -- 172.21.15.111:0/340409226 >> v1:172.21.15.149:6789/0 conn(0x7fbd600cc550 legacy=0x7fbd600af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.370+0000 7fbd650ea700 1 -- 172.21.15.111:0/340409226 >> v1:172.21.15.111:6789/0 conn(0x7fbd600b1140 legacy=0x7fbd600ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.370+0000 7fbd650ea700 1 -- 172.21.15.111:0/340409226 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd60160b40 con 0x7fbd600b0d70 2023-11-28T13:01:38.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.370+0000 7fbd57fff700 1 -- 172.21.15.111:0/340409226 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272256698 0 0) 0x7fbd50001110 con 0x7fbd600b1140 2023-11-28T13:01:38.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.370+0000 7fbd650ea700 1 --2- 172.21.15.111:0/340409226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd600b0d70 0x7fbd601604b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009820 tx=0x7fbd5c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6608703d4fe15989 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:38.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.371+0000 7fbd57fff700 1 -- 172.21.15.111:0/340409226 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c005e30 con 0x7fbd600b0d70 2023-11-28T13:01:38.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.371+0000 7fbd57fff700 1 -- 172.21.15.111:0/340409226 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5c006030 con 0x7fbd600b0d70 2023-11-28T13:01:38.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.371+0000 7fbd57fff700 1 -- 172.21.15.111:0/340409226 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c0063e0 con 0x7fbd600b0d70 2023-11-28T13:01:38.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.371+0000 7fbd6734e700 1 -- 172.21.15.111:0/340409226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd600b0d70 msgr2=0x7fbd601604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.371+0000 7fbd6734e700 1 --2- 172.21.15.111:0/340409226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd600b0d70 0x7fbd601604b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009820 tx=0x7fbd5c009b30 comp rx=0 tx=0).stop 2023-11-28T13:01:38.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.371+0000 7fbd6734e700 1 -- 172.21.15.111:0/340409226 shutdown_connections 2023-11-28T13:01:38.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.371+0000 7fbd6734e700 1 --2- 172.21.15.111:0/340409226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd600b0d70 0x7fbd601604b0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.371+0000 7fbd6734e700 1 -- 172.21.15.111:0/340409226 >> 172.21.15.111:0/340409226 conn(0x7fbd60098ee0 msgr2=0x7fbd6006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:38.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.371+0000 7fbd6734e700 1 -- 172.21.15.111:0/340409226 shutdown_connections 2023-11-28T13:01:38.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.371+0000 7fbd6734e700 1 -- 172.21.15.111:0/340409226 wait complete. 2023-11-28T13:01:38.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.372+0000 7fbd6734e700 1 Processor -- start 2023-11-28T13:01:38.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.372+0000 7fbd6734e700 1 -- start start 2023-11-28T13:01:38.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd6734e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbd600b0d70 0x7fbd601e6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:38.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd6734e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd600b1140 0x7fbd60155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:38.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd6734e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbd600cc550 0x7fbd600c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:38.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd6734e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd600b2970 con 0x7fbd600b1140 2023-11-28T13:01:38.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd6734e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd600b2c10 con 0x7fbd600b0d70 2023-11-28T13:01:38.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd6734e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd600b2ef0 con 0x7fbd600cc550 2023-11-28T13:01:38.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd648e9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd600b1140 0x7fbd60155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:38.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd648e9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd600b1140 0x7fbd60155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36710/0 (socket says 172.21.15.111:36710) 2023-11-28T13:01:38.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd648e9700 1 -- 172.21.15.111:0/2146754976 learned_addr learned my addr 172.21.15.111:0/2146754976 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:38.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd650ea700 1 --2- 172.21.15.111:0/2146754976 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbd600b0d70 0x7fbd601e6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:38.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd648e9700 1 -- 172.21.15.111:0/2146754976 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbd600cc550 msgr2=0x7fbd600c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd648e9700 1 --2- 172.21.15.111:0/2146754976 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbd600cc550 0x7fbd600c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd648e9700 1 -- 172.21.15.111:0/2146754976 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbd600b0d70 msgr2=0x7fbd601e6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd648e9700 1 --2- 172.21.15.111:0/2146754976 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbd600b0d70 0x7fbd601e6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.373+0000 7fbd648e9700 1 -- 172.21.15.111:0/2146754976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c009430 con 0x7fbd600b1140 2023-11-28T13:01:38.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.374+0000 7fbd648e9700 1 --2- 172.21.15.111:0/2146754976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd600b1140 0x7fbd60155ab0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000c520 tx=0x7fbd5000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:38.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.374+0000 7fbd55ffb700 1 -- 172.21.15.111:0/2146754976 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50004c80 con 0x7fbd600b1140 2023-11-28T13:01:38.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.374+0000 7fbd55ffb700 1 -- 172.21.15.111:0/2146754976 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd50004e20 con 0x7fbd600b1140 2023-11-28T13:01:38.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.374+0000 7fbd55ffb700 1 -- 172.21.15.111:0/2146754976 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50007310 con 0x7fbd600b1140 2023-11-28T13:01:38.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.374+0000 7fbd6734e700 1 -- 172.21.15.111:0/2146754976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd60161100 con 0x7fbd600b1140 2023-11-28T13:01:38.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.374+0000 7fbd6734e700 1 -- 172.21.15.111:0/2146754976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd60160e20 con 0x7fbd600b1140 2023-11-28T13:01:38.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.375+0000 7fbd6734e700 1 -- 172.21.15.111:0/2146754976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd600af340 con 0x7fbd600b1140 2023-11-28T13:01:38.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.375+0000 7fbd55ffb700 1 -- 172.21.15.111:0/2146754976 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbd5001a020 con 0x7fbd600b1140 2023-11-28T13:01:38.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.376+0000 7fbd55ffb700 1 --2- 172.21.15.111:0/2146754976 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbd44068320 0x7fbd4406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:38.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.376+0000 7fbd55ffb700 1 -- 172.21.15.111:0/2146754976 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fbd50085960 con 0x7fbd600b1140 2023-11-28T13:01:38.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.376+0000 7fbd650ea700 1 --2- 172.21.15.111:0/2146754976 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbd44068320 0x7fbd4406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:38.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.376+0000 7fbd650ea700 1 --2- 172.21.15.111:0/2146754976 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbd44068320 0x7fbd4406a7d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbd600c46f0 tx=0x7fbd5c008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:38.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.380+0000 7fbd55ffb700 1 -- 172.21.15.111:0/2146754976 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbd50051f30 con 0x7fbd600b1140 2023-11-28T13:01:38.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.608+0000 7fbd6734e700 1 -- 172.21.15.111:0/2146754976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd6015cd90 con 0x7fbd600b1140 2023-11-28T13:01:38.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.608+0000 7fbd55ffb700 1 -- 172.21.15.111:0/2146754976 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4660 (secure 0 0 0) 0x7fbd6015cd90 con 0x7fbd600b1140 2023-11-28T13:01:38.610 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:38.610 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2072621958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2072621958},{"type":"v1","addr":"172.21.15.149:6834","nonce":2072621958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3348427170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3348427170},{"type":"v1","addr":"172.21.15.149:6835","nonce":3348427170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1745443298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1745443298},{"type":"v1","addr":"172.21.15.111:6836","nonce":1745443298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:56.272312+0000","modified":"2023-11-28T13:00:59.284265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5814":{"gid":5814,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/4146274267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":4146274267},{"type":"v1","addr":"172.21.15.111:6837","nonce":4146274267}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T13:01:38.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.610+0000 7fbd6734e700 1 -- 172.21.15.111:0/2146754976 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbd44068320 msgr2=0x7fbd4406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.610+0000 7fbd6734e700 1 --2- 172.21.15.111:0/2146754976 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbd44068320 0x7fbd4406a7d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbd600c46f0 tx=0x7fbd5c008070 comp rx=0 tx=0).stop 2023-11-28T13:01:38.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.610+0000 7fbd6734e700 1 -- 172.21.15.111:0/2146754976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd600b1140 msgr2=0x7fbd60155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.610+0000 7fbd6734e700 1 --2- 172.21.15.111:0/2146754976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd600b1140 0x7fbd60155ab0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000c520 tx=0x7fbd5000c8e0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.610+0000 7fbd6734e700 1 -- 172.21.15.111:0/2146754976 shutdown_connections 2023-11-28T13:01:38.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.610+0000 7fbd6734e700 1 --2- 172.21.15.111:0/2146754976 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbd600cc550 0x7fbd600c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.610+0000 7fbd6734e700 1 --2- 172.21.15.111:0/2146754976 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbd44068320 0x7fbd4406a7d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.610+0000 7fbd6734e700 1 --2- 172.21.15.111:0/2146754976 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbd600b0d70 0x7fbd601e6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.610+0000 7fbd6734e700 1 --2- 172.21.15.111:0/2146754976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd600b1140 0x7fbd60155ab0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.610+0000 7fbd6734e700 1 -- 172.21.15.111:0/2146754976 >> 172.21.15.111:0/2146754976 conn(0x7fbd60098ee0 msgr2=0x7fbd6006e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:38.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.610+0000 7fbd6734e700 1 -- 172.21.15.111:0/2146754976 shutdown_connections 2023-11-28T13:01:38.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.610+0000 7fbd6734e700 1 -- 172.21.15.111:0/2146754976 wait complete. 2023-11-28T13:01:38.613 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 25 2023-11-28T13:01:38.624 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:01:38.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.743+0000 7fe3a1c6a700 1 Processor -- start 2023-11-28T13:01:38.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.743+0000 7fe3a1c6a700 1 -- start start 2023-11-28T13:01:38.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe3a1c6a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe39c15c6c0 0x7fe39c0afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:38.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe3a1c6a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe39c0b0440 0x7fe39c0b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:38.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe3a1c6a700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c0cc750 con 0x7fe39c1602a0 2023-11-28T13:01:38.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe3a1c6a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39c0b0df0 con 0x7fe39c15c6c0 2023-11-28T13:01:38.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe3a1c6a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c0cca30 con 0x7fe39c0b0440 2023-11-28T13:01:38.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe39b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe39c15c6c0 0x7fe39c0afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:38.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe39b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe39c15c6c0 0x7fe39c0afb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36716/0 (socket says 172.21.15.111:36716) 2023-11-28T13:01:38.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe39b7fe700 1 -- 172.21.15.111:0/533068380 learned_addr learned my addr 172.21.15.111:0/533068380 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:38.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe39affd700 1 --2- 172.21.15.111:0/533068380 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe39c0b0440 0x7fe39c0b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:38.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe39b7fe700 1 -- 172.21.15.111:0/533068380 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe39c0b0440 msgr2=0x7fe39c0b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:38.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe39b7fe700 1 --2- 172.21.15.111:0/533068380 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe39c0b0440 0x7fe39c0b08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe39b7fe700 1 -- 172.21.15.111:0/533068380 >> v1:172.21.15.149:6789/0 conn(0x7fe39c1602a0 legacy=0x7fe39c160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:38.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe39b7fe700 1 -- 172.21.15.111:0/533068380 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c0ccd10 con 0x7fe39c15c6c0 2023-11-28T13:01:38.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe39b7fe700 1 --2- 172.21.15.111:0/533068380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe39c15c6c0 0x7fe39c0afb50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe3900098f0 tx=0x7fe390009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7a85a523ceb02e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:38.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.744+0000 7fe39a7fc700 1 -- 172.21.15.111:0/533068380 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe390004230 con 0x7fe39c15c6c0 2023-11-28T13:01:38.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.745+0000 7fe39a7fc700 1 -- 172.21.15.111:0/533068380 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe390004390 con 0x7fe39c15c6c0 2023-11-28T13:01:38.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.745+0000 7fe39a7fc700 1 -- 172.21.15.111:0/533068380 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3900036a0 con 0x7fe39c15c6c0 2023-11-28T13:01:38.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.745+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/533068380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe39c15c6c0 msgr2=0x7fe39c0afb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.745+0000 7fe3a1c6a700 1 --2- 172.21.15.111:0/533068380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe39c15c6c0 0x7fe39c0afb50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe3900098f0 tx=0x7fe390009c00 comp rx=0 tx=0).stop 2023-11-28T13:01:38.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.745+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/533068380 shutdown_connections 2023-11-28T13:01:38.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.745+0000 7fe3a1c6a700 1 --2- 172.21.15.111:0/533068380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe39c15c6c0 0x7fe39c0afb50 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.745+0000 7fe3a1c6a700 1 --2- 172.21.15.111:0/533068380 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe39c0b0440 0x7fe39c0b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.745+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/533068380 >> 172.21.15.111:0/533068380 conn(0x7fe39c098ee0 msgr2=0x7fe39c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:38.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.745+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/533068380 shutdown_connections 2023-11-28T13:01:38.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.745+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/533068380 wait complete. 2023-11-28T13:01:38.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.746+0000 7fe3a1c6a700 1 Processor -- start 2023-11-28T13:01:38.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.746+0000 7fe3a1c6a700 1 -- start start 2023-11-28T13:01:38.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.747+0000 7fe3a1c6a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe39c0b0440 0x7fe39c1f1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:38.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.747+0000 7fe3a1c6a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe39c15c6c0 0x7fe39c15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:38.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.747+0000 7fe3a1c6a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe39c1602a0 0x7fe39c15d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:38.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.747+0000 7fe3a1c6a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c152980 con 0x7fe39c15c6c0 2023-11-28T13:01:38.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.747+0000 7fe3a1c6a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c15e540 con 0x7fe39c0b0440 2023-11-28T13:01:38.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.747+0000 7fe3a1c6a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c1f2250 con 0x7fe39c1602a0 2023-11-28T13:01:38.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.747+0000 7fe39affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe39c15c6c0 0x7fe39c15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:38.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.747+0000 7fe39affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe39c15c6c0 0x7fe39c15cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36722/0 (socket says 172.21.15.111:36722) 2023-11-28T13:01:38.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.747+0000 7fe39affd700 1 -- 172.21.15.111:0/475531291 learned_addr learned my addr 172.21.15.111:0/475531291 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:38.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.747+0000 7fe39b7fe700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe39c0b0440 0x7fe39c1f1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:38.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.747+0000 7fe39bfff700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe39c1602a0 0x7fe39c15d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:38.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.748+0000 7fe39affd700 1 -- 172.21.15.111:0/475531291 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe39c1602a0 msgr2=0x7fe39c15d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.748+0000 7fe39affd700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe39c1602a0 0x7fe39c15d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.748+0000 7fe39affd700 1 -- 172.21.15.111:0/475531291 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe39c0b0440 msgr2=0x7fe39c1f1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.748+0000 7fe39affd700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe39c0b0440 0x7fe39c1f1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.748+0000 7fe39affd700 1 -- 172.21.15.111:0/475531291 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe390009550 con 0x7fe39c15c6c0 2023-11-28T13:01:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.748+0000 7fe39b7fe700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe39c0b0440 0x7fe39c1f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.748+0000 7fe39bfff700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe39c1602a0 0x7fe39c15d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.748+0000 7fe39affd700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe39c15c6c0 0x7fe39c15cd70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe38c00d830 tx=0x7fe38c00dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.748+0000 7fe383fff700 1 -- 172.21.15.111:0/475531291 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38c0125f0 con 0x7fe39c15c6c0 2023-11-28T13:01:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.748+0000 7fe383fff700 1 -- 172.21.15.111:0/475531291 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38c012750 con 0x7fe39c15c6c0 2023-11-28T13:01:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.748+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/475531291 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39c0cd2d0 con 0x7fe39c15c6c0 2023-11-28T13:01:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.748+0000 7fe383fff700 1 -- 172.21.15.111:0/475531291 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38c012b00 con 0x7fe39c15c6c0 2023-11-28T13:01:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.748+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/475531291 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39c0ccff0 con 0x7fe39c15c6c0 2023-11-28T13:01:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.749+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/475531291 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39c15ca90 con 0x7fe39c15c6c0 2023-11-28T13:01:38.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.749+0000 7fe383fff700 1 -- 172.21.15.111:0/475531291 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe38c012c60 con 0x7fe39c15c6c0 2023-11-28T13:01:38.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.750+0000 7fe383fff700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe3880682f0 0x7fe38806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:38.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.750+0000 7fe383fff700 1 -- 172.21.15.111:0/475531291 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe38c086290 con 0x7fe39c15c6c0 2023-11-28T13:01:38.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.750+0000 7fe39b7fe700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe3880682f0 0x7fe38806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:38.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.750+0000 7fe39b7fe700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe3880682f0 0x7fe38806a7a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe390009130 tx=0x7fe390008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:38.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.754+0000 7fe383fff700 1 -- 172.21.15.111:0/475531291 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe38c0529b0 con 0x7fe39c15c6c0 2023-11-28T13:01:38.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.937+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/475531291 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe39c0cdf80 con 0x7fe39c15c6c0 2023-11-28T13:01:38.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.938+0000 7fe383fff700 1 -- 172.21.15.111:0/475531291 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+18228 (secure 0 0 0) 0x7fe38c055fd0 con 0x7fe39c15c6c0 2023-11-28T13:01:38.939 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:38.940 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":55,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:57.274026+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:53.965825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:55.242768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:01:38.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.940+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/475531291 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe3880682f0 msgr2=0x7fe38806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.940+0000 7fe3a1c6a700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe3880682f0 0x7fe38806a7a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe390009130 tx=0x7fe390008030 comp rx=0 tx=0).stop 2023-11-28T13:01:38.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.940+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/475531291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe39c15c6c0 msgr2=0x7fe39c15cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:38.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.940+0000 7fe3a1c6a700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe39c15c6c0 0x7fe39c15cd70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe38c00d830 tx=0x7fe38c00dbf0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.940+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/475531291 shutdown_connections 2023-11-28T13:01:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.940+0000 7fe3a1c6a700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe39c1602a0 0x7fe39c15d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.940+0000 7fe3a1c6a700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe3880682f0 0x7fe38806a7a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.940+0000 7fe3a1c6a700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe39c0b0440 0x7fe39c1f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.940+0000 7fe3a1c6a700 1 --2- 172.21.15.111:0/475531291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe39c15c6c0 0x7fe39c15cd70 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.940+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/475531291 >> 172.21.15.111:0/475531291 conn(0x7fe39c098ee0 msgr2=0x7fe39c06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.940+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/475531291 shutdown_connections 2023-11-28T13:01:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:38.940+0000 7fe3a1c6a700 1 -- 172.21.15.111:0/475531291 wait complete. 2023-11-28T13:01:38.955 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T13:00:53.965825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T13:00:55.242768+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T13:01:38.956 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T13:01:39.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.077+0000 7fe821a3b700 1 Processor -- start 2023-11-28T13:01:39.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.077+0000 7fe821a3b700 1 -- start start 2023-11-28T13:01:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe821a3b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe81c0b8510 0x7fe81c06b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe821a3b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe81c0b88e0 con 0x7fe81c0cd460 2023-11-28T13:01:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe821a3b700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe81c0cd8f0 con 0x7fe81c0d0d60 2023-11-28T13:01:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe821a3b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c0cdbd0 con 0x7fe81c0b8510 2023-11-28T13:01:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe81a7fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe81c0cd460 0x7fe81c06beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37508/0 (socket says 172.21.15.111:37508) 2023-11-28T13:01:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe81a7fc700 1 -- 172.21.15.111:0/1587977930 learned_addr learned my addr 172.21.15.111:0/1587977930 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe81affd700 1 --2- 172.21.15.111:0/1587977930 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe81c0b8510 0x7fe81c06b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408648129 0 0) 0x7fe81c0b88e0 con 0x7fe81c0cd460 2023-11-28T13:01:39.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8040035b0 con 0x7fe81c0cd460 2023-11-28T13:01:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 757885233 0 0) 0x7fe8040035b0 con 0x7fe81c0cd460 2023-11-28T13:01:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe81c0b8510 msgr2=0x7fe81c06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe819ffb700 1 --2- 172.21.15.111:0/1587977930 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe81c0b8510 0x7fe81c06b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 >> v1:172.21.15.149:6789/0 conn(0x7fe81c0d0d60 legacy=0x7fe81c0d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c0cfd10 con 0x7fe81c0cd460 2023-11-28T13:01:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fe80c001670 con 0x7fe81c0cd460 2023-11-28T13:01:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 >> v1:172.21.15.111:6789/0 conn(0x7fe81c0cd460 legacy=0x7fe81c06beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe819ffb700 1 --2- 172.21.15.111:0/1587977930 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe81c0b8510 0x7fe804003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.078+0000 7fe819ffb700 1 --2- 172.21.15.111:0/1587977930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe804003f70 0x7fe804004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe819ffb700 1 --2- 172.21.15.111:0/1587977930 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe804004d90 0x7fe804005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c0cdbd0 con 0x7fe804003f70 2023-11-28T13:01:39.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80400ba40 con 0x7fe804004d90 2023-11-28T13:01:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80400bc00 con 0x7fe81c0b8510 2023-11-28T13:01:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe81a7fc700 1 --2- 172.21.15.111:0/1587977930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe804003f70 0x7fe804004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe81affd700 1 --2- 172.21.15.111:0/1587977930 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe81c0b8510 0x7fe804003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe81b7fe700 1 --2- 172.21.15.111:0/1587977930 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe804004d90 0x7fe804005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe81a7fc700 1 -- 172.21.15.111:0/1587977930 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe81c0b8510 msgr2=0x7fe804003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe81a7fc700 1 --2- 172.21.15.111:0/1587977930 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe81c0b8510 0x7fe804003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe81a7fc700 1 -- 172.21.15.111:0/1587977930 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe804004d90 msgr2=0x7fe804005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe81a7fc700 1 --2- 172.21.15.111:0/1587977930 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe804004d90 0x7fe804005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe81a7fc700 1 -- 172.21.15.111:0/1587977930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8040035b0 con 0x7fe804003f70 2023-11-28T13:01:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe81a7fc700 1 --2- 172.21.15.111:0/1587977930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe804003f70 0x7fe804004850 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00c240 tx=0x7fe804005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c4c06d2358b5b2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c00cd80 con 0x7fe804003f70 2023-11-28T13:01:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80c005960 con 0x7fe804003f70 2023-11-28T13:01:39.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.079+0000 7fe819ffb700 1 -- 172.21.15.111:0/1587977930 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c005d10 con 0x7fe804003f70 2023-11-28T13:01:39.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.080+0000 7fe821a3b700 1 -- 172.21.15.111:0/1587977930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe804003f70 msgr2=0x7fe804004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.080+0000 7fe821a3b700 1 --2- 172.21.15.111:0/1587977930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe804003f70 0x7fe804004850 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00c240 tx=0x7fe804005880 comp rx=0 tx=0).stop 2023-11-28T13:01:39.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.080+0000 7fe81b7fe700 1 -- 172.21.15.111:0/1587977930 reap_dead start 2023-11-28T13:01:39.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.080+0000 7fe821a3b700 1 -- 172.21.15.111:0/1587977930 shutdown_connections 2023-11-28T13:01:39.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.080+0000 7fe821a3b700 1 -- 172.21.15.111:0/1587977930 >> 172.21.15.111:0/1587977930 conn(0x7fe81c0719a0 msgr2=0x7fe81c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:39.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.080+0000 7fe821a3b700 1 -- 172.21.15.111:0/1587977930 shutdown_connections 2023-11-28T13:01:39.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.080+0000 7fe821a3b700 1 -- 172.21.15.111:0/1587977930 wait complete. 2023-11-28T13:01:39.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.081+0000 7fe821a3b700 1 Processor -- start 2023-11-28T13:01:39.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.081+0000 7fe821a3b700 1 -- start start 2023-11-28T13:01:39.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe821a3b700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe804003f70 0x7fe81c1544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe821a3b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c1578b0 0x7fe81c0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe821a3b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe81c0cc5f0 0x7fe81c159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe821a3b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c0cca90 con 0x7fe81c1578b0 2023-11-28T13:01:39.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe821a3b700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c1565c0 con 0x7fe804003f70 2023-11-28T13:01:39.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe821a3b700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c156810 con 0x7fe81c0cc5f0 2023-11-28T13:01:39.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe81a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c1578b0 0x7fe81c0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe81a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c1578b0 0x7fe81c0cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36742/0 (socket says 172.21.15.111:36742) 2023-11-28T13:01:39.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe81a7fc700 1 -- 172.21.15.111:0/4085845324 learned_addr learned my addr 172.21.15.111:0/4085845324 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:39.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe81affd700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe804003f70 0x7fe81c1544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe81b7fe700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe81c0cc5f0 0x7fe81c159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe81a7fc700 1 -- 172.21.15.111:0/4085845324 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe81c0cc5f0 msgr2=0x7fe81c159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe81a7fc700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe81c0cc5f0 0x7fe81c159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe81a7fc700 1 -- 172.21.15.111:0/4085845324 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe804003f70 msgr2=0x7fe81c1544e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe81a7fc700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe804003f70 0x7fe81c1544e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe81a7fc700 1 -- 172.21.15.111:0/4085845324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c0d29c0 con 0x7fe81c1578b0 2023-11-28T13:01:39.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe81a7fc700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c1578b0 0x7fe81c0cc0b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00c830 tx=0x7fe80c001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:39.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.082+0000 7fe803fff700 1 -- 172.21.15.111:0/4085845324 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c001780 con 0x7fe81c1578b0 2023-11-28T13:01:39.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.083+0000 7fe803fff700 1 -- 172.21.15.111:0/4085845324 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80c000ed0 con 0x7fe81c1578b0 2023-11-28T13:01:39.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.083+0000 7fe821a3b700 1 -- 172.21.15.111:0/4085845324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81c0cfff0 con 0x7fe81c1578b0 2023-11-28T13:01:39.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.083+0000 7fe803fff700 1 -- 172.21.15.111:0/4085845324 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c007310 con 0x7fe81c1578b0 2023-11-28T13:01:39.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.083+0000 7fe821a3b700 1 -- 172.21.15.111:0/4085845324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81c0d2b50 con 0x7fe81c1578b0 2023-11-28T13:01:39.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.084+0000 7fe821a3b700 1 -- 172.21.15.111:0/4085845324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81c0b43c0 con 0x7fe81c1578b0 2023-11-28T13:01:39.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.084+0000 7fe803fff700 1 -- 172.21.15.111:0/4085845324 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe80c010070 con 0x7fe81c1578b0 2023-11-28T13:01:39.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.084+0000 7fe803fff700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe808068320 0x7fe80806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.084+0000 7fe803fff700 1 -- 172.21.15.111:0/4085845324 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe80c087420 con 0x7fe81c1578b0 2023-11-28T13:01:39.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.085+0000 7fe81affd700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe808068320 0x7fe80806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.085+0000 7fe81affd700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe808068320 0x7fe80806a7d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe810009c60 tx=0x7fe810009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:39.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.089+0000 7fe803fff700 1 -- 172.21.15.111:0/4085845324 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe80c053b40 con 0x7fe81c1578b0 2023-11-28T13:01:39.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.270+0000 7fe821a3b700 1 -- 172.21.15.111:0/4085845324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe81c067880 con 0x7fe81c1578b0 2023-11-28T13:01:39.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.271+0000 7fe803fff700 1 -- 172.21.15.111:0/4085845324 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe80c057160 con 0x7fe81c1578b0 2023-11-28T13:01:39.272 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T13:01:39.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.273+0000 7fe821a3b700 1 -- 172.21.15.111:0/4085845324 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe808068320 msgr2=0x7fe80806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.273+0000 7fe821a3b700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe808068320 0x7fe80806a7d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe810009c60 tx=0x7fe810009210 comp rx=0 tx=0).stop 2023-11-28T13:01:39.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.273+0000 7fe821a3b700 1 -- 172.21.15.111:0/4085845324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c1578b0 msgr2=0x7fe81c0cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.273+0000 7fe821a3b700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c1578b0 0x7fe81c0cc0b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00c830 tx=0x7fe80c001980 comp rx=0 tx=0).stop 2023-11-28T13:01:39.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.273+0000 7fe821a3b700 1 -- 172.21.15.111:0/4085845324 shutdown_connections 2023-11-28T13:01:39.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.273+0000 7fe821a3b700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe81c0cc5f0 0x7fe81c159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.273+0000 7fe821a3b700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe808068320 0x7fe80806a7d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.273+0000 7fe821a3b700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe804003f70 0x7fe81c1544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.273+0000 7fe821a3b700 1 --2- 172.21.15.111:0/4085845324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe81c1578b0 0x7fe81c0cc0b0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.273+0000 7fe821a3b700 1 -- 172.21.15.111:0/4085845324 >> 172.21.15.111:0/4085845324 conn(0x7fe81c0719a0 msgr2=0x7fe81c0b77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:39.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.273+0000 7fe821a3b700 1 -- 172.21.15.111:0/4085845324 shutdown_connections 2023-11-28T13:01:39.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.273+0000 7fe821a3b700 1 -- 172.21.15.111:0/4085845324 wait complete. 2023-11-28T13:01:39.286 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T13:01:39.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.408+0000 7fc4aec20700 1 Processor -- start 2023-11-28T13:01:39.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.408+0000 7fc4aec20700 1 -- start start 2023-11-28T13:01:39.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4aec20700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc4a80aecc0 0x7fc4a80b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4aec20700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a80ad420 con 0x7fc4a8159960 2023-11-28T13:01:39.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4aec20700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a80ac810 con 0x7fc4a80cdd90 2023-11-28T13:01:39.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4aec20700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a8159d30 con 0x7fc4a80aecc0 2023-11-28T13:01:39.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4ac9bc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc4a8159960 0x7fc4a80cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37526/0 (socket says 172.21.15.111:37526) 2023-11-28T13:01:39.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4ac9bc700 1 -- 172.21.15.111:0/3424561315 learned_addr learned my addr 172.21.15.111:0/3424561315 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:39.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4ad1bd700 1 --2- 172.21.15.111:0/3424561315 >> v2:172.21.15.149:3300/0 conn(0x7fc4a80aecc0 0x7fc4a80b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114191048 0 0) 0x7fc4a80ad420 con 0x7fc4a8159960 2023-11-28T13:01:39.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4980035b0 con 0x7fc4a8159960 2023-11-28T13:01:39.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1921435739 0 0) 0x7fc4980035b0 con 0x7fc4a8159960 2023-11-28T13:01:39.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 >> v2:172.21.15.149:3300/0 conn(0x7fc4a80aecc0 msgr2=0x7fc4a80b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:39.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4a77fe700 1 --2- 172.21.15.111:0/3424561315 >> v2:172.21.15.149:3300/0 conn(0x7fc4a80aecc0 0x7fc4a80b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 >> v1:172.21.15.149:6789/0 conn(0x7fc4a80cdd90 legacy=0x7fc4a80ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a80acaf0 con 0x7fc4a8159960 2023-11-28T13:01:39.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.409+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fc49c0017f0 con 0x7fc4a8159960 2023-11-28T13:01:39.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 >> v1:172.21.15.111:6789/0 conn(0x7fc4a8159960 legacy=0x7fc4a80cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4a77fe700 1 --2- 172.21.15.111:0/3424561315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc498003a00 0x7fc498003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4a77fe700 1 --2- 172.21.15.111:0/3424561315 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc498004850 0x7fc498004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4a77fe700 1 --2- 172.21.15.111:0/3424561315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc498005200 0x7fc49800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a8159d30 con 0x7fc498003a00 2023-11-28T13:01:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49800c080 con 0x7fc498005200 2023-11-28T13:01:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49800c240 con 0x7fc498004850 2023-11-28T13:01:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4ac9bc700 1 --2- 172.21.15.111:0/3424561315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc498003a00 0x7fc498003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4a7fff700 1 --2- 172.21.15.111:0/3424561315 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc498004850 0x7fc498004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4ad1bd700 1 --2- 172.21.15.111:0/3424561315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc498005200 0x7fc49800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4ac9bc700 1 -- 172.21.15.111:0/3424561315 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc498004850 msgr2=0x7fc498004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4ac9bc700 1 --2- 172.21.15.111:0/3424561315 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc498004850 0x7fc498004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4ac9bc700 1 -- 172.21.15.111:0/3424561315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc498005200 msgr2=0x7fc49800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:39.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4ac9bc700 1 --2- 172.21.15.111:0/3424561315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc498005200 0x7fc49800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4ac9bc700 1 -- 172.21.15.111:0/3424561315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4980035b0 con 0x7fc498003a00 2023-11-28T13:01:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4ad1bd700 1 --2- 172.21.15.111:0/3424561315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc498005200 0x7fc49800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4ad1bd700 1 -- 172.21.15.111:0/3424561315 reap_dead start 2023-11-28T13:01:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4ac9bc700 1 --2- 172.21.15.111:0/3424561315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc498003a00 0x7fc498003e10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00c240 tx=0x7fc498005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df173bbbe04bedab server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c00cde0 con 0x7fc498003a00 2023-11-28T13:01:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c007b80 con 0x7fc498003a00 2023-11-28T13:01:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.410+0000 7fc4a77fe700 1 -- 172.21.15.111:0/3424561315 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c0072f0 con 0x7fc498003a00 2023-11-28T13:01:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.411+0000 7fc4aec20700 1 -- 172.21.15.111:0/3424561315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc498003a00 msgr2=0x7fc498003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.411+0000 7fc4aec20700 1 --2- 172.21.15.111:0/3424561315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc498003a00 0x7fc498003e10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc49c00c240 tx=0x7fc498005880 comp rx=0 tx=0).stop 2023-11-28T13:01:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.411+0000 7fc4aec20700 1 -- 172.21.15.111:0/3424561315 shutdown_connections 2023-11-28T13:01:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.411+0000 7fc4aec20700 1 --2- 172.21.15.111:0/3424561315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc498003a00 0x7fc498003e10 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.411+0000 7fc4aec20700 1 -- 172.21.15.111:0/3424561315 >> 172.21.15.111:0/3424561315 conn(0x7fc4a8098ee0 msgr2=0x7fc4a806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:39.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.411+0000 7fc4aec20700 1 -- 172.21.15.111:0/3424561315 shutdown_connections 2023-11-28T13:01:39.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.411+0000 7fc4aec20700 1 -- 172.21.15.111:0/3424561315 wait complete. 2023-11-28T13:01:39.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.412+0000 7fc4aec20700 1 Processor -- start 2023-11-28T13:01:39.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.412+0000 7fc4aec20700 1 -- start start 2023-11-28T13:01:39.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4aec20700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc498003a00 0x7fc4a809cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4aec20700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4a80c5d80 0x7fc4a80c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4aec20700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4a806dc90 0x7fc4a806e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4aec20700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a806ea30 con 0x7fc498003a00 2023-11-28T13:01:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4aec20700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a806ed10 con 0x7fc4a80c5d80 2023-11-28T13:01:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4aec20700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a806eff0 con 0x7fc4a806dc90 2023-11-28T13:01:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4ac9bc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc498003a00 0x7fc4a809cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4ac9bc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc498003a00 0x7fc4a809cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36760/0 (socket says 172.21.15.111:36760) 2023-11-28T13:01:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4ac9bc700 1 -- 172.21.15.111:0/2713538314 learned_addr learned my addr 172.21.15.111:0/2713538314 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4a7fff700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4a80c5d80 0x7fc4a80c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4ad1bd700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4a806dc90 0x7fc4a806e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4ac9bc700 1 -- 172.21.15.111:0/2713538314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4a806dc90 msgr2=0x7fc4a806e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4ac9bc700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4a806dc90 0x7fc4a806e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4ac9bc700 1 -- 172.21.15.111:0/2713538314 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4a80c5d80 msgr2=0x7fc4a80c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4ac9bc700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4a80c5d80 0x7fc4a80c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4ac9bc700 1 -- 172.21.15.111:0/2713538314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a80ad0b0 con 0x7fc498003a00 2023-11-28T13:01:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.413+0000 7fc4ac9bc700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc498003a00 0x7fc4a809cf60 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc49c000c00 tx=0x7fc49c001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.414+0000 7fc4a57fa700 1 -- 172.21.15.111:0/2713538314 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c000f70 con 0x7fc498003a00 2023-11-28T13:01:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.414+0000 7fc4a57fa700 1 -- 172.21.15.111:0/2713538314 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c005830 con 0x7fc498003a00 2023-11-28T13:01:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.414+0000 7fc4aec20700 1 -- 172.21.15.111:0/2713538314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a80acdd0 con 0x7fc498003a00 2023-11-28T13:01:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.414+0000 7fc4a57fa700 1 -- 172.21.15.111:0/2713538314 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c005be0 con 0x7fc498003a00 2023-11-28T13:01:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.414+0000 7fc4aec20700 1 -- 172.21.15.111:0/2713538314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a80b1fd0 con 0x7fc498003a00 2023-11-28T13:01:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.415+0000 7fc4aec20700 1 -- 172.21.15.111:0/2713538314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a8067170 con 0x7fc498003a00 2023-11-28T13:01:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.415+0000 7fc4a57fa700 1 -- 172.21.15.111:0/2713538314 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc49c010070 con 0x7fc498003a00 2023-11-28T13:01:39.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.415+0000 7fc4a57fa700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc48c0682f0 0x7fc48c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.415+0000 7fc4a57fa700 1 -- 172.21.15.111:0/2713538314 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fc49c0863e0 con 0x7fc498003a00 2023-11-28T13:01:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.415+0000 7fc4a7fff700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc48c0682f0 0x7fc48c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.416+0000 7fc4a7fff700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc48c0682f0 0x7fc48c06a7a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc4a81f21b0 tx=0x7fc494009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:39.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.420+0000 7fc4a57fa700 1 -- 172.21.15.111:0/2713538314 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc49c052b00 con 0x7fc498003a00 2023-11-28T13:01:39.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.601+0000 7fc4aec20700 1 -- 172.21.15.111:0/2713538314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc4a806c4f0 con 0x7fc498003a00 2023-11-28T13:01:39.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.601+0000 7fc4a57fa700 1 -- 172.21.15.111:0/2713538314 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v55) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc49c056120 con 0x7fc498003a00 2023-11-28T13:01:39.603 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T13:01:39.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.603+0000 7fc4aec20700 1 -- 172.21.15.111:0/2713538314 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc48c0682f0 msgr2=0x7fc48c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.603+0000 7fc4aec20700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc48c0682f0 0x7fc48c06a7a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc4a81f21b0 tx=0x7fc494009040 comp rx=0 tx=0).stop 2023-11-28T13:01:39.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.603+0000 7fc4aec20700 1 -- 172.21.15.111:0/2713538314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc498003a00 msgr2=0x7fc4a809cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.603+0000 7fc4aec20700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc498003a00 0x7fc4a809cf60 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc49c000c00 tx=0x7fc49c001b80 comp rx=0 tx=0).stop 2023-11-28T13:01:39.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.604+0000 7fc4aec20700 1 -- 172.21.15.111:0/2713538314 shutdown_connections 2023-11-28T13:01:39.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.604+0000 7fc4aec20700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4a806dc90 0x7fc4a806e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.604+0000 7fc4aec20700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc498003a00 0x7fc4a809cf60 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.604+0000 7fc4aec20700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc48c0682f0 0x7fc48c06a7a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.604+0000 7fc4aec20700 1 --2- 172.21.15.111:0/2713538314 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4a80c5d80 0x7fc4a80c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.604+0000 7fc4aec20700 1 -- 172.21.15.111:0/2713538314 >> 172.21.15.111:0/2713538314 conn(0x7fc4a8098ee0 msgr2=0x7fc4a80cfb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:39.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.604+0000 7fc4aec20700 1 -- 172.21.15.111:0/2713538314 shutdown_connections 2023-11-28T13:01:39.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.604+0000 7fc4aec20700 1 -- 172.21.15.111:0/2713538314 wait complete. 2023-11-28T13:01:39.617 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T13:01:39.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.739+0000 7fe6d122c700 1 Processor -- start 2023-11-28T13:01:39.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.739+0000 7fe6d122c700 1 -- start start 2023-11-28T13:01:39.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.740+0000 7fe6d122c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe6cc15a6a0 0x7fe6cc06a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.740+0000 7fe6d122c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6cc15aa70 con 0x7fe6cc0af160 2023-11-28T13:01:39.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.740+0000 7fe6d122c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6cc0ad280 con 0x7fe6cc0ae720 2023-11-28T13:01:39.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.740+0000 7fe6d122c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6cc0ac950 con 0x7fe6cc15a6a0 2023-11-28T13:01:39.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.740+0000 7fe6cad9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe6cc15a6a0 0x7fe6cc06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.740+0000 7fe6ca59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe6cc0af160 0x7fe6cc0cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37546/0 (socket says 172.21.15.111:37546) 2023-11-28T13:01:39.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.740+0000 7fe6ca59c700 1 -- 172.21.15.111:0/4243617897 learned_addr learned my addr 172.21.15.111:0/4243617897 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:39.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6cad9d700 1 -- 172.21.15.111:0/4243617897 >> v1:172.21.15.149:6789/0 conn(0x7fe6cc0ae720 legacy=0x7fe6cc0d0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:39.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6cad9d700 1 -- 172.21.15.111:0/4243617897 >> v1:172.21.15.111:6789/0 conn(0x7fe6cc0af160 legacy=0x7fe6cc0cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6c9d9b700 1 -- 172.21.15.111:0/4243617897 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669307517 0 0) 0x7fe6bc001110 con 0x7fe6cc0af160 2023-11-28T13:01:39.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6cad9d700 1 -- 172.21.15.111:0/4243617897 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc0acc30 con 0x7fe6cc15a6a0 2023-11-28T13:01:39.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6cad9d700 1 --2- 172.21.15.111:0/4243617897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6cc15a6a0 0x7fe6cc06a270 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009840 tx=0x7fe6c0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5fda86b48b628e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:39.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6c9d9b700 1 -- 172.21.15.111:0/4243617897 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0004230 con 0x7fe6cc15a6a0 2023-11-28T13:01:39.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6c9d9b700 1 -- 172.21.15.111:0/4243617897 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c0004390 con 0x7fe6cc15a6a0 2023-11-28T13:01:39.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6c9d9b700 1 -- 172.21.15.111:0/4243617897 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c00036a0 con 0x7fe6cc15a6a0 2023-11-28T13:01:39.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6d122c700 1 -- 172.21.15.111:0/4243617897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6cc15a6a0 msgr2=0x7fe6cc06a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6d122c700 1 --2- 172.21.15.111:0/4243617897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6cc15a6a0 0x7fe6cc06a270 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009840 tx=0x7fe6c0009b50 comp rx=0 tx=0).stop 2023-11-28T13:01:39.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6d122c700 1 -- 172.21.15.111:0/4243617897 shutdown_connections 2023-11-28T13:01:39.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6d122c700 1 --2- 172.21.15.111:0/4243617897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6cc15a6a0 0x7fe6cc06a270 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6d122c700 1 -- 172.21.15.111:0/4243617897 >> 172.21.15.111:0/4243617897 conn(0x7fe6cc098ee0 msgr2=0x7fe6cc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:39.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6d122c700 1 -- 172.21.15.111:0/4243617897 shutdown_connections 2023-11-28T13:01:39.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.741+0000 7fe6d122c700 1 -- 172.21.15.111:0/4243617897 wait complete. 2023-11-28T13:01:39.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.743+0000 7fe6d122c700 1 Processor -- start 2023-11-28T13:01:39.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.743+0000 7fe6d122c700 1 -- start start 2023-11-28T13:01:39.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6d122c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6cc0ae720 0x7fe6cc15ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6d122c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe6cc0af160 0x7fe6cc14cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6d122c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6cc0c3740 0x7fe6cc15d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6d122c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc15dac0 con 0x7fe6cc0c3740 2023-11-28T13:01:39.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6d122c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc15beb0 con 0x7fe6cc0af160 2023-11-28T13:01:39.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6d122c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc15b2a0 con 0x7fe6cc0ae720 2023-11-28T13:01:39.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6cb59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6cc0c3740 0x7fe6cc15d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6cb59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6cc0c3740 0x7fe6cc15d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36778/0 (socket says 172.21.15.111:36778) 2023-11-28T13:01:39.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6cb59e700 1 -- 172.21.15.111:0/671889749 learned_addr learned my addr 172.21.15.111:0/671889749 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6ca59c700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe6cc0af160 0x7fe6cc14cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6cad9d700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6cc0ae720 0x7fe6cc15ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6cb59e700 1 -- 172.21.15.111:0/671889749 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6cc0ae720 msgr2=0x7fe6cc15ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6cb59e700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6cc0ae720 0x7fe6cc15ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6cb59e700 1 -- 172.21.15.111:0/671889749 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe6cc0af160 msgr2=0x7fe6cc14cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6cb59e700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe6cc0af160 0x7fe6cc14cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6cb59e700 1 -- 172.21.15.111:0/671889749 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c00094a0 con 0x7fe6cc0c3740 2023-11-28T13:01:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.744+0000 7fe6cb59e700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6cc0c3740 0x7fe6cc15d580 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400cc40 tx=0x7fe6c4004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.745+0000 7fe6b37fe700 1 -- 172.21.15.111:0/671889749 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c400aae0 con 0x7fe6cc0c3740 2023-11-28T13:01:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.745+0000 7fe6b37fe700 1 -- 172.21.15.111:0/671889749 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c400ac40 con 0x7fe6cc0c3740 2023-11-28T13:01:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.745+0000 7fe6d122c700 1 -- 172.21.15.111:0/671889749 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6cc0d0950 con 0x7fe6cc0c3740 2023-11-28T13:01:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.745+0000 7fe6b37fe700 1 -- 172.21.15.111:0/671889749 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4012420 con 0x7fe6cc0c3740 2023-11-28T13:01:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.745+0000 7fe6d122c700 1 -- 172.21.15.111:0/671889749 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6cc0acf10 con 0x7fe6cc0c3740 2023-11-28T13:01:39.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.746+0000 7fe6d122c700 1 -- 172.21.15.111:0/671889749 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6cc0992e0 con 0x7fe6cc0c3740 2023-11-28T13:01:39.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.746+0000 7fe6b37fe700 1 -- 172.21.15.111:0/671889749 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe6c4010050 con 0x7fe6cc0c3740 2023-11-28T13:01:39.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.746+0000 7fe6b37fe700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe6b80682f0 0x7fe6b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:39.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.746+0000 7fe6b37fe700 1 -- 172.21.15.111:0/671889749 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe6c40860c0 con 0x7fe6cc0c3740 2023-11-28T13:01:39.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.746+0000 7fe6cad9d700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe6b80682f0 0x7fe6b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:39.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.747+0000 7fe6cad9d700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe6b80682f0 0x7fe6b806a7a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0669f0 tx=0x7fe6c0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:39.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.751+0000 7fe6b37fe700 1 -- 172.21.15.111:0/671889749 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe6c4052830 con 0x7fe6cc0c3740 2023-11-28T13:01:39.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.934+0000 7fe6d122c700 1 -- 172.21.15.111:0/671889749 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe6cc067170 con 0x7fe6cc0c3740 2023-11-28T13:01:39.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.934+0000 7fe6b37fe700 1 -- 172.21.15.111:0/671889749 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe6c4055e50 con 0x7fe6cc0c3740 2023-11-28T13:01:39.936 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T13:01:39.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.936+0000 7fe6d122c700 1 -- 172.21.15.111:0/671889749 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe6b80682f0 msgr2=0x7fe6b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.936+0000 7fe6d122c700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe6b80682f0 0x7fe6b806a7a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0669f0 tx=0x7fe6c0008030 comp rx=0 tx=0).stop 2023-11-28T13:01:39.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.936+0000 7fe6d122c700 1 -- 172.21.15.111:0/671889749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6cc0c3740 msgr2=0x7fe6cc15d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:39.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.936+0000 7fe6d122c700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6cc0c3740 0x7fe6cc15d580 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400cc40 tx=0x7fe6c4004990 comp rx=0 tx=0).stop 2023-11-28T13:01:39.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.936+0000 7fe6d122c700 1 -- 172.21.15.111:0/671889749 shutdown_connections 2023-11-28T13:01:39.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.936+0000 7fe6d122c700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6cc0c3740 0x7fe6cc15d580 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.936+0000 7fe6d122c700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6cc0ae720 0x7fe6cc15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.936+0000 7fe6d122c700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe6b80682f0 0x7fe6b806a7a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.936+0000 7fe6d122c700 1 --2- 172.21.15.111:0/671889749 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe6cc0af160 0x7fe6cc14cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:39.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.936+0000 7fe6d122c700 1 -- 172.21.15.111:0/671889749 >> 172.21.15.111:0/671889749 conn(0x7fe6cc098ee0 msgr2=0x7fe6cc157d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:39.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.936+0000 7fe6d122c700 1 -- 172.21.15.111:0/671889749 shutdown_connections 2023-11-28T13:01:39.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:39.936+0000 7fe6d122c700 1 -- 172.21.15.111:0/671889749 wait complete. 2023-11-28T13:01:39.949 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T13:01:40.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.070+0000 7ff8d4bd5700 1 Processor -- start 2023-11-28T13:01:40.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.070+0000 7ff8d4bd5700 1 -- start start 2023-11-28T13:01:40.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.071+0000 7ff8d4bd5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8d015c7b0 0x7ff8d00cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.071+0000 7ff8d4bd5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d00cb810 con 0x7ff8d00d0490 2023-11-28T13:01:40.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.071+0000 7ff8d4bd5700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d00cc650 con 0x7ff8d006b380 2023-11-28T13:01:40.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.071+0000 7ff8d4bd5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d00cba40 con 0x7ff8d015c7b0 2023-11-28T13:01:40.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.071+0000 7ff8cdd9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff8d00d0490 0x7ff8d00ad8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37560/0 (socket says 172.21.15.111:37560) 2023-11-28T13:01:40.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cdd9b700 1 -- 172.21.15.111:0/2556113863 learned_addr learned my addr 172.21.15.111:0/2556113863 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:40.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8ce59c700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8d015c7b0 0x7ff8d00cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608608676 0 0) 0x7ff8d00cb810 con 0x7ff8d00d0490 2023-11-28T13:01:40.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b80035b0 con 0x7ff8d00d0490 2023-11-28T13:01:40.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767622173 0 0) 0x7ff8d00cc650 con 0x7ff8d006b380 2023-11-28T13:01:40.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d00cb810 con 0x7ff8d006b380 2023-11-28T13:01:40.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3131121717 0 0) 0x7ff8b80035b0 con 0x7ff8d00d0490 2023-11-28T13:01:40.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8d015c7b0 msgr2=0x7ff8d00cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:40.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8d015c7b0 0x7ff8d00cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 >> v1:172.21.15.149:6789/0 conn(0x7ff8d006b380 legacy=0x7ff8d00ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d00cbd20 con 0x7ff8d00d0490 2023-11-28T13:01:40.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7ff8c0001670 con 0x7ff8d00d0490 2023-11-28T13:01:40.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 >> v1:172.21.15.111:6789/0 conn(0x7ff8d00d0490 legacy=0x7ff8d00ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b8003bc0 0x7ff8b8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8b8004850 0x7ff8b8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8d015c7b0 0x7ff8b8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d00cba40 con 0x7ff8b8004850 2023-11-28T13:01:40.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b800baf0 con 0x7ff8b8003bc0 2023-11-28T13:01:40.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b800bcb0 con 0x7ff8d015c7b0 2023-11-28T13:01:40.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cdd9b700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8b8004850 0x7ff8b8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8ce59c700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b8003bc0 0x7ff8b8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8ced9d700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8d015c7b0 0x7ff8b8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cdd9b700 1 -- 172.21.15.111:0/2556113863 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8d015c7b0 msgr2=0x7ff8b8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:40.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cdd9b700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8d015c7b0 0x7ff8b8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cdd9b700 1 -- 172.21.15.111:0/2556113863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b8003bc0 msgr2=0x7ff8b8003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:40.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cdd9b700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b8003bc0 0x7ff8b8003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8cdd9b700 1 -- 172.21.15.111:0/2556113863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b80035b0 con 0x7ff8b8004850 2023-11-28T13:01:40.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8ce59c700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b8003bc0 0x7ff8b8003fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:40.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.072+0000 7ff8ced9d700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8d015c7b0 0x7ff8b8005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:40.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.073+0000 7ff8cdd9b700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8b8004850 0x7ff8b8004cc0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000c240 tx=0x7ff8b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5dcf62a03da640f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:40.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.073+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c000cd80 con 0x7ff8b8004850 2023-11-28T13:01:40.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.073+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c0005960 con 0x7ff8b8004850 2023-11-28T13:01:40.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.073+0000 7ff8cd59a700 1 -- 172.21.15.111:0/2556113863 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0005d10 con 0x7ff8b8004850 2023-11-28T13:01:40.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.073+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/2556113863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8b8004850 msgr2=0x7ff8b8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.073+0000 7ff8d4bd5700 1 --2- 172.21.15.111:0/2556113863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8b8004850 0x7ff8b8004cc0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000c240 tx=0x7ff8b8005880 comp rx=0 tx=0).stop 2023-11-28T13:01:40.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.073+0000 7ff8ced9d700 1 -- 172.21.15.111:0/2556113863 reap_dead start 2023-11-28T13:01:40.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.073+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/2556113863 shutdown_connections 2023-11-28T13:01:40.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.073+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/2556113863 >> 172.21.15.111:0/2556113863 conn(0x7ff8d009a040 msgr2=0x7ff8d00d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:40.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.073+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/2556113863 shutdown_connections 2023-11-28T13:01:40.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.073+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/2556113863 wait complete. 2023-11-28T13:01:40.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.075+0000 7ff8d4bd5700 1 Processor -- start 2023-11-28T13:01:40.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.075+0000 7ff8d4bd5700 1 -- start start 2023-11-28T13:01:40.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.075+0000 7ff8d4bd5700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b8004850 0x7ff8d00b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.075+0000 7ff8d4bd5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8d00b4770 0x7ff8d00b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.075+0000 7ff8d4bd5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8d00c6e40 0x7ff8d00c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.075+0000 7ff8d4bd5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d00c5560 con 0x7ff8d00c6e40 2023-11-28T13:01:40.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.075+0000 7ff8d4bd5700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d00c4950 con 0x7ff8b8004850 2023-11-28T13:01:40.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.075+0000 7ff8d4bd5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d00c4c30 con 0x7ff8d00b4770 2023-11-28T13:01:40.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8ced9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8d00c6e40 0x7ff8d00c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8ced9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8d00c6e40 0x7ff8d00c72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36798/0 (socket says 172.21.15.111:36798) 2023-11-28T13:01:40.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8ced9d700 1 -- 172.21.15.111:0/665611339 learned_addr learned my addr 172.21.15.111:0/665611339 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:40.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8ced9d700 1 -- 172.21.15.111:0/665611339 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8d00b4770 msgr2=0x7ff8d00b4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8ced9d700 1 --2- 172.21.15.111:0/665611339 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8d00b4770 0x7ff8d00b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8ced9d700 1 -- 172.21.15.111:0/665611339 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b8004850 msgr2=0x7ff8d00b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8ced9d700 1 --2- 172.21.15.111:0/665611339 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b8004850 0x7ff8d00b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8ced9d700 1 -- 172.21.15.111:0/665611339 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d00cc2e0 con 0x7ff8d00c6e40 2023-11-28T13:01:40.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8ced9d700 1 --2- 172.21.15.111:0/665611339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8d00c6e40 0x7ff8d00c72b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8001a50 tx=0x7ff8c800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:40.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8beffd700 1 -- 172.21.15.111:0/665611339 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8003c10 con 0x7ff8d00c6e40 2023-11-28T13:01:40.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8beffd700 1 -- 172.21.15.111:0/665611339 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c8003d90 con 0x7ff8d00c6e40 2023-11-28T13:01:40.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/665611339 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d00cc000 con 0x7ff8d00c6e40 2023-11-28T13:01:40.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8beffd700 1 -- 172.21.15.111:0/665611339 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8005310 con 0x7ff8d00c6e40 2023-11-28T13:01:40.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.076+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/665611339 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d015cb80 con 0x7ff8d00c6e40 2023-11-28T13:01:40.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.077+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/665611339 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d0067170 con 0x7ff8d00c6e40 2023-11-28T13:01:40.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.078+0000 7ff8beffd700 1 -- 172.21.15.111:0/665611339 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff8c8005470 con 0x7ff8d00c6e40 2023-11-28T13:01:40.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.078+0000 7ff8beffd700 1 --2- 172.21.15.111:0/665611339 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8b00682f0 0x7ff8b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.078+0000 7ff8beffd700 1 -- 172.21.15.111:0/665611339 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff8c8086090 con 0x7ff8d00c6e40 2023-11-28T13:01:40.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.078+0000 7ff8ce59c700 1 --2- 172.21.15.111:0/665611339 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8b00682f0 0x7ff8b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.078+0000 7ff8ce59c700 1 --2- 172.21.15.111:0/665611339 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8b00682f0 0x7ff8b006a7a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40098d0 tx=0x7ff8c4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:40.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.082+0000 7ff8beffd700 1 -- 172.21.15.111:0/665611339 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff8c80527b0 con 0x7ff8d00c6e40 2023-11-28T13:01:40.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.263+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/665611339 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff8d00cde70 con 0x7ff8d00c6e40 2023-11-28T13:01:40.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.263+0000 7ff8beffd700 1 -- 172.21.15.111:0/665611339 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff8c8055dd0 con 0x7ff8d00c6e40 2023-11-28T13:01:40.265 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T13:01:40.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.265+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/665611339 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8b00682f0 msgr2=0x7ff8b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.265+0000 7ff8d4bd5700 1 --2- 172.21.15.111:0/665611339 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8b00682f0 0x7ff8b006a7a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40098d0 tx=0x7ff8c4008040 comp rx=0 tx=0).stop 2023-11-28T13:01:40.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.265+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/665611339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8d00c6e40 msgr2=0x7ff8d00c72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.265+0000 7ff8d4bd5700 1 --2- 172.21.15.111:0/665611339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8d00c6e40 0x7ff8d00c72b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8001a50 tx=0x7ff8c800ef40 comp rx=0 tx=0).stop 2023-11-28T13:01:40.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.265+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/665611339 shutdown_connections 2023-11-28T13:01:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.265+0000 7ff8d4bd5700 1 --2- 172.21.15.111:0/665611339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8d00c6e40 0x7ff8d00c72b0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.265+0000 7ff8d4bd5700 1 --2- 172.21.15.111:0/665611339 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8b00682f0 0x7ff8b006a7a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.265+0000 7ff8d4bd5700 1 --2- 172.21.15.111:0/665611339 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8b8004850 0x7ff8d00b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.265+0000 7ff8d4bd5700 1 --2- 172.21.15.111:0/665611339 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8d00b4770 0x7ff8d00b4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.265+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/665611339 >> 172.21.15.111:0/665611339 conn(0x7ff8d009a040 msgr2=0x7ff8d00cf6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.265+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/665611339 shutdown_connections 2023-11-28T13:01:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.265+0000 7ff8d4bd5700 1 -- 172.21.15.111:0/665611339 wait complete. 2023-11-28T13:01:40.278 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:01:40.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.398+0000 7f81b71ea700 1 Processor -- start 2023-11-28T13:01:40.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.398+0000 7f81b71ea700 1 -- start start 2023-11-28T13:01:40.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.398+0000 7f81b71ea700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f81b015cc50 0x7f81b00712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.398+0000 7f81b71ea700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b00717f0 con 0x7f81b00cceb0 2023-11-28T13:01:40.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.398+0000 7f81b71ea700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b006fb90 con 0x7f81b0072300 2023-11-28T13:01:40.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.398+0000 7f81b71ea700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b00726d0 con 0x7f81b015cc50 2023-11-28T13:01:40.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81b5787700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f81b00cceb0 0x7f81b00cd2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37584/0 (socket says 172.21.15.111:37584) 2023-11-28T13:01:40.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81b5787700 1 -- 172.21.15.111:0/2498915557 learned_addr learned my addr 172.21.15.111:0/2498915557 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:40.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81affff700 1 --2- 172.21.15.111:0/2498915557 >> v2:172.21.15.149:3300/0 conn(0x7f81b015cc50 0x7f81b00712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634661079 0 0) 0x7f81b00717f0 con 0x7f81b00cceb0 2023-11-28T13:01:40.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a00035b0 con 0x7f81b00cceb0 2023-11-28T13:01:40.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3107339618 0 0) 0x7f81a00035b0 con 0x7f81b00cceb0 2023-11-28T13:01:40.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 >> v2:172.21.15.149:3300/0 conn(0x7f81b015cc50 msgr2=0x7f81b00712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:40.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 --2- 172.21.15.111:0/2498915557 >> v2:172.21.15.149:3300/0 conn(0x7f81b015cc50 0x7f81b00712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 >> v1:172.21.15.149:6789/0 conn(0x7f81b0072300 legacy=0x7f81b006b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b015d0f0 con 0x7f81b00cceb0 2023-11-28T13:01:40.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f81a80019f0 con 0x7f81b00cceb0 2023-11-28T13:01:40.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 >> v1:172.21.15.111:6789/0 conn(0x7f81b00cceb0 legacy=0x7f81b00cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 --2- 172.21.15.111:0/2498915557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81a0003a00 0x7f81a0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 --2- 172.21.15.111:0/2498915557 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81a0004850 0x7f81a0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 --2- 172.21.15.111:0/2498915557 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81a0005200 0x7f81a000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b00726d0 con 0x7f81a0003a00 2023-11-28T13:01:40.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a000c110 con 0x7f81a0004850 2023-11-28T13:01:40.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a000c2d0 con 0x7f81a0005200 2023-11-28T13:01:40.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81b4f86700 1 --2- 172.21.15.111:0/2498915557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81a0003a00 0x7f81a0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81affff700 1 --2- 172.21.15.111:0/2498915557 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81a0004850 0x7f81a0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81b5787700 1 --2- 172.21.15.111:0/2498915557 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81a0005200 0x7f81a000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81b4f86700 1 -- 172.21.15.111:0/2498915557 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81a0005200 msgr2=0x7f81a000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:40.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81b4f86700 1 --2- 172.21.15.111:0/2498915557 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81a0005200 0x7f81a000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81b4f86700 1 -- 172.21.15.111:0/2498915557 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81a0004850 msgr2=0x7f81a0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:40.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.399+0000 7f81b4f86700 1 --2- 172.21.15.111:0/2498915557 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81a0004850 0x7f81a0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81b4f86700 1 -- 172.21.15.111:0/2498915557 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a00035b0 con 0x7f81a0003a00 2023-11-28T13:01:40.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81b5787700 1 -- 172.21.15.111:0/2498915557 reap_dead start 2023-11-28T13:01:40.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81affff700 1 --2- 172.21.15.111:0/2498915557 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81a0004850 0x7f81a0004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:40.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81b4f86700 1 --2- 172.21.15.111:0/2498915557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81a0003a00 0x7f81a0003e10 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f81a40017c0 tx=0x7f81a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ce018e0e5bd9e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:40.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a40052d0 con 0x7f81a0003a00 2023-11-28T13:01:40.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a4005430 con 0x7f81a0003a00 2023-11-28T13:01:40.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81af7fe700 1 -- 172.21.15.111:0/2498915557 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a40038c0 con 0x7f81a0003a00 2023-11-28T13:01:40.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81b71ea700 1 -- 172.21.15.111:0/2498915557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81a0003a00 msgr2=0x7f81a0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81b71ea700 1 --2- 172.21.15.111:0/2498915557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81a0003a00 0x7f81a0003e10 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f81a40017c0 tx=0x7f81a0005880 comp rx=0 tx=0).stop 2023-11-28T13:01:40.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81b71ea700 1 -- 172.21.15.111:0/2498915557 shutdown_connections 2023-11-28T13:01:40.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81b71ea700 1 --2- 172.21.15.111:0/2498915557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81a0003a00 0x7f81a0003e10 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81b71ea700 1 -- 172.21.15.111:0/2498915557 >> 172.21.15.111:0/2498915557 conn(0x7f81b00cf2e0 msgr2=0x7f81b00cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:40.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81b71ea700 1 -- 172.21.15.111:0/2498915557 shutdown_connections 2023-11-28T13:01:40.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.400+0000 7f81b71ea700 1 -- 172.21.15.111:0/2498915557 wait complete. 2023-11-28T13:01:40.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.401+0000 7f81b71ea700 1 Processor -- start 2023-11-28T13:01:40.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.401+0000 7f81b71ea700 1 -- start start 2023-11-28T13:01:40.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.402+0000 7f81b71ea700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81a0003a00 0x7f81b00b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.402+0000 7f81b71ea700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81b0152d90 0x7f81b00c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.402+0000 7f81b71ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00c7dd0 0x7f81b00b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.402+0000 7f81b71ea700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b00c8240 con 0x7f81b00c7dd0 2023-11-28T13:01:40.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.402+0000 7f81b71ea700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b00b0bd0 con 0x7f81b0152d90 2023-11-28T13:01:40.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.402+0000 7f81b71ea700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b00afb30 con 0x7f81a0003a00 2023-11-28T13:01:40.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.402+0000 7f81b5787700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00c7dd0 0x7f81b00b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.402+0000 7f81b5787700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00c7dd0 0x7f81b00b2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36818/0 (socket says 172.21.15.111:36818) 2023-11-28T13:01:40.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81b5787700 1 -- 172.21.15.111:0/683651323 learned_addr learned my addr 172.21.15.111:0/683651323 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:40.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81affff700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81b0152d90 0x7f81b00c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81b4f86700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81a0003a00 0x7f81b00b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81b5787700 1 -- 172.21.15.111:0/683651323 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81a0003a00 msgr2=0x7f81b00b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81b5787700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81a0003a00 0x7f81b00b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81b5787700 1 -- 172.21.15.111:0/683651323 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81b0152d90 msgr2=0x7f81b00c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81b5787700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81b0152d90 0x7f81b00c7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81b5787700 1 -- 172.21.15.111:0/683651323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b015d6b0 con 0x7f81b00c7dd0 2023-11-28T13:01:40.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81b5787700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00c7dd0 0x7f81b00b2110 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f81b00ce8c0 tx=0x7f81a800e4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:40.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81ad7fa700 1 -- 172.21.15.111:0/683651323 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a8017070 con 0x7f81b00c7dd0 2023-11-28T13:01:40.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81ad7fa700 1 -- 172.21.15.111:0/683651323 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a800ee00 con 0x7f81b00c7dd0 2023-11-28T13:01:40.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81b71ea700 1 -- 172.21.15.111:0/683651323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b015d3d0 con 0x7f81b00c7dd0 2023-11-28T13:01:40.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81ad7fa700 1 -- 172.21.15.111:0/683651323 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a8013420 con 0x7f81b00c7dd0 2023-11-28T13:01:40.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.403+0000 7f81b71ea700 1 -- 172.21.15.111:0/683651323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b015d990 con 0x7f81b00c7dd0 2023-11-28T13:01:40.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.404+0000 7f81b71ea700 1 -- 172.21.15.111:0/683651323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b0067170 con 0x7f81b00c7dd0 2023-11-28T13:01:40.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.404+0000 7f81ad7fa700 1 -- 172.21.15.111:0/683651323 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f81a8013580 con 0x7f81b00c7dd0 2023-11-28T13:01:40.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.405+0000 7f81ad7fa700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f81940682f0 0x7f819406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.405+0000 7f81ad7fa700 1 -- 172.21.15.111:0/683651323 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f81a8085da0 con 0x7f81b00c7dd0 2023-11-28T13:01:40.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.405+0000 7f81b4f86700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f81940682f0 0x7f819406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.405+0000 7f81b4f86700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f81940682f0 0x7f819406a7a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f81a4015040 tx=0x7f81a4009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:40.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.409+0000 7f81ad7fa700 1 -- 172.21.15.111:0/683651323 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f81a80524c0 con 0x7f81b00c7dd0 2023-11-28T13:01:40.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.639+0000 7f81b71ea700 1 -- 172.21.15.111:0/683651323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81b0066390 con 0x7f81b00c7dd0 2023-11-28T13:01:40.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.640+0000 7f81ad7fa700 1 -- 172.21.15.111:0/683651323 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f81a8055ae0 con 0x7f81b00c7dd0 2023-11-28T13:01:40.642 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:40.642 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2072621958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2072621958},{"type":"v1","addr":"172.21.15.149:6834","nonce":2072621958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3348427170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3348427170},{"type":"v1","addr":"172.21.15.149:6835","nonce":3348427170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1745443298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1745443298},{"type":"v1","addr":"172.21.15.111:6836","nonce":1745443298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:56.272312+0000","modified":"2023-11-28T13:00:59.284265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5814":{"gid":5814,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/4146274267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":4146274267},{"type":"v1","addr":"172.21.15.111:6837","nonce":4146274267}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T13:01:40.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.642+0000 7f81b71ea700 1 -- 172.21.15.111:0/683651323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f81940682f0 msgr2=0x7f819406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.642+0000 7f81b71ea700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f81940682f0 0x7f819406a7a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f81a4015040 tx=0x7f81a4009030 comp rx=0 tx=0).stop 2023-11-28T13:01:40.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.642+0000 7f81b71ea700 1 -- 172.21.15.111:0/683651323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00c7dd0 msgr2=0x7f81b00b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.642+0000 7f81b71ea700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00c7dd0 0x7f81b00b2110 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f81b00ce8c0 tx=0x7f81a800e4f0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.642+0000 7f81b71ea700 1 -- 172.21.15.111:0/683651323 shutdown_connections 2023-11-28T13:01:40.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.642+0000 7f81b71ea700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81b00c7dd0 0x7f81b00b2110 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.642+0000 7f81b71ea700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f81a0003a00 0x7f81b00b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.642+0000 7f81b71ea700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f81940682f0 0x7f819406a7a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.642+0000 7f81b71ea700 1 --2- 172.21.15.111:0/683651323 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f81b0152d90 0x7f81b00c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.642+0000 7f81b71ea700 1 -- 172.21.15.111:0/683651323 >> 172.21.15.111:0/683651323 conn(0x7f81b00cf2e0 msgr2=0x7f81b00caed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:40.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.642+0000 7f81b71ea700 1 -- 172.21.15.111:0/683651323 shutdown_connections 2023-11-28T13:01:40.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.642+0000 7f81b71ea700 1 -- 172.21.15.111:0/683651323 wait complete. 2023-11-28T13:01:40.645 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 25 2023-11-28T13:01:40.655 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:01:40.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.774+0000 7f63db6f5700 1 Processor -- start 2023-11-28T13:01:40.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.774+0000 7f63db6f5700 1 -- start start 2023-11-28T13:01:40.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.775+0000 7f63db6f5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63d40cdd90 0x7f63d40b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.775+0000 7f63db6f5700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f63d40aecc0 0x7f63d40ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.775+0000 7f63db6f5700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d40ad420 con 0x7f63d4159960 2023-11-28T13:01:40.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.775+0000 7f63db6f5700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d4159d30 con 0x7f63d40aecc0 2023-11-28T13:01:40.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.775+0000 7f63db6f5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40ac810 con 0x7f63d40cdd90 2023-11-28T13:01:40.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63d8c90700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63d40cdd90 0x7f63d40b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63d9c92700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f63d40aecc0 0x7f63d40ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63d9491700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f63d4159960 0x7f63d40cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:46142/0 (socket says 172.21.15.111:46142) 2023-11-28T13:01:40.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63d9491700 1 -- 172.21.15.111:0/2186279865 learned_addr learned my addr 172.21.15.111:0/2186279865 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:40.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63cbfff700 1 -- 172.21.15.111:0/2186279865 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939305155 0 0) 0x7f63d40ad420 con 0x7f63d4159960 2023-11-28T13:01:40.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63cbfff700 1 -- 172.21.15.111:0/2186279865 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c00035b0 con 0x7f63d4159960 2023-11-28T13:01:40.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63d9c92700 1 -- 172.21.15.111:0/2186279865 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63d40cdd90 msgr2=0x7f63d40b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:40.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63d9c92700 1 --2- 172.21.15.111:0/2186279865 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63d40cdd90 0x7f63d40b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63d9c92700 1 -- 172.21.15.111:0/2186279865 >> v1:172.21.15.149:6789/0 conn(0x7f63d4159960 legacy=0x7f63d40cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63d9c92700 1 -- 172.21.15.111:0/2186279865 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d40acaf0 con 0x7f63d40aecc0 2023-11-28T13:01:40.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63d8c90700 1 --2- 172.21.15.111:0/2186279865 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63d40cdd90 0x7f63d40b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:01:40.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63d9c92700 1 --2- 172.21.15.111:0/2186279865 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63d40aecc0 0x7f63d40ad8b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f63cc009d90 tx=0x7f63cc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94ad89f447c4623a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:40.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63cbfff700 1 -- 172.21.15.111:0/2186279865 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc00e070 con 0x7f63d40aecc0 2023-11-28T13:01:40.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63cbfff700 1 -- 172.21.15.111:0/2186279865 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc00bb20 con 0x7f63d40aecc0 2023-11-28T13:01:40.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.776+0000 7f63cbfff700 1 -- 172.21.15.111:0/2186279865 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc012430 con 0x7f63d40aecc0 2023-11-28T13:01:40.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.777+0000 7f63db6f5700 1 -- 172.21.15.111:0/2186279865 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63d40aecc0 msgr2=0x7f63d40ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.777+0000 7f63db6f5700 1 --2- 172.21.15.111:0/2186279865 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63d40aecc0 0x7f63d40ad8b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f63cc009d90 tx=0x7f63cc00b330 comp rx=0 tx=0).stop 2023-11-28T13:01:40.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.777+0000 7f63db6f5700 1 -- 172.21.15.111:0/2186279865 shutdown_connections 2023-11-28T13:01:40.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.777+0000 7f63db6f5700 1 --2- 172.21.15.111:0/2186279865 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63d40aecc0 0x7f63d40ad8b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.777+0000 7f63db6f5700 1 --2- 172.21.15.111:0/2186279865 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63d40cdd90 0x7f63d40b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.777+0000 7f63db6f5700 1 -- 172.21.15.111:0/2186279865 >> 172.21.15.111:0/2186279865 conn(0x7f63d4098ee0 msgr2=0x7f63d406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:40.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.777+0000 7f63db6f5700 1 -- 172.21.15.111:0/2186279865 shutdown_connections 2023-11-28T13:01:40.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.777+0000 7f63db6f5700 1 -- 172.21.15.111:0/2186279865 wait complete. 2023-11-28T13:01:40.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.778+0000 7f63db6f5700 1 Processor -- start 2023-11-28T13:01:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.778+0000 7f63db6f5700 1 -- start start 2023-11-28T13:01:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63db6f5700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63d40aecc0 0x7f63d4159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63db6f5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63d40cdd90 0x7f63d4145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63db6f5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63d4159960 0x7f63d414ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63db6f5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40c09b0 con 0x7f63d4159960 2023-11-28T13:01:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63db6f5700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d406f780 con 0x7f63d40aecc0 2023-11-28T13:01:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63db6f5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d415c610 con 0x7f63d40cdd90 2023-11-28T13:01:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63d9c92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63d4159960 0x7f63d414ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63d9c92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63d4159960 0x7f63d414ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36832/0 (socket says 172.21.15.111:36832) 2023-11-28T13:01:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63d9c92700 1 -- 172.21.15.111:0/2771602757 learned_addr learned my addr 172.21.15.111:0/2771602757 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63d9491700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63d40aecc0 0x7f63d4159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63d8c90700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63d40cdd90 0x7f63d4145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63d9c92700 1 -- 172.21.15.111:0/2771602757 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63d40cdd90 msgr2=0x7f63d4145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63d9c92700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63d40cdd90 0x7f63d4145cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63d9c92700 1 -- 172.21.15.111:0/2771602757 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63d40aecc0 msgr2=0x7f63d4159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63d9c92700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63d40aecc0 0x7f63d4159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63d9c92700 1 -- 172.21.15.111:0/2771602757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d00094a0 con 0x7f63d4159960 2023-11-28T13:01:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.779+0000 7f63d9c92700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63d4159960 0x7f63d414ca90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00b370 tx=0x7f63cc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.780+0000 7f63c9ffb700 1 -- 172.21.15.111:0/2771602757 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc012430 con 0x7f63d4159960 2023-11-28T13:01:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.780+0000 7f63c9ffb700 1 -- 172.21.15.111:0/2771602757 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc020070 con 0x7f63d4159960 2023-11-28T13:01:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.780+0000 7f63db6f5700 1 -- 172.21.15.111:0/2771602757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc009a70 con 0x7f63d4159960 2023-11-28T13:01:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.780+0000 7f63c9ffb700 1 -- 172.21.15.111:0/2771602757 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc0098f0 con 0x7f63d4159960 2023-11-28T13:01:40.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.780+0000 7f63db6f5700 1 -- 172.21.15.111:0/2771602757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d40ad0b0 con 0x7f63d4159960 2023-11-28T13:01:40.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.781+0000 7f63db6f5700 1 -- 172.21.15.111:0/2771602757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d40acdd0 con 0x7f63d4159960 2023-11-28T13:01:40.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.781+0000 7f63c9ffb700 1 -- 172.21.15.111:0/2771602757 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f63cc02e070 con 0x7f63d4159960 2023-11-28T13:01:40.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.782+0000 7f63c9ffb700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63b80682f0 0x7f63b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:40.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.782+0000 7f63c9ffb700 1 -- 172.21.15.111:0/2771602757 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f63cc022070 con 0x7f63d4159960 2023-11-28T13:01:40.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.782+0000 7f63d9491700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63b80682f0 0x7f63b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:40.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.782+0000 7f63d9491700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63b80682f0 0x7f63b806a7a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f63c4001230 tx=0x7f63c4007df0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:40.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.785+0000 7f63c9ffb700 1 -- 172.21.15.111:0/2771602757 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f63cc058170 con 0x7f63d4159960 2023-11-28T13:01:40.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.969+0000 7f63db6f5700 1 -- 172.21.15.111:0/2771602757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63d40addf0 con 0x7f63d4159960 2023-11-28T13:01:40.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.970+0000 7f63c9ffb700 1 -- 172.21.15.111:0/2771602757 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+18228 (secure 0 0 0) 0x7f63cc02c020 con 0x7f63d4159960 2023-11-28T13:01:40.972 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:40.972 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":55,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:57.274026+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:53.965825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:55.242768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:01:40.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.972+0000 7f63db6f5700 1 -- 172.21.15.111:0/2771602757 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63b80682f0 msgr2=0x7f63b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.972+0000 7f63db6f5700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63b80682f0 0x7f63b806a7a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f63c4001230 tx=0x7f63c4007df0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.972+0000 7f63db6f5700 1 -- 172.21.15.111:0/2771602757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63d4159960 msgr2=0x7f63d414ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:40.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.972+0000 7f63db6f5700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63d4159960 0x7f63d414ca90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00b370 tx=0x7f63cc004ad0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.972+0000 7f63db6f5700 1 -- 172.21.15.111:0/2771602757 shutdown_connections 2023-11-28T13:01:40.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.972+0000 7f63db6f5700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63d4159960 0x7f63d414ca90 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.972+0000 7f63db6f5700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63b80682f0 0x7f63b806a7a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.972+0000 7f63db6f5700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63d40aecc0 0x7f63d4159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.972+0000 7f63db6f5700 1 --2- 172.21.15.111:0/2771602757 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63d40cdd90 0x7f63d4145cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:40.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.972+0000 7f63db6f5700 1 -- 172.21.15.111:0/2771602757 >> 172.21.15.111:0/2771602757 conn(0x7f63d4098ee0 msgr2=0x7f63d406ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:40.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.972+0000 7f63db6f5700 1 -- 172.21.15.111:0/2771602757 shutdown_connections 2023-11-28T13:01:40.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:40.972+0000 7f63db6f5700 1 -- 172.21.15.111:0/2771602757 wait complete. 2023-11-28T13:01:40.986 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T13:01:40.986 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:01:41.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.103+0000 7f30f4a67700 1 Processor -- start 2023-11-28T13:01:41.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.103+0000 7f30f4a67700 1 -- start start 2023-11-28T13:01:41.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.104+0000 7f30f4a67700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f30f00cfc90 0x7f30f00d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.104+0000 7f30f4a67700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f00d05a0 con 0x7f30f00cc5a0 2023-11-28T13:01:41.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.104+0000 7f30f4a67700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f00ca260 con 0x7f30f00cdef0 2023-11-28T13:01:41.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.104+0000 7f30f4a67700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f00ccaa0 con 0x7f30f00cfc90 2023-11-28T13:01:41.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.104+0000 7f30ee59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f30f00cfc90 0x7f30f00d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.105+0000 7f30edd9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f30f00cc5a0 0x7f30f00cd7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37616/0 (socket says 172.21.15.111:37616) 2023-11-28T13:01:41.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.105+0000 7f30edd9b700 1 -- 172.21.15.111:0/4214641821 learned_addr learned my addr 172.21.15.111:0/4214641821 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:41.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.105+0000 7f30ee59c700 1 -- 172.21.15.111:0/4214641821 >> v1:172.21.15.149:6789/0 conn(0x7f30f00cdef0 legacy=0x7f30f00ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.105+0000 7f30ee59c700 1 -- 172.21.15.111:0/4214641821 >> v1:172.21.15.111:6789/0 conn(0x7f30f00cc5a0 legacy=0x7f30f00cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.105+0000 7f30ee59c700 1 -- 172.21.15.111:0/4214641821 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f00ccd80 con 0x7f30f00cfc90 2023-11-28T13:01:41.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.105+0000 7f30ee59c700 1 --2- 172.21.15.111:0/4214641821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30f00cfc90 0x7f30f00d0060 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f30e400bae0 tx=0x7f30e400bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f02a8add3162dc2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:41.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.105+0000 7f30ed59a700 1 -- 172.21.15.111:0/4214641821 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e400dd30 con 0x7f30f00cfc90 2023-11-28T13:01:41.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.105+0000 7f30ed59a700 1 -- 172.21.15.111:0/4214641821 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e400de90 con 0x7f30f00cfc90 2023-11-28T13:01:41.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.105+0000 7f30ed59a700 1 -- 172.21.15.111:0/4214641821 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e4010690 con 0x7f30f00cfc90 2023-11-28T13:01:41.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.106+0000 7f30f4a67700 1 -- 172.21.15.111:0/4214641821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30f00cfc90 msgr2=0x7f30f00d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.106+0000 7f30f4a67700 1 --2- 172.21.15.111:0/4214641821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30f00cfc90 0x7f30f00d0060 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f30e400bae0 tx=0x7f30e400bdf0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.106+0000 7f30f4a67700 1 -- 172.21.15.111:0/4214641821 shutdown_connections 2023-11-28T13:01:41.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.106+0000 7f30f4a67700 1 --2- 172.21.15.111:0/4214641821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30f00cfc90 0x7f30f00d0060 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.106+0000 7f30f4a67700 1 -- 172.21.15.111:0/4214641821 >> 172.21.15.111:0/4214641821 conn(0x7f30f0098ee0 msgr2=0x7f30f006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:41.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.106+0000 7f30f4a67700 1 -- 172.21.15.111:0/4214641821 shutdown_connections 2023-11-28T13:01:41.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.106+0000 7f30f4a67700 1 -- 172.21.15.111:0/4214641821 wait complete. 2023-11-28T13:01:41.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.107+0000 7f30f4a67700 1 Processor -- start 2023-11-28T13:01:41.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.107+0000 7f30f4a67700 1 -- start start 2023-11-28T13:01:41.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30f4a67700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f30f00cc5a0 0x7f30f00c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30f4a67700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30f00cdef0 0x7f30f00c2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30f4a67700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f30f00cfc90 0x7f30f00b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30f4a67700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f00c9440 con 0x7f30f00cdef0 2023-11-28T13:01:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30f4a67700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f00c8b10 con 0x7f30f00cc5a0 2023-11-28T13:01:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30f4a67700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f00c8df0 con 0x7f30f00cfc90 2023-11-28T13:01:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30edd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30f00cdef0 0x7f30f00c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30edd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30f00cdef0 0x7f30f00c2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36844/0 (socket says 172.21.15.111:36844) 2023-11-28T13:01:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30edd9b700 1 -- 172.21.15.111:0/1630555836 learned_addr learned my addr 172.21.15.111:0/1630555836 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30ee59c700 1 --2- 172.21.15.111:0/1630555836 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f30f00cc5a0 0x7f30f00c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30edd9b700 1 -- 172.21.15.111:0/1630555836 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f30f00cfc90 msgr2=0x7f30f00b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30edd9b700 1 --2- 172.21.15.111:0/1630555836 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f30f00cfc90 0x7f30f00b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30edd9b700 1 -- 172.21.15.111:0/1630555836 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f30f00cc5a0 msgr2=0x7f30f00c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30edd9b700 1 --2- 172.21.15.111:0/1630555836 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f30f00cc5a0 0x7f30f00c98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30edd9b700 1 -- 172.21.15.111:0/1630555836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e400b740 con 0x7f30f00cdef0 2023-11-28T13:01:41.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30edd9b700 1 --2- 172.21.15.111:0/1630555836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30f00cdef0 0x7f30f00c2900 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f30e000c450 tx=0x7f30e000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:41.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30deffd700 1 -- 172.21.15.111:0/1630555836 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e0005d70 con 0x7f30f00cdef0 2023-11-28T13:01:41.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30deffd700 1 -- 172.21.15.111:0/1630555836 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e00070d0 con 0x7f30f00cdef0 2023-11-28T13:01:41.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.108+0000 7f30f4a67700 1 -- 172.21.15.111:0/1630555836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f00cd340 con 0x7f30f00cdef0 2023-11-28T13:01:41.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.109+0000 7f30deffd700 1 -- 172.21.15.111:0/1630555836 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e0007440 con 0x7f30f00cdef0 2023-11-28T13:01:41.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.109+0000 7f30f4a67700 1 -- 172.21.15.111:0/1630555836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f00cd060 con 0x7f30f00cdef0 2023-11-28T13:01:41.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.109+0000 7f30f4a67700 1 -- 172.21.15.111:0/1630555836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f0143280 con 0x7f30f00cdef0 2023-11-28T13:01:41.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.110+0000 7f30deffd700 1 -- 172.21.15.111:0/1630555836 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f30e0019030 con 0x7f30f00cdef0 2023-11-28T13:01:41.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.110+0000 7f30deffd700 1 --2- 172.21.15.111:0/1630555836 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f30d00682f0 0x7f30d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.110+0000 7f30ee59c700 1 --2- 172.21.15.111:0/1630555836 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f30d00682f0 0x7f30d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.110+0000 7f30deffd700 1 -- 172.21.15.111:0/1630555836 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f30e0084e50 con 0x7f30f00cdef0 2023-11-28T13:01:41.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.111+0000 7f30ee59c700 1 --2- 172.21.15.111:0/1630555836 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f30d00682f0 0x7f30d006a7a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f30e400be30 tx=0x7f30e4013000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:41.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.115+0000 7f30deffd700 1 -- 172.21.15.111:0/1630555836 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f30e004b5e0 con 0x7f30f00cdef0 2023-11-28T13:01:41.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.341+0000 7f30f4a67700 1 -- 172.21.15.111:0/1630555836 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30f00d05a0 con 0x7f30f00cdef0 2023-11-28T13:01:41.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.341+0000 7f30deffd700 1 -- 172.21.15.111:0/1630555836 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4660 (secure 0 0 0) 0x7f30f00d05a0 con 0x7f30f00cdef0 2023-11-28T13:01:41.343 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:41.343 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2072621958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2072621958},{"type":"v1","addr":"172.21.15.149:6834","nonce":2072621958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3348427170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3348427170},{"type":"v1","addr":"172.21.15.149:6835","nonce":3348427170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1745443298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1745443298},{"type":"v1","addr":"172.21.15.111:6836","nonce":1745443298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:56.272312+0000","modified":"2023-11-28T13:00:59.284265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5814":{"gid":5814,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/4146274267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":4146274267},{"type":"v1","addr":"172.21.15.111:6837","nonce":4146274267}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T13:01:41.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.343+0000 7f30f4a67700 1 -- 172.21.15.111:0/1630555836 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f30d00682f0 msgr2=0x7f30d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.343+0000 7f30f4a67700 1 --2- 172.21.15.111:0/1630555836 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f30d00682f0 0x7f30d006a7a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f30e400be30 tx=0x7f30e4013000 comp rx=0 tx=0).stop 2023-11-28T13:01:41.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.343+0000 7f30f4a67700 1 -- 172.21.15.111:0/1630555836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30f00cdef0 msgr2=0x7f30f00c2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.343+0000 7f30f4a67700 1 --2- 172.21.15.111:0/1630555836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30f00cdef0 0x7f30f00c2900 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f30e000c450 tx=0x7f30e000c810 comp rx=0 tx=0).stop 2023-11-28T13:01:41.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.343+0000 7f30f4a67700 1 -- 172.21.15.111:0/1630555836 shutdown_connections 2023-11-28T13:01:41.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.343+0000 7f30f4a67700 1 --2- 172.21.15.111:0/1630555836 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f30f00cfc90 0x7f30f00b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.343+0000 7f30f4a67700 1 --2- 172.21.15.111:0/1630555836 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f30d00682f0 0x7f30d006a7a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.343+0000 7f30f4a67700 1 --2- 172.21.15.111:0/1630555836 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f30f00cc5a0 0x7f30f00c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.343+0000 7f30f4a67700 1 --2- 172.21.15.111:0/1630555836 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30f00cdef0 0x7f30f00c2900 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.343+0000 7f30f4a67700 1 -- 172.21.15.111:0/1630555836 >> 172.21.15.111:0/1630555836 conn(0x7f30f0098ee0 msgr2=0x7f30f00ca5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:41.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.343+0000 7f30f4a67700 1 -- 172.21.15.111:0/1630555836 shutdown_connections 2023-11-28T13:01:41.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.343+0000 7f30f4a67700 1 -- 172.21.15.111:0/1630555836 wait complete. 2023-11-28T13:01:41.348 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 25 2023-11-28T13:01:41.357 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:01:41.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.475+0000 7fb2d26c8700 1 Processor -- start 2023-11-28T13:01:41.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.475+0000 7fb2d26c8700 1 -- start start 2023-11-28T13:01:41.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.476+0000 7fb2d26c8700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb2cc0b0440 0x7fb2cc0b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.476+0000 7fb2d26c8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb2cc0ca2f0 0x7fb2cc0ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.476+0000 7fb2d26c8700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc0cc750 con 0x7fb2cc15c6c0 2023-11-28T13:01:41.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.476+0000 7fb2d26c8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc0b0260 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.476+0000 7fb2d26c8700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc0b0e20 con 0x7fb2cc0b0440 2023-11-28T13:01:41.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.476+0000 7fb2d0c65700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb2cc0ca2f0 0x7fb2cc0ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.476+0000 7fb2d0c65700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb2cc0ca2f0 0x7fb2cc0ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36854/0 (socket says 172.21.15.111:36854) 2023-11-28T13:01:41.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.476+0000 7fb2d0c65700 1 -- 172.21.15.111:0/4104172073 learned_addr learned my addr 172.21.15.111:0/4104172073 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:41.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2cb7fe700 1 --2- 172.21.15.111:0/4104172073 >> v2:172.21.15.149:3300/0 conn(0x7fb2cc0b0440 0x7fb2cc0b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2d0c65700 1 -- 172.21.15.111:0/4104172073 >> v2:172.21.15.149:3300/0 conn(0x7fb2cc0b0440 msgr2=0x7fb2cc0b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:41.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2d0c65700 1 --2- 172.21.15.111:0/4104172073 >> v2:172.21.15.149:3300/0 conn(0x7fb2cc0b0440 0x7fb2cc0b08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2d0c65700 1 -- 172.21.15.111:0/4104172073 >> v1:172.21.15.149:6789/0 conn(0x7fb2cc15c6c0 legacy=0x7fb2cc0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2d0c65700 1 -- 172.21.15.111:0/4104172073 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc0cca30 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2d0c65700 1 --2- 172.21.15.111:0/4104172073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc0ca2f0 0x7fb2cc0ca750 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc066980 tx=0x7fb2c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634d5d81cb37fa09 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:41.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2caffd700 1 -- 172.21.15.111:0/4104172073 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c400e070 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2caffd700 1 -- 172.21.15.111:0/4104172073 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c400b9c0 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2caffd700 1 -- 172.21.15.111:0/4104172073 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c400bd70 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2d26c8700 1 -- 172.21.15.111:0/4104172073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc0ca2f0 msgr2=0x7fb2cc0ca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2d26c8700 1 --2- 172.21.15.111:0/4104172073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc0ca2f0 0x7fb2cc0ca750 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc066980 tx=0x7fb2c400b330 comp rx=0 tx=0).stop 2023-11-28T13:01:41.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2d26c8700 1 -- 172.21.15.111:0/4104172073 shutdown_connections 2023-11-28T13:01:41.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2d26c8700 1 --2- 172.21.15.111:0/4104172073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc0ca2f0 0x7fb2cc0ca750 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.477+0000 7fb2d26c8700 1 --2- 172.21.15.111:0/4104172073 >> v2:172.21.15.149:3300/0 conn(0x7fb2cc0b0440 0x7fb2cc0b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.478+0000 7fb2d26c8700 1 -- 172.21.15.111:0/4104172073 >> 172.21.15.111:0/4104172073 conn(0x7fb2cc098ee0 msgr2=0x7fb2cc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:41.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.478+0000 7fb2d26c8700 1 -- 172.21.15.111:0/4104172073 shutdown_connections 2023-11-28T13:01:41.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.478+0000 7fb2d26c8700 1 -- 172.21.15.111:0/4104172073 wait complete. 2023-11-28T13:01:41.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.479+0000 7fb2d26c8700 1 Processor -- start 2023-11-28T13:01:41.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.479+0000 7fb2d26c8700 1 -- start start 2023-11-28T13:01:41.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2d26c8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb2cc0b0440 0x7fb2cc155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2d26c8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc0ca2f0 0x7fb2cc1f5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2d26c8700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb2cc15c6c0 0x7fb2cc1f6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2d26c8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc0b4660 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2d26c8700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc0b4380 con 0x7fb2cc15c6c0 2023-11-28T13:01:41.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2d26c8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc0c6e10 con 0x7fb2cc0b0440 2023-11-28T13:01:41.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc0ca2f0 0x7fb2cc1f5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc0ca2f0 0x7fb2cc1f5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36858/0 (socket says 172.21.15.111:36858) 2023-11-28T13:01:41.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2cb7fe700 1 -- 172.21.15.111:0/1016484061 learned_addr learned my addr 172.21.15.111:0/1016484061 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:41.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2cb7fe700 1 -- 172.21.15.111:0/1016484061 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb2cc0b0440 msgr2=0x7fb2cc155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2cb7fe700 1 --2- 172.21.15.111:0/1016484061 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb2cc0b0440 0x7fb2cc155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2d0c65700 1 --2- 172.21.15.111:0/1016484061 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb2cc15c6c0 0x7fb2cc1f6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2cb7fe700 1 -- 172.21.15.111:0/1016484061 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb2cc15c6c0 msgr2=0x7fb2cc1f6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2cb7fe700 1 --2- 172.21.15.111:0/1016484061 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb2cc15c6c0 0x7fb2cc1f6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2cb7fe700 1 -- 172.21.15.111:0/1016484061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c4009b30 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2cb7fe700 1 --2- 172.21.15.111:0/1016484061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc0ca2f0 0x7fb2cc1f5d20 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00d800 tx=0x7fb2bc00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:41.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2c8ff9700 1 -- 172.21.15.111:0/1016484061 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc0125f0 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2c8ff9700 1 -- 172.21.15.111:0/1016484061 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2bc012750 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2d26c8700 1 -- 172.21.15.111:0/1016484061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2cc0ccff0 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2c8ff9700 1 -- 172.21.15.111:0/1016484061 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc012b00 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.480+0000 7fb2d26c8700 1 -- 172.21.15.111:0/1016484061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2cc0ccd10 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.481+0000 7fb2d26c8700 1 -- 172.21.15.111:0/1016484061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2cc0cd2d0 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.482+0000 7fb2c8ff9700 1 -- 172.21.15.111:0/1016484061 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb2bc012c60 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.482+0000 7fb2c8ff9700 1 --2- 172.21.15.111:0/1016484061 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb2b80682f0 0x7fb2b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.482+0000 7fb2cbfff700 1 --2- 172.21.15.111:0/1016484061 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb2b80682f0 0x7fb2b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.482+0000 7fb2c8ff9700 1 -- 172.21.15.111:0/1016484061 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb2bc086290 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.482+0000 7fb2cbfff700 1 --2- 172.21.15.111:0/1016484061 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb2b80682f0 0x7fb2b806a7a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0000f60 tx=0x7fb2c0007a40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:41.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.486+0000 7fb2c8ff9700 1 -- 172.21.15.111:0/1016484061 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb2bc0529b0 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.715+0000 7fb2d26c8700 1 -- 172.21.15.111:0/1016484061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2cc0cdf80 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.715+0000 7fb2c8ff9700 1 -- 172.21.15.111:0/1016484061 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4660 (secure 0 0 0) 0x7fb2bc055fd0 con 0x7fb2cc0ca2f0 2023-11-28T13:01:41.717 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:41.717 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2072621958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2072621958},{"type":"v1","addr":"172.21.15.149:6834","nonce":2072621958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3348427170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3348427170},{"type":"v1","addr":"172.21.15.149:6835","nonce":3348427170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1745443298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1745443298},{"type":"v1","addr":"172.21.15.111:6836","nonce":1745443298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:00:56.272312+0000","modified":"2023-11-28T13:00:59.284265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5814},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5814":{"gid":5814,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/4146274267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":4146274267},{"type":"v1","addr":"172.21.15.111:6837","nonce":4146274267}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T13:01:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.717+0000 7fb2d26c8700 1 -- 172.21.15.111:0/1016484061 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb2b80682f0 msgr2=0x7fb2b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.717+0000 7fb2d26c8700 1 --2- 172.21.15.111:0/1016484061 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb2b80682f0 0x7fb2b806a7a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0000f60 tx=0x7fb2c0007a40 comp rx=0 tx=0).stop 2023-11-28T13:01:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.717+0000 7fb2d26c8700 1 -- 172.21.15.111:0/1016484061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc0ca2f0 msgr2=0x7fb2cc1f5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.717+0000 7fb2d26c8700 1 --2- 172.21.15.111:0/1016484061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc0ca2f0 0x7fb2cc1f5d20 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00d800 tx=0x7fb2bc00dbc0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.718+0000 7fb2d26c8700 1 -- 172.21.15.111:0/1016484061 shutdown_connections 2023-11-28T13:01:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.718+0000 7fb2d26c8700 1 --2- 172.21.15.111:0/1016484061 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb2b80682f0 0x7fb2b806a7a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.718+0000 7fb2d26c8700 1 --2- 172.21.15.111:0/1016484061 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb2cc15c6c0 0x7fb2cc1f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.718+0000 7fb2d26c8700 1 --2- 172.21.15.111:0/1016484061 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb2cc0b0440 0x7fb2cc155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.718+0000 7fb2d26c8700 1 --2- 172.21.15.111:0/1016484061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2cc0ca2f0 0x7fb2cc1f5d20 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.718+0000 7fb2d26c8700 1 -- 172.21.15.111:0/1016484061 >> 172.21.15.111:0/1016484061 conn(0x7fb2cc098ee0 msgr2=0x7fb2cc06ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.718+0000 7fb2d26c8700 1 -- 172.21.15.111:0/1016484061 shutdown_connections 2023-11-28T13:01:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.718+0000 7fb2d26c8700 1 -- 172.21.15.111:0/1016484061 wait complete. 2023-11-28T13:01:41.720 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 25 2023-11-28T13:01:41.731 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:01:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.850+0000 7ffa9fe28700 1 Processor -- start 2023-11-28T13:01:41.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.850+0000 7ffa9fe28700 1 -- start start 2023-11-28T13:01:41.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9fe28700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ffa980b0d70 0x7ffa981604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9fe28700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa980b1140 0x7ffa980ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9fe28700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9815cd90 con 0x7ffa980b15b0 2023-11-28T13:01:41.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9fe28700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa981613e0 con 0x7ffa980b0d70 2023-11-28T13:01:41.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9fe28700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa98160b40 con 0x7ffa980b1140 2023-11-28T13:01:41.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9d3c3700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa980b1140 0x7ffa980ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9dbc4700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ffa980b0d70 0x7ffa981604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9d3c3700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa980b1140 0x7ffa980ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:58612/0 (socket says 172.21.15.111:58612) 2023-11-28T13:01:41.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9d3c3700 1 -- 172.21.15.111:0/4150137934 learned_addr learned my addr 172.21.15.111:0/4150137934 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:41.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9e3c5700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7ffa980b15b0 0x7ffa980cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:46188/0 (socket says 172.21.15.111:46188) 2023-11-28T13:01:41.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9dbc4700 1 -- 172.21.15.111:0/4150137934 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa980b1140 msgr2=0x7ffa980ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:41.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9dbc4700 1 --2- 172.21.15.111:0/4150137934 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa980b1140 0x7ffa980ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9dbc4700 1 -- 172.21.15.111:0/4150137934 >> v1:172.21.15.149:6789/0 conn(0x7ffa980b15b0 legacy=0x7ffa980cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9cbc2700 1 -- 172.21.15.111:0/4150137934 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945723645 0 0) 0x7ffa90001490 con 0x7ffa980b15b0 2023-11-28T13:01:41.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9dbc4700 1 -- 172.21.15.111:0/4150137934 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa98160e20 con 0x7ffa980b0d70 2023-11-28T13:01:41.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.851+0000 7ffa9d3c3700 1 --2- 172.21.15.111:0/4150137934 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa980b1140 0x7ffa980ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:01:41.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.852+0000 7ffa9dbc4700 1 --2- 172.21.15.111:0/4150137934 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa980b0d70 0x7ffa981604b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009820 tx=0x7ffa94009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16af7896d96e7992 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:41.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.852+0000 7ffa9cbc2700 1 -- 172.21.15.111:0/4150137934 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa94005e30 con 0x7ffa980b0d70 2023-11-28T13:01:41.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.852+0000 7ffa9cbc2700 1 -- 172.21.15.111:0/4150137934 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa94006030 con 0x7ffa980b0d70 2023-11-28T13:01:41.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.852+0000 7ffa9cbc2700 1 -- 172.21.15.111:0/4150137934 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa940063c0 con 0x7ffa980b0d70 2023-11-28T13:01:41.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.852+0000 7ffa9fe28700 1 -- 172.21.15.111:0/4150137934 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa980b0d70 msgr2=0x7ffa981604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.852+0000 7ffa9fe28700 1 --2- 172.21.15.111:0/4150137934 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa980b0d70 0x7ffa981604b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009820 tx=0x7ffa94009b30 comp rx=0 tx=0).stop 2023-11-28T13:01:41.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.852+0000 7ffa9fe28700 1 -- 172.21.15.111:0/4150137934 shutdown_connections 2023-11-28T13:01:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.852+0000 7ffa9fe28700 1 --2- 172.21.15.111:0/4150137934 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa980b0d70 0x7ffa981604b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.852+0000 7ffa9fe28700 1 --2- 172.21.15.111:0/4150137934 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa980b1140 0x7ffa980ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.852+0000 7ffa9fe28700 1 -- 172.21.15.111:0/4150137934 >> 172.21.15.111:0/4150137934 conn(0x7ffa98098ee0 msgr2=0x7ffa9806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:41.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.852+0000 7ffa9fe28700 1 -- 172.21.15.111:0/4150137934 shutdown_connections 2023-11-28T13:01:41.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.852+0000 7ffa9fe28700 1 -- 172.21.15.111:0/4150137934 wait complete. 2023-11-28T13:01:41.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.854+0000 7ffa9fe28700 1 Processor -- start 2023-11-28T13:01:41.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.854+0000 7ffa9fe28700 1 -- start start 2023-11-28T13:01:41.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.854+0000 7ffa9fe28700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa980b0d70 0x7ffa981f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.854+0000 7ffa9fe28700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa980b1140 0x7ffa980b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.854+0000 7ffa9fe28700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa980b15b0 0x7ffa9815bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.854+0000 7ffa9fe28700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa981f34a0 con 0x7ffa980b0d70 2023-11-28T13:01:41.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.854+0000 7ffa9fe28700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9815e2b0 con 0x7ffa980b15b0 2023-11-28T13:01:41.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.854+0000 7ffa9fe28700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9815df80 con 0x7ffa980b1140 2023-11-28T13:01:41.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.854+0000 7ffa9dbc4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa980b0d70 0x7ffa981f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.854+0000 7ffa9dbc4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa980b0d70 0x7ffa981f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36870/0 (socket says 172.21.15.111:36870) 2023-11-28T13:01:41.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.854+0000 7ffa9dbc4700 1 -- 172.21.15.111:0/3761270923 learned_addr learned my addr 172.21.15.111:0/3761270923 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:41.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.855+0000 7ffa9e3c5700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa980b15b0 0x7ffa9815bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.855+0000 7ffa9d3c3700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa980b1140 0x7ffa980b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.855+0000 7ffa9dbc4700 1 -- 172.21.15.111:0/3761270923 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa980b1140 msgr2=0x7ffa980b2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.855+0000 7ffa9dbc4700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa980b1140 0x7ffa980b2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.855+0000 7ffa9dbc4700 1 -- 172.21.15.111:0/3761270923 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa980b15b0 msgr2=0x7ffa9815bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:41.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.855+0000 7ffa9dbc4700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa980b15b0 0x7ffa9815bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:41.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.855+0000 7ffa9dbc4700 1 -- 172.21.15.111:0/3761270923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa88009550 con 0x7ffa980b0d70 2023-11-28T13:01:41.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.855+0000 7ffa9dbc4700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa980b0d70 0x7ffa981f22b0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009b70 tx=0x7ffa94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:41.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.855+0000 7ffa8e7fc700 1 -- 172.21.15.111:0/3761270923 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa940038a0 con 0x7ffa980b0d70 2023-11-28T13:01:41.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.855+0000 7ffa8e7fc700 1 -- 172.21.15.111:0/3761270923 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa94003a00 con 0x7ffa980b0d70 2023-11-28T13:01:41.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.855+0000 7ffa9fe28700 1 -- 172.21.15.111:0/3761270923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa94009430 con 0x7ffa980b0d70 2023-11-28T13:01:41.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.855+0000 7ffa8e7fc700 1 -- 172.21.15.111:0/3761270923 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa94003db0 con 0x7ffa980b0d70 2023-11-28T13:01:41.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.855+0000 7ffa9fe28700 1 -- 172.21.15.111:0/3761270923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa980af340 con 0x7ffa980b0d70 2023-11-28T13:01:41.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.856+0000 7ffa9fe28700 1 -- 172.21.15.111:0/3761270923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa98161100 con 0x7ffa980b0d70 2023-11-28T13:01:41.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.856+0000 7ffa8e7fc700 1 -- 172.21.15.111:0/3761270923 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffa9400f050 con 0x7ffa980b0d70 2023-11-28T13:01:41.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.857+0000 7ffa8e7fc700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffa7c0682f0 0x7ffa7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:41.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.857+0000 7ffa8e7fc700 1 -- 172.21.15.111:0/3761270923 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ffa94087830 con 0x7ffa980b0d70 2023-11-28T13:01:41.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.857+0000 7ffa9d3c3700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffa7c0682f0 0x7ffa7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:41.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.857+0000 7ffa9d3c3700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffa7c0682f0 0x7ffa7c06a7a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ffa980b0a30 tx=0x7ffa880092c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:41.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:41.861+0000 7ffa8e7fc700 1 -- 172.21.15.111:0/3761270923 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffa94053ef0 con 0x7ffa980b0d70 2023-11-28T13:01:42.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.044+0000 7ffa9fe28700 1 -- 172.21.15.111:0/3761270923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa9815e590 con 0x7ffa980b0d70 2023-11-28T13:01:42.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.045+0000 7ffa8e7fc700 1 -- 172.21.15.111:0/3761270923 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+18228 (secure 0 0 0) 0x7ffa94057510 con 0x7ffa980b0d70 2023-11-28T13:01:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:42.047 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":55,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:00:57.274026+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:00:53.965825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-11-28T13:00:55.242768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:01:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.047+0000 7ffa9fe28700 1 -- 172.21.15.111:0/3761270923 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffa7c0682f0 msgr2=0x7ffa7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.047+0000 7ffa9fe28700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffa7c0682f0 0x7ffa7c06a7a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ffa980b0a30 tx=0x7ffa880092c0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.047+0000 7ffa9fe28700 1 -- 172.21.15.111:0/3761270923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa980b0d70 msgr2=0x7ffa981f22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.047+0000 7ffa9fe28700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa980b0d70 0x7ffa981f22b0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009b70 tx=0x7ffa94004750 comp rx=0 tx=0).stop 2023-11-28T13:01:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.047+0000 7ffa9fe28700 1 -- 172.21.15.111:0/3761270923 shutdown_connections 2023-11-28T13:01:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.047+0000 7ffa9fe28700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffa7c0682f0 0x7ffa7c06a7a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.047+0000 7ffa9fe28700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffa980b15b0 0x7ffa9815bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.047+0000 7ffa9fe28700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa980b0d70 0x7ffa981f22b0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.047+0000 7ffa9fe28700 1 --2- 172.21.15.111:0/3761270923 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffa980b1140 0x7ffa980b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.047+0000 7ffa9fe28700 1 -- 172.21.15.111:0/3761270923 >> 172.21.15.111:0/3761270923 conn(0x7ffa98098ee0 msgr2=0x7ffa980ac670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:42.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.047+0000 7ffa9fe28700 1 -- 172.21.15.111:0/3761270923 shutdown_connections 2023-11-28T13:01:42.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.047+0000 7ffa9fe28700 1 -- 172.21.15.111:0/3761270923 wait complete. 2023-11-28T13:01:42.061 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T13:01:42.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.179+0000 7f800745a700 1 Processor -- start 2023-11-28T13:01:42.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.179+0000 7f800745a700 1 -- start start 2023-11-28T13:01:42.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.180+0000 7f800745a700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f80000d2df0 0x7f80000cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.180+0000 7f800745a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f80000b4fc0 0x7f80000d31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.180+0000 7f800745a700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80000d1950 con 0x7f80000b1c60 2023-11-28T13:01:42.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.180+0000 7f800745a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80000d3d30 con 0x7f80000b4fc0 2023-11-28T13:01:42.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.180+0000 7f800745a700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80000b2500 con 0x7f80000d2df0 2023-11-28T13:01:42.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.180+0000 7f80049f5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f80000b4fc0 0x7f80000d31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.180+0000 7f80049f5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f80000b4fc0 0x7f80000d31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36878/0 (socket says 172.21.15.111:36878) 2023-11-28T13:01:42.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.180+0000 7f80049f5700 1 -- 172.21.15.111:0/2387703640 learned_addr learned my addr 172.21.15.111:0/2387703640 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:42.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.180+0000 7f80051f6700 1 --2- 172.21.15.111:0/2387703640 >> v2:172.21.15.149:3300/0 conn(0x7f80000d2df0 0x7f80000cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f80049f5700 1 -- 172.21.15.111:0/2387703640 >> v2:172.21.15.149:3300/0 conn(0x7f80000d2df0 msgr2=0x7f80000cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:42.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f80049f5700 1 --2- 172.21.15.111:0/2387703640 >> v2:172.21.15.149:3300/0 conn(0x7f80000d2df0 0x7f80000cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f80049f5700 1 -- 172.21.15.111:0/2387703640 >> v1:172.21.15.149:6789/0 conn(0x7f80000b1c60 legacy=0x7f80000cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:42.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f80049f5700 1 -- 172.21.15.111:0/2387703640 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80000d3a50 con 0x7f80000b4fc0 2023-11-28T13:01:42.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f80049f5700 1 --2- 172.21.15.111:0/2387703640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80000b4fc0 0x7f80000d31c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f80000d27f0 tx=0x7f7ffc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4bdc961dcec22ea server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:42.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f7ff7fff700 1 -- 172.21.15.111:0/2387703640 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc011830 con 0x7f80000b4fc0 2023-11-28T13:01:42.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f7ff7fff700 1 -- 172.21.15.111:0/2387703640 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc011990 con 0x7f80000b4fc0 2023-11-28T13:01:42.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f7ff7fff700 1 -- 172.21.15.111:0/2387703640 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc011d20 con 0x7f80000b4fc0 2023-11-28T13:01:42.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f800745a700 1 -- 172.21.15.111:0/2387703640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80000b4fc0 msgr2=0x7f80000d31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:42.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f800745a700 1 --2- 172.21.15.111:0/2387703640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80000b4fc0 0x7f80000d31c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f80000d27f0 tx=0x7f7ffc009bd0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f800745a700 1 -- 172.21.15.111:0/2387703640 shutdown_connections 2023-11-28T13:01:42.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f800745a700 1 --2- 172.21.15.111:0/2387703640 >> v2:172.21.15.149:3300/0 conn(0x7f80000d2df0 0x7f80000cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f800745a700 1 --2- 172.21.15.111:0/2387703640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80000b4fc0 0x7f80000d31c0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.181+0000 7f800745a700 1 -- 172.21.15.111:0/2387703640 >> 172.21.15.111:0/2387703640 conn(0x7f80000719a0 msgr2=0x7f8000071da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:42.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.182+0000 7f800745a700 1 -- 172.21.15.111:0/2387703640 shutdown_connections 2023-11-28T13:01:42.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.182+0000 7f800745a700 1 -- 172.21.15.111:0/2387703640 wait complete. 2023-11-28T13:01:42.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.183+0000 7f800745a700 1 Processor -- start 2023-11-28T13:01:42.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.183+0000 7f800745a700 1 -- start start 2023-11-28T13:01:42.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f800745a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f80000b1c60 0x7f80001e5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f800745a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80000b4fc0 0x7f80001584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f800745a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80000d2df0 0x7f800015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f800745a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80001e78c0 con 0x7f80000d2df0 2023-11-28T13:01:42.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f800745a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80001e7b60 con 0x7f80000b4fc0 2023-11-28T13:01:42.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f800745a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80001e7e40 con 0x7f80000b1c60 2023-11-28T13:01:42.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f80059f7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80000d2df0 0x7f800015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f80059f7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80000d2df0 0x7f800015ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36888/0 (socket says 172.21.15.111:36888) 2023-11-28T13:01:42.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f80059f7700 1 -- 172.21.15.111:0/1352869855 learned_addr learned my addr 172.21.15.111:0/1352869855 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:42.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f80051f6700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f80000b1c60 0x7f80001e5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f80049f5700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80000b4fc0 0x7f80001584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f80059f7700 1 -- 172.21.15.111:0/1352869855 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f80000b1c60 msgr2=0x7f80001e5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:42.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f80059f7700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f80000b1c60 0x7f80001e5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f80059f7700 1 -- 172.21.15.111:0/1352869855 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80000b4fc0 msgr2=0x7f80001584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:42.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f80059f7700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80000b4fc0 0x7f80001584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.184+0000 7f80059f7700 1 -- 172.21.15.111:0/1352869855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc009550 con 0x7f80000d2df0 2023-11-28T13:01:42.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.185+0000 7f80059f7700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80000d2df0 0x7f800015ee80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800cb50 tx=0x7f7ff800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:42.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.185+0000 7f7ff5ffb700 1 -- 172.21.15.111:0/1352869855 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8003c00 con 0x7f80000d2df0 2023-11-28T13:01:42.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.185+0000 7f7ff5ffb700 1 -- 172.21.15.111:0/1352869855 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff8003d60 con 0x7f80000d2df0 2023-11-28T13:01:42.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.185+0000 7f7ff5ffb700 1 -- 172.21.15.111:0/1352869855 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff800aa20 con 0x7f80000d2df0 2023-11-28T13:01:42.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.185+0000 7f800745a700 1 -- 172.21.15.111:0/1352869855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80000b88f0 con 0x7f80000d2df0 2023-11-28T13:01:42.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.185+0000 7f800745a700 1 -- 172.21.15.111:0/1352869855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80000d15e0 con 0x7f80000d2df0 2023-11-28T13:01:42.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.186+0000 7f800745a700 1 -- 172.21.15.111:0/1352869855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80000b21b0 con 0x7f80000d2df0 2023-11-28T13:01:42.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.186+0000 7f7ff5ffb700 1 -- 172.21.15.111:0/1352869855 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7ff8010050 con 0x7f80000d2df0 2023-11-28T13:01:42.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.187+0000 7f7ff5ffb700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7fe40682f0 0x7f7fe406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.187+0000 7f7ff5ffb700 1 -- 172.21.15.111:0/1352869855 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f7ff8085c40 con 0x7f80000d2df0 2023-11-28T13:01:42.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.187+0000 7f80051f6700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7fe40682f0 0x7f7fe406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.187+0000 7f80051f6700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7fe40682f0 0x7f7fe406a7a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f80000b3d50 tx=0x7f7ff0009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:42.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.191+0000 7f7ff5ffb700 1 -- 172.21.15.111:0/1352869855 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7ff8052360 con 0x7f80000d2df0 2023-11-28T13:01:42.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.416+0000 7f800745a700 1 -- 172.21.15.111:0/1352869855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8000067170 con 0x7f80000d2df0 2023-11-28T13:01:42.694 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-11-28T13:01:42.693+0000 7fb379352700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T13:01:42.706 INFO:tasks.ceph.mds.c.smithi111.stderr: -7673> 2023-11-28T13:00:50.814+0000 7fa2fdc87700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-11-28T13:01:42.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.706+0000 7f7ff5ffb700 1 -- 172.21.15.111:0/1352869855 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7ff8055980 con 0x7f80000d2df0 2023-11-28T13:01:42.712 INFO:tasks.ceph.mds.c.smithi111.stderr: -6452> 2023-11-28T13:01:01.762+0000 7fa2f7c7b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T13:01:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.709+0000 7f800745a700 1 -- 172.21.15.111:0/1352869855 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7fe40682f0 msgr2=0x7f7fe406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.709+0000 7f800745a700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7fe40682f0 0x7f7fe406a7a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f80000b3d50 tx=0x7f7ff0009210 comp rx=0 tx=0).stop 2023-11-28T13:01:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.709+0000 7f800745a700 1 -- 172.21.15.111:0/1352869855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80000d2df0 msgr2=0x7f800015ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.709+0000 7f800745a700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80000d2df0 0x7f800015ee80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800cb50 tx=0x7f7ff800ce60 comp rx=0 tx=0).stop 2023-11-28T13:01:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.711+0000 7f800745a700 1 -- 172.21.15.111:0/1352869855 shutdown_connections 2023-11-28T13:01:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.711+0000 7f800745a700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80000d2df0 0x7f800015ee80 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.711+0000 7f800745a700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f80000b1c60 0x7f80001e5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.711+0000 7f800745a700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7fe40682f0 0x7f7fe406a7a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.711+0000 7f800745a700 1 --2- 172.21.15.111:0/1352869855 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f80000b4fc0 0x7f80001584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.711+0000 7f800745a700 1 -- 172.21.15.111:0/1352869855 >> 172.21.15.111:0/1352869855 conn(0x7f80000719a0 msgr2=0x7f80000b2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:42.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.711+0000 7f800745a700 1 -- 172.21.15.111:0/1352869855 shutdown_connections 2023-11-28T13:01:42.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.711+0000 7f800745a700 1 -- 172.21.15.111:0/1352869855 wait complete. 2023-11-28T13:01:42.714 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T13:01:42.714 INFO:tasks.ceph.mds.c.smithi111.stderr: -2943> 2023-11-28T13:01:04.660+0000 7fa2f4474700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5219=0-4194304@1} caps={5219=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5219 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x55abdfdffb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-28T13:01:42.725 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T13:01:42.741 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-11-28T13:01:42.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.847+0000 7f09fce18700 1 Processor -- start 2023-11-28T13:01:42.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.847+0000 7f09fce18700 1 -- start start 2023-11-28T13:01:42.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.848+0000 7f09fce18700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09f80b08e0 0x7f09f80caa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.848+0000 7f09fce18700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f80b0d60 con 0x7f09f815c6c0 2023-11-28T13:01:42.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.848+0000 7f09fce18700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f80cc750 con 0x7f09f80b0440 2023-11-28T13:01:42.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.848+0000 7f09fce18700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f80b0260 con 0x7f09f80b08e0 2023-11-28T13:01:42.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.848+0000 7f09f659c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f09f815c6c0 0x7f09f80afb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37682/0 (socket says 172.21.15.111:37682) 2023-11-28T13:01:42.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.848+0000 7f09f659c700 1 -- 172.21.15.111:0/1798223233 learned_addr learned my addr 172.21.15.111:0/1798223233 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:42.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f6d9d700 1 --2- 172.21.15.111:0/1798223233 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09f80b08e0 0x7f09f80caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797418808 0 0) 0x7f09f80b0d60 con 0x7f09f815c6c0 2023-11-28T13:01:42.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e00035b0 con 0x7f09f815c6c0 2023-11-28T13:01:42.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271540388 0 0) 0x7f09f80cc750 con 0x7f09f80b0440 2023-11-28T13:01:42.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f80b0d60 con 0x7f09f80b0440 2023-11-28T13:01:42.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1483755311 0 0) 0x7f09e00035b0 con 0x7f09f815c6c0 2023-11-28T13:01:42.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09f80b08e0 msgr2=0x7f09f80caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:42.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 --2- 172.21.15.111:0/1798223233 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09f80b08e0 0x7f09f80caa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 >> v1:172.21.15.149:6789/0 conn(0x7f09f80b0440 legacy=0x7f09f80ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:42.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f80cca30 con 0x7f09f815c6c0 2023-11-28T13:01:42.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f09ec001670 con 0x7f09f815c6c0 2023-11-28T13:01:42.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 >> v1:172.21.15.111:6789/0 conn(0x7f09f815c6c0 legacy=0x7f09f80afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:42.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 --2- 172.21.15.111:0/1798223233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09e0003bc0 0x7f09e0003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 --2- 172.21.15.111:0/1798223233 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09e0004850 0x7f09e0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 --2- 172.21.15.111:0/1798223233 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09f80b08e0 0x7f09e0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f80b0260 con 0x7f09e0003bc0 2023-11-28T13:01:42.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e000bac0 con 0x7f09e0004850 2023-11-28T13:01:42.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e000bc30 con 0x7f09f80b08e0 2023-11-28T13:01:42.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f659c700 1 --2- 172.21.15.111:0/1798223233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09e0003bc0 0x7f09e0003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f5d9b700 1 --2- 172.21.15.111:0/1798223233 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09e0004850 0x7f09e0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f6d9d700 1 --2- 172.21.15.111:0/1798223233 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09f80b08e0 0x7f09e0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f659c700 1 -- 172.21.15.111:0/1798223233 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09f80b08e0 msgr2=0x7f09e0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:42.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f659c700 1 --2- 172.21.15.111:0/1798223233 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09f80b08e0 0x7f09e0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f659c700 1 -- 172.21.15.111:0/1798223233 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09e0004850 msgr2=0x7f09e0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:42.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f659c700 1 --2- 172.21.15.111:0/1798223233 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09e0004850 0x7f09e0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f659c700 1 -- 172.21.15.111:0/1798223233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e00035b0 con 0x7f09e0003bc0 2023-11-28T13:01:42.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.849+0000 7f09f6d9d700 1 --2- 172.21.15.111:0/1798223233 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09f80b08e0 0x7f09e0005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:42.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.850+0000 7f09f659c700 1 --2- 172.21.15.111:0/1798223233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09e0003bc0 0x7f09e0003fb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00c210 tx=0x7f09e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=304061e364b46b0e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:42.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.850+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec00cdb0 con 0x7f09e0003bc0 2023-11-28T13:01:42.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.850+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ec001af0 con 0x7f09e0003bc0 2023-11-28T13:01:42.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.850+0000 7f09f559a700 1 -- 172.21.15.111:0/1798223233 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec005ba0 con 0x7f09e0003bc0 2023-11-28T13:01:42.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.850+0000 7f09fce18700 1 -- 172.21.15.111:0/1798223233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09e0003bc0 msgr2=0x7f09e0003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:42.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.850+0000 7f09fce18700 1 --2- 172.21.15.111:0/1798223233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09e0003bc0 0x7f09e0003fb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00c210 tx=0x7f09e0005880 comp rx=0 tx=0).stop 2023-11-28T13:01:42.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.850+0000 7f09f6d9d700 1 -- 172.21.15.111:0/1798223233 reap_dead start 2023-11-28T13:01:42.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.850+0000 7f09fce18700 1 -- 172.21.15.111:0/1798223233 shutdown_connections 2023-11-28T13:01:42.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.850+0000 7f09fce18700 1 -- 172.21.15.111:0/1798223233 >> 172.21.15.111:0/1798223233 conn(0x7f09f8098ee0 msgr2=0x7f09f806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:42.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.850+0000 7f09fce18700 1 -- 172.21.15.111:0/1798223233 shutdown_connections 2023-11-28T13:01:42.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.850+0000 7f09fce18700 1 -- 172.21.15.111:0/1798223233 wait complete. 2023-11-28T13:01:42.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.852+0000 7f09fce18700 1 Processor -- start 2023-11-28T13:01:42.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.852+0000 7f09fce18700 1 -- start start 2023-11-28T13:01:42.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.853+0000 7f09fce18700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09e0003bc0 0x7f09f8155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.853+0000 7f09fce18700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09f80c77a0 0x7f09f80cf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.853+0000 7f09fce18700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09f80cff30 0x7f09f80b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.853+0000 7f09fce18700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f80d03a0 con 0x7f09f80cff30 2023-11-28T13:01:42.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.853+0000 7f09fce18700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f80c7310 con 0x7f09f80c77a0 2023-11-28T13:01:42.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.853+0000 7f09fce18700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f81556d0 con 0x7f09e0003bc0 2023-11-28T13:01:42.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.853+0000 7f09f6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09f80cff30 0x7f09f80b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.853+0000 7f09f6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09f80cff30 0x7f09f80b3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36920/0 (socket says 172.21.15.111:36920) 2023-11-28T13:01:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.853+0000 7f09f6d9d700 1 -- 172.21.15.111:0/64972151 learned_addr learned my addr 172.21.15.111:0/64972151 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.853+0000 7f09f659c700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09e0003bc0 0x7f09f8155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.853+0000 7f09f5d9b700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09f80c77a0 0x7f09f80cf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.854+0000 7f09f6d9d700 1 -- 172.21.15.111:0/64972151 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09e0003bc0 msgr2=0x7f09f8155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.854+0000 7f09f6d9d700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09e0003bc0 0x7f09f8155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.854+0000 7f09f6d9d700 1 -- 172.21.15.111:0/64972151 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09f80c77a0 msgr2=0x7f09f80cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.854+0000 7f09f6d9d700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09f80c77a0 0x7f09f80cf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.854+0000 7f09f6d9d700 1 -- 172.21.15.111:0/64972151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f80ccff0 con 0x7f09f80cff30 2023-11-28T13:01:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.854+0000 7f09f6d9d700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09f80cff30 0x7f09f80b3a90 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f09f809db20 tx=0x7f09f00104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.854+0000 7f09e6ffd700 1 -- 172.21.15.111:0/64972151 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f001b070 con 0x7f09f80cff30 2023-11-28T13:01:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.854+0000 7f09e6ffd700 1 -- 172.21.15.111:0/64972151 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f000f040 con 0x7f09f80cff30 2023-11-28T13:01:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.854+0000 7f09e6ffd700 1 -- 172.21.15.111:0/64972151 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f000e050 con 0x7f09f80cff30 2023-11-28T13:01:42.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.854+0000 7f09fce18700 1 -- 172.21.15.111:0/64972151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f80ccd10 con 0x7f09f80cff30 2023-11-28T13:01:42.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.854+0000 7f09fce18700 1 -- 172.21.15.111:0/64972151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f80cd2d0 con 0x7f09f80cff30 2023-11-28T13:01:42.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.855+0000 7f09fce18700 1 -- 172.21.15.111:0/64972151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f80c74a0 con 0x7f09f80cff30 2023-11-28T13:01:42.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.855+0000 7f09e6ffd700 1 -- 172.21.15.111:0/64972151 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f09f0015440 con 0x7f09f80cff30 2023-11-28T13:01:42.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.856+0000 7f09e6ffd700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f09d80682f0 0x7f09d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:42.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.856+0000 7f09e6ffd700 1 -- 172.21.15.111:0/64972151 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f09f0085fc0 con 0x7f09f80cff30 2023-11-28T13:01:42.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.856+0000 7f09f659c700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f09d80682f0 0x7f09d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:42.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.856+0000 7f09f659c700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f09d80682f0 0x7f09d806a7a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f09ec015040 tx=0x7f09ec00a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:42.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:42.860+0000 7f09e6ffd700 1 -- 172.21.15.111:0/64972151 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f09f0052530 con 0x7f09f80cff30 2023-11-28T13:01:43.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.087+0000 7f09fce18700 1 -- 172.21.15.111:0/64972151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f09f8067170 con 0x7f09f80cff30 2023-11-28T13:01:43.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.709+0000 7f09e6ffd700 1 -- 172.21.15.111:0/64972151 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f09f0055b50 con 0x7f09f80cff30 2023-11-28T13:01:43.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.712+0000 7f09fce18700 1 -- 172.21.15.111:0/64972151 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f09d80682f0 msgr2=0x7f09d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:43.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.712+0000 7f09fce18700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f09d80682f0 0x7f09d806a7a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f09ec015040 tx=0x7f09ec00a000 comp rx=0 tx=0).stop 2023-11-28T13:01:43.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.712+0000 7f09fce18700 1 -- 172.21.15.111:0/64972151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09f80cff30 msgr2=0x7f09f80b3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:43.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.712+0000 7f09fce18700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09f80cff30 0x7f09f80b3a90 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f09f809db20 tx=0x7f09f00104a0 comp rx=0 tx=0).stop 2023-11-28T13:01:43.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.712+0000 7f09fce18700 1 -- 172.21.15.111:0/64972151 shutdown_connections 2023-11-28T13:01:43.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.712+0000 7f09fce18700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09f80cff30 0x7f09f80b3a90 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:43.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.712+0000 7f09fce18700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09e0003bc0 0x7f09f8155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:43.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.712+0000 7f09fce18700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f09d80682f0 0x7f09d806a7a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:43.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.712+0000 7f09fce18700 1 --2- 172.21.15.111:0/64972151 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09f80c77a0 0x7f09f80cf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:43.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.712+0000 7f09fce18700 1 -- 172.21.15.111:0/64972151 >> 172.21.15.111:0/64972151 conn(0x7f09f8098ee0 msgr2=0x7f09f80af550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:43.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.712+0000 7f09fce18700 1 -- 172.21.15.111:0/64972151 shutdown_connections 2023-11-28T13:01:43.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.712+0000 7f09fce18700 1 -- 172.21.15.111:0/64972151 wait complete. 2023-11-28T13:01:43.727 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T13:01:43.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.894+0000 7f8ab0ea6700 1 Processor -- start 2023-11-28T13:01:43.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.894+0000 7f8ab0ea6700 1 -- start start 2023-11-28T13:01:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8ab0ea6700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f8aac0cd460 0x7f8aac06b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8ab0ea6700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8aac06be80 0x7f8aac0d0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8ab0ea6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aac0b5240 con 0x7f8aac0b8790 2023-11-28T13:01:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8ab0ea6700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aac06c300 con 0x7f8aac0cd460 2023-11-28T13:01:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8ab0ea6700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0b5520 con 0x7f8aac06be80 2023-11-28T13:01:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aaa59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8aac0b8790 0x7f8aac0cfb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37704/0 (socket says 172.21.15.111:37704) 2023-11-28T13:01:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aaa59c700 1 -- 172.21.15.111:0/3081902826 learned_addr learned my addr 172.21.15.111:0/3081902826 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aaad9d700 1 --2- 172.21.15.111:0/3081902826 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8aac06be80 0x7f8aac0d0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa9d9b700 1 --2- 172.21.15.111:0/3081902826 >> v2:172.21.15.149:3300/0 conn(0x7f8aac0cd460 0x7f8aac06b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492733291 0 0) 0x7f8aac0b5240 con 0x7f8aac0b8790 2023-11-28T13:01:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a940035b0 con 0x7f8aac0b8790 2023-11-28T13:01:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3887044686 0 0) 0x7f8a940035b0 con 0x7f8aac0b8790 2023-11-28T13:01:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8aac06be80 msgr2=0x7f8aac0d0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 --2- 172.21.15.111:0/3081902826 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8aac06be80 0x7f8aac0d0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 >> v2:172.21.15.149:3300/0 conn(0x7f8aac0cd460 msgr2=0x7f8aac06b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 --2- 172.21.15.111:0/3081902826 >> v2:172.21.15.149:3300/0 conn(0x7f8aac0cd460 0x7f8aac06b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac0d0a80 con 0x7f8aac0b8790 2023-11-28T13:01:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f8a9c001670 con 0x7f8aac0b8790 2023-11-28T13:01:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 >> v1:172.21.15.111:6789/0 conn(0x7f8aac0b8790 legacy=0x7f8aac0cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 --2- 172.21.15.111:0/3081902826 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8aac06be80 0x7f8a94003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 --2- 172.21.15.111:0/3081902826 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a94003f80 0x7f8a94004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:43.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 --2- 172.21.15.111:0/3081902826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a94004d90 0x7f8a94005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac06c300 con 0x7f8a94004d90 2023-11-28T13:01:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0b5520 con 0x7f8a94003f80 2023-11-28T13:01:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.895+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9400ba10 con 0x7f8aac06be80 2023-11-28T13:01:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.896+0000 7f8aaad9d700 1 --2- 172.21.15.111:0/3081902826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a94004d90 0x7f8a94005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.896+0000 7f8aaa59c700 1 --2- 172.21.15.111:0/3081902826 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8aac06be80 0x7f8a94003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.896+0000 7f8aa9d9b700 1 --2- 172.21.15.111:0/3081902826 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a94003f80 0x7f8a94004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.896+0000 7f8aaad9d700 1 -- 172.21.15.111:0/3081902826 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8aac06be80 msgr2=0x7f8a94003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.896+0000 7f8aaad9d700 1 --2- 172.21.15.111:0/3081902826 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8aac06be80 0x7f8a94003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.896+0000 7f8aaad9d700 1 -- 172.21.15.111:0/3081902826 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a94003f80 msgr2=0x7f8a94004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.896+0000 7f8aaad9d700 1 --2- 172.21.15.111:0/3081902826 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8a94003f80 0x7f8a94004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.896+0000 7f8aaad9d700 1 -- 172.21.15.111:0/3081902826 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a940035b0 con 0x7f8a94004d90 2023-11-28T13:01:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.896+0000 7f8aaad9d700 1 --2- 172.21.15.111:0/3081902826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a94004d90 0x7f8a94005200 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009fc0 tx=0x7f8a94005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32de06045142c247 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.896+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa400ec30 con 0x7f8a94004d90 2023-11-28T13:01:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.896+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa400f040 con 0x7f8a94004d90 2023-11-28T13:01:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.896+0000 7f8aa959a700 1 -- 172.21.15.111:0/3081902826 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4014480 con 0x7f8a94004d90 2023-11-28T13:01:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.897+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/3081902826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a94004d90 msgr2=0x7f8a94005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.897+0000 7f8ab0ea6700 1 --2- 172.21.15.111:0/3081902826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a94004d90 0x7f8a94005200 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009fc0 tx=0x7f8a94005880 comp rx=0 tx=0).stop 2023-11-28T13:01:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.897+0000 7f8aaad9d700 1 -- 172.21.15.111:0/3081902826 reap_dead start 2023-11-28T13:01:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.897+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/3081902826 shutdown_connections 2023-11-28T13:01:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.897+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/3081902826 >> 172.21.15.111:0/3081902826 conn(0x7f8aac0719a0 msgr2=0x7f8aac071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.897+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/3081902826 shutdown_connections 2023-11-28T13:01:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.897+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/3081902826 wait complete. 2023-11-28T13:01:43.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.898+0000 7f8ab0ea6700 1 Processor -- start 2023-11-28T13:01:43.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.898+0000 7f8ab0ea6700 1 -- start start 2023-11-28T13:01:43.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8ab0ea6700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a94004d90 0x7f8aac0c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:43.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8ab0ea6700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8aac0c7cf0 0x7f8aac1f60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:43.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8ab0ea6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8aac0b7010 0x7f8aac0b7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8ab0ea6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0b79c0 con 0x7f8aac0b7010 2023-11-28T13:01:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8ab0ea6700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac1f6600 con 0x7f8aac0c7cf0 2023-11-28T13:01:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8ab0ea6700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0c8160 con 0x7f8a94004d90 2023-11-28T13:01:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8aaad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8aac0b7010 0x7f8aac0b7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8aaad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8aac0b7010 0x7f8aac0b7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36942/0 (socket says 172.21.15.111:36942) 2023-11-28T13:01:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8aaad9d700 1 -- 172.21.15.111:0/1174023857 learned_addr learned my addr 172.21.15.111:0/1174023857 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8aa9d9b700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8aac0c7cf0 0x7f8aac1f60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8aaa59c700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a94004d90 0x7f8aac0c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8aaad9d700 1 -- 172.21.15.111:0/1174023857 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a94004d90 msgr2=0x7f8aac0c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8aaad9d700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a94004d90 0x7f8aac0c8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8aaad9d700 1 -- 172.21.15.111:0/1174023857 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8aac0c7cf0 msgr2=0x7f8aac1f60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8aaad9d700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8aac0c7cf0 0x7f8aac1f60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8aaad9d700 1 -- 172.21.15.111:0/1174023857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac0d0fa0 con 0x7f8aac0b7010 2023-11-28T13:01:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.899+0000 7f8aa9d9b700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8aac0c7cf0 0x7f8aac1f60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.900+0000 7f8aaad9d700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8aac0b7010 0x7f8aac0b7480 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4016000 tx=0x7f8aa4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.900+0000 7f8a9affd700 1 -- 172.21.15.111:0/1174023857 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4007000 con 0x7f8aac0b7010 2023-11-28T13:01:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.900+0000 7f8a9affd700 1 -- 172.21.15.111:0/1174023857 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa4003e50 con 0x7f8aac0b7010 2023-11-28T13:01:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.900+0000 7f8a9affd700 1 -- 172.21.15.111:0/1174023857 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4014b50 con 0x7f8aac0b7010 2023-11-28T13:01:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.900+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/1174023857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aac0b5800 con 0x7f8aac0b7010 2023-11-28T13:01:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.900+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/1174023857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac0d1130 con 0x7f8aac0b7010 2023-11-28T13:01:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.901+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/1174023857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac1f30c0 con 0x7f8aac0b7010 2023-11-28T13:01:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.901+0000 7f8a9affd700 1 -- 172.21.15.111:0/1174023857 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8aa402d070 con 0x7f8aac0b7010 2023-11-28T13:01:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.902+0000 7f8a9affd700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8a8c0682f0 0x7f8a8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.902+0000 7f8aaa59c700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8a8c0682f0 0x7f8a8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.902+0000 7f8a9affd700 1 -- 172.21.15.111:0/1174023857 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6468+0+0 (secure 0 0 0) 0x7f8aa4087590 con 0x7f8aac0b7010 2023-11-28T13:01:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.902+0000 7f8aaa59c700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8a8c0682f0 0x7f8a8c06a7a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8aac157db0 tx=0x7f8a9c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:43.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:43.906+0000 7f8a9affd700 1 -- 172.21.15.111:0/1174023857 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8aac1f30c0 con 0x7f8aac0b7010 2023-11-28T13:01:44.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.090+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/1174023857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8aac1f30c0 con 0x7f8aac0b7010 2023-11-28T13:01:44.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.720+0000 7f8a9affd700 1 -- 172.21.15.111:0/1174023857 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v57) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8aac1f30c0 con 0x7f8aac0b7010 2023-11-28T13:01:44.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.723+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/1174023857 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8a8c0682f0 msgr2=0x7f8a8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:44.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.723+0000 7f8ab0ea6700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8a8c0682f0 0x7f8a8c06a7a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8aac157db0 tx=0x7f8a9c009040 comp rx=0 tx=0).stop 2023-11-28T13:01:44.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.723+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/1174023857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8aac0b7010 msgr2=0x7f8aac0b7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:44.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.723+0000 7f8ab0ea6700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8aac0b7010 0x7f8aac0b7480 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4016000 tx=0x7f8aa4009750 comp rx=0 tx=0).stop 2023-11-28T13:01:44.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.724+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/1174023857 shutdown_connections 2023-11-28T13:01:44.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.724+0000 7f8ab0ea6700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8aac0b7010 0x7f8aac0b7480 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:44.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.724+0000 7f8ab0ea6700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8a94004d90 0x7f8aac0c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:44.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.724+0000 7f8ab0ea6700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8a8c0682f0 0x7f8a8c06a7a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:44.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.724+0000 7f8ab0ea6700 1 --2- 172.21.15.111:0/1174023857 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8aac0c7cf0 0x7f8aac1f60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:44.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.724+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/1174023857 >> 172.21.15.111:0/1174023857 conn(0x7f8aac0719a0 msgr2=0x7f8aac15b050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:44.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.724+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/1174023857 shutdown_connections 2023-11-28T13:01:44.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.724+0000 7f8ab0ea6700 1 -- 172.21.15.111:0/1174023857 wait complete. 2023-11-28T13:01:44.727 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-11-28T13:01:44.739 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T13:01:44.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.863+0000 7fb555322700 1 Processor -- start 2023-11-28T13:01:44.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.863+0000 7fb555322700 1 -- start start 2023-11-28T13:01:44.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.864+0000 7fb555322700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb55006a980 0x7fb5501679f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:44.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.864+0000 7fb555322700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5500ac6d0 con 0x7fb55015cca0 2023-11-28T13:01:44.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.864+0000 7fb555322700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5500acf70 con 0x7fb5500ae900 2023-11-28T13:01:44.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.864+0000 7fb555322700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55015d070 con 0x7fb55006a980 2023-11-28T13:01:44.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.864+0000 7fb54effd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb55015cca0 0x7fb5500cf470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37724/0 (socket says 172.21.15.111:37724) 2023-11-28T13:01:44.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.864+0000 7fb54effd700 1 -- 172.21.15.111:0/815334782 learned_addr learned my addr 172.21.15.111:0/815334782 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:44.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208034828 0 0) 0x7fb5500ac6d0 con 0x7fb55015cca0 2023-11-28T13:01:44.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54f7fe700 1 --2- 172.21.15.111:0/815334782 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb55006a980 0x7fb5501679f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:44.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5380035b0 con 0x7fb55015cca0 2023-11-28T13:01:44.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3716386689 0 0) 0x7fb5380035b0 con 0x7fb55015cca0 2023-11-28T13:01:44.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb55006a980 msgr2=0x7fb5501679f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:44.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 --2- 172.21.15.111:0/815334782 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb55006a980 0x7fb5501679f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:44.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 >> v1:172.21.15.149:6789/0 conn(0x7fb5500ae900 legacy=0x7fb55006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:44.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5500af310 con 0x7fb55015cca0 2023-11-28T13:01:44.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fb5440017f0 con 0x7fb55015cca0 2023-11-28T13:01:44.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 >> v1:172.21.15.111:6789/0 conn(0x7fb55015cca0 legacy=0x7fb5500cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:44.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 --2- 172.21.15.111:0/815334782 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb538003a30 0x7fb538003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:44.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 --2- 172.21.15.111:0/815334782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb538004850 0x7fb538004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:44.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 --2- 172.21.15.111:0/815334782 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb55006a980 0x7fb538005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:44.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53800bb40 con 0x7fb538004850 2023-11-28T13:01:44.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53800bcb0 con 0x7fb538003a30 2023-11-28T13:01:44.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53800be70 con 0x7fb55006a980 2023-11-28T13:01:44.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54e7fc700 1 --2- 172.21.15.111:0/815334782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb538004850 0x7fb538004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:44.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54effd700 1 --2- 172.21.15.111:0/815334782 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb538003a30 0x7fb538003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:44.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54f7fe700 1 --2- 172.21.15.111:0/815334782 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb55006a980 0x7fb538005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:44.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54e7fc700 1 -- 172.21.15.111:0/815334782 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb55006a980 msgr2=0x7fb538005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:44.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54e7fc700 1 --2- 172.21.15.111:0/815334782 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb55006a980 0x7fb538005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:44.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54e7fc700 1 -- 172.21.15.111:0/815334782 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb538003a30 msgr2=0x7fb538003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:44.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54e7fc700 1 --2- 172.21.15.111:0/815334782 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb538003a30 0x7fb538003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:44.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54e7fc700 1 -- 172.21.15.111:0/815334782 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5380035b0 con 0x7fb538004850 2023-11-28T13:01:44.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.865+0000 7fb54e7fc700 1 --2- 172.21.15.111:0/815334782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb538004850 0x7fb538004cc0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb550071290 tx=0x7fb538005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b782fe891428298 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:44.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.866+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5400012f0 con 0x7fb538004850 2023-11-28T13:01:44.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.866+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5400052f0 con 0x7fb538004850 2023-11-28T13:01:44.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.866+0000 7fb54dffb700 1 -- 172.21.15.111:0/815334782 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540006860 con 0x7fb538004850 2023-11-28T13:01:44.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.866+0000 7fb555322700 1 -- 172.21.15.111:0/815334782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb538004850 msgr2=0x7fb538004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:44.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.866+0000 7fb555322700 1 --2- 172.21.15.111:0/815334782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb538004850 0x7fb538004cc0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb550071290 tx=0x7fb538005880 comp rx=0 tx=0).stop 2023-11-28T13:01:44.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.866+0000 7fb54f7fe700 1 -- 172.21.15.111:0/815334782 reap_dead start 2023-11-28T13:01:44.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.866+0000 7fb555322700 1 -- 172.21.15.111:0/815334782 shutdown_connections 2023-11-28T13:01:44.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.866+0000 7fb555322700 1 -- 172.21.15.111:0/815334782 >> 172.21.15.111:0/815334782 conn(0x7fb55006bca0 msgr2=0x7fb5500cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:44.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.866+0000 7fb555322700 1 -- 172.21.15.111:0/815334782 shutdown_connections 2023-11-28T13:01:44.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.866+0000 7fb555322700 1 -- 172.21.15.111:0/815334782 wait complete. 2023-11-28T13:01:44.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.868+0000 7fb555322700 1 Processor -- start 2023-11-28T13:01:44.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.868+0000 7fb555322700 1 -- start start 2023-11-28T13:01:44.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.868+0000 7fb555322700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb538004850 0x7fb5500c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:44.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.868+0000 7fb555322700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5500cc410 0x7fb5500cc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:44.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.868+0000 7fb555322700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5501f5fe0 0x7fb5501f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:44.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.868+0000 7fb555322700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5500ccdc0 con 0x7fb5500cc410 2023-11-28T13:01:44.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.868+0000 7fb555322700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5501e83b0 con 0x7fb5501f5fe0 2023-11-28T13:01:44.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.868+0000 7fb555322700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5501f6990 con 0x7fb538004850 2023-11-28T13:01:44.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.868+0000 7fb54e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5500cc410 0x7fb5500cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:44.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.868+0000 7fb54e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5500cc410 0x7fb5500cc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36960/0 (socket says 172.21.15.111:36960) 2023-11-28T13:01:44.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.868+0000 7fb54e7fc700 1 -- 172.21.15.111:0/2445375137 learned_addr learned my addr 172.21.15.111:0/2445375137 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:44.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb54f7fe700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5501f5fe0 0x7fb5501f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:44.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb54effd700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb538004850 0x7fb5500c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:44.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb54e7fc700 1 -- 172.21.15.111:0/2445375137 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb538004850 msgr2=0x7fb5500c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:44.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb54e7fc700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb538004850 0x7fb5500c5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:44.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb54e7fc700 1 -- 172.21.15.111:0/2445375137 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5501f5fe0 msgr2=0x7fb5501f6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:44.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb54e7fc700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5501f5fe0 0x7fb5501f6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:44.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb54e7fc700 1 -- 172.21.15.111:0/2445375137 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5500ac920 con 0x7fb5500cc410 2023-11-28T13:01:44.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb54f7fe700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5501f5fe0 0x7fb5501f6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:44.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb54effd700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb538004850 0x7fb5500c5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:44.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb54e7fc700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5500cc410 0x7fb5500cc880 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb540001dc0 tx=0x7fb540001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:44.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb5377fe700 1 -- 172.21.15.111:0/2445375137 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540006e30 con 0x7fb5500cc410 2023-11-28T13:01:44.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb5377fe700 1 -- 172.21.15.111:0/2445375137 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb540007020 con 0x7fb5500cc410 2023-11-28T13:01:44.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb555322700 1 -- 172.21.15.111:0/2445375137 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5500aef70 con 0x7fb5500cc410 2023-11-28T13:01:44.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb5377fe700 1 -- 172.21.15.111:0/2445375137 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5400073d0 con 0x7fb5500cc410 2023-11-28T13:01:44.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.869+0000 7fb555322700 1 -- 172.21.15.111:0/2445375137 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5500acc00 con 0x7fb5500cc410 2023-11-28T13:01:44.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.870+0000 7fb555322700 1 -- 172.21.15.111:0/2445375137 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb550067170 con 0x7fb5500cc410 2023-11-28T13:01:44.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.870+0000 7fb5377fe700 1 -- 172.21.15.111:0/2445375137 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb54002e070 con 0x7fb5500cc410 2023-11-28T13:01:44.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.871+0000 7fb5377fe700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb53c0682f0 0x7fb53c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:44.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.871+0000 7fb5377fe700 1 -- 172.21.15.111:0/2445375137 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6030+0+0 (secure 0 0 0) 0x7fb540086f90 con 0x7fb5500cc410 2023-11-28T13:01:44.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.871+0000 7fb54effd700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb53c0682f0 0x7fb53c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:44.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.871+0000 7fb54effd700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb53c0682f0 0x7fb53c06a7a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb5440011d0 tx=0x7fb544009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:44.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:44.875+0000 7fb5377fe700 1 -- 172.21.15.111:0/2445375137 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb540053720 con 0x7fb5500cc410 2023-11-28T13:01:45.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.066+0000 7fb555322700 1 -- 172.21.15.111:0/2445375137 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb55015a8f0 con 0x7fb5500cc410 2023-11-28T13:01:45.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.722+0000 7fb5377fe700 1 -- 172.21.15.111:0/2445375137 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v58) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb54002c0d0 con 0x7fb5500cc410 2023-11-28T13:01:45.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.725+0000 7fb555322700 1 -- 172.21.15.111:0/2445375137 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb53c0682f0 msgr2=0x7fb53c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.725+0000 7fb555322700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb53c0682f0 0x7fb53c06a7a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb5440011d0 tx=0x7fb544009040 comp rx=0 tx=0).stop 2023-11-28T13:01:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.725+0000 7fb555322700 1 -- 172.21.15.111:0/2445375137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5500cc410 msgr2=0x7fb5500cc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.725+0000 7fb555322700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5500cc410 0x7fb5500cc880 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb540001dc0 tx=0x7fb540001120 comp rx=0 tx=0).stop 2023-11-28T13:01:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.725+0000 7fb555322700 1 -- 172.21.15.111:0/2445375137 shutdown_connections 2023-11-28T13:01:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.725+0000 7fb555322700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb53c0682f0 0x7fb53c06a7a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.725+0000 7fb555322700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb5501f5fe0 0x7fb5501f6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.725+0000 7fb555322700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb538004850 0x7fb5500c5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.725+0000 7fb555322700 1 --2- 172.21.15.111:0/2445375137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5500cc410 0x7fb5500cc880 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.725+0000 7fb555322700 1 -- 172.21.15.111:0/2445375137 >> 172.21.15.111:0/2445375137 conn(0x7fb55006bca0 msgr2=0x7fb5500afe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:45.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.725+0000 7fb555322700 1 -- 172.21.15.111:0/2445375137 shutdown_connections 2023-11-28T13:01:45.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.725+0000 7fb555322700 1 -- 172.21.15.111:0/2445375137 wait complete. 2023-11-28T13:01:45.728 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-11-28T13:01:45.740 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T13:01:45.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.863+0000 7fb7fb963700 1 Processor -- start 2023-11-28T13:01:45.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.863+0000 7fb7fb963700 1 -- start start 2023-11-28T13:01:45.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.864+0000 7fb7fb963700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7f40b0d70 0x7fb7f41604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.864+0000 7fb7fb963700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb7f40b1140 0x7fb7f40ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.864+0000 7fb7fb963700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f415cd90 con 0x7fb7f40b15b0 2023-11-28T13:01:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.864+0000 7fb7fb963700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f41613e0 con 0x7fb7f40b1140 2023-11-28T13:01:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.864+0000 7fb7fb963700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f4160b40 con 0x7fb7f40b0d70 2023-11-28T13:01:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.864+0000 7fb7f8efe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb7f40b1140 0x7fb7f40ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.864+0000 7fb7f8efe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb7f40b1140 0x7fb7f40ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36966/0 (socket says 172.21.15.111:36966) 2023-11-28T13:01:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.864+0000 7fb7f8efe700 1 -- 172.21.15.111:0/628810623 learned_addr learned my addr 172.21.15.111:0/628810623 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.864+0000 7fb7f96ff700 1 --2- 172.21.15.111:0/628810623 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7f40b0d70 0x7fb7f41604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7f8efe700 1 -- 172.21.15.111:0/628810623 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7f40b0d70 msgr2=0x7fb7f41604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7f8efe700 1 --2- 172.21.15.111:0/628810623 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7f40b0d70 0x7fb7f41604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7f8efe700 1 -- 172.21.15.111:0/628810623 >> v1:172.21.15.149:6789/0 conn(0x7fb7f40b15b0 legacy=0x7fb7f40cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7f8efe700 1 -- 172.21.15.111:0/628810623 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f4160e20 con 0x7fb7f40b1140 2023-11-28T13:01:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7f8efe700 1 --2- 172.21.15.111:0/628810623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f40b1140 0x7fb7f40ce6c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00098f0 tx=0x7fb7f0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da09caa1098865ae server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7ebfff700 1 -- 172.21.15.111:0/628810623 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f000b8a0 con 0x7fb7f40b1140 2023-11-28T13:01:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7ebfff700 1 -- 172.21.15.111:0/628810623 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f000ba00 con 0x7fb7f40b1140 2023-11-28T13:01:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7ebfff700 1 -- 172.21.15.111:0/628810623 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f000bd90 con 0x7fb7f40b1140 2023-11-28T13:01:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7fb963700 1 -- 172.21.15.111:0/628810623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f40b1140 msgr2=0x7fb7f40ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7fb963700 1 --2- 172.21.15.111:0/628810623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f40b1140 0x7fb7f40ce6c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00098f0 tx=0x7fb7f0009c00 comp rx=0 tx=0).stop 2023-11-28T13:01:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7fb963700 1 -- 172.21.15.111:0/628810623 shutdown_connections 2023-11-28T13:01:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7fb963700 1 --2- 172.21.15.111:0/628810623 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7f40b0d70 0x7fb7f41604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7fb963700 1 --2- 172.21.15.111:0/628810623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f40b1140 0x7fb7f40ce6c0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7fb963700 1 -- 172.21.15.111:0/628810623 >> 172.21.15.111:0/628810623 conn(0x7fb7f4098ee0 msgr2=0x7fb7f406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:45.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.865+0000 7fb7fb963700 1 -- 172.21.15.111:0/628810623 shutdown_connections 2023-11-28T13:01:45.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.866+0000 7fb7fb963700 1 -- 172.21.15.111:0/628810623 wait complete. 2023-11-28T13:01:45.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.867+0000 7fb7fb963700 1 Processor -- start 2023-11-28T13:01:45.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.867+0000 7fb7fb963700 1 -- start start 2023-11-28T13:01:45.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.867+0000 7fb7fb963700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7f40b0d70 0x7fb7f4157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:45.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.867+0000 7fb7fb963700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f40b1140 0x7fb7f40c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:45.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.867+0000 7fb7fb963700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7f40b15b0 0x7fb7f40c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:45.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.867+0000 7fb7fb963700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f4157c60 con 0x7fb7f40b1140 2023-11-28T13:01:45.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.867+0000 7fb7fb963700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f40b5950 con 0x7fb7f40b15b0 2023-11-28T13:01:45.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.867+0000 7fb7fb963700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f4158000 con 0x7fb7f40b0d70 2023-11-28T13:01:45.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7f8efe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f40b1140 0x7fb7f40c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:45.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7f8efe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f40b1140 0x7fb7f40c93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36974/0 (socket says 172.21.15.111:36974) 2023-11-28T13:01:45.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7f8efe700 1 -- 172.21.15.111:0/2015129502 learned_addr learned my addr 172.21.15.111:0/2015129502 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:45.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7f96ff700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7f40b0d70 0x7fb7f4157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:45.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7f9f00700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7f40b15b0 0x7fb7f40c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:45.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7f8efe700 1 -- 172.21.15.111:0/2015129502 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7f40b0d70 msgr2=0x7fb7f4157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:45.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7f8efe700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7f40b0d70 0x7fb7f4157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:45.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7f8efe700 1 -- 172.21.15.111:0/2015129502 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7f40b15b0 msgr2=0x7fb7f40c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:45.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7f8efe700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7f40b15b0 0x7fb7f40c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:45.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7f8efe700 1 -- 172.21.15.111:0/2015129502 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f0009550 con 0x7fb7f40b1140 2023-11-28T13:01:45.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7f8efe700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f40b1140 0x7fb7f40c93c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0000c00 tx=0x7fb7f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:45.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7e9ffb700 1 -- 172.21.15.111:0/2015129502 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0004480 con 0x7fb7f40b1140 2023-11-28T13:01:45.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7e9ffb700 1 -- 172.21.15.111:0/2015129502 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f001f070 con 0x7fb7f40b1140 2023-11-28T13:01:45.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7e9ffb700 1 -- 172.21.15.111:0/2015129502 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f00038c0 con 0x7fb7f40b1140 2023-11-28T13:01:45.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7fb963700 1 -- 172.21.15.111:0/2015129502 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f40af340 con 0x7fb7f40b1140 2023-11-28T13:01:45.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.868+0000 7fb7fb963700 1 -- 172.21.15.111:0/2015129502 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f4161100 con 0x7fb7f40b1140 2023-11-28T13:01:45.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.869+0000 7fb7fb963700 1 -- 172.21.15.111:0/2015129502 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f40cec00 con 0x7fb7f40b1140 2023-11-28T13:01:45.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.870+0000 7fb7e9ffb700 1 -- 172.21.15.111:0/2015129502 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb7f002a020 con 0x7fb7f40b1140 2023-11-28T13:01:45.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.870+0000 7fb7e9ffb700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7d80682f0 0x7fb7d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:45.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.870+0000 7fb7e9ffb700 1 -- 172.21.15.111:0/2015129502 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fb7f00871f0 con 0x7fb7f40b1140 2023-11-28T13:01:45.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.870+0000 7fb7f96ff700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7d80682f0 0x7fb7d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:45.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.871+0000 7fb7f96ff700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7d80682f0 0x7fb7d806a7a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4006fd0 tx=0x7fb7e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:45.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:45.874+0000 7fb7e9ffb700 1 -- 172.21.15.111:0/2015129502 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb7f0053c10 con 0x7fb7f40b1140 2023-11-28T13:01:46.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.057+0000 7fb7fb963700 1 -- 172.21.15.111:0/2015129502 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb7f4067170 con 0x7fb7f40b1140 2023-11-28T13:01:46.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.060+0000 7fb7e9ffb700 1 -- 172.21.15.111:0/2015129502 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb7f0057230 con 0x7fb7f40b1140 2023-11-28T13:01:46.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.062+0000 7fb7fb963700 1 -- 172.21.15.111:0/2015129502 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7d80682f0 msgr2=0x7fb7d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:46.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.062+0000 7fb7fb963700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7d80682f0 0x7fb7d806a7a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4006fd0 tx=0x7fb7e4008040 comp rx=0 tx=0).stop 2023-11-28T13:01:46.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.062+0000 7fb7fb963700 1 -- 172.21.15.111:0/2015129502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f40b1140 msgr2=0x7fb7f40c93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:46.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.062+0000 7fb7fb963700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f40b1140 0x7fb7f40c93c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0000c00 tx=0x7fb7f0004750 comp rx=0 tx=0).stop 2023-11-28T13:01:46.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.062+0000 7fb7fb963700 1 -- 172.21.15.111:0/2015129502 shutdown_connections 2023-11-28T13:01:46.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.062+0000 7fb7fb963700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb7d80682f0 0x7fb7d806a7a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.062+0000 7fb7fb963700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb7f40b15b0 0x7fb7f40c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.062+0000 7fb7fb963700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb7f40b0d70 0x7fb7f4157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.062+0000 7fb7fb963700 1 --2- 172.21.15.111:0/2015129502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7f40b1140 0x7fb7f40c93c0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.062+0000 7fb7fb963700 1 -- 172.21.15.111:0/2015129502 >> 172.21.15.111:0/2015129502 conn(0x7fb7f4098ee0 msgr2=0x7fb7f415a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:46.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.062+0000 7fb7fb963700 1 -- 172.21.15.111:0/2015129502 shutdown_connections 2023-11-28T13:01:46.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.062+0000 7fb7fb963700 1 -- 172.21.15.111:0/2015129502 wait complete. 2023-11-28T13:01:46.076 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2023-11-28T13:01:46.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.194+0000 7fe70100a700 1 Processor -- start 2023-11-28T13:01:46.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.194+0000 7fe70100a700 1 -- start start 2023-11-28T13:01:46.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe70100a700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe6fc15c7b0 0x7fe6fc0cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:46.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe70100a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6fc0d0490 0x7fe6fc0ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:46.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe70100a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe6fc06b380 0x7fe6fc06b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:46.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe70100a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6fc06bd40 con 0x7fe6fc06b380 2023-11-28T13:01:46.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe70100a700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6fc0cb810 con 0x7fe6fc15c7b0 2023-11-28T13:01:46.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe70100a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0cc650 con 0x7fe6fc0d0490 2023-11-28T13:01:46.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe6fb59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe6fc06b380 0x7fe6fc06b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:46.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe6fb59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe6fc06b380 0x7fe6fc06b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36984/0 (socket says 172.21.15.111:36984) 2023-11-28T13:01:46.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe6fb59e700 1 -- 172.21.15.111:0/3309296246 learned_addr learned my addr 172.21.15.111:0/3309296246 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:46.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe6fa59c700 1 --2- 172.21.15.111:0/3309296246 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6fc0d0490 0x7fe6fc0ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:46.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe6fad9d700 1 --2- 172.21.15.111:0/3309296246 >> v2:172.21.15.149:3300/0 conn(0x7fe6fc15c7b0 0x7fe6fc0cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:46.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe6fb59e700 1 -- 172.21.15.111:0/3309296246 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6fc0d0490 msgr2=0x7fe6fc0ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:46.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe6fb59e700 1 --2- 172.21.15.111:0/3309296246 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6fc0d0490 0x7fe6fc0ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe6fb59e700 1 -- 172.21.15.111:0/3309296246 >> v2:172.21.15.149:3300/0 conn(0x7fe6fc15c7b0 msgr2=0x7fe6fc0cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:46.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe6fb59e700 1 --2- 172.21.15.111:0/3309296246 >> v2:172.21.15.149:3300/0 conn(0x7fe6fc15c7b0 0x7fe6fc0cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.195+0000 7fe6fb59e700 1 -- 172.21.15.111:0/3309296246 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc0cba40 con 0x7fe6fc06b380 2023-11-28T13:01:46.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.196+0000 7fe6fb59e700 1 --2- 172.21.15.111:0/3309296246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6fc06b380 0x7fe6fc06b800 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc0726d0 tx=0x7fe6f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f734eef18b967604 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:46.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.196+0000 7fe6f9d9b700 1 -- 172.21.15.111:0/3309296246 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f400e070 con 0x7fe6fc06b380 2023-11-28T13:01:46.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.196+0000 7fe6f9d9b700 1 -- 172.21.15.111:0/3309296246 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f400b910 con 0x7fe6fc06b380 2023-11-28T13:01:46.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.196+0000 7fe6f9d9b700 1 -- 172.21.15.111:0/3309296246 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f400bcc0 con 0x7fe6fc06b380 2023-11-28T13:01:46.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.196+0000 7fe70100a700 1 -- 172.21.15.111:0/3309296246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6fc06b380 msgr2=0x7fe6fc06b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:46.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.196+0000 7fe70100a700 1 --2- 172.21.15.111:0/3309296246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6fc06b380 0x7fe6fc06b800 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc0726d0 tx=0x7fe6f400b330 comp rx=0 tx=0).stop 2023-11-28T13:01:46.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.196+0000 7fe70100a700 1 -- 172.21.15.111:0/3309296246 shutdown_connections 2023-11-28T13:01:46.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.196+0000 7fe70100a700 1 --2- 172.21.15.111:0/3309296246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6fc06b380 0x7fe6fc06b800 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.196+0000 7fe70100a700 1 --2- 172.21.15.111:0/3309296246 >> v2:172.21.15.149:3300/0 conn(0x7fe6fc15c7b0 0x7fe6fc0cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.196+0000 7fe70100a700 1 --2- 172.21.15.111:0/3309296246 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6fc0d0490 0x7fe6fc0ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.196+0000 7fe70100a700 1 -- 172.21.15.111:0/3309296246 >> 172.21.15.111:0/3309296246 conn(0x7fe6fc09a040 msgr2=0x7fe6fc0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:46.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.196+0000 7fe70100a700 1 -- 172.21.15.111:0/3309296246 shutdown_connections 2023-11-28T13:01:46.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.196+0000 7fe70100a700 1 -- 172.21.15.111:0/3309296246 wait complete. 2023-11-28T13:01:46.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.198+0000 7fe70100a700 1 Processor -- start 2023-11-28T13:01:46.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.198+0000 7fe70100a700 1 -- start start 2023-11-28T13:01:46.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.199+0000 7fe70100a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6fc06b380 0x7fe6fc0c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:46.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.199+0000 7fe70100a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6fc0d0490 0x7fe6fc0b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:46.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.199+0000 7fe70100a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe6fc15c7b0 0x7fe6fc158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:46.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.199+0000 7fe70100a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0b2090 con 0x7fe6fc06b380 2023-11-28T13:01:46.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.199+0000 7fe70100a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0c8470 con 0x7fe6fc15c7b0 2023-11-28T13:01:46.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.199+0000 7fe70100a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0c7ef0 con 0x7fe6fc0d0490 2023-11-28T13:01:46.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.199+0000 7fe6fad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6fc06b380 0x7fe6fc0c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:46.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.199+0000 7fe6fad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6fc06b380 0x7fe6fc0c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36986/0 (socket says 172.21.15.111:36986) 2023-11-28T13:01:46.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.199+0000 7fe6fad9d700 1 -- 172.21.15.111:0/1950788511 learned_addr learned my addr 172.21.15.111:0/1950788511 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:46.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.199+0000 7fe6fb59e700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe6fc15c7b0 0x7fe6fc158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:46.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.199+0000 7fe6fa59c700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6fc0d0490 0x7fe6fc0b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:46.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.200+0000 7fe6fad9d700 1 -- 172.21.15.111:0/1950788511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6fc0d0490 msgr2=0x7fe6fc0b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:46.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.200+0000 7fe6fad9d700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6fc0d0490 0x7fe6fc0b2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.200+0000 7fe6fad9d700 1 -- 172.21.15.111:0/1950788511 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe6fc15c7b0 msgr2=0x7fe6fc158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:46.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.200+0000 7fe6fad9d700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe6fc15c7b0 0x7fe6fc158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.200+0000 7fe6fad9d700 1 -- 172.21.15.111:0/1950788511 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f4009b30 con 0x7fe6fc06b380 2023-11-28T13:01:46.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.200+0000 7fe6fa59c700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6fc0d0490 0x7fe6fc0b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:46.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.200+0000 7fe6fb59e700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe6fc15c7b0 0x7fe6fc158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:46.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.200+0000 7fe6fad9d700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6fc06b380 0x7fe6fc0c8760 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000e9c0 tx=0x7fe6f000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:46.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.200+0000 7fe6e37fe700 1 -- 172.21.15.111:0/1950788511 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f00096f0 con 0x7fe6fc06b380 2023-11-28T13:01:46.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.200+0000 7fe6e37fe700 1 -- 172.21.15.111:0/1950788511 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f000c850 con 0x7fe6fc06b380 2023-11-28T13:01:46.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.200+0000 7fe70100a700 1 -- 172.21.15.111:0/1950788511 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6fc0cc000 con 0x7fe6fc06b380 2023-11-28T13:01:46.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.200+0000 7fe6e37fe700 1 -- 172.21.15.111:0/1950788511 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f000cbc0 con 0x7fe6fc06b380 2023-11-28T13:01:46.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.200+0000 7fe70100a700 1 -- 172.21.15.111:0/1950788511 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6fc0cbd20 con 0x7fe6fc06b380 2023-11-28T13:01:46.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.201+0000 7fe70100a700 1 -- 172.21.15.111:0/1950788511 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6fc0cc2e0 con 0x7fe6fc06b380 2023-11-28T13:01:46.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.201+0000 7fe6e37fe700 1 -- 172.21.15.111:0/1950788511 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe6f0023070 con 0x7fe6fc06b380 2023-11-28T13:01:46.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.201+0000 7fe6e37fe700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe6e80682f0 0x7fe6e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:46.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.202+0000 7fe6e37fe700 1 -- 172.21.15.111:0/1950788511 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fe6f0085ea0 con 0x7fe6fc06b380 2023-11-28T13:01:46.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.202+0000 7fe6fa59c700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe6e80682f0 0x7fe6e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:46.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.202+0000 7fe6fa59c700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe6e80682f0 0x7fe6e806a7a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc0cb760 tx=0x7fe6ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:46.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.206+0000 7fe6e37fe700 1 -- 172.21.15.111:0/1950788511 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe6f00527f0 con 0x7fe6fc06b380 2023-11-28T13:01:46.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.470+0000 7fe70100a700 1 -- 172.21.15.111:0/1950788511 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) v1 -- 0x7fe6fc0b2880 con 0x7fe6fc06b380 2023-11-28T13:01:46.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.736+0000 7fe6e37fe700 1 -- 172.21.15.111:0/1950788511 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v159) v1 ==== 149+0+0 (secure 0 0 0) 0x7fe6f0055e10 con 0x7fe6fc06b380 2023-11-28T13:01:46.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.738+0000 7fe70100a700 1 -- 172.21.15.111:0/1950788511 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe6e80682f0 msgr2=0x7fe6e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:46.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.738+0000 7fe70100a700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe6e80682f0 0x7fe6e806a7a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc0cb760 tx=0x7fe6ec008040 comp rx=0 tx=0).stop 2023-11-28T13:01:46.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.738+0000 7fe70100a700 1 -- 172.21.15.111:0/1950788511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6fc06b380 msgr2=0x7fe6fc0c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:46.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.738+0000 7fe70100a700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6fc06b380 0x7fe6fc0c8760 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000e9c0 tx=0x7fe6f000ed80 comp rx=0 tx=0).stop 2023-11-28T13:01:46.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.738+0000 7fe70100a700 1 -- 172.21.15.111:0/1950788511 shutdown_connections 2023-11-28T13:01:46.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.738+0000 7fe70100a700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe6e80682f0 0x7fe6e806a7a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.738+0000 7fe70100a700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe6fc15c7b0 0x7fe6fc158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.738+0000 7fe70100a700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6fc06b380 0x7fe6fc0c8760 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.738+0000 7fe70100a700 1 --2- 172.21.15.111:0/1950788511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe6fc0d0490 0x7fe6fc0b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.738+0000 7fe70100a700 1 -- 172.21.15.111:0/1950788511 >> 172.21.15.111:0/1950788511 conn(0x7fe6fc09a040 msgr2=0x7fe6fc0cf6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:46.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.739+0000 7fe70100a700 1 -- 172.21.15.111:0/1950788511 shutdown_connections 2023-11-28T13:01:46.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.739+0000 7fe70100a700 1 -- 172.21.15.111:0/1950788511 wait complete. 2023-11-28T13:01:46.753 INFO:tasks.cephfs_test_runner:That the MDS can repair an inodes backtrace in the data pool ... ok 2023-11-28T13:01:46.754 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2023-11-28T13:01:46.754 INFO:tasks.cephfs_test_runner:test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-28T13:01:46.754 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2023-11-28T13:01:46.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.883+0000 7f2f5e1a0700 1 Processor -- start 2023-11-28T13:01:46.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.883+0000 7f2f5e1a0700 1 -- start start 2023-11-28T13:01:46.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.884+0000 7f2f5e1a0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2f5815f960 0x7f2f5815d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:46.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.884+0000 7f2f5e1a0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f580d15c0 0x7f2f580b7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:46.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.884+0000 7f2f5e1a0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f580cdb10 con 0x7f2f580d1b30 2023-11-28T13:01:46.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.884+0000 7f2f5e1a0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5815daf0 con 0x7f2f5815f960 2023-11-28T13:01:46.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.884+0000 7f2f5e1a0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580cd500 con 0x7f2f580d15c0 2023-11-28T13:01:46.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.884+0000 7f2f5d19e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2f5815f960 0x7f2f5815d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:46.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.884+0000 7f2f5c99d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2f580d1b30 0x7f2f580d0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37770/0 (socket says 172.21.15.111:37770) 2023-11-28T13:01:46.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.884+0000 7f2f5c99d700 1 -- 172.21.15.111:0/983341881 learned_addr learned my addr 172.21.15.111:0/983341881 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:46.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f5d99f700 1 --2- 172.21.15.111:0/983341881 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f580d15c0 0x7f2f580b7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:46.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f5d19e700 1 -- 172.21.15.111:0/983341881 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f580d15c0 msgr2=0x7f2f580b7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:46.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f5d19e700 1 --2- 172.21.15.111:0/983341881 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f580d15c0 0x7f2f580b7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f5d19e700 1 -- 172.21.15.111:0/983341881 >> v1:172.21.15.111:6789/0 conn(0x7f2f580d1b30 legacy=0x7f2f580d0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:46.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f4ffff700 1 -- 172.21.15.111:0/983341881 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927913750 0 0) 0x7f2f580cdb10 con 0x7f2f580d1b30 2023-11-28T13:01:46.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f5d19e700 1 -- 172.21.15.111:0/983341881 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f580d25f0 con 0x7f2f5815f960 2023-11-28T13:01:46.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f5d19e700 1 --2- 172.21.15.111:0/983341881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f5815f960 0x7f2f5815d5b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f2f48009840 tx=0x7f2f48009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2606a0a2630f90 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:46.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f4ffff700 1 -- 172.21.15.111:0/983341881 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f48011830 con 0x7f2f5815f960 2023-11-28T13:01:46.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f4ffff700 1 -- 172.21.15.111:0/983341881 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f48011990 con 0x7f2f5815f960 2023-11-28T13:01:46.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f4ffff700 1 -- 172.21.15.111:0/983341881 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f48011d20 con 0x7f2f5815f960 2023-11-28T13:01:46.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/983341881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f5815f960 msgr2=0x7f2f5815d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:46.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f5e1a0700 1 --2- 172.21.15.111:0/983341881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f5815f960 0x7f2f5815d5b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f2f48009840 tx=0x7f2f48009b50 comp rx=0 tx=0).stop 2023-11-28T13:01:46.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/983341881 shutdown_connections 2023-11-28T13:01:46.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f5e1a0700 1 --2- 172.21.15.111:0/983341881 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f580d15c0 0x7f2f580b7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f5e1a0700 1 --2- 172.21.15.111:0/983341881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f5815f960 0x7f2f5815d5b0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/983341881 >> 172.21.15.111:0/983341881 conn(0x7f2f580719a0 msgr2=0x7f2f58071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:46.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.885+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/983341881 shutdown_connections 2023-11-28T13:01:46.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.886+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/983341881 wait complete. 2023-11-28T13:01:46.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.887+0000 7f2f5e1a0700 1 Processor -- start 2023-11-28T13:01:46.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.887+0000 7f2f5e1a0700 1 -- start start 2023-11-28T13:01:46.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.887+0000 7f2f5e1a0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f580d15c0 0x7f2f581f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:46.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.887+0000 7f2f5e1a0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f580d1b30 0x7f2f580b8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:46.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.887+0000 7f2f5e1a0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f5815f960 0x7f2f581551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:46.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.887+0000 7f2f5e1a0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580ccc20 con 0x7f2f580d1b30 2023-11-28T13:01:46.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.887+0000 7f2f5e1a0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f581f2a10 con 0x7f2f580d15c0 2023-11-28T13:01:46.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.887+0000 7f2f5e1a0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580b6100 con 0x7f2f5815f960 2023-11-28T13:01:46.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.887+0000 7f2f5c99d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f580d1b30 0x7f2f580b8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:46.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5c99d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f580d1b30 0x7f2f580b8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37000/0 (socket says 172.21.15.111:37000) 2023-11-28T13:01:46.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5c99d700 1 -- 172.21.15.111:0/2812482846 learned_addr learned my addr 172.21.15.111:0/2812482846 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:46.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5d19e700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f580d15c0 0x7f2f581f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:46.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5d99f700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f5815f960 0x7f2f581551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:46.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5c99d700 1 -- 172.21.15.111:0/2812482846 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f5815f960 msgr2=0x7f2f581551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:46.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5c99d700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f5815f960 0x7f2f581551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5c99d700 1 -- 172.21.15.111:0/2812482846 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f580d15c0 msgr2=0x7f2f581f24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:46.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5c99d700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f580d15c0 0x7f2f581f24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:46.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5c99d700 1 -- 172.21.15.111:0/2812482846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f480094a0 con 0x7f2f580d1b30 2023-11-28T13:01:46.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5d19e700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f580d15c0 0x7f2f581f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:46.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5d99f700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f5815f960 0x7f2f581551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:46.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5c99d700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f580d1b30 0x7f2f580b8110 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2f540011e0 tx=0x7f2f5400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:46.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f4dffb700 1 -- 172.21.15.111:0/2812482846 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f54004c80 con 0x7f2f580d1b30 2023-11-28T13:01:46.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f4dffb700 1 -- 172.21.15.111:0/2812482846 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f54004e00 con 0x7f2f580d1b30 2023-11-28T13:01:46.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/2812482846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f580cd7a0 con 0x7f2f580d1b30 2023-11-28T13:01:46.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f4dffb700 1 -- 172.21.15.111:0/2812482846 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f54007310 con 0x7f2f580d1b30 2023-11-28T13:01:46.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.888+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/2812482846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f580d28d0 con 0x7f2f580d1b30 2023-11-28T13:01:46.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.889+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/2812482846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f580d2bb0 con 0x7f2f580d1b30 2023-11-28T13:01:46.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.890+0000 7f2f4dffb700 1 -- 172.21.15.111:0/2812482846 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f5401a020 con 0x7f2f580d1b30 2023-11-28T13:01:46.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.890+0000 7f2f4dffb700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f3c0682f0 0x7f2f3c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:46.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.890+0000 7f2f4dffb700 1 -- 172.21.15.111:0/2812482846 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f2f54085750 con 0x7f2f580d1b30 2023-11-28T13:01:46.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.890+0000 7f2f5d19e700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f3c0682f0 0x7f2f3c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:46.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.890+0000 7f2f5d19e700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f3c0682f0 0x7f2f3c06a7a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2f480090b0 tx=0x7f2f48008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:46.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:46.894+0000 7f2f4dffb700 1 -- 172.21.15.111:0/2812482846 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f54052170 con 0x7f2f580d1b30 2023-11-28T13:01:47.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.151+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/2812482846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) v1 -- 0x7f2f5815d6e0 con 0x7f2f580d1b30 2023-11-28T13:01:47.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.738+0000 7f2f4dffb700 1 -- 172.21.15.111:0/2812482846 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v160) v1 ==== 172+0+0 (secure 0 0 0) 0x7f2f54055790 con 0x7f2f580d1b30 2023-11-28T13:01:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.741+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/2812482846 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f3c0682f0 msgr2=0x7f2f3c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.741+0000 7f2f5e1a0700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f3c0682f0 0x7f2f3c06a7a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2f480090b0 tx=0x7f2f48008030 comp rx=0 tx=0).stop 2023-11-28T13:01:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.741+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/2812482846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f580d1b30 msgr2=0x7f2f580b8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.741+0000 7f2f5e1a0700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f580d1b30 0x7f2f580b8110 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2f540011e0 tx=0x7f2f5400cae0 comp rx=0 tx=0).stop 2023-11-28T13:01:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.741+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/2812482846 shutdown_connections 2023-11-28T13:01:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.741+0000 7f2f5e1a0700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f5815f960 0x7f2f581551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.741+0000 7f2f5e1a0700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f3c0682f0 0x7f2f3c06a7a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.741+0000 7f2f5e1a0700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f580d15c0 0x7f2f581f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.741+0000 7f2f5e1a0700 1 --2- 172.21.15.111:0/2812482846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f580d1b30 0x7f2f580b8110 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.741+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/2812482846 >> 172.21.15.111:0/2812482846 conn(0x7f2f580719a0 msgr2=0x7f2f580c8950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.741+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/2812482846 shutdown_connections 2023-11-28T13:01:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.741+0000 7f2f5e1a0700 1 -- 172.21.15.111:0/2812482846 wait complete. 2023-11-28T13:01:47.756 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T13:01:47.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.881+0000 7f451bc79700 1 Processor -- start 2023-11-28T13:01:47.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.881+0000 7f451bc79700 1 -- start start 2023-11-28T13:01:47.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.882+0000 7f451bc79700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f451415a9e0 0x7f45140ac670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:47.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.882+0000 7f451bc79700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f45140b15c0 0x7f45140caef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:47.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.882+0000 7f451bc79700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45140cb430 con 0x7f45140b0250 2023-11-28T13:01:47.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.882+0000 7f451bc79700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45140af650 con 0x7f451415a9e0 2023-11-28T13:01:47.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.882+0000 7f451bc79700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45140cb6d0 con 0x7f45140b15c0 2023-11-28T13:01:47.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.882+0000 7f4519a15700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f451415a9e0 0x7f45140ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:47.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.882+0000 7f451a216700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f45140b0250 0x7f45140cc770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37786/0 (socket says 172.21.15.111:37786) 2023-11-28T13:01:47.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.882+0000 7f451a216700 1 -- 172.21.15.111:0/1052226865 learned_addr learned my addr 172.21.15.111:0/1052226865 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.882+0000 7f4519a15700 1 -- 172.21.15.111:0/1052226865 >> v2:172.21.15.149:3300/0 conn(0x7f45140b15c0 msgr2=0x7f45140caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.882+0000 7f4519a15700 1 --2- 172.21.15.111:0/1052226865 >> v2:172.21.15.149:3300/0 conn(0x7f45140b15c0 0x7f45140caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.882+0000 7f4519a15700 1 -- 172.21.15.111:0/1052226865 >> v1:172.21.15.111:6789/0 conn(0x7f45140b0250 legacy=0x7f45140cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.882+0000 7f4519a15700 1 -- 172.21.15.111:0/1052226865 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140cb9b0 con 0x7f451415a9e0 2023-11-28T13:01:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.882+0000 7f4518a13700 1 -- 172.21.15.111:0/1052226865 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483544155 0 0) 0x7f45140cb430 con 0x7f45140b0250 2023-11-28T13:01:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.883+0000 7f4519a15700 1 --2- 172.21.15.111:0/1052226865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f451415a9e0 0x7f45140ac670 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f451406b310 tx=0x7f4504009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95d24675e90db75 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.883+0000 7f4518a13700 1 -- 172.21.15.111:0/1052226865 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4504005e30 con 0x7f451415a9e0 2023-11-28T13:01:47.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.883+0000 7f4518a13700 1 -- 172.21.15.111:0/1052226865 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4504006030 con 0x7f451415a9e0 2023-11-28T13:01:47.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.883+0000 7f4518a13700 1 -- 172.21.15.111:0/1052226865 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45040063c0 con 0x7f451415a9e0 2023-11-28T13:01:47.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.883+0000 7f451bc79700 1 -- 172.21.15.111:0/1052226865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f451415a9e0 msgr2=0x7f45140ac670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:47.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.883+0000 7f451bc79700 1 --2- 172.21.15.111:0/1052226865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f451415a9e0 0x7f45140ac670 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f451406b310 tx=0x7f4504009b00 comp rx=0 tx=0).stop 2023-11-28T13:01:47.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.883+0000 7f451bc79700 1 -- 172.21.15.111:0/1052226865 shutdown_connections 2023-11-28T13:01:47.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.883+0000 7f451bc79700 1 --2- 172.21.15.111:0/1052226865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f451415a9e0 0x7f45140ac670 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:47.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.883+0000 7f451bc79700 1 --2- 172.21.15.111:0/1052226865 >> v2:172.21.15.149:3300/0 conn(0x7f45140b15c0 0x7f45140caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:47.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.883+0000 7f451bc79700 1 -- 172.21.15.111:0/1052226865 >> 172.21.15.111:0/1052226865 conn(0x7f4514098ee0 msgr2=0x7f451406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:47.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.883+0000 7f451bc79700 1 -- 172.21.15.111:0/1052226865 shutdown_connections 2023-11-28T13:01:47.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.883+0000 7f451bc79700 1 -- 172.21.15.111:0/1052226865 wait complete. 2023-11-28T13:01:47.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.885+0000 7f451bc79700 1 Processor -- start 2023-11-28T13:01:47.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.885+0000 7f451bc79700 1 -- start start 2023-11-28T13:01:47.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.885+0000 7f451bc79700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f45140b0250 0x7f45140b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:47.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.885+0000 7f451bc79700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45140b15c0 0x7f45141521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:47.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.885+0000 7f451bc79700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f451415a9e0 0x7f45141f57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:47.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.885+0000 7f451bc79700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45140b5980 con 0x7f45140b15c0 2023-11-28T13:01:47.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.885+0000 7f451bc79700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45140b50e0 con 0x7f451415a9e0 2023-11-28T13:01:47.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.885+0000 7f451bc79700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45140b53c0 con 0x7f45140b0250 2023-11-28T13:01:47.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.885+0000 7f4519214700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45140b15c0 0x7f45141521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:47.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.885+0000 7f4519214700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45140b15c0 0x7f45141521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37012/0 (socket says 172.21.15.111:37012) 2023-11-28T13:01:47.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.885+0000 7f4519214700 1 -- 172.21.15.111:0/3901897900 learned_addr learned my addr 172.21.15.111:0/3901897900 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:47.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.886+0000 7f4519a15700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f45140b0250 0x7f45140b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:47.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.886+0000 7f451a216700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f451415a9e0 0x7f45141f57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:47.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.886+0000 7f4519214700 1 -- 172.21.15.111:0/3901897900 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f45140b0250 msgr2=0x7f45140b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:47.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.886+0000 7f4519214700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f45140b0250 0x7f45140b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:47.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.886+0000 7f4519214700 1 -- 172.21.15.111:0/3901897900 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f451415a9e0 msgr2=0x7f45141f57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:47.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.886+0000 7f4519214700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f451415a9e0 0x7f45141f57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:47.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.886+0000 7f4519214700 1 -- 172.21.15.111:0/3901897900 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4504009430 con 0x7f45140b15c0 2023-11-28T13:01:47.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.886+0000 7f4519214700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45140b15c0 0x7f45141521a0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f451000b530 tx=0x7f451000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:47.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.886+0000 7f450a7fc700 1 -- 172.21.15.111:0/3901897900 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4510011830 con 0x7f45140b15c0 2023-11-28T13:01:47.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.886+0000 7f450a7fc700 1 -- 172.21.15.111:0/3901897900 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45100119b0 con 0x7f45140b15c0 2023-11-28T13:01:47.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.886+0000 7f451bc79700 1 -- 172.21.15.111:0/3901897900 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45140cbf70 con 0x7f45140b15c0 2023-11-28T13:01:47.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.886+0000 7f450a7fc700 1 -- 172.21.15.111:0/3901897900 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4510011d60 con 0x7f45140b15c0 2023-11-28T13:01:47.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.886+0000 7f451bc79700 1 -- 172.21.15.111:0/3901897900 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45140cbc90 con 0x7f45140b15c0 2023-11-28T13:01:47.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.887+0000 7f451bc79700 1 -- 172.21.15.111:0/3901897900 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45140cc250 con 0x7f45140b15c0 2023-11-28T13:01:47.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.887+0000 7f450a7fc700 1 -- 172.21.15.111:0/3901897900 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4510021070 con 0x7f45140b15c0 2023-11-28T13:01:47.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.888+0000 7f450a7fc700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f44f80682f0 0x7f44f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:47.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.888+0000 7f450a7fc700 1 -- 172.21.15.111:0/3901897900 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f45100859e0 con 0x7f45140b15c0 2023-11-28T13:01:47.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.888+0000 7f4519a15700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f44f80682f0 0x7f44f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:47.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.888+0000 7f4519a15700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f44f80682f0 0x7f44f806a7a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4504013040 tx=0x7f4504008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:47.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:47.892+0000 7f450a7fc700 1 -- 172.21.15.111:0/3901897900 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4510052400 con 0x7f45140b15c0 2023-11-28T13:01:48.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.076+0000 7f451bc79700 1 -- 172.21.15.111:0/3901897900 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4514067170 con 0x7f45140b15c0 2023-11-28T13:01:48.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.079+0000 7f450a7fc700 1 -- 172.21.15.111:0/3901897900 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4510015020 con 0x7f45140b15c0 2023-11-28T13:01:48.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.081+0000 7f451bc79700 1 -- 172.21.15.111:0/3901897900 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f44f80682f0 msgr2=0x7f44f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:48.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.081+0000 7f451bc79700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f44f80682f0 0x7f44f806a7a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4504013040 tx=0x7f4504008070 comp rx=0 tx=0).stop 2023-11-28T13:01:48.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.081+0000 7f451bc79700 1 -- 172.21.15.111:0/3901897900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45140b15c0 msgr2=0x7f45141521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:48.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.081+0000 7f451bc79700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45140b15c0 0x7f45141521a0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f451000b530 tx=0x7f451000b8f0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.081+0000 7f451bc79700 1 -- 172.21.15.111:0/3901897900 shutdown_connections 2023-11-28T13:01:48.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.081+0000 7f451bc79700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f44f80682f0 0x7f44f806a7a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.081+0000 7f451bc79700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f451415a9e0 0x7f45141f57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.081+0000 7f451bc79700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f45140b0250 0x7f45140b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.081+0000 7f451bc79700 1 --2- 172.21.15.111:0/3901897900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45140b15c0 0x7f45141521a0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.081+0000 7f451bc79700 1 -- 172.21.15.111:0/3901897900 >> 172.21.15.111:0/3901897900 conn(0x7f4514098ee0 msgr2=0x7f451406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:48.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.081+0000 7f451bc79700 1 -- 172.21.15.111:0/3901897900 shutdown_connections 2023-11-28T13:01:48.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.082+0000 7f451bc79700 1 -- 172.21.15.111:0/3901897900 wait complete. 2023-11-28T13:01:48.095 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T13:01:48.095 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T13:01:48.137 DEBUG:teuthology.orchestra.run.smithi149:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T13:01:48.179 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T13:01:48.179 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:01:48.179 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:01:48.196 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284268k,nr_inodes=4071067,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18839 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-28T13:01:48.197 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T13:01:48.197 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:01:48.239 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8026740k,nr_inodes=2006685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16040 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=1609048k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-28T13:01:48.239 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:01:48.239 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:01:48.256 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284268k,nr_inodes=4071067,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18839 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-28T13:01:48.256 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T13:01:48.256 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:01:48.299 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8026740k,nr_inodes=2006685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16040 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=1609048k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-28T13:01:48.299 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T13:01:48.299 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:01:48.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.417+0000 7f49dc70f700 1 Processor -- start 2023-11-28T13:01:48.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.417+0000 7f49dc70f700 1 -- start start 2023-11-28T13:01:48.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.418+0000 7f49dc70f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f49d40cdd90 0x7f49d40b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:48.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.418+0000 7f49dc70f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f49d40aecc0 0x7f49d40ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:48.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.418+0000 7f49dc70f700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d40ad420 con 0x7f49d4159960 2023-11-28T13:01:48.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.418+0000 7f49dc70f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d4159d30 con 0x7f49d40aecc0 2023-11-28T13:01:48.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.418+0000 7f49dc70f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d40ac810 con 0x7f49d40cdd90 2023-11-28T13:01:48.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.418+0000 7f49dacac700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f49d40aecc0 0x7f49d40ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:48.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.418+0000 7f49dacac700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f49d40aecc0 0x7f49d40ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37022/0 (socket says 172.21.15.111:37022) 2023-11-28T13:01:48.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.418+0000 7f49dacac700 1 -- 172.21.15.111:0/407013952 learned_addr learned my addr 172.21.15.111:0/407013952 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:48.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.418+0000 7f49dacac700 1 -- 172.21.15.111:0/407013952 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f49d40cdd90 msgr2=0x7f49d40b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:48.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.418+0000 7f49d9caa700 1 --2- 172.21.15.111:0/407013952 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f49d40cdd90 0x7f49d40b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:48.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.418+0000 7f49dacac700 1 --2- 172.21.15.111:0/407013952 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f49d40cdd90 0x7f49d40b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.418+0000 7f49dacac700 1 -- 172.21.15.111:0/407013952 >> v1:172.21.15.149:6789/0 conn(0x7f49d4159960 legacy=0x7f49d40cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:48.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49dacac700 1 -- 172.21.15.111:0/407013952 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d40acaf0 con 0x7f49d40aecc0 2023-11-28T13:01:48.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49d9caa700 1 --2- 172.21.15.111:0/407013952 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f49d40cdd90 0x7f49d40b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:48.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49dacac700 1 --2- 172.21.15.111:0/407013952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49d40aecc0 0x7f49d40ad8b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00d150 tx=0x7f49cc00d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd62fcc4660ebd3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:48.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49d94a9700 1 -- 172.21.15.111:0/407013952 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49cc010070 con 0x7f49d40aecc0 2023-11-28T13:01:48.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49d94a9700 1 -- 172.21.15.111:0/407013952 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49cc00dd80 con 0x7f49d40aecc0 2023-11-28T13:01:48.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49d94a9700 1 -- 172.21.15.111:0/407013952 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49cc00b310 con 0x7f49d40aecc0 2023-11-28T13:01:48.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49dc70f700 1 -- 172.21.15.111:0/407013952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49d40aecc0 msgr2=0x7f49d40ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:48.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49dc70f700 1 --2- 172.21.15.111:0/407013952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49d40aecc0 0x7f49d40ad8b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00d150 tx=0x7f49cc00d460 comp rx=0 tx=0).stop 2023-11-28T13:01:48.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49dc70f700 1 -- 172.21.15.111:0/407013952 shutdown_connections 2023-11-28T13:01:48.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49dc70f700 1 --2- 172.21.15.111:0/407013952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49d40aecc0 0x7f49d40ad8b0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49dc70f700 1 --2- 172.21.15.111:0/407013952 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f49d40cdd90 0x7f49d40b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49dc70f700 1 -- 172.21.15.111:0/407013952 >> 172.21.15.111:0/407013952 conn(0x7f49d4098ee0 msgr2=0x7f49d406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:48.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49dc70f700 1 -- 172.21.15.111:0/407013952 shutdown_connections 2023-11-28T13:01:48.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.419+0000 7f49dc70f700 1 -- 172.21.15.111:0/407013952 wait complete. 2023-11-28T13:01:48.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.421+0000 7f49dc70f700 1 Processor -- start 2023-11-28T13:01:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.421+0000 7f49dc70f700 1 -- start start 2023-11-28T13:01:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.421+0000 7f49dc70f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49d40aecc0 0x7f49d40b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.421+0000 7f49dc70f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f49d40cdd90 0x7f49d415c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.421+0000 7f49dc70f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f49d4159960 0x7f49d406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.421+0000 7f49dc70f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d40b4aa0 con 0x7f49d40aecc0 2023-11-28T13:01:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.421+0000 7f49dc70f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d40b3e90 con 0x7f49d40cdd90 2023-11-28T13:01:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.421+0000 7f49dc70f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d40b4170 con 0x7f49d4159960 2023-11-28T13:01:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.421+0000 7f49da4ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49d40aecc0 0x7f49d40b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.421+0000 7f49da4ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49d40aecc0 0x7f49d40b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37024/0 (socket says 172.21.15.111:37024) 2023-11-28T13:01:48.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.421+0000 7f49da4ab700 1 -- 172.21.15.111:0/738974 learned_addr learned my addr 172.21.15.111:0/738974 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.422+0000 7f49d9caa700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f49d40cdd90 0x7f49d415c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.422+0000 7f49dacac700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f49d4159960 0x7f49d406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.422+0000 7f49da4ab700 1 -- 172.21.15.111:0/738974 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f49d4159960 msgr2=0x7f49d406ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.422+0000 7f49da4ab700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f49d4159960 0x7f49d406ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.422+0000 7f49da4ab700 1 -- 172.21.15.111:0/738974 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f49d40cdd90 msgr2=0x7f49d415c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.422+0000 7f49da4ab700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f49d40cdd90 0x7f49d415c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.422+0000 7f49da4ab700 1 -- 172.21.15.111:0/738974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc00bd40 con 0x7f49d40aecc0 2023-11-28T13:01:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.422+0000 7f49da4ab700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49d40aecc0 0x7f49d40b37d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f49c400b450 tx=0x7f49c400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.422+0000 7f49caffd700 1 -- 172.21.15.111:0/738974 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c4004020 con 0x7f49d40aecc0 2023-11-28T13:01:48.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.422+0000 7f49caffd700 1 -- 172.21.15.111:0/738974 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c4004180 con 0x7f49d40aecc0 2023-11-28T13:01:48.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.422+0000 7f49dc70f700 1 -- 172.21.15.111:0/738974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d40ad0b0 con 0x7f49d40aecc0 2023-11-28T13:01:48.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.422+0000 7f49caffd700 1 -- 172.21.15.111:0/738974 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c4004530 con 0x7f49d40aecc0 2023-11-28T13:01:48.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.422+0000 7f49dc70f700 1 -- 172.21.15.111:0/738974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d40acdd0 con 0x7f49d40aecc0 2023-11-28T13:01:48.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.423+0000 7f49dc70f700 1 -- 172.21.15.111:0/738974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d40addf0 con 0x7f49d40aecc0 2023-11-28T13:01:48.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.423+0000 7f49caffd700 1 -- 172.21.15.111:0/738974 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f49c400f050 con 0x7f49d40aecc0 2023-11-28T13:01:48.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.424+0000 7f49caffd700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f49b80682f0 0x7f49b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:48.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.424+0000 7f49caffd700 1 -- 172.21.15.111:0/738974 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f49c4085560 con 0x7f49d40aecc0 2023-11-28T13:01:48.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.424+0000 7f49d9caa700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f49b80682f0 0x7f49b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:48.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.424+0000 7f49d9caa700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f49b80682f0 0x7f49b806a7a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f49d406f930 tx=0x7f49d0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:48.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.428+0000 7f49caffd700 1 -- 172.21.15.111:0/738974 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f49c4051f80 con 0x7f49d40aecc0 2023-11-28T13:01:48.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.652+0000 7f49dc70f700 1 -- 172.21.15.111:0/738974 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49d40b4df0 con 0x7f49d40aecc0 2023-11-28T13:01:48.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.653+0000 7f49caffd700 1 -- 172.21.15.111:0/738974 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+3541 (secure 0 0 0) 0x7f49c4015df0 con 0x7f49d40aecc0 2023-11-28T13:01:48.655 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:48.655 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5383,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/2072621958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":2072621958},{"type":"v1","addr":"172.21.15.149:6834","nonce":2072621958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5386,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/3348427170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":3348427170},{"type":"v1","addr":"172.21.15.149:6835","nonce":3348427170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":5811,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/1745443298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1745443298},{"type":"v1","addr":"172.21.15.111:6836","nonce":1745443298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20},{"gid":6189,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1995337380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1995337380},{"type":"v1","addr":"172.21.15.111:6837","nonce":1995337380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[]} 2023-11-28T13:01:48.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.655+0000 7f49dc70f700 1 -- 172.21.15.111:0/738974 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f49b80682f0 msgr2=0x7f49b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:48.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.655+0000 7f49dc70f700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f49b80682f0 0x7f49b806a7a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f49d406f930 tx=0x7f49d0008040 comp rx=0 tx=0).stop 2023-11-28T13:01:48.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.655+0000 7f49dc70f700 1 -- 172.21.15.111:0/738974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49d40aecc0 msgr2=0x7f49d40b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:48.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.655+0000 7f49dc70f700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49d40aecc0 0x7f49d40b37d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f49c400b450 tx=0x7f49c400b810 comp rx=0 tx=0).stop 2023-11-28T13:01:48.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.655+0000 7f49dc70f700 1 -- 172.21.15.111:0/738974 shutdown_connections 2023-11-28T13:01:48.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.655+0000 7f49dc70f700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f49d4159960 0x7f49d406ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.655+0000 7f49dc70f700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49d40aecc0 0x7f49d40b37d0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.655+0000 7f49dc70f700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f49b80682f0 0x7f49b806a7a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.655+0000 7f49dc70f700 1 --2- 172.21.15.111:0/738974 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f49d40cdd90 0x7f49d415c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.655+0000 7f49dc70f700 1 -- 172.21.15.111:0/738974 >> 172.21.15.111:0/738974 conn(0x7f49d4098ee0 msgr2=0x7f49d40ae1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:48.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.655+0000 7f49dc70f700 1 -- 172.21.15.111:0/738974 shutdown_connections 2023-11-28T13:01:48.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.655+0000 7f49dc70f700 1 -- 172.21.15.111:0/738974 wait complete. 2023-11-28T13:01:48.658 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 27 2023-11-28T13:01:48.668 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T13:01:48.668 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T13:01:48.669 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T13:01:48.669 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:01:48.669 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T13:01:48.669 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T13:01:48.669 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T13:01:48.669 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:01:48.669 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T13:01:48.669 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T13:01:48.670 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T13:01:48.670 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:01:48.670 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T13:01:48.670 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T13:01:48.670 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T13:01:48.670 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:01:48.677 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T13:01:48.678 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T13:01:48.678 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T13:01:48.679 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T13:01:48.683 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T13:01:48.683 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T13:01:48.684 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T13:01:48.684 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T13:01:48.720 INFO:tasks.ceph.mds.a:Started 2023-11-28T13:01:48.721 DEBUG:teuthology.parallel:result is None 2023-11-28T13:01:48.721 INFO:tasks.ceph.mds.c:Started 2023-11-28T13:01:48.722 DEBUG:teuthology.parallel:result is None 2023-11-28T13:01:48.726 INFO:tasks.ceph.mds.b:Started 2023-11-28T13:01:48.726 DEBUG:teuthology.parallel:result is None 2023-11-28T13:01:48.727 INFO:tasks.ceph.mds.d:Started 2023-11-28T13:01:48.727 DEBUG:teuthology.parallel:result is None 2023-11-28T13:01:48.727 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T13:01:48.800 INFO:tasks.ceph.mds.a.smithi111.stdout:starting mds.a at 2023-11-28T13:01:48.800 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-11-28T13:01:48.814 INFO:tasks.ceph.mds.d.smithi149.stdout:starting mds.d at 2023-11-28T13:01:48.814 INFO:tasks.ceph.mds.b.smithi149.stdout:starting mds.b at 2023-11-28T13:01:48.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.858+0000 7f2d9b9b9700 1 Processor -- start 2023-11-28T13:01:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.858+0000 7f2d9b9b9700 1 -- start start 2023-11-28T13:01:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d9b9b9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d94142e80 0x7f2d9406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d9b9b9700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f2d9406a7b0 0x7f2d940c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d9b9b9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2d940ca430 0x7f2d9406d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d9b9b9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d940c9380 con 0x7f2d940ca430 2023-11-28T13:01:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d9b9b9700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d9406ac00 con 0x7f2d9406a7b0 2023-11-28T13:01:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d9b9b9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d94159b10 con 0x7f2d94142e80 2023-11-28T13:01:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d99f56700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2d940ca430 0x7f2d9406d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:48.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d99f56700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2d940ca430 0x7f2d9406d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37060/0 (socket says 172.21.15.111:37060) 2023-11-28T13:01:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d99f56700 1 -- 172.21.15.111:0/4063379233 learned_addr learned my addr 172.21.15.111:0/4063379233 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d99755700 1 --2- 172.21.15.111:0/4063379233 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d94142e80 0x7f2d9406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d98f54700 1 --2- 172.21.15.111:0/4063379233 >> v2:172.21.15.149:3300/0 conn(0x7f2d9406a7b0 0x7f2d940c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d99f56700 1 -- 172.21.15.111:0/4063379233 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d94142e80 msgr2=0x7f2d9406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d99f56700 1 --2- 172.21.15.111:0/4063379233 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d94142e80 0x7f2d9406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d99f56700 1 -- 172.21.15.111:0/4063379233 >> v2:172.21.15.149:3300/0 conn(0x7f2d9406a7b0 msgr2=0x7f2d940c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d99f56700 1 --2- 172.21.15.111:0/4063379233 >> v2:172.21.15.149:3300/0 conn(0x7f2d9406a7b0 0x7f2d940c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d99f56700 1 -- 172.21.15.111:0/4063379233 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d940ca890 con 0x7f2d940ca430 2023-11-28T13:01:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d99755700 1 --2- 172.21.15.111:0/4063379233 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d94142e80 0x7f2d9406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d99f56700 1 --2- 172.21.15.111:0/4063379233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d940ca430 0x7f2d9406d2d0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2d9406eee0 tx=0x7f2d8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a71f04f32f3eba server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d8bfff700 1 -- 172.21.15.111:0/4063379233 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c00e070 con 0x7f2d940ca430 2023-11-28T13:01:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d8bfff700 1 -- 172.21.15.111:0/4063379233 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8c00b9c0 con 0x7f2d940ca430 2023-11-28T13:01:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.859+0000 7f2d8bfff700 1 -- 172.21.15.111:0/4063379233 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c00bd70 con 0x7f2d940ca430 2023-11-28T13:01:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.860+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/4063379233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d940ca430 msgr2=0x7f2d9406d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.860+0000 7f2d9b9b9700 1 --2- 172.21.15.111:0/4063379233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d940ca430 0x7f2d9406d2d0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2d9406eee0 tx=0x7f2d8c00b330 comp rx=0 tx=0).stop 2023-11-28T13:01:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.860+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/4063379233 shutdown_connections 2023-11-28T13:01:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.860+0000 7f2d9b9b9700 1 --2- 172.21.15.111:0/4063379233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d940ca430 0x7f2d9406d2d0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.860+0000 7f2d9b9b9700 1 --2- 172.21.15.111:0/4063379233 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d94142e80 0x7f2d9406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.860+0000 7f2d9b9b9700 1 --2- 172.21.15.111:0/4063379233 >> v2:172.21.15.149:3300/0 conn(0x7f2d9406a7b0 0x7f2d940c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.860+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/4063379233 >> 172.21.15.111:0/4063379233 conn(0x7f2d940cb490 msgr2=0x7f2d940cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.860+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/4063379233 shutdown_connections 2023-11-28T13:01:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.860+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/4063379233 wait complete. 2023-11-28T13:01:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.861+0000 7f2d9b9b9700 1 Processor -- start 2023-11-28T13:01:48.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.861+0000 7f2d9b9b9700 1 -- start start 2023-11-28T13:01:48.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d9b9b9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d9406a7b0 0x7f2d94159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:48.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d9b9b9700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2d940ca430 0x7f2d941f3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:48.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d9b9b9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d94142e80 0x7f2d9415c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:48.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d9b9b9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d941f4360 con 0x7f2d9406a7b0 2023-11-28T13:01:48.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d9b9b9700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9415a330 con 0x7f2d940ca430 2023-11-28T13:01:48.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d9b9b9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9415bd90 con 0x7f2d94142e80 2023-11-28T13:01:48.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d99755700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d9406a7b0 0x7f2d94159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:48.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d99755700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d9406a7b0 0x7f2d94159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37062/0 (socket says 172.21.15.111:37062) 2023-11-28T13:01:48.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d99755700 1 -- 172.21.15.111:0/1088193134 learned_addr learned my addr 172.21.15.111:0/1088193134 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:48.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d99f56700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d94142e80 0x7f2d9415c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:48.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d98f54700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2d940ca430 0x7f2d941f3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:48.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d99755700 1 -- 172.21.15.111:0/1088193134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d94142e80 msgr2=0x7f2d9415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:48.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d99755700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d94142e80 0x7f2d9415c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d99755700 1 -- 172.21.15.111:0/1088193134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2d940ca430 msgr2=0x7f2d941f3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:48.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d99755700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2d940ca430 0x7f2d941f3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:48.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d99755700 1 -- 172.21.15.111:0/1088193134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c009b30 con 0x7f2d9406a7b0 2023-11-28T13:01:48.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.862+0000 7f2d99755700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d9406a7b0 0x7f2d94159ce0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000e760 tx=0x7f2d9000ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:48.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.863+0000 7f2d89ffb700 1 -- 172.21.15.111:0/1088193134 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9000c070 con 0x7f2d9406a7b0 2023-11-28T13:01:48.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.863+0000 7f2d89ffb700 1 -- 172.21.15.111:0/1088193134 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d90013610 con 0x7f2d9406a7b0 2023-11-28T13:01:48.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.863+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/1088193134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d940cab00 con 0x7f2d9406a7b0 2023-11-28T13:01:48.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.863+0000 7f2d89ffb700 1 -- 172.21.15.111:0/1088193134 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d900139c0 con 0x7f2d9406a7b0 2023-11-28T13:01:48.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.863+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/1088193134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d94143250 con 0x7f2d9406a7b0 2023-11-28T13:01:48.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.864+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/1088193134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d940cc5f0 con 0x7f2d9406a7b0 2023-11-28T13:01:48.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.864+0000 7f2d89ffb700 1 -- 172.21.15.111:0/1088193134 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2d90023070 con 0x7f2d9406a7b0 2023-11-28T13:01:48.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.864+0000 7f2d89ffb700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2d780682f0 0x7f2d7806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:48.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.864+0000 7f2d89ffb700 1 -- 172.21.15.111:0/1088193134 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f2d90085650 con 0x7f2d9406a7b0 2023-11-28T13:01:48.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.864+0000 7f2d98f54700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2d780682f0 0x7f2d7806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:48.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.865+0000 7f2d98f54700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2d780682f0 0x7f2d7806a7a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2d9406cd70 tx=0x7f2d84008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:48.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:48.869+0000 7f2d89ffb700 1 -- 172.21.15.111:0/1088193134 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2d90052120 con 0x7f2d9406a7b0 2023-11-28T13:01:49.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.067+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/1088193134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2d9415bbf0 con 0x7f2d9406a7b0 2023-11-28T13:01:49.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.848+0000 7f2d89ffb700 1 -- 172.21.15.111:0/1088193134 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v63) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2d9415bbf0 con 0x7f2d9406a7b0 2023-11-28T13:01:49.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.851+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/1088193134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2d780682f0 msgr2=0x7f2d7806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:49.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.851+0000 7f2d9b9b9700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2d780682f0 0x7f2d7806a7a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2d9406cd70 tx=0x7f2d84008040 comp rx=0 tx=0).stop 2023-11-28T13:01:49.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.851+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/1088193134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d9406a7b0 msgr2=0x7f2d94159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:49.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.851+0000 7f2d9b9b9700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d9406a7b0 0x7f2d94159ce0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000e760 tx=0x7f2d9000ea70 comp rx=0 tx=0).stop 2023-11-28T13:01:49.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.851+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/1088193134 shutdown_connections 2023-11-28T13:01:49.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.851+0000 7f2d9b9b9700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2d94142e80 0x7f2d9415c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:49.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.851+0000 7f2d9b9b9700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d9406a7b0 0x7f2d94159ce0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:49.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.851+0000 7f2d9b9b9700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2d780682f0 0x7f2d7806a7a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:49.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.851+0000 7f2d9b9b9700 1 --2- 172.21.15.111:0/1088193134 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2d940ca430 0x7f2d941f3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:49.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.851+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/1088193134 >> 172.21.15.111:0/1088193134 conn(0x7f2d940cb490 msgr2=0x7f2d940ae6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:49.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.851+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/1088193134 shutdown_connections 2023-11-28T13:01:49.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.851+0000 7f2d9b9b9700 1 -- 172.21.15.111:0/1088193134 wait complete. 2023-11-28T13:01:49.854 INFO:teuthology.orchestra.run.smithi111.stderr: removed all blocklist entries 2023-11-28T13:01:49.867 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T13:01:49.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.991+0000 7f6969ae4700 1 Processor -- start 2023-11-28T13:01:49.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.991+0000 7f6969ae4700 1 -- start start 2023-11-28T13:01:49.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.991+0000 7f6969ae4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f696406ba90 0x7f69640cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:49.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.991+0000 7f6969ae4700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69640cb810 con 0x7f696415c7b0 2023-11-28T13:01:49.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.991+0000 7f6969ae4700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69640cc650 con 0x7f69640d0490 2023-11-28T13:01:49.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.991+0000 7f6969ae4700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69640cba40 con 0x7f696406ba90 2023-11-28T13:01:49.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.991+0000 7f6963fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f696406ba90 0x7f69640cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:49.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.991+0000 7f6963fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f696406ba90 0x7f69640cb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37078/0 (socket says 172.21.15.111:37078) 2023-11-28T13:01:49.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.991+0000 7f6963fff700 1 -- 172.21.15.111:0/1218521798 learned_addr learned my addr 172.21.15.111:0/1218521798 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:49.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.991+0000 7f69637fe700 1 --1- 172.21.15.111:0/1218521798 >> v1:172.21.15.111:6789/0 conn(0x7f696415c7b0 0x7f69640ad8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37852/0 (socket says 172.21.15.111:37852) 2023-11-28T13:01:49.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.992+0000 7f6963fff700 1 -- 172.21.15.111:0/1218521798 >> v1:172.21.15.149:6789/0 conn(0x7f69640d0490 legacy=0x7f696406b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.992+0000 7f6963fff700 1 -- 172.21.15.111:0/1218521798 >> v1:172.21.15.111:6789/0 conn(0x7f696415c7b0 legacy=0x7f69640ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.992+0000 7f6963fff700 1 -- 172.21.15.111:0/1218521798 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69640cbd20 con 0x7f696406ba90 2023-11-28T13:01:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.992+0000 7f69627fc700 1 -- 172.21.15.111:0/1218521798 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098568721 0 0) 0x7f69640cb810 con 0x7f696415c7b0 2023-11-28T13:01:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.992+0000 7f6963fff700 1 --2- 172.21.15.111:0/1218521798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f696406ba90 0x7f69640cb2d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f695c009e50 tx=0x7f695c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd83ca6858af5f01 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.992+0000 7f69627fc700 1 -- 172.21.15.111:0/1218521798 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c00e070 con 0x7f696406ba90 2023-11-28T13:01:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.992+0000 7f69627fc700 1 -- 172.21.15.111:0/1218521798 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695c003a20 con 0x7f696406ba90 2023-11-28T13:01:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.992+0000 7f69627fc700 1 -- 172.21.15.111:0/1218521798 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c003dd0 con 0x7f696406ba90 2023-11-28T13:01:49.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.993+0000 7f6969ae4700 1 -- 172.21.15.111:0/1218521798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f696406ba90 msgr2=0x7f69640cb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:49.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.993+0000 7f6969ae4700 1 --2- 172.21.15.111:0/1218521798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f696406ba90 0x7f69640cb2d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f695c009e50 tx=0x7f695c00b330 comp rx=0 tx=0).stop 2023-11-28T13:01:49.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.993+0000 7f6969ae4700 1 -- 172.21.15.111:0/1218521798 shutdown_connections 2023-11-28T13:01:49.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.993+0000 7f6969ae4700 1 --2- 172.21.15.111:0/1218521798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f696406ba90 0x7f69640cb2d0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:49.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.993+0000 7f6969ae4700 1 -- 172.21.15.111:0/1218521798 >> 172.21.15.111:0/1218521798 conn(0x7f696409a040 msgr2=0x7f69640d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:49.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.993+0000 7f6969ae4700 1 -- 172.21.15.111:0/1218521798 shutdown_connections 2023-11-28T13:01:49.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.993+0000 7f6969ae4700 1 -- 172.21.15.111:0/1218521798 wait complete. 2023-11-28T13:01:50.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.994+0000 7f6969ae4700 1 Processor -- start 2023-11-28T13:01:50.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.994+0000 7f6969ae4700 1 -- start start 2023-11-28T13:01:50.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f6969ae4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f696406ba90 0x7f6964154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f6969ae4700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69640d0490 0x7f69640b5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f6969ae4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f696415c7b0 0x7f69640b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f6969ae4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69640b0cc0 con 0x7f696406ba90 2023-11-28T13:01:50.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f6969ae4700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69641547b0 con 0x7f69640d0490 2023-11-28T13:01:50.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f6969ae4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69640c9ea0 con 0x7f696415c7b0 2023-11-28T13:01:50.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f69637fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f696406ba90 0x7f6964154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f69637fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f696406ba90 0x7f6964154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37080/0 (socket says 172.21.15.111:37080) 2023-11-28T13:01:50.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f69637fe700 1 -- 172.21.15.111:0/431615619 learned_addr learned my addr 172.21.15.111:0/431615619 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:50.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f6962ffd700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69640d0490 0x7f69640b5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f6963fff700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f696415c7b0 0x7f69640b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f69637fe700 1 -- 172.21.15.111:0/431615619 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f696415c7b0 msgr2=0x7f69640b62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f69637fe700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f696415c7b0 0x7f69640b62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f69637fe700 1 -- 172.21.15.111:0/431615619 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69640d0490 msgr2=0x7f69640b5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f69637fe700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69640d0490 0x7f69640b5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f69637fe700 1 -- 172.21.15.111:0/431615619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c009b30 con 0x7f696406ba90 2023-11-28T13:01:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f69637fe700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f696406ba90 0x7f6964154270 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f695800c520 tx=0x7f695800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f694bfff700 1 -- 172.21.15.111:0/431615619 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69580074e0 con 0x7f696406ba90 2023-11-28T13:01:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f694bfff700 1 -- 172.21.15.111:0/431615619 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6958007640 con 0x7f696406ba90 2023-11-28T13:01:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f6969ae4700 1 -- 172.21.15.111:0/431615619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69640cc2e0 con 0x7f696406ba90 2023-11-28T13:01:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f694bfff700 1 -- 172.21.15.111:0/431615619 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6958005a50 con 0x7f696406ba90 2023-11-28T13:01:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.995+0000 7f6969ae4700 1 -- 172.21.15.111:0/431615619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69640cc000 con 0x7f696406ba90 2023-11-28T13:01:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.996+0000 7f6969ae4700 1 -- 172.21.15.111:0/431615619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696415cb80 con 0x7f696406ba90 2023-11-28T13:01:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.997+0000 7f694bfff700 1 -- 172.21.15.111:0/431615619 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6958019030 con 0x7f696406ba90 2023-11-28T13:01:50.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.997+0000 7f694bfff700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f69500682f0 0x7f695006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.997+0000 7f694bfff700 1 -- 172.21.15.111:0/431615619 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f6958084ea0 con 0x7f696406ba90 2023-11-28T13:01:50.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.997+0000 7f6962ffd700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f69500682f0 0x7f695006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:49.997+0000 7f6962ffd700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f69500682f0 0x7f695006a7a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6964070ab0 tx=0x7f6954009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:50.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.001+0000 7f694bfff700 1 -- 172.21.15.111:0/431615619 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6958051910 con 0x7f696406ba90 2023-11-28T13:01:50.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.257+0000 7f6969ae4700 1 -- 172.21.15.111:0/431615619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6964067170 con 0x7f696406ba90 2023-11-28T13:01:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.258+0000 7f694bfff700 1 -- 172.21.15.111:0/431615619 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5298 (secure 0 0 0) 0x7f6958054f30 con 0x7f696406ba90 2023-11-28T13:01:50.262 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:50.262 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:01:50.262 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_dump": [ 2023-11-28T13:01:50.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.262 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.a", 2023-11-28T13:01:50.262 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlH823BRAArK8kE5qUgPoqFZPxIudN+w==", 2023-11-28T13:01:50.262 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.262 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:01:50.262 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:01:50.262 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:01:50.262 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.263 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.263 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.b", 2023-11-28T13:01:50.263 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlIo9ZEhAA5NmyiPE68OmMcsG1hTiTgw==", 2023-11-28T13:01:50.263 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.263 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:01:50.263 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:01:50.263 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:01:50.263 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.263 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.264 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.264 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.c", 2023-11-28T13:01:50.264 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlQqHxDBAA8RDJ2PETpTHNjnn1i1T3Ow==", 2023-11-28T13:01:50.264 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.264 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:01:50.264 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:01:50.264 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:01:50.264 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.264 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.264 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.264 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mds.d", 2023-11-28T13:01:50.265 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42Vl8RDmGRAAkiGKEQMr9rHahKJ/3lw9BA==", 2023-11-28T13:01:50.265 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.265 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:01:50.265 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:01:50.265 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:01:50.265 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.265 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.265 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.0", 2023-11-28T13:01:50.265 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDc42VlfB0eNRAAJEa0AL3iDLArHc4n6CWK9g==", 2023-11-28T13:01:50.265 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.265 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:01:50.266 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:01:50.266 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.266 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.266 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.1", 2023-11-28T13:01:50.266 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDf42VlnrbLNBAAU+sVJ/6EsadNo3BXpyrbSw==", 2023-11-28T13:01:50.266 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.266 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:01:50.266 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:01:50.266 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.266 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.2", 2023-11-28T13:01:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDi42VlwCUsNRAAwZKJ/o1qTBtWwR35B1r2zQ==", 2023-11-28T13:01:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:01:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:01:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.3", 2023-11-28T13:01:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDl42Vl3/t2NBAAMPuhlpUi37Fpd/6NclsErw==", 2023-11-28T13:01:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.268 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:01:50.268 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:01:50.268 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.268 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.268 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.4", 2023-11-28T13:01:50.268 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDr42VlkrWgNxAAuu2dXZZ3U/XRrq3cWCEa8w==", 2023-11-28T13:01:50.268 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.268 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:01:50.268 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:01:50.268 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.269 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.5", 2023-11-28T13:01:50.269 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDu42VldDFENxAAYKD2Dqgka5G+XxGjOR3M6w==", 2023-11-28T13:01:50.269 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.269 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:01:50.269 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:01:50.269 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.269 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.269 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.6", 2023-11-28T13:01:50.270 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDx42VlvjotNhAAWD2E1sgAZLdxb5CG1Rg26g==", 2023-11-28T13:01:50.270 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.270 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:01:50.270 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:01:50.270 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.270 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.270 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "osd.7", 2023-11-28T13:01:50.270 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQD042VlZheVNBAAlMIKaDmgUMmt/hA+vx2/PA==", 2023-11-28T13:01:50.270 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.270 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow profile osd", 2023-11-28T13:01:50.271 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile osd", 2023-11-28T13:01:50.271 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.271 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.271 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.0", 2023-11-28T13:01:50.271 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42Vl3MswHhAA/oDIgj9lzhcnU7f0ksZ9cw==", 2023-11-28T13:01:50.271 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.271 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:01:50.271 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow r", 2023-11-28T13:01:50.271 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rw pool=cephfs_data" 2023-11-28T13:01:50.271 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.272 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.1", 2023-11-28T13:01:50.272 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlV3dQJxAAdzQk1q+DoxFDEr/TYJi/6A==", 2023-11-28T13:01:50.272 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.272 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:01:50.272 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-11-28T13:01:50.272 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-11-28T13:01:50.272 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-11-28T13:01:50.272 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.272 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.2", 2023-11-28T13:01:50.272 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlJiXlIxAAYuFMFLUqIDN5f0E008GKZw==", 2023-11-28T13:01:50.273 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.273 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:01:50.273 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-11-28T13:01:50.273 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-11-28T13:01:50.273 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-11-28T13:01:50.273 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.273 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.273 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.273 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.3", 2023-11-28T13:01:50.273 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42VlagSZKhAAkLS8RwTZZTDyO2slb0MzIQ==", 2023-11-28T13:01:50.273 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.273 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow", 2023-11-28T13:01:50.274 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow r", 2023-11-28T13:01:50.274 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow rw", 2023-11-28T13:01:50.274 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow rwx" 2023-11-28T13:01:50.274 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.274 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "client.admin", 2023-11-28T13:01:50.274 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDY42Vl74R5IhAADPhHAlk7DX23esg8UFYW/Q==", 2023-11-28T13:01:50.274 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.274 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-11-28T13:01:50.274 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": "allow *", 2023-11-28T13:01:50.274 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow *", 2023-11-28T13:01:50.274 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.275 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.275 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.x", 2023-11-28T13:01:50.275 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDZ42Vl+AxXAhAADTJ8Jt/WNo8qr5zAyQJ4rw==", 2023-11-28T13:01:50.275 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.275 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-11-28T13:01:50.275 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-11-28T13:01:50.275 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.275 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:50.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:50.275 INFO:teuthology.orchestra.run.smithi111.stdout: "entity": "mgr.y", 2023-11-28T13:01:50.276 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "AQDY42Vl6WCiOBAALRDd8ZjOOlxxLKQsgXP9Jg==", 2023-11-28T13:01:50.276 INFO:teuthology.orchestra.run.smithi111.stdout: "caps": { 2023-11-28T13:01:50.276 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": "allow *", 2023-11-28T13:01:50.276 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": "allow profile mgr", 2023-11-28T13:01:50.276 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": "allow *" 2023-11-28T13:01:50.276 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.276 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:50.276 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-11-28T13:01:50.276 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:01:50.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.260+0000 7f6969ae4700 1 -- 172.21.15.111:0/431615619 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f69500682f0 msgr2=0x7f695006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.260+0000 7f6969ae4700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f69500682f0 0x7f695006a7a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6964070ab0 tx=0x7f6954009040 comp rx=0 tx=0).stop 2023-11-28T13:01:50.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.260+0000 7f6969ae4700 1 -- 172.21.15.111:0/431615619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f696406ba90 msgr2=0x7f6964154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.260+0000 7f6969ae4700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f696406ba90 0x7f6964154270 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f695800c520 tx=0x7f695800c8e0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.260+0000 7f6969ae4700 1 -- 172.21.15.111:0/431615619 shutdown_connections 2023-11-28T13:01:50.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.260+0000 7f6969ae4700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f696415c7b0 0x7f69640b62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.260+0000 7f6969ae4700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f696406ba90 0x7f6964154270 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.260+0000 7f6969ae4700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f69500682f0 0x7f695006a7a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.260+0000 7f6969ae4700 1 --2- 172.21.15.111:0/431615619 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f69640d0490 0x7f69640b5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.260+0000 7f6969ae4700 1 -- 172.21.15.111:0/431615619 >> 172.21.15.111:0/431615619 conn(0x7f696409a040 msgr2=0x7f69640d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:50.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.260+0000 7f6969ae4700 1 -- 172.21.15.111:0/431615619 shutdown_connections 2023-11-28T13:01:50.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.260+0000 7f6969ae4700 1 -- 172.21.15.111:0/431615619 wait complete. 2023-11-28T13:01:50.279 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:01:50.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.397+0000 7fad6da5e700 1 Processor -- start 2023-11-28T13:01:50.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.397+0000 7fad6da5e700 1 -- start start 2023-11-28T13:01:50.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.398+0000 7fad6da5e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fad680b8510 0x7fad6806b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.398+0000 7fad6da5e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fad680cd460 0x7fad6806beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.398+0000 7fad6da5e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad680b88e0 con 0x7fad6806c3f0 2023-11-28T13:01:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.398+0000 7fad6da5e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad680cdbd0 con 0x7fad680cd460 2023-11-28T13:01:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.398+0000 7fad6da5e700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad680d02d0 con 0x7fad680b8510 2023-11-28T13:01:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.398+0000 7fad667fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fad680cd460 0x7fad6806beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.398+0000 7fad677fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fad6806c3f0 0x7fad6815cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37870/0 (socket says 172.21.15.111:37870) 2023-11-28T13:01:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.398+0000 7fad667fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fad680cd460 0x7fad6806beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37088/0 (socket says 172.21.15.111:37088) 2023-11-28T13:01:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.398+0000 7fad677fe700 1 -- 172.21.15.111:0/3490582397 learned_addr learned my addr 172.21.15.111:0/3490582397 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.399+0000 7fad66ffd700 1 --2- 172.21.15.111:0/3490582397 >> v2:172.21.15.149:3300/0 conn(0x7fad680b8510 0x7fad6806b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.399+0000 7fad667fc700 1 -- 172.21.15.111:0/3490582397 >> v2:172.21.15.149:3300/0 conn(0x7fad680b8510 msgr2=0x7fad6806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.399+0000 7fad667fc700 1 --2- 172.21.15.111:0/3490582397 >> v2:172.21.15.149:3300/0 conn(0x7fad680b8510 0x7fad6806b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.399+0000 7fad667fc700 1 -- 172.21.15.111:0/3490582397 >> v1:172.21.15.111:6789/0 conn(0x7fad6806c3f0 legacy=0x7fad6815cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.399+0000 7fad667fc700 1 -- 172.21.15.111:0/3490582397 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad680cd8f0 con 0x7fad680cd460 2023-11-28T13:01:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.399+0000 7fad65ffb700 1 -- 172.21.15.111:0/3490582397 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810896477 0 0) 0x7fad680b88e0 con 0x7fad6806c3f0 2023-11-28T13:01:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.399+0000 7fad66ffd700 1 --2- 172.21.15.111:0/3490582397 >> v2:172.21.15.149:3300/0 conn(0x7fad680b8510 0x7fad6806b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.399+0000 7fad667fc700 1 --2- 172.21.15.111:0/3490582397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad680cd460 0x7fad6806beb0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fad58009870 tx=0x7fad58009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979a1e3e3009209e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.399+0000 7fad65ffb700 1 -- 172.21.15.111:0/3490582397 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad58011830 con 0x7fad680cd460 2023-11-28T13:01:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.399+0000 7fad65ffb700 1 -- 172.21.15.111:0/3490582397 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad58011990 con 0x7fad680cd460 2023-11-28T13:01:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.399+0000 7fad65ffb700 1 -- 172.21.15.111:0/3490582397 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad58011d20 con 0x7fad680cd460 2023-11-28T13:01:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.400+0000 7fad6da5e700 1 -- 172.21.15.111:0/3490582397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad680cd460 msgr2=0x7fad6806beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.400+0000 7fad6da5e700 1 --2- 172.21.15.111:0/3490582397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad680cd460 0x7fad6806beb0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fad58009870 tx=0x7fad58009b80 comp rx=0 tx=0).stop 2023-11-28T13:01:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.400+0000 7fad6da5e700 1 -- 172.21.15.111:0/3490582397 shutdown_connections 2023-11-28T13:01:50.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.400+0000 7fad6da5e700 1 --2- 172.21.15.111:0/3490582397 >> v2:172.21.15.149:3300/0 conn(0x7fad680b8510 0x7fad6806b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.400+0000 7fad6da5e700 1 --2- 172.21.15.111:0/3490582397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad680cd460 0x7fad6806beb0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.400+0000 7fad6da5e700 1 -- 172.21.15.111:0/3490582397 >> 172.21.15.111:0/3490582397 conn(0x7fad680719a0 msgr2=0x7fad68071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:50.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.400+0000 7fad6da5e700 1 -- 172.21.15.111:0/3490582397 shutdown_connections 2023-11-28T13:01:50.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.400+0000 7fad6da5e700 1 -- 172.21.15.111:0/3490582397 wait complete. 2023-11-28T13:01:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.401+0000 7fad6da5e700 1 Processor -- start 2023-11-28T13:01:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.401+0000 7fad6da5e700 1 -- start start 2023-11-28T13:01:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad6da5e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad6806c3f0 0x7fad680cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad6da5e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fad680b8510 0x7fad680b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad6da5e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fad680cd460 0x7fad681f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad6da5e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad680cc440 con 0x7fad6806c3f0 2023-11-28T13:01:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad6da5e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad680cb140 con 0x7fad680cd460 2023-11-28T13:01:50.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad6da5e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad680cb420 con 0x7fad680b8510 2023-11-28T13:01:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad66ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad6806c3f0 0x7fad680cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad66ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad6806c3f0 0x7fad680cbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37094/0 (socket says 172.21.15.111:37094) 2023-11-28T13:01:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad66ffd700 1 -- 172.21.15.111:0/954039224 learned_addr learned my addr 172.21.15.111:0/954039224 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad677fe700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fad680cd460 0x7fad681f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad667fc700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fad680b8510 0x7fad680b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad66ffd700 1 -- 172.21.15.111:0/954039224 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fad680b8510 msgr2=0x7fad680b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad66ffd700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fad680b8510 0x7fad680b6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad66ffd700 1 -- 172.21.15.111:0/954039224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fad680cd460 msgr2=0x7fad681f1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad66ffd700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fad680cd460 0x7fad681f1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad66ffd700 1 -- 172.21.15.111:0/954039224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad580094a0 con 0x7fad6806c3f0 2023-11-28T13:01:50.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad677fe700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fad680cd460 0x7fad681f1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:50.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad66ffd700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad6806c3f0 0x7fad680cbf00 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fad5c00c750 tx=0x7fad5c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:50.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad4ffff700 1 -- 172.21.15.111:0/954039224 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c011800 con 0x7fad6806c3f0 2023-11-28T13:01:50.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad4ffff700 1 -- 172.21.15.111:0/954039224 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5c011960 con 0x7fad6806c3f0 2023-11-28T13:01:50.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad6da5e700 1 -- 172.21.15.111:0/954039224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad680cfff0 con 0x7fad6806c3f0 2023-11-28T13:01:50.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad4ffff700 1 -- 172.21.15.111:0/954039224 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c011cd0 con 0x7fad6806c3f0 2023-11-28T13:01:50.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.402+0000 7fad6da5e700 1 -- 172.21.15.111:0/954039224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad680cfd10 con 0x7fad6806c3f0 2023-11-28T13:01:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.403+0000 7fad6da5e700 1 -- 172.21.15.111:0/954039224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad68067880 con 0x7fad6806c3f0 2023-11-28T13:01:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.404+0000 7fad4ffff700 1 -- 172.21.15.111:0/954039224 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fad5c020070 con 0x7fad6806c3f0 2023-11-28T13:01:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.404+0000 7fad4ffff700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fad540682f0 0x7fad5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.404+0000 7fad4ffff700 1 -- 172.21.15.111:0/954039224 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fad5c085900 con 0x7fad6806c3f0 2023-11-28T13:01:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.404+0000 7fad667fc700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fad540682f0 0x7fad5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.405+0000 7fad667fc700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fad540682f0 0x7fad5406a7a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fad680b1de0 tx=0x7fad5801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.409+0000 7fad4ffff700 1 -- 172.21.15.111:0/954039224 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fad5c052370 con 0x7fad6806c3f0 2023-11-28T13:01:50.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.591+0000 7fad6da5e700 1 -- 172.21.15.111:0/954039224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad680d2940 con 0x7fad6806c3f0 2023-11-28T13:01:50.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.592+0000 7fad4ffff700 1 -- 172.21.15.111:0/954039224 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+14520 (secure 0 0 0) 0x7fad5c055990 con 0x7fad6806c3f0 2023-11-28T13:01:50.594 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:50.594 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":63,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:01:49.845577+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:01:50.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.594+0000 7fad6da5e700 1 -- 172.21.15.111:0/954039224 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fad540682f0 msgr2=0x7fad5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.594+0000 7fad6da5e700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fad540682f0 0x7fad5406a7a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fad680b1de0 tx=0x7fad5801c040 comp rx=0 tx=0).stop 2023-11-28T13:01:50.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.594+0000 7fad6da5e700 1 -- 172.21.15.111:0/954039224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad6806c3f0 msgr2=0x7fad680cbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.594+0000 7fad6da5e700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad6806c3f0 0x7fad680cbf00 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fad5c00c750 tx=0x7fad5c00cb10 comp rx=0 tx=0).stop 2023-11-28T13:01:50.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.594+0000 7fad6da5e700 1 -- 172.21.15.111:0/954039224 shutdown_connections 2023-11-28T13:01:50.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.594+0000 7fad6da5e700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fad540682f0 0x7fad5406a7a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.594+0000 7fad6da5e700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fad680cd460 0x7fad681f1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.594+0000 7fad6da5e700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad6806c3f0 0x7fad680cbf00 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.594+0000 7fad6da5e700 1 --2- 172.21.15.111:0/954039224 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fad680b8510 0x7fad680b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.594+0000 7fad6da5e700 1 -- 172.21.15.111:0/954039224 >> 172.21.15.111:0/954039224 conn(0x7fad680719a0 msgr2=0x7fad680b5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:50.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.594+0000 7fad6da5e700 1 -- 172.21.15.111:0/954039224 shutdown_connections 2023-11-28T13:01:50.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.594+0000 7fad6da5e700 1 -- 172.21.15.111:0/954039224 wait complete. 2023-11-28T13:01:50.607 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T13:01:50.608 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T13:01:50.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.729+0000 7fc7ec3f6700 1 Processor -- start 2023-11-28T13:01:50.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.729+0000 7fc7ec3f6700 1 -- start start 2023-11-28T13:01:50.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.730+0000 7fc7ec3f6700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc7e41604b0 0x7fc7e40ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.730+0000 7fc7ec3f6700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7e40cc550 0x7fc7e40cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.730+0000 7fc7ec3f6700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e415cd90 con 0x7fc7e40b0d70 2023-11-28T13:01:50.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.730+0000 7fc7ec3f6700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e4160950 con 0x7fc7e41604b0 2023-11-28T13:01:50.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.730+0000 7fc7ec3f6700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e40ccf10 con 0x7fc7e40cc550 2023-11-28T13:01:50.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.730+0000 7fc7ea993700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7e40cc550 0x7fc7e40cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.730+0000 7fc7e9991700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc7e41604b0 0x7fc7e40ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.730+0000 7fc7ea993700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7e40cc550 0x7fc7e40cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:58854/0 (socket says 172.21.15.111:58854) 2023-11-28T13:01:50.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.730+0000 7fc7ea993700 1 -- 172.21.15.111:0/2496900511 learned_addr learned my addr 172.21.15.111:0/2496900511 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:50.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.730+0000 7fc7e9991700 1 -- 172.21.15.111:0/2496900511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7e40cc550 msgr2=0x7fc7e40cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:50.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.730+0000 7fc7e9991700 1 --2- 172.21.15.111:0/2496900511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7e40cc550 0x7fc7e40cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7e9991700 1 -- 172.21.15.111:0/2496900511 >> v1:172.21.15.149:6789/0 conn(0x7fc7e40b0d70 legacy=0x7fc7e40b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7e9991700 1 -- 172.21.15.111:0/2496900511 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e41613e0 con 0x7fc7e41604b0 2023-11-28T13:01:50.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7e9190700 1 -- 172.21.15.111:0/2496900511 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693261714 0 0) 0x7fc7d4001160 con 0x7fc7e40b0d70 2023-11-28T13:01:50.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7ea993700 1 --2- 172.21.15.111:0/2496900511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7e40cc550 0x7fc7e40cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:01:50.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7e9991700 1 --2- 172.21.15.111:0/2496900511 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7e41604b0 0x7fc7e40ce6c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009840 tx=0x7fc7e0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fa782293e07f327 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:50.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7e9190700 1 -- 172.21.15.111:0/2496900511 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0011830 con 0x7fc7e41604b0 2023-11-28T13:01:50.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7e9190700 1 -- 172.21.15.111:0/2496900511 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e0011990 con 0x7fc7e41604b0 2023-11-28T13:01:50.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7e9190700 1 -- 172.21.15.111:0/2496900511 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0011d20 con 0x7fc7e41604b0 2023-11-28T13:01:50.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/2496900511 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7e41604b0 msgr2=0x7fc7e40ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7ec3f6700 1 --2- 172.21.15.111:0/2496900511 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7e41604b0 0x7fc7e40ce6c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009840 tx=0x7fc7e0009b50 comp rx=0 tx=0).stop 2023-11-28T13:01:50.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/2496900511 shutdown_connections 2023-11-28T13:01:50.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7ec3f6700 1 --2- 172.21.15.111:0/2496900511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7e40cc550 0x7fc7e40cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7ec3f6700 1 --2- 172.21.15.111:0/2496900511 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7e41604b0 0x7fc7e40ce6c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.731+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/2496900511 >> 172.21.15.111:0/2496900511 conn(0x7fc7e4098ee0 msgr2=0x7fc7e406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:50.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.732+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/2496900511 shutdown_connections 2023-11-28T13:01:50.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.732+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/2496900511 wait complete. 2023-11-28T13:01:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.733+0000 7fc7ec3f6700 1 Processor -- start 2023-11-28T13:01:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.733+0000 7fc7ec3f6700 1 -- start start 2023-11-28T13:01:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.733+0000 7fc7ec3f6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e40b0d70 0x7fc7e40c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.733+0000 7fc7ec3f6700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7e40cc550 0x7fc7e40b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.733+0000 7fc7ec3f6700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7e41604b0 0x7fc7e40c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.733+0000 7fc7ec3f6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e40c4460 con 0x7fc7e40b0d70 2023-11-28T13:01:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.733+0000 7fc7ec3f6700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e41f59f0 con 0x7fc7e40cc550 2023-11-28T13:01:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.733+0000 7fc7ec3f6700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e40c3bc0 con 0x7fc7e41604b0 2023-11-28T13:01:50.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.733+0000 7fc7ea192700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e40b0d70 0x7fc7e40c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7ea192700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e40b0d70 0x7fc7e40c35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37108/0 (socket says 172.21.15.111:37108) 2023-11-28T13:01:50.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7ea192700 1 -- 172.21.15.111:0/3943385875 learned_addr learned my addr 172.21.15.111:0/3943385875 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:50.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7e9991700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7e40cc550 0x7fc7e40b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7ea993700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7e41604b0 0x7fc7e40c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7ea192700 1 -- 172.21.15.111:0/3943385875 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7e41604b0 msgr2=0x7fc7e40c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7ea192700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7e41604b0 0x7fc7e40c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7ea192700 1 -- 172.21.15.111:0/3943385875 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7e40cc550 msgr2=0x7fc7e40b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7ea192700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7e40cc550 0x7fc7e40b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7ea192700 1 -- 172.21.15.111:0/3943385875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc009b30 con 0x7fc7e40b0d70 2023-11-28T13:01:50.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7ea993700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7e41604b0 0x7fc7e40c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:50.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7ea192700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e40b0d70 0x7fc7e40c35e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400c750 tx=0x7fc7d400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:50.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7daffd700 1 -- 172.21.15.111:0/3943385875 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40054e0 con 0x7fc7e40b0d70 2023-11-28T13:01:50.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7daffd700 1 -- 172.21.15.111:0/3943385875 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d40036d0 con 0x7fc7e40b0d70 2023-11-28T13:01:50.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/3943385875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e00094a0 con 0x7fc7e40b0d70 2023-11-28T13:01:50.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7daffd700 1 -- 172.21.15.111:0/3943385875 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4003a80 con 0x7fc7e40b0d70 2023-11-28T13:01:50.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.734+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/3943385875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e4160e20 con 0x7fc7e40b0d70 2023-11-28T13:01:50.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.735+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/3943385875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e4160b40 con 0x7fc7e40b0d70 2023-11-28T13:01:50.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.735+0000 7fc7daffd700 1 -- 172.21.15.111:0/3943385875 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc7d4019030 con 0x7fc7e40b0d70 2023-11-28T13:01:50.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.736+0000 7fc7daffd700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7c80682f0 0x7fc7c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:50.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.736+0000 7fc7daffd700 1 -- 172.21.15.111:0/3943385875 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fc7d4084ce0 con 0x7fc7e40b0d70 2023-11-28T13:01:50.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.736+0000 7fc7e9991700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7c80682f0 0x7fc7c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:50.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.736+0000 7fc7e9991700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7c80682f0 0x7fc7c806a7a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0000c00 tx=0x7fc7e0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:50.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.740+0000 7fc7daffd700 1 -- 172.21.15.111:0/3943385875 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc7d4051750 con 0x7fc7e40b0d70 2023-11-28T13:01:50.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.925+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/3943385875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7e4161100 con 0x7fc7e40b0d70 2023-11-28T13:01:50.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.925+0000 7fc7daffd700 1 -- 172.21.15.111:0/3943385875 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc7d4017070 con 0x7fc7e40b0d70 2023-11-28T13:01:50.927 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T13:01:50.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.927+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/3943385875 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7c80682f0 msgr2=0x7fc7c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.927+0000 7fc7ec3f6700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7c80682f0 0x7fc7c806a7a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0000c00 tx=0x7fc7e0008030 comp rx=0 tx=0).stop 2023-11-28T13:01:50.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.927+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/3943385875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e40b0d70 msgr2=0x7fc7e40c35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:50.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.927+0000 7fc7ec3f6700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e40b0d70 0x7fc7e40c35e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400c750 tx=0x7fc7d400ca60 comp rx=0 tx=0).stop 2023-11-28T13:01:50.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.927+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/3943385875 shutdown_connections 2023-11-28T13:01:50.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.927+0000 7fc7ec3f6700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc7e41604b0 0x7fc7e40c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.927+0000 7fc7ec3f6700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e40b0d70 0x7fc7e40c35e0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.927+0000 7fc7ec3f6700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc7c80682f0 0x7fc7c806a7a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.927+0000 7fc7ec3f6700 1 --2- 172.21.15.111:0/3943385875 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc7e40cc550 0x7fc7e40b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:50.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.927+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/3943385875 >> 172.21.15.111:0/3943385875 conn(0x7fc7e4098ee0 msgr2=0x7fc7e4072cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:50.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.927+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/3943385875 shutdown_connections 2023-11-28T13:01:50.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:50.927+0000 7fc7ec3f6700 1 -- 172.21.15.111:0/3943385875 wait complete. 2023-11-28T13:01:50.940 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T13:01:51.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.061+0000 7f24e44c5700 1 Processor -- start 2023-11-28T13:01:51.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.061+0000 7f24e44c5700 1 -- start start 2023-11-28T13:01:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.062+0000 7f24e44c5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24dc0cb830 0x7f24dc06a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.062+0000 7f24e44c5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f24dc06a7b0 0x7f24dc15df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.062+0000 7f24e44c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24dc06b490 con 0x7f24dc154b60 2023-11-28T13:01:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.062+0000 7f24e44c5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24dc0c9930 con 0x7f24dc06a7b0 2023-11-28T13:01:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.062+0000 7f24e44c5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc0c9e80 con 0x7f24dc0cb830 2023-11-28T13:01:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.062+0000 7f24e2a62700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f24dc06a7b0 0x7f24dc15df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.062+0000 7f24e2261700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f24dc154b60 0x7f24dc15b350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37894/0 (socket says 172.21.15.111:37894) 2023-11-28T13:01:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.062+0000 7f24e2261700 1 -- 172.21.15.111:0/820287254 learned_addr learned my addr 172.21.15.111:0/820287254 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.063+0000 7f24e1a60700 1 --2- 172.21.15.111:0/820287254 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24dc0cb830 0x7f24dc06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.063+0000 7f24e2a62700 1 -- 172.21.15.111:0/820287254 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24dc0cb830 msgr2=0x7f24dc06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.063+0000 7f24e2a62700 1 --2- 172.21.15.111:0/820287254 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24dc0cb830 0x7f24dc06a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.063+0000 7f24e2a62700 1 -- 172.21.15.111:0/820287254 >> v1:172.21.15.111:6789/0 conn(0x7f24dc154b60 legacy=0x7f24dc15b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.063+0000 7f24e125f700 1 -- 172.21.15.111:0/820287254 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305247164 0 0) 0x7f24dc06b490 con 0x7f24dc154b60 2023-11-28T13:01:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.063+0000 7f24e2a62700 1 -- 172.21.15.111:0/820287254 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc06abf0 con 0x7f24dc06a7b0 2023-11-28T13:01:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.063+0000 7f24e2a62700 1 --2- 172.21.15.111:0/820287254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc06a7b0 0x7f24dc15df10 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009d90 tx=0x7f24d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd176162df9b42f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.063+0000 7f24e125f700 1 -- 172.21.15.111:0/820287254 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d400e070 con 0x7f24dc06a7b0 2023-11-28T13:01:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.063+0000 7f24e125f700 1 -- 172.21.15.111:0/820287254 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d400bb20 con 0x7f24dc06a7b0 2023-11-28T13:01:51.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.063+0000 7f24e125f700 1 -- 172.21.15.111:0/820287254 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d4012430 con 0x7f24dc06a7b0 2023-11-28T13:01:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.064+0000 7f24e44c5700 1 -- 172.21.15.111:0/820287254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc06a7b0 msgr2=0x7f24dc15df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.064+0000 7f24e44c5700 1 --2- 172.21.15.111:0/820287254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc06a7b0 0x7f24dc15df10 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009d90 tx=0x7f24d400b330 comp rx=0 tx=0).stop 2023-11-28T13:01:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.064+0000 7f24e44c5700 1 -- 172.21.15.111:0/820287254 shutdown_connections 2023-11-28T13:01:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.064+0000 7f24e44c5700 1 --2- 172.21.15.111:0/820287254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc06a7b0 0x7f24dc15df10 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.064+0000 7f24e44c5700 1 --2- 172.21.15.111:0/820287254 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24dc0cb830 0x7f24dc06a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.064+0000 7f24e44c5700 1 -- 172.21.15.111:0/820287254 >> 172.21.15.111:0/820287254 conn(0x7f24dc0cf2e0 msgr2=0x7f24dc0cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.064+0000 7f24e44c5700 1 -- 172.21.15.111:0/820287254 shutdown_connections 2023-11-28T13:01:51.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.064+0000 7f24e44c5700 1 -- 172.21.15.111:0/820287254 wait complete. 2023-11-28T13:01:51.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.065+0000 7f24e44c5700 1 Processor -- start 2023-11-28T13:01:51.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.065+0000 7f24e44c5700 1 -- start start 2023-11-28T13:01:51.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e44c5700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24dc06a7b0 0x7f24dc0abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e44c5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24dc0cb830 0x7f24dc0c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e44c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc154b60 0x7f24dc0ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e44c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc14fb00 con 0x7f24dc154b60 2023-11-28T13:01:51.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e44c5700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc0c4d40 con 0x7f24dc06a7b0 2023-11-28T13:01:51.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e44c5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc0b22b0 con 0x7f24dc0cb830 2023-11-28T13:01:51.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e2a62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc154b60 0x7f24dc0ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e2a62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc154b60 0x7f24dc0ae3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37126/0 (socket says 172.21.15.111:37126) 2023-11-28T13:01:51.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e2a62700 1 -- 172.21.15.111:0/4016485878 learned_addr learned my addr 172.21.15.111:0/4016485878 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:51.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e2261700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24dc06a7b0 0x7f24dc0abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e2a62700 1 -- 172.21.15.111:0/4016485878 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24dc0cb830 msgr2=0x7f24dc0c46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e2a62700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24dc0cb830 0x7f24dc0c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e2a62700 1 -- 172.21.15.111:0/4016485878 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24dc06a7b0 msgr2=0x7f24dc0abbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e2a62700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24dc06a7b0 0x7f24dc0abbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e2a62700 1 -- 172.21.15.111:0/4016485878 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d4009a70 con 0x7f24dc154b60 2023-11-28T13:01:51.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e2261700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24dc06a7b0 0x7f24dc0abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:51.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e2a62700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc154b60 0x7f24dc0ae3a0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f24d400b680 tx=0x7f24d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:51.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24d2ffd700 1 -- 172.21.15.111:0/4016485878 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d4012430 con 0x7f24dc154b60 2023-11-28T13:01:51.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24d2ffd700 1 -- 172.21.15.111:0/4016485878 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d4020070 con 0x7f24dc154b60 2023-11-28T13:01:51.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e44c5700 1 -- 172.21.15.111:0/4016485878 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24dc06b9c0 con 0x7f24dc154b60 2023-11-28T13:01:51.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24d2ffd700 1 -- 172.21.15.111:0/4016485878 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d4004280 con 0x7f24dc154b60 2023-11-28T13:01:51.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.066+0000 7f24e44c5700 1 -- 172.21.15.111:0/4016485878 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24dc06b6e0 con 0x7f24dc154b60 2023-11-28T13:01:51.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.067+0000 7f24e44c5700 1 -- 172.21.15.111:0/4016485878 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24dc0c9b10 con 0x7f24dc154b60 2023-11-28T13:01:51.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.068+0000 7f24d2ffd700 1 -- 172.21.15.111:0/4016485878 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f24d40043e0 con 0x7f24dc154b60 2023-11-28T13:01:51.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.068+0000 7f24d2ffd700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24c00682f0 0x7f24c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.068+0000 7f24d2ffd700 1 -- 172.21.15.111:0/4016485878 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f24d4022070 con 0x7f24dc154b60 2023-11-28T13:01:51.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.068+0000 7f24e2261700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24c00682f0 0x7f24c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.068+0000 7f24e2261700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24c00682f0 0x7f24c006a7a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f24cc000f60 tx=0x7f24cc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:51.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.072+0000 7f24d2ffd700 1 -- 172.21.15.111:0/4016485878 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f24d40548a0 con 0x7f24dc154b60 2023-11-28T13:01:51.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.251+0000 7f24e44c5700 1 -- 172.21.15.111:0/4016485878 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f24dc067170 con 0x7f24dc154b60 2023-11-28T13:01:51.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.251+0000 7f24d2ffd700 1 -- 172.21.15.111:0/4016485878 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v63) v1 ==== 92+0+10 (secure 0 0 0) 0x7f24d4057ec0 con 0x7f24dc154b60 2023-11-28T13:01:51.253 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T13:01:51.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.254+0000 7f24e44c5700 1 -- 172.21.15.111:0/4016485878 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24c00682f0 msgr2=0x7f24c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.254+0000 7f24e44c5700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24c00682f0 0x7f24c006a7a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f24cc000f60 tx=0x7f24cc009040 comp rx=0 tx=0).stop 2023-11-28T13:01:51.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.254+0000 7f24e44c5700 1 -- 172.21.15.111:0/4016485878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc154b60 msgr2=0x7f24dc0ae3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.254+0000 7f24e44c5700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc154b60 0x7f24dc0ae3a0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f24d400b680 tx=0x7f24d4004ad0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.254+0000 7f24e44c5700 1 -- 172.21.15.111:0/4016485878 shutdown_connections 2023-11-28T13:01:51.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.254+0000 7f24e44c5700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc154b60 0x7f24dc0ae3a0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.254+0000 7f24e44c5700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24c00682f0 0x7f24c006a7a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.254+0000 7f24e44c5700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24dc06a7b0 0x7f24dc0abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.254+0000 7f24e44c5700 1 --2- 172.21.15.111:0/4016485878 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24dc0cb830 0x7f24dc0c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.254+0000 7f24e44c5700 1 -- 172.21.15.111:0/4016485878 >> 172.21.15.111:0/4016485878 conn(0x7f24dc0cf2e0 msgr2=0x7f24dc06e990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:51.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.254+0000 7f24e44c5700 1 -- 172.21.15.111:0/4016485878 shutdown_connections 2023-11-28T13:01:51.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.254+0000 7f24e44c5700 1 -- 172.21.15.111:0/4016485878 wait complete. 2023-11-28T13:01:51.267 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T13:01:51.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.386+0000 7fe490b50700 1 Processor -- start 2023-11-28T13:01:51.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.386+0000 7fe490b50700 1 -- start start 2023-11-28T13:01:51.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.387+0000 7fe490b50700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe48c0cef60 0x7fe48c0ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.387+0000 7fe490b50700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c0cc390 con 0x7fe48c0b0160 2023-11-28T13:01:51.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.387+0000 7fe490b50700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c0cc670 con 0x7fe48c15a5d0 2023-11-28T13:01:51.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.387+0000 7fe490b50700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48c0cc1b0 con 0x7fe48c0cef60 2023-11-28T13:01:51.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.387+0000 7fe48ad9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe48c0b0160 0x7fe48c0adde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37912/0 (socket says 172.21.15.111:37912) 2023-11-28T13:01:51.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.387+0000 7fe48ad9d700 1 -- 172.21.15.111:0/1660717967 learned_addr learned my addr 172.21.15.111:0/1660717967 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:51.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.387+0000 7fe489d9b700 1 --2- 172.21.15.111:0/1660717967 >> v2:172.21.15.149:3300/0 conn(0x7fe48c0cef60 0x7fe48c0ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638974775 0 0) 0x7fe48c0cc390 con 0x7fe48c0b0160 2023-11-28T13:01:51.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4740035b0 con 0x7fe48c0b0160 2023-11-28T13:01:51.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638469723 0 0) 0x7fe48c0cc670 con 0x7fe48c15a5d0 2023-11-28T13:01:51.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe48c0cc390 con 0x7fe48c15a5d0 2023-11-28T13:01:51.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 187666725 0 0) 0x7fe4740035b0 con 0x7fe48c0b0160 2023-11-28T13:01:51.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 >> v2:172.21.15.149:3300/0 conn(0x7fe48c0cef60 msgr2=0x7fe48c0ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:51.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 --2- 172.21.15.111:0/1660717967 >> v2:172.21.15.149:3300/0 conn(0x7fe48c0cef60 0x7fe48c0ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 >> v1:172.21.15.149:6789/0 conn(0x7fe48c15a5d0 legacy=0x7fe48c0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c0cc950 con 0x7fe48c0b0160 2023-11-28T13:01:51.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fe4840019f0 con 0x7fe48c0b0160 2023-11-28T13:01:51.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 >> v1:172.21.15.111:6789/0 conn(0x7fe48c0b0160 legacy=0x7fe48c0adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 --2- 172.21.15.111:0/1660717967 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe474003b90 0x7fe474003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 --2- 172.21.15.111:0/1660717967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe474004850 0x7fe474004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 --2- 172.21.15.111:0/1660717967 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe474005200 0x7fe47400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0cc1b0 con 0x7fe474004850 2023-11-28T13:01:51.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47400c110 con 0x7fe474003b90 2023-11-28T13:01:51.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47400c2d0 con 0x7fe474005200 2023-11-28T13:01:51.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe489d9b700 1 --2- 172.21.15.111:0/1660717967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe474004850 0x7fe474004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48a59c700 1 --2- 172.21.15.111:0/1660717967 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe474003b90 0x7fe474003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48ad9d700 1 --2- 172.21.15.111:0/1660717967 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe474005200 0x7fe47400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe489d9b700 1 -- 172.21.15.111:0/1660717967 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe474005200 msgr2=0x7fe47400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:51.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe489d9b700 1 --2- 172.21.15.111:0/1660717967 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe474005200 0x7fe47400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe489d9b700 1 -- 172.21.15.111:0/1660717967 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe474003b90 msgr2=0x7fe474003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:51.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe489d9b700 1 --2- 172.21.15.111:0/1660717967 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe474003b90 0x7fe474003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48ad9d700 1 -- 172.21.15.111:0/1660717967 reap_dead start 2023-11-28T13:01:51.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe489d9b700 1 -- 172.21.15.111:0/1660717967 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4740035b0 con 0x7fe474004850 2023-11-28T13:01:51.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe489d9b700 1 --2- 172.21.15.111:0/1660717967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe474004850 0x7fe474004cc0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0cf350 tx=0x7fe474005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d1ef279a1550c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:51.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.388+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00e6c0 con 0x7fe474004850 2023-11-28T13:01:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.389+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c00e820 con 0x7fe474004850 2023-11-28T13:01:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.389+0000 7fe48959a700 1 -- 172.21.15.111:0/1660717967 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00ebb0 con 0x7fe474004850 2023-11-28T13:01:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.389+0000 7fe490b50700 1 -- 172.21.15.111:0/1660717967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe474004850 msgr2=0x7fe474004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.389+0000 7fe490b50700 1 --2- 172.21.15.111:0/1660717967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe474004850 0x7fe474004cc0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0cf350 tx=0x7fe474005880 comp rx=0 tx=0).stop 2023-11-28T13:01:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.389+0000 7fe490b50700 1 -- 172.21.15.111:0/1660717967 shutdown_connections 2023-11-28T13:01:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.389+0000 7fe490b50700 1 --2- 172.21.15.111:0/1660717967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe474004850 0x7fe474004cc0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.389+0000 7fe490b50700 1 -- 172.21.15.111:0/1660717967 >> 172.21.15.111:0/1660717967 conn(0x7fe48c09a040 msgr2=0x7fe48c0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.389+0000 7fe490b50700 1 -- 172.21.15.111:0/1660717967 shutdown_connections 2023-11-28T13:01:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.389+0000 7fe490b50700 1 -- 172.21.15.111:0/1660717967 wait complete. 2023-11-28T13:01:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.390+0000 7fe490b50700 1 Processor -- start 2023-11-28T13:01:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.391+0000 7fe490b50700 1 -- start start 2023-11-28T13:01:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.391+0000 7fe490b50700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe474004850 0x7fe48c0ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.391+0000 7fe490b50700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe48c0c4750 0x7fe48c0c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.391+0000 7fe490b50700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe48c0c67f0 0x7fe48c0c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.391+0000 7fe490b50700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c06c6a0 con 0x7fe474004850 2023-11-28T13:01:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.391+0000 7fe490b50700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c06bd90 con 0x7fe48c0c67f0 2023-11-28T13:01:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.391+0000 7fe490b50700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0c71a0 con 0x7fe48c0c4750 2023-11-28T13:01:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.391+0000 7fe48a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe474004850 0x7fe48c0ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.391+0000 7fe48a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe474004850 0x7fe48c0ca640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37142/0 (socket says 172.21.15.111:37142) 2023-11-28T13:01:51.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.391+0000 7fe48a59c700 1 -- 172.21.15.111:0/3810731722 learned_addr learned my addr 172.21.15.111:0/3810731722 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:51.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.392+0000 7fe48a59c700 1 -- 172.21.15.111:0/3810731722 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe48c0c4750 msgr2=0x7fe48c0c62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.392+0000 7fe48ad9d700 1 --2- 172.21.15.111:0/3810731722 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe48c0c67f0 0x7fe48c0c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.392+0000 7fe48a59c700 1 --2- 172.21.15.111:0/3810731722 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe48c0c4750 0x7fe48c0c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.392+0000 7fe48a59c700 1 -- 172.21.15.111:0/3810731722 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe48c0c67f0 msgr2=0x7fe48c0c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.392+0000 7fe48a59c700 1 --2- 172.21.15.111:0/3810731722 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe48c0c67f0 0x7fe48c0c6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.392+0000 7fe48a59c700 1 -- 172.21.15.111:0/3810731722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c0ccf10 con 0x7fe474004850 2023-11-28T13:01:51.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.392+0000 7fe48a59c700 1 --2- 172.21.15.111:0/3810731722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe474004850 0x7fe48c0ca640 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe48000f7d0 tx=0x7fe48000fb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:51.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.392+0000 7fe47affd700 1 -- 172.21.15.111:0/3810731722 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4800052e0 con 0x7fe474004850 2023-11-28T13:01:51.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.392+0000 7fe47affd700 1 -- 172.21.15.111:0/3810731722 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe480005460 con 0x7fe474004850 2023-11-28T13:01:51.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.392+0000 7fe490b50700 1 -- 172.21.15.111:0/3810731722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48c0ccc30 con 0x7fe474004850 2023-11-28T13:01:51.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.392+0000 7fe47affd700 1 -- 172.21.15.111:0/3810731722 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe480003910 con 0x7fe474004850 2023-11-28T13:01:51.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.392+0000 7fe490b50700 1 -- 172.21.15.111:0/3810731722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48c15a9a0 con 0x7fe474004850 2023-11-28T13:01:51.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.393+0000 7fe490b50700 1 -- 172.21.15.111:0/3810731722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48c067170 con 0x7fe474004850 2023-11-28T13:01:51.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.393+0000 7fe47affd700 1 -- 172.21.15.111:0/3810731722 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe480003a70 con 0x7fe474004850 2023-11-28T13:01:51.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.393+0000 7fe47affd700 1 --2- 172.21.15.111:0/3810731722 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe46c068320 0x7fe46c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.394+0000 7fe47affd700 1 -- 172.21.15.111:0/3810731722 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5530+0+0 (secure 0 0 0) 0x7fe480085950 con 0x7fe474004850 2023-11-28T13:01:51.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.394+0000 7fe489d9b700 1 --2- 172.21.15.111:0/3810731722 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe46c068320 0x7fe46c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.394+0000 7fe489d9b700 1 --2- 172.21.15.111:0/3810731722 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe46c068320 0x7fe46c06a7d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0c7850 tx=0x7fe47c013000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:51.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.398+0000 7fe47affd700 1 -- 172.21.15.111:0/3810731722 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe4800523c0 con 0x7fe474004850 2023-11-28T13:01:51.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.582+0000 7fe490b50700 1 -- 172.21.15.111:0/3810731722 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe48c0cb500 con 0x7fe474004850 2023-11-28T13:01:51.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.582+0000 7fe47affd700 1 -- 172.21.15.111:0/3810731722 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v63) v1 ==== 84+0+126 (secure 0 0 0) 0x7fe4800559e0 con 0x7fe474004850 2023-11-28T13:01:51.584 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:51.584 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-11-28T13:01:51.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:51.584 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-11-28T13:01:51.584 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-11-28T13:01:51.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:01:51.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:01:51.584 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 2, 2023-11-28T13:01:51.584 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": "rbd" 2023-11-28T13:01:51.584 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:01:51.584 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-11-28T13:01:51.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.584+0000 7fe490b50700 1 -- 172.21.15.111:0/3810731722 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe46c068320 msgr2=0x7fe46c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.584+0000 7fe490b50700 1 --2- 172.21.15.111:0/3810731722 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe46c068320 0x7fe46c06a7d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0c7850 tx=0x7fe47c013000 comp rx=0 tx=0).stop 2023-11-28T13:01:51.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.584+0000 7fe490b50700 1 -- 172.21.15.111:0/3810731722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe474004850 msgr2=0x7fe48c0ca640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.584+0000 7fe490b50700 1 --2- 172.21.15.111:0/3810731722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe474004850 0x7fe48c0ca640 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe48000f7d0 tx=0x7fe48000fb90 comp rx=0 tx=0).stop 2023-11-28T13:01:51.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.584+0000 7fe490b50700 1 -- 172.21.15.111:0/3810731722 shutdown_connections 2023-11-28T13:01:51.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.584+0000 7fe490b50700 1 --2- 172.21.15.111:0/3810731722 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe46c068320 0x7fe46c06a7d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.584+0000 7fe490b50700 1 --2- 172.21.15.111:0/3810731722 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe48c0c67f0 0x7fe48c0c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.584+0000 7fe490b50700 1 --2- 172.21.15.111:0/3810731722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe474004850 0x7fe48c0ca640 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.584+0000 7fe490b50700 1 --2- 172.21.15.111:0/3810731722 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe48c0c4750 0x7fe48c0c62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.584+0000 7fe490b50700 1 -- 172.21.15.111:0/3810731722 >> 172.21.15.111:0/3810731722 conn(0x7fe48c09a040 msgr2=0x7fe48c0af670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:51.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.584+0000 7fe490b50700 1 -- 172.21.15.111:0/3810731722 shutdown_connections 2023-11-28T13:01:51.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.584+0000 7fe490b50700 1 -- 172.21.15.111:0/3810731722 wait complete. 2023-11-28T13:01:51.597 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T13:01:51.598 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T13:01:51.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.720+0000 7f4dfbd48700 1 Processor -- start 2023-11-28T13:01:51.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.720+0000 7f4dfbd48700 1 -- start start 2023-11-28T13:01:51.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.721+0000 7f4dfbd48700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4df415a3c0 0x7f4df406f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.721+0000 7f4dfbd48700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df406b490 con 0x7f4df40cb790 2023-11-28T13:01:51.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.721+0000 7f4dfbd48700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df406f680 con 0x7f4df406a980 2023-11-28T13:01:51.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.721+0000 7f4dfbd48700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df415a790 con 0x7f4df415a3c0 2023-11-28T13:01:51.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.721+0000 7f4df9ae4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4df415a3c0 0x7f4df406f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.721+0000 7f4df92e3700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4df40cb790 0x7f4df406a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37930/0 (socket says 172.21.15.111:37930) 2023-11-28T13:01:51.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.721+0000 7f4df92e3700 1 -- 172.21.15.111:0/1464453651 learned_addr learned my addr 172.21.15.111:0/1464453651 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:51.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.721+0000 7f4df9ae4700 1 -- 172.21.15.111:0/1464453651 >> v1:172.21.15.149:6789/0 conn(0x7f4df406a980 legacy=0x7f4df40712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.721+0000 7f4df9ae4700 1 -- 172.21.15.111:0/1464453651 >> v1:172.21.15.111:6789/0 conn(0x7f4df40cb790 legacy=0x7f4df406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.721+0000 7f4df9ae4700 1 -- 172.21.15.111:0/1464453651 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df406b6e0 con 0x7f4df415a3c0 2023-11-28T13:01:51.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.722+0000 7f4df9ae4700 1 --2- 172.21.15.111:0/1464453651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df415a3c0 0x7f4df406f140 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009840 tx=0x7f4df0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6ab8441d24b121 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:51.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.722+0000 7f4df8ae2700 1 -- 172.21.15.111:0/1464453651 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0011830 con 0x7f4df415a3c0 2023-11-28T13:01:51.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.722+0000 7f4df8ae2700 1 -- 172.21.15.111:0/1464453651 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df0011990 con 0x7f4df415a3c0 2023-11-28T13:01:51.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.722+0000 7f4df8ae2700 1 -- 172.21.15.111:0/1464453651 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0011d20 con 0x7f4df415a3c0 2023-11-28T13:01:51.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.722+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1464453651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df415a3c0 msgr2=0x7f4df406f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.722+0000 7f4dfbd48700 1 --2- 172.21.15.111:0/1464453651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df415a3c0 0x7f4df406f140 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009840 tx=0x7f4df0009b50 comp rx=0 tx=0).stop 2023-11-28T13:01:51.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.722+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1464453651 shutdown_connections 2023-11-28T13:01:51.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.722+0000 7f4dfbd48700 1 --2- 172.21.15.111:0/1464453651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df415a3c0 0x7f4df406f140 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.722+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1464453651 >> 172.21.15.111:0/1464453651 conn(0x7f4df40b0db0 msgr2=0x7f4df40b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:51.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.722+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1464453651 shutdown_connections 2023-11-28T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.722+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1464453651 wait complete. 2023-11-28T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.723+0000 7f4dfbd48700 1 Processor -- start 2023-11-28T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.724+0000 7f4dfbd48700 1 -- start start 2023-11-28T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.724+0000 7f4dfbd48700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4df406a980 0x7f4df4159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.724+0000 7f4dfbd48700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4df40cb790 0x7f4df41f01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.724+0000 7f4dfbd48700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df415a3c0 0x7f4df40c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.724+0000 7f4dfbd48700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df41f06f0 con 0x7f4df415a3c0 2023-11-28T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.724+0000 7f4dfbd48700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df41f34e0 con 0x7f4df406a980 2023-11-28T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.724+0000 7f4dfbd48700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df415ae30 con 0x7f4df40cb790 2023-11-28T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.724+0000 7f4dfa2e5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df415a3c0 0x7f4df40c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.724+0000 7f4dfa2e5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df415a3c0 0x7f4df40c4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37160/0 (socket says 172.21.15.111:37160) 2023-11-28T13:01:51.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.724+0000 7f4dfa2e5700 1 -- 172.21.15.111:0/1360994893 learned_addr learned my addr 172.21.15.111:0/1360994893 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:51.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.725+0000 7f4df9ae4700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4df406a980 0x7f4df4159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.725+0000 7f4df92e3700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4df40cb790 0x7f4df41f01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.725+0000 7f4dfa2e5700 1 -- 172.21.15.111:0/1360994893 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4df40cb790 msgr2=0x7f4df41f01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.725+0000 7f4dfa2e5700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4df40cb790 0x7f4df41f01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.725+0000 7f4dfa2e5700 1 -- 172.21.15.111:0/1360994893 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4df406a980 msgr2=0x7f4df4159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:51.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.725+0000 7f4dfa2e5700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4df406a980 0x7f4df4159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:51.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.725+0000 7f4dfa2e5700 1 -- 172.21.15.111:0/1360994893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df00094a0 con 0x7f4df415a3c0 2023-11-28T13:01:51.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.725+0000 7f4dfa2e5700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df415a3c0 0x7f4df40c4850 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00cbf0 tx=0x7f4dec00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:51.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.725+0000 7f4dea7fc700 1 -- 172.21.15.111:0/1360994893 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec003c00 con 0x7f4df415a3c0 2023-11-28T13:01:51.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.725+0000 7f4dea7fc700 1 -- 172.21.15.111:0/1360994893 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dec003d60 con 0x7f4df415a3c0 2023-11-28T13:01:51.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.725+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1360994893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df40c9ab0 con 0x7f4df415a3c0 2023-11-28T13:01:51.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.725+0000 7f4dea7fc700 1 -- 172.21.15.111:0/1360994893 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec00aaa0 con 0x7f4df415a3c0 2023-11-28T13:01:51.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.725+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1360994893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df406b9c0 con 0x7f4df415a3c0 2023-11-28T13:01:51.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.726+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1360994893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df40c9d90 con 0x7f4df415a3c0 2023-11-28T13:01:51.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.726+0000 7f4dea7fc700 1 -- 172.21.15.111:0/1360994893 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4dec010050 con 0x7f4df415a3c0 2023-11-28T13:01:51.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.727+0000 7f4dea7fc700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4dd80682f0 0x7f4dd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:51.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.727+0000 7f4dea7fc700 1 -- 172.21.15.111:0/1360994893 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5530+0+0 (secure 0 0 0) 0x7f4dec0858c0 con 0x7f4df415a3c0 2023-11-28T13:01:51.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.727+0000 7f4df9ae4700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4dd80682f0 0x7f4dd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:51.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.727+0000 7f4df9ae4700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4dd80682f0 0x7f4dd806a7a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4df00090b0 tx=0x7f4df001c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:51.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.731+0000 7f4dea7fc700 1 -- 172.21.15.111:0/1360994893 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4dec052330 con 0x7f4df415a3c0 2023-11-28T13:01:51.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:51.915+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1360994893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4df4067170 con 0x7f4df415a3c0 2023-11-28T13:01:52.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:52.856+0000 7f4dea7fc700 1 -- 172.21.15.111:0/1360994893 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v64) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4dec055950 con 0x7f4df415a3c0 2023-11-28T13:01:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:52.859+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1360994893 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4dd80682f0 msgr2=0x7f4dd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:52.859+0000 7f4dfbd48700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4dd80682f0 0x7f4dd806a7a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4df00090b0 tx=0x7f4df001c040 comp rx=0 tx=0).stop 2023-11-28T13:01:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:52.859+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1360994893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df415a3c0 msgr2=0x7f4df40c4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:52.859+0000 7f4dfbd48700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df415a3c0 0x7f4df40c4850 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00cbf0 tx=0x7f4dec00cf00 comp rx=0 tx=0).stop 2023-11-28T13:01:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:52.860+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1360994893 shutdown_connections 2023-11-28T13:01:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:52.860+0000 7f4dfbd48700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df415a3c0 0x7f4df40c4850 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:52.860+0000 7f4dfbd48700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4dd80682f0 0x7f4dd806a7a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:52.860+0000 7f4dfbd48700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4df406a980 0x7f4df4159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:52.860+0000 7f4dfbd48700 1 --2- 172.21.15.111:0/1360994893 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4df40cb790 0x7f4df41f01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:52.860+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1360994893 >> 172.21.15.111:0/1360994893 conn(0x7f4df40b0db0 msgr2=0x7f4df40b0410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:52.860+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1360994893 shutdown_connections 2023-11-28T13:01:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:52.860+0000 7f4dfbd48700 1 -- 172.21.15.111:0/1360994893 wait complete. 2023-11-28T13:01:52.869 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-11-28T13:01:52.879 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T13:01:53.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.009+0000 7f4c97026700 1 Processor -- start 2023-11-28T13:01:53.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.009+0000 7f4c97026700 1 -- start start 2023-11-28T13:01:53.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.010+0000 7f4c97026700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4c9015d820 0x7f4c90160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:53.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.010+0000 7f4c97026700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c900b1d50 con 0x7f4c900cde60 2023-11-28T13:01:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.010+0000 7f4c97026700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9015cd40 con 0x7f4c9006bff0 2023-11-28T13:01:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.010+0000 7f4c97026700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c9015d020 con 0x7f4c9015d820 2023-11-28T13:01:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.010+0000 7f4c955c3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4c9015d820 0x7f4c90160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.010+0000 7f4c8ffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4c900cde60 0x7f4c9006a700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37944/0 (socket says 172.21.15.111:37944) 2023-11-28T13:01:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.010+0000 7f4c8ffff700 1 -- 172.21.15.111:0/1269458446 learned_addr learned my addr 172.21.15.111:0/1269458446 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.010+0000 7f4c955c3700 1 -- 172.21.15.111:0/1269458446 >> v1:172.21.15.149:6789/0 conn(0x7f4c9006bff0 legacy=0x7f4c900cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.010+0000 7f4c955c3700 1 -- 172.21.15.111:0/1269458446 >> v1:172.21.15.111:6789/0 conn(0x7f4c900cde60 legacy=0x7f4c9006a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.010+0000 7f4c955c3700 1 -- 172.21.15.111:0/1269458446 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9015d300 con 0x7f4c9015d820 2023-11-28T13:01:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.010+0000 7f4c955c3700 1 --2- 172.21.15.111:0/1269458446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c9015d820 0x7f4c90160ca0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009d90 tx=0x7f4c8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2891aebc481e1a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.010+0000 7f4c8f7fe700 1 -- 172.21.15.111:0/1269458446 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8800e070 con 0x7f4c9015d820 2023-11-28T13:01:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.011+0000 7f4c8f7fe700 1 -- 172.21.15.111:0/1269458446 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8800bbd0 con 0x7f4c9015d820 2023-11-28T13:01:53.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.011+0000 7f4c8f7fe700 1 -- 172.21.15.111:0/1269458446 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88012430 con 0x7f4c9015d820 2023-11-28T13:01:53.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.011+0000 7f4c97026700 1 -- 172.21.15.111:0/1269458446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c9015d820 msgr2=0x7f4c90160ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:53.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.011+0000 7f4c97026700 1 --2- 172.21.15.111:0/1269458446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c9015d820 0x7f4c90160ca0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009d90 tx=0x7f4c8800b330 comp rx=0 tx=0).stop 2023-11-28T13:01:53.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.011+0000 7f4c97026700 1 -- 172.21.15.111:0/1269458446 shutdown_connections 2023-11-28T13:01:53.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.011+0000 7f4c97026700 1 --2- 172.21.15.111:0/1269458446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c9015d820 0x7f4c90160ca0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:53.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.011+0000 7f4c97026700 1 -- 172.21.15.111:0/1269458446 >> 172.21.15.111:0/1269458446 conn(0x7f4c9006e790 msgr2=0x7f4c900cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:53.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.011+0000 7f4c97026700 1 -- 172.21.15.111:0/1269458446 shutdown_connections 2023-11-28T13:01:53.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.011+0000 7f4c97026700 1 -- 172.21.15.111:0/1269458446 wait complete. 2023-11-28T13:01:53.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.012+0000 7f4c97026700 1 Processor -- start 2023-11-28T13:01:53.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.012+0000 7f4c97026700 1 -- start start 2023-11-28T13:01:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.013+0000 7f4c97026700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4c9006bff0 0x7f4c90155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.013+0000 7f4c97026700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4c900cde60 0x7f4c9015af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.013+0000 7f4c97026700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c9015d820 0x7f4c9015b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.013+0000 7f4c97026700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c90155f40 con 0x7f4c9015d820 2023-11-28T13:01:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.013+0000 7f4c97026700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c900b2dd0 con 0x7f4c900cde60 2023-11-28T13:01:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.013+0000 7f4c97026700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c900ad010 con 0x7f4c9006bff0 2023-11-28T13:01:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.013+0000 7f4c955c3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c9015d820 0x7f4c9015b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.013+0000 7f4c955c3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c9015d820 0x7f4c9015b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37174/0 (socket says 172.21.15.111:37174) 2023-11-28T13:01:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.013+0000 7f4c955c3700 1 -- 172.21.15.111:0/1126487642 learned_addr learned my addr 172.21.15.111:0/1126487642 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.013+0000 7f4c8ffff700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4c900cde60 0x7f4c9015af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.013+0000 7f4c94dc2700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4c9006bff0 0x7f4c90155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:53.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.014+0000 7f4c955c3700 1 -- 172.21.15.111:0/1126487642 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4c9006bff0 msgr2=0x7f4c90155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.014+0000 7f4c955c3700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4c9006bff0 0x7f4c90155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.014+0000 7f4c955c3700 1 -- 172.21.15.111:0/1126487642 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4c900cde60 msgr2=0x7f4c9015af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.014+0000 7f4c955c3700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4c900cde60 0x7f4c9015af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.014+0000 7f4c955c3700 1 -- 172.21.15.111:0/1126487642 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c88009a70 con 0x7f4c9015d820 2023-11-28T13:01:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.014+0000 7f4c8ffff700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4c900cde60 0x7f4c9015af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.014+0000 7f4c955c3700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c9015d820 0x7f4c9015b490 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f4c88014040 tx=0x7f4c88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.014+0000 7f4c8d7fa700 1 -- 172.21.15.111:0/1126487642 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c880098f0 con 0x7f4c9015d820 2023-11-28T13:01:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.014+0000 7f4c8d7fa700 1 -- 172.21.15.111:0/1126487642 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c88020070 con 0x7f4c9015d820 2023-11-28T13:01:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.014+0000 7f4c97026700 1 -- 172.21.15.111:0/1126487642 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c900b1600 con 0x7f4c9015d820 2023-11-28T13:01:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.014+0000 7f4c8d7fa700 1 -- 172.21.15.111:0/1126487642 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88004280 con 0x7f4c9015d820 2023-11-28T13:01:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.014+0000 7f4c97026700 1 -- 172.21.15.111:0/1126487642 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c900b2030 con 0x7f4c9015d820 2023-11-28T13:01:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.015+0000 7f4c97026700 1 -- 172.21.15.111:0/1126487642 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c900b1790 con 0x7f4c9015d820 2023-11-28T13:01:53.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.015+0000 7f4c8d7fa700 1 -- 172.21.15.111:0/1126487642 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4c8802e070 con 0x7f4c9015d820 2023-11-28T13:01:53.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.016+0000 7f4c8d7fa700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4c740682f0 0x7f4c7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:53.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.016+0000 7f4c8d7fa700 1 -- 172.21.15.111:0/1126487642 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5900+0+0 (secure 0 0 0) 0x7f4c88022070 con 0x7f4c9015d820 2023-11-28T13:01:53.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.016+0000 7f4c94dc2700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4c740682f0 0x7f4c7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:53.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.016+0000 7f4c94dc2700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4c740682f0 0x7f4c7406a7a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4c84000f60 tx=0x7f4c84009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:53.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.020+0000 7f4c8d7fa700 1 -- 172.21.15.111:0/1126487642 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4c88054410 con 0x7f4c9015d820 2023-11-28T13:01:53.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.200+0000 7f4c97026700 1 -- 172.21.15.111:0/1126487642 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4c90067170 con 0x7f4c9015d820 2023-11-28T13:01:53.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.863+0000 7f4c8d7fa700 1 -- 172.21.15.111:0/1126487642 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v65) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4c8802c020 con 0x7f4c9015d820 2023-11-28T13:01:53.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.867+0000 7f4c97026700 1 -- 172.21.15.111:0/1126487642 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4c740682f0 msgr2=0x7f4c7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:53.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.867+0000 7f4c97026700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4c740682f0 0x7f4c7406a7a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4c84000f60 tx=0x7f4c84009040 comp rx=0 tx=0).stop 2023-11-28T13:01:53.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.867+0000 7f4c97026700 1 -- 172.21.15.111:0/1126487642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c9015d820 msgr2=0x7f4c9015b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:53.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.867+0000 7f4c97026700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c9015d820 0x7f4c9015b490 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f4c88014040 tx=0x7f4c88004ad0 comp rx=0 tx=0).stop 2023-11-28T13:01:53.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.868+0000 7f4c97026700 1 -- 172.21.15.111:0/1126487642 shutdown_connections 2023-11-28T13:01:53.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.868+0000 7f4c97026700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c9015d820 0x7f4c9015b490 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:53.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.868+0000 7f4c97026700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4c9006bff0 0x7f4c90155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:53.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.868+0000 7f4c97026700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4c740682f0 0x7f4c7406a7a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:53.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.868+0000 7f4c97026700 1 --2- 172.21.15.111:0/1126487642 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4c900cde60 0x7f4c9015af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:53.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.868+0000 7f4c97026700 1 -- 172.21.15.111:0/1126487642 >> 172.21.15.111:0/1126487642 conn(0x7f4c9006e790 msgr2=0x7f4c9006af90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:53.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.868+0000 7f4c97026700 1 -- 172.21.15.111:0/1126487642 shutdown_connections 2023-11-28T13:01:53.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:53.869+0000 7f4c97026700 1 -- 172.21.15.111:0/1126487642 wait complete. 2023-11-28T13:01:53.873 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-11-28T13:01:53.894 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T13:01:54.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.017+0000 7fec457cd700 1 Processor -- start 2023-11-28T13:01:54.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.017+0000 7fec457cd700 1 -- start start 2023-11-28T13:01:54.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.018+0000 7fec457cd700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fec4006e730 0x7fec4006eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.018+0000 7fec457cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec400cd540 con 0x7fec4015d270 2023-11-28T13:01:54.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.018+0000 7fec457cd700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec401597b0 con 0x7fec400cb240 2023-11-28T13:01:54.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.018+0000 7fec457cd700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4006f060 con 0x7fec4006e730 2023-11-28T13:01:54.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.018+0000 7fec3e7fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fec4015d270 0x7fec400cd9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37956/0 (socket says 172.21.15.111:37956) 2023-11-28T13:01:54.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.018+0000 7fec3e7fc700 1 -- 172.21.15.111:0/1232381285 learned_addr learned my addr 172.21.15.111:0/1232381285 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:54.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3f7fe700 1 --2- 172.21.15.111:0/1232381285 >> v2:172.21.15.149:3300/0 conn(0x7fec4006e730 0x7fec4006eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28593780 0 0) 0x7fec400cd540 con 0x7fec4015d270 2023-11-28T13:01:54.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec280046c0 con 0x7fec4015d270 2023-11-28T13:01:54.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 392951761 0 0) 0x7fec280046c0 con 0x7fec4015d270 2023-11-28T13:01:54.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 >> v2:172.21.15.149:3300/0 conn(0x7fec4006e730 msgr2=0x7fec4006eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:54.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 --2- 172.21.15.111:0/1232381285 >> v2:172.21.15.149:3300/0 conn(0x7fec4006e730 0x7fec4006eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 >> v1:172.21.15.149:6789/0 conn(0x7fec400cb240 legacy=0x7fec401590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec400ce0e0 con 0x7fec4015d270 2023-11-28T13:01:54.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fec30001670 con 0x7fec4015d270 2023-11-28T13:01:54.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 >> v1:172.21.15.111:6789/0 conn(0x7fec4015d270 legacy=0x7fec400cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 --2- 172.21.15.111:0/1232381285 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fec28004b10 0x7fec28004f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 --2- 172.21.15.111:0/1232381285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28005960 0x7fec28005dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 --2- 172.21.15.111:0/1232381285 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fec28006310 0x7fec280035b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4006f060 con 0x7fec28005960 2023-11-28T13:01:54.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec28003d00 con 0x7fec28006310 2023-11-28T13:01:54.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec28003e70 con 0x7fec28004b10 2023-11-28T13:01:54.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3e7fc700 1 --2- 172.21.15.111:0/1232381285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28005960 0x7fec28005dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3effd700 1 --2- 172.21.15.111:0/1232381285 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fec28004b10 0x7fec28004f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3f7fe700 1 --2- 172.21.15.111:0/1232381285 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fec28006310 0x7fec280035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3e7fc700 1 -- 172.21.15.111:0/1232381285 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fec28004b10 msgr2=0x7fec28004f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:54.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3e7fc700 1 --2- 172.21.15.111:0/1232381285 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fec28004b10 0x7fec28004f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3e7fc700 1 -- 172.21.15.111:0/1232381285 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fec28006310 msgr2=0x7fec280035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:54.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3e7fc700 1 --2- 172.21.15.111:0/1232381285 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fec28006310 0x7fec280035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3e7fc700 1 -- 172.21.15.111:0/1232381285 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec280046c0 con 0x7fec28005960 2023-11-28T13:01:54.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.019+0000 7fec3f7fe700 1 -- 172.21.15.111:0/1232381285 reap_dead start 2023-11-28T13:01:54.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.020+0000 7fec3e7fc700 1 --2- 172.21.15.111:0/1232381285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28005960 0x7fec28005dd0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fec3000c240 tx=0x7fec28006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=668567c966adce27 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:54.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.020+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30007c10 con 0x7fec28005960 2023-11-28T13:01:54.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.020+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3000cd80 con 0x7fec28005960 2023-11-28T13:01:54.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.020+0000 7fec3dffb700 1 -- 172.21.15.111:0/1232381285 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30005ba0 con 0x7fec28005960 2023-11-28T13:01:54.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.020+0000 7fec457cd700 1 -- 172.21.15.111:0/1232381285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28005960 msgr2=0x7fec28005dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.020+0000 7fec457cd700 1 --2- 172.21.15.111:0/1232381285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28005960 0x7fec28005dd0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fec3000c240 tx=0x7fec28006990 comp rx=0 tx=0).stop 2023-11-28T13:01:54.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.020+0000 7fec457cd700 1 -- 172.21.15.111:0/1232381285 shutdown_connections 2023-11-28T13:01:54.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.020+0000 7fec457cd700 1 --2- 172.21.15.111:0/1232381285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28005960 0x7fec28005dd0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.020+0000 7fec457cd700 1 -- 172.21.15.111:0/1232381285 >> 172.21.15.111:0/1232381285 conn(0x7fec400b0db0 msgr2=0x7fec400b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:54.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.020+0000 7fec457cd700 1 -- 172.21.15.111:0/1232381285 shutdown_connections 2023-11-28T13:01:54.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.020+0000 7fec457cd700 1 -- 172.21.15.111:0/1232381285 wait complete. 2023-11-28T13:01:54.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.021+0000 7fec457cd700 1 Processor -- start 2023-11-28T13:01:54.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.022+0000 7fec457cd700 1 -- start start 2023-11-28T13:01:54.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.022+0000 7fec457cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28005960 0x7fec40154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.022+0000 7fec457cd700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fec401562b0 0x7fec4015c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.022+0000 7fec457cd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fec4014e670 0x7fec4014eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.022+0000 7fec457cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec40155790 con 0x7fec28005960 2023-11-28T13:01:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.022+0000 7fec457cd700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec40154c10 con 0x7fec401562b0 2023-11-28T13:01:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.022+0000 7fec457cd700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec40154ef0 con 0x7fec4014e670 2023-11-28T13:01:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.022+0000 7fec3effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28005960 0x7fec40154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.022+0000 7fec3effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28005960 0x7fec40154630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37190/0 (socket says 172.21.15.111:37190) 2023-11-28T13:01:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.022+0000 7fec3effd700 1 -- 172.21.15.111:0/3501078703 learned_addr learned my addr 172.21.15.111:0/3501078703 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.023+0000 7fec3effd700 1 -- 172.21.15.111:0/3501078703 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fec4014e670 msgr2=0x7fec4014eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.023+0000 7fec3e7fc700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fec401562b0 0x7fec4015c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.023+0000 7fec3f7fe700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fec4014e670 0x7fec4014eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.023+0000 7fec3effd700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fec4014e670 0x7fec4014eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.023+0000 7fec3effd700 1 -- 172.21.15.111:0/3501078703 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fec401562b0 msgr2=0x7fec4015c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.023+0000 7fec3effd700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fec401562b0 0x7fec4015c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.023+0000 7fec3effd700 1 -- 172.21.15.111:0/3501078703 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec40159d00 con 0x7fec28005960 2023-11-28T13:01:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.023+0000 7fec3effd700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28005960 0x7fec40154630 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fec3400f820 tx=0x7fec3400fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.023+0000 7fec277fe700 1 -- 172.21.15.111:0/3501078703 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec340052e0 con 0x7fec28005960 2023-11-28T13:01:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.023+0000 7fec277fe700 1 -- 172.21.15.111:0/3501078703 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec34005460 con 0x7fec28005960 2023-11-28T13:01:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.023+0000 7fec457cd700 1 -- 172.21.15.111:0/3501078703 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec40159a20 con 0x7fec28005960 2023-11-28T13:01:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.023+0000 7fec277fe700 1 -- 172.21.15.111:0/3501078703 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec340038c0 con 0x7fec28005960 2023-11-28T13:01:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.023+0000 7fec457cd700 1 -- 172.21.15.111:0/3501078703 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec40159fe0 con 0x7fec28005960 2023-11-28T13:01:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.024+0000 7fec457cd700 1 -- 172.21.15.111:0/3501078703 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec40067170 con 0x7fec28005960 2023-11-28T13:01:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.024+0000 7fec277fe700 1 -- 172.21.15.111:0/3501078703 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fec34003a20 con 0x7fec28005960 2023-11-28T13:01:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.025+0000 7fec277fe700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fec2c0682f0 0x7fec2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.025+0000 7fec277fe700 1 -- 172.21.15.111:0/3501078703 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6282+0+0 (secure 0 0 0) 0x7fec34085ca0 con 0x7fec28005960 2023-11-28T13:01:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.025+0000 7fec3e7fc700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fec2c0682f0 0x7fec2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.025+0000 7fec3e7fc700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fec2c0682f0 0x7fec2c06a7a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fec40151060 tx=0x7fec30007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:54.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.029+0000 7fec277fe700 1 -- 172.21.15.111:0/3501078703 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fec3408a070 con 0x7fec28005960 2023-11-28T13:01:54.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.255+0000 7fec457cd700 1 -- 172.21.15.111:0/3501078703 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fec40157970 con 0x7fec28005960 2023-11-28T13:01:54.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.262+0000 7fec277fe700 1 -- 172.21.15.111:0/3501078703 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v29) v1 ==== 174+0+0 (secure 0 0 0) 0x7fec34052420 con 0x7fec28005960 2023-11-28T13:01:54.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.270+0000 7fec457cd700 1 -- 172.21.15.111:0/3501078703 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fec2c0682f0 msgr2=0x7fec2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.270+0000 7fec457cd700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fec2c0682f0 0x7fec2c06a7a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fec40151060 tx=0x7fec30007000 comp rx=0 tx=0).stop 2023-11-28T13:01:54.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.270+0000 7fec457cd700 1 -- 172.21.15.111:0/3501078703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28005960 msgr2=0x7fec40154630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.270+0000 7fec457cd700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28005960 0x7fec40154630 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fec3400f820 tx=0x7fec3400fbe0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.274+0000 7fec457cd700 1 -- 172.21.15.111:0/3501078703 shutdown_connections 2023-11-28T13:01:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.274+0000 7fec457cd700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fec4014e670 0x7fec4014eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.274+0000 7fec457cd700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28005960 0x7fec40154630 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.274+0000 7fec457cd700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fec2c0682f0 0x7fec2c06a7a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.274+0000 7fec457cd700 1 --2- 172.21.15.111:0/3501078703 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fec401562b0 0x7fec4015c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.274+0000 7fec457cd700 1 -- 172.21.15.111:0/3501078703 >> 172.21.15.111:0/3501078703 conn(0x7fec400b0db0 msgr2=0x7fec401e2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:54.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.276+0000 7fec457cd700 1 -- 172.21.15.111:0/3501078703 shutdown_connections 2023-11-28T13:01:54.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.276+0000 7fec457cd700 1 -- 172.21.15.111:0/3501078703 wait complete. 2023-11-28T13:01:54.278 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 10 and data pool 11 2023-11-28T13:01:54.298 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:01:54.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.417+0000 7fac7e769700 1 Processor -- start 2023-11-28T13:01:54.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.417+0000 7fac7e769700 1 -- start start 2023-11-28T13:01:54.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac7e769700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fac780b0d70 0x7fac781604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac7e769700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fac780b1140 0x7fac780ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac7e769700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7815cd90 con 0x7fac780b15b0 2023-11-28T13:01:54.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac7e769700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac781613e0 con 0x7fac780b0d70 2023-11-28T13:01:54.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac7e769700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac78160b40 con 0x7fac780b1140 2023-11-28T13:01:54.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac77fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fac780b0d70 0x7fac781604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac7cd06700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fac780b15b0 0x7fac780cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:37992/0 (socket says 172.21.15.111:37992) 2023-11-28T13:01:54.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac7cd06700 1 -- 172.21.15.111:0/3939114664 learned_addr learned my addr 172.21.15.111:0/3939114664 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:54.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac777fe700 1 --2- 172.21.15.111:0/3939114664 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fac780b1140 0x7fac780ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac77fff700 1 -- 172.21.15.111:0/3939114664 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fac780b1140 msgr2=0x7fac780ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:54.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac76ffd700 1 -- 172.21.15.111:0/3939114664 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148847081 0 0) 0x7fac70001490 con 0x7fac780b15b0 2023-11-28T13:01:54.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac77fff700 1 --2- 172.21.15.111:0/3939114664 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fac780b1140 0x7fac780ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac77fff700 1 -- 172.21.15.111:0/3939114664 >> v1:172.21.15.111:6789/0 conn(0x7fac780b15b0 legacy=0x7fac780cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac77fff700 1 -- 172.21.15.111:0/3939114664 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac78160e20 con 0x7fac780b0d70 2023-11-28T13:01:54.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac77fff700 1 --2- 172.21.15.111:0/3939114664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac780b0d70 0x7fac781604b0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fac6c009820 tx=0x7fac6c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779e9927bc398082 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:54.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.418+0000 7fac76ffd700 1 -- 172.21.15.111:0/3939114664 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6c005e30 con 0x7fac780b0d70 2023-11-28T13:01:54.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.419+0000 7fac76ffd700 1 -- 172.21.15.111:0/3939114664 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6c006030 con 0x7fac780b0d70 2023-11-28T13:01:54.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.419+0000 7fac76ffd700 1 -- 172.21.15.111:0/3939114664 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6c0063e0 con 0x7fac780b0d70 2023-11-28T13:01:54.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.419+0000 7fac7e769700 1 -- 172.21.15.111:0/3939114664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac780b0d70 msgr2=0x7fac781604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.419+0000 7fac7e769700 1 --2- 172.21.15.111:0/3939114664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac780b0d70 0x7fac781604b0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fac6c009820 tx=0x7fac6c009b30 comp rx=0 tx=0).stop 2023-11-28T13:01:54.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.419+0000 7fac7e769700 1 -- 172.21.15.111:0/3939114664 shutdown_connections 2023-11-28T13:01:54.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.419+0000 7fac7e769700 1 --2- 172.21.15.111:0/3939114664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac780b0d70 0x7fac781604b0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.419+0000 7fac7e769700 1 --2- 172.21.15.111:0/3939114664 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fac780b1140 0x7fac780ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.419+0000 7fac7e769700 1 -- 172.21.15.111:0/3939114664 >> 172.21.15.111:0/3939114664 conn(0x7fac78098ee0 msgr2=0x7fac7806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:54.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.419+0000 7fac7e769700 1 -- 172.21.15.111:0/3939114664 shutdown_connections 2023-11-28T13:01:54.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.419+0000 7fac7e769700 1 -- 172.21.15.111:0/3939114664 wait complete. 2023-11-28T13:01:54.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.420+0000 7fac7e769700 1 Processor -- start 2023-11-28T13:01:54.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.420+0000 7fac7e769700 1 -- start start 2023-11-28T13:01:54.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.421+0000 7fac7e769700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fac780b0d70 0x7fac781f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.421+0000 7fac7e769700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac780b1140 0x7fac780b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.421+0000 7fac7e769700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac780b15b0 0x7fac7815bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.421+0000 7fac7e769700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac781f34a0 con 0x7fac780b15b0 2023-11-28T13:01:54.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.421+0000 7fac7e769700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7815e2b0 con 0x7fac780b1140 2023-11-28T13:01:54.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.421+0000 7fac7e769700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7815df80 con 0x7fac780b0d70 2023-11-28T13:01:54.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.421+0000 7fac7cd06700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac780b15b0 0x7fac7815bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.421+0000 7fac7cd06700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac780b15b0 0x7fac7815bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37220/0 (socket says 172.21.15.111:37220) 2023-11-28T13:01:54.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.421+0000 7fac7cd06700 1 -- 172.21.15.111:0/1166602270 learned_addr learned my addr 172.21.15.111:0/1166602270 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:54.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.422+0000 7fac777fe700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac780b1140 0x7fac780b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.422+0000 7fac77fff700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fac780b0d70 0x7fac781f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.422+0000 7fac7cd06700 1 -- 172.21.15.111:0/1166602270 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fac780b0d70 msgr2=0x7fac781f22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.422+0000 7fac7cd06700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fac780b0d70 0x7fac781f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.422+0000 7fac7cd06700 1 -- 172.21.15.111:0/1166602270 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac780b1140 msgr2=0x7fac780b2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.422+0000 7fac7cd06700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac780b1140 0x7fac780b2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.422+0000 7fac7cd06700 1 -- 172.21.15.111:0/1166602270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c009430 con 0x7fac780b15b0 2023-11-28T13:01:54.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.422+0000 7fac7cd06700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac780b15b0 0x7fac7815bbf0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fac7000cb60 tx=0x7fac7000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:54.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.422+0000 7fac74ff9700 1 -- 172.21.15.111:0/1166602270 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7000aaf0 con 0x7fac780b15b0 2023-11-28T13:01:54.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.422+0000 7fac74ff9700 1 -- 172.21.15.111:0/1166602270 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7000ac50 con 0x7fac780b15b0 2023-11-28T13:01:54.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.422+0000 7fac7e769700 1 -- 172.21.15.111:0/1166602270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac780af340 con 0x7fac780b15b0 2023-11-28T13:01:54.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.422+0000 7fac74ff9700 1 -- 172.21.15.111:0/1166602270 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac700070d0 con 0x7fac780b15b0 2023-11-28T13:01:54.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.422+0000 7fac7e769700 1 -- 172.21.15.111:0/1166602270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac78161100 con 0x7fac780b15b0 2023-11-28T13:01:54.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.423+0000 7fac7e769700 1 -- 172.21.15.111:0/1166602270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac780cec00 con 0x7fac780b15b0 2023-11-28T13:01:54.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.423+0000 7fac74ff9700 1 -- 172.21.15.111:0/1166602270 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fac70010070 con 0x7fac780b15b0 2023-11-28T13:01:54.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.424+0000 7fac74ff9700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fac640682f0 0x7fac6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.424+0000 7fac74ff9700 1 -- 172.21.15.111:0/1166602270 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fac700856a0 con 0x7fac780b15b0 2023-11-28T13:01:54.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.424+0000 7fac77fff700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fac640682f0 0x7fac6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.424+0000 7fac77fff700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fac640682f0 0x7fac6406a7a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fac6c013040 tx=0x7fac6c008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:54.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.428+0000 7fac74ff9700 1 -- 172.21.15.111:0/1166602270 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fac70051d40 con 0x7fac780b15b0 2023-11-28T13:01:54.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.613+0000 7fac7e769700 1 -- 172.21.15.111:0/1166602270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac7815e460 con 0x7fac780b15b0 2023-11-28T13:01:54.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.614+0000 7fac74ff9700 1 -- 172.21.15.111:0/1166602270 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+18244 (secure 0 0 0) 0x7fac7815e460 con 0x7fac780b15b0 2023-11-28T13:01:54.616 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:54.616 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":66,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:01:54.257328+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:01:51.934388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-28T13:01:53.219817+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:01:54.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.617+0000 7fac7e769700 1 -- 172.21.15.111:0/1166602270 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fac640682f0 msgr2=0x7fac6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.617+0000 7fac7e769700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fac640682f0 0x7fac6406a7a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fac6c013040 tx=0x7fac6c008070 comp rx=0 tx=0).stop 2023-11-28T13:01:54.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.617+0000 7fac7e769700 1 -- 172.21.15.111:0/1166602270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac780b15b0 msgr2=0x7fac7815bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.617+0000 7fac7e769700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac780b15b0 0x7fac7815bbf0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fac7000cb60 tx=0x7fac7000cf20 comp rx=0 tx=0).stop 2023-11-28T13:01:54.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.617+0000 7fac7e769700 1 -- 172.21.15.111:0/1166602270 shutdown_connections 2023-11-28T13:01:54.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.617+0000 7fac7e769700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac780b15b0 0x7fac7815bbf0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.617+0000 7fac7e769700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fac780b0d70 0x7fac781f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.617+0000 7fac7e769700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fac640682f0 0x7fac6406a7a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.617+0000 7fac7e769700 1 --2- 172.21.15.111:0/1166602270 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fac780b1140 0x7fac780b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.617+0000 7fac7e769700 1 -- 172.21.15.111:0/1166602270 >> 172.21.15.111:0/1166602270 conn(0x7fac78098ee0 msgr2=0x7fac780ac670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:54.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.617+0000 7fac7e769700 1 -- 172.21.15.111:0/1166602270 shutdown_connections 2023-11-28T13:01:54.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.617+0000 7fac7e769700 1 -- 172.21.15.111:0/1166602270 wait complete. 2023-11-28T13:01:54.631 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:01:54.749 INFO:tasks.ceph.mds.c.smithi111.stderr:2023-11-28T13:01:54.748+0000 7f60ff06f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T13:01:54.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.748+0000 7f1e1a079700 1 Processor -- start 2023-11-28T13:01:54.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.748+0000 7f1e1a079700 1 -- start start 2023-11-28T13:01:54.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.749+0000 7f1e1a079700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e1415cc50 0x7f1e140712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.749+0000 7f1e1a079700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e140717f0 con 0x7f1e140cceb0 2023-11-28T13:01:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.749+0000 7f1e1a079700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1406fb90 con 0x7f1e14072300 2023-11-28T13:01:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.749+0000 7f1e1a079700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e140726d0 con 0x7f1e1415cc50 2023-11-28T13:01:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.749+0000 7f1e13fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1e140cceb0 0x7f1e1406a610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38006/0 (socket says 172.21.15.111:38006) 2023-11-28T13:01:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.749+0000 7f1e13fff700 1 -- 172.21.15.111:0/1258744532 learned_addr learned my addr 172.21.15.111:0/1258744532 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.749+0000 7f1e12ffd700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e1415cc50 0x7f1e140712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.749+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874602718 0 0) 0x7f1e140717f0 con 0x7f1e140cceb0 2023-11-28T13:01:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.749+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dfc0035b0 con 0x7f1e140cceb0 2023-11-28T13:01:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405452274 0 0) 0x7f1e1406fb90 con 0x7f1e14072300 2023-11-28T13:01:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e140717f0 con 0x7f1e14072300 2023-11-28T13:01:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2807132568 0 0) 0x7f1dfc0035b0 con 0x7f1e140cceb0 2023-11-28T13:01:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e1415cc50 msgr2=0x7f1e140712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e1415cc50 0x7f1e140712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 >> v1:172.21.15.149:6789/0 conn(0x7f1e14072300 legacy=0x7f1e1406b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1415d0f0 con 0x7f1e140cceb0 2023-11-28T13:01:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f1e0c0019f0 con 0x7f1e140cceb0 2023-11-28T13:01:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 >> v1:172.21.15.111:6789/0 conn(0x7f1e140cceb0 legacy=0x7f1e1406a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1dfc003bc0 0x7f1dfc003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e1415cc50 0x7f1dfc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dfc004d90 0x7f1dfc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e140726d0 con 0x7f1dfc004d90 2023-11-28T13:01:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc00bb10 con 0x7f1dfc003bc0 2023-11-28T13:01:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc00bcd0 con 0x7f1e1415cc50 2023-11-28T13:01:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e13fff700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dfc004d90 0x7f1dfc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e137fe700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1dfc003bc0 0x7f1dfc003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e12ffd700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e1415cc50 0x7f1dfc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e13fff700 1 -- 172.21.15.111:0/1258744532 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e1415cc50 msgr2=0x7f1dfc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e13fff700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e1415cc50 0x7f1dfc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e13fff700 1 -- 172.21.15.111:0/1258744532 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1dfc003bc0 msgr2=0x7f1dfc003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e13fff700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1dfc003bc0 0x7f1dfc003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e13fff700 1 -- 172.21.15.111:0/1258744532 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dfc0035b0 con 0x7f1dfc004d90 2023-11-28T13:01:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e12ffd700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e1415cc50 0x7f1dfc004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e137fe700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1dfc003bc0 0x7f1dfc003fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e13fff700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dfc004d90 0x7f1dfc005200 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00a7c0 tx=0x7f1dfc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d445f29127a490 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c012600 con 0x7f1dfc004d90 2023-11-28T13:01:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0c012780 con 0x7f1dfc004d90 2023-11-28T13:01:54.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.750+0000 7f1e127fc700 1 -- 172.21.15.111:0/1258744532 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c012b30 con 0x7f1dfc004d90 2023-11-28T13:01:54.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.751+0000 7f1e1a079700 1 -- 172.21.15.111:0/1258744532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dfc004d90 msgr2=0x7f1dfc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.751+0000 7f1e1a079700 1 --2- 172.21.15.111:0/1258744532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dfc004d90 0x7f1dfc005200 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00a7c0 tx=0x7f1dfc005880 comp rx=0 tx=0).stop 2023-11-28T13:01:54.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.751+0000 7f1e13fff700 1 -- 172.21.15.111:0/1258744532 reap_dead start 2023-11-28T13:01:54.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.751+0000 7f1e1a079700 1 -- 172.21.15.111:0/1258744532 shutdown_connections 2023-11-28T13:01:54.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.751+0000 7f1e1a079700 1 -- 172.21.15.111:0/1258744532 >> 172.21.15.111:0/1258744532 conn(0x7f1e140cf2e0 msgr2=0x7f1e140cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:54.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.751+0000 7f1e1a079700 1 -- 172.21.15.111:0/1258744532 shutdown_connections 2023-11-28T13:01:54.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.751+0000 7f1e1a079700 1 -- 172.21.15.111:0/1258744532 wait complete. 2023-11-28T13:01:54.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.752+0000 7f1e1a079700 1 Processor -- start 2023-11-28T13:01:54.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.752+0000 7f1e1a079700 1 -- start start 2023-11-28T13:01:54.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.753+0000 7f1e1a079700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e141528d0 0x7f1e141e7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.753+0000 7f1e1a079700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e141e8550 0x7f1e14154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.753+0000 7f1e1a079700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e14154e50 0x7f1e140b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.753+0000 7f1e1a079700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e141552f0 con 0x7f1e14154e50 2023-11-28T13:01:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.753+0000 7f1e1a079700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e141e6e30 con 0x7f1e141528d0 2023-11-28T13:01:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.753+0000 7f1e1a079700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e141e7dc0 con 0x7f1e141e8550 2023-11-28T13:01:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.753+0000 7f1e13fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e14154e50 0x7f1e140b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.753+0000 7f1e13fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e14154e50 0x7f1e140b2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37240/0 (socket says 172.21.15.111:37240) 2023-11-28T13:01:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.753+0000 7f1e13fff700 1 -- 172.21.15.111:0/271460626 learned_addr learned my addr 172.21.15.111:0/271460626 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.753+0000 7f1e137fe700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e141528d0 0x7f1e141e7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.753+0000 7f1e12ffd700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e141e8550 0x7f1e14154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.754+0000 7f1e13fff700 1 -- 172.21.15.111:0/271460626 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e141e8550 msgr2=0x7f1e14154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.754+0000 7f1e13fff700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e141e8550 0x7f1e14154910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.754+0000 7f1e13fff700 1 -- 172.21.15.111:0/271460626 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e141528d0 msgr2=0x7f1e141e7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.754+0000 7f1e13fff700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e141528d0 0x7f1e141e7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.754+0000 7f1e13fff700 1 -- 172.21.15.111:0/271460626 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1415d6b0 con 0x7f1e14154e50 2023-11-28T13:01:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.754+0000 7f1e13fff700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e14154e50 0x7f1e140b2110 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c015040 tx=0x7f1e0c001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.754+0000 7f1dfbfff700 1 -- 172.21.15.111:0/271460626 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c012400 con 0x7f1e14154e50 2023-11-28T13:01:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.754+0000 7f1dfbfff700 1 -- 172.21.15.111:0/271460626 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0c012560 con 0x7f1e14154e50 2023-11-28T13:01:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.754+0000 7f1e1a079700 1 -- 172.21.15.111:0/271460626 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1415d3d0 con 0x7f1e14154e50 2023-11-28T13:01:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.754+0000 7f1dfbfff700 1 -- 172.21.15.111:0/271460626 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c0019d0 con 0x7f1e14154e50 2023-11-28T13:01:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.754+0000 7f1e1a079700 1 -- 172.21.15.111:0/271460626 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1415d990 con 0x7f1e14154e50 2023-11-28T13:01:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.755+0000 7f1e1a079700 1 -- 172.21.15.111:0/271460626 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e14067170 con 0x7f1e14154e50 2023-11-28T13:01:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.755+0000 7f1dfbfff700 1 -- 172.21.15.111:0/271460626 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1e0c02e070 con 0x7f1e14154e50 2023-11-28T13:01:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.756+0000 7f1dfbfff700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1e00068320 0x7f1e0006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:54.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.756+0000 7f1dfbfff700 1 -- 172.21.15.111:0/271460626 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f1e0c087240 con 0x7f1e14154e50 2023-11-28T13:01:54.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.756+0000 7f1e137fe700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1e00068320 0x7f1e0006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:54.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.756+0000 7f1e137fe700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1e00068320 0x7f1e0006a7d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1e08000f60 tx=0x7f1e08009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:54.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.760+0000 7f1dfbfff700 1 -- 172.21.15.111:0/271460626 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1e0c053960 con 0x7f1e14154e50 2023-11-28T13:01:54.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.946+0000 7f1e1a079700 1 -- 172.21.15.111:0/271460626 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e141e7ff0 con 0x7f1e14154e50 2023-11-28T13:01:54.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.947+0000 7f1dfbfff700 1 -- 172.21.15.111:0/271460626 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+18244 (secure 0 0 0) 0x7f1e0c022090 con 0x7f1e14154e50 2023-11-28T13:01:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:54.949 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":66,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:01:54.257328+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:01:51.934388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-28T13:01:53.219817+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:01:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.949+0000 7f1e1a079700 1 -- 172.21.15.111:0/271460626 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1e00068320 msgr2=0x7f1e0006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.949+0000 7f1e1a079700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1e00068320 0x7f1e0006a7d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1e08000f60 tx=0x7f1e08009040 comp rx=0 tx=0).stop 2023-11-28T13:01:54.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.949+0000 7f1e1a079700 1 -- 172.21.15.111:0/271460626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e14154e50 msgr2=0x7f1e140b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:54.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.949+0000 7f1e1a079700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e14154e50 0x7f1e140b2110 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c015040 tx=0x7f1e0c001d00 comp rx=0 tx=0).stop 2023-11-28T13:01:54.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.949+0000 7f1e1a079700 1 -- 172.21.15.111:0/271460626 shutdown_connections 2023-11-28T13:01:54.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.949+0000 7f1e1a079700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e14154e50 0x7f1e140b2110 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.949+0000 7f1e1a079700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1e00068320 0x7f1e0006a7d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.949+0000 7f1e1a079700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e141528d0 0x7f1e141e7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.949+0000 7f1e1a079700 1 --2- 172.21.15.111:0/271460626 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e141e8550 0x7f1e14154910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:54.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.949+0000 7f1e1a079700 1 -- 172.21.15.111:0/271460626 >> 172.21.15.111:0/271460626 conn(0x7f1e140cf2e0 msgr2=0x7f1e140caf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:54.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.949+0000 7f1e1a079700 1 -- 172.21.15.111:0/271460626 shutdown_connections 2023-11-28T13:01:54.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:54.949+0000 7f1e1a079700 1 -- 172.21.15.111:0/271460626 wait complete. 2023-11-28T13:01:54.963 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T13:01:55.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.083+0000 7fbbe7a12700 1 Processor -- start 2023-11-28T13:01:55.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.084+0000 7fbbe7a12700 1 -- start start 2023-11-28T13:01:55.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.084+0000 7fbbe7a12700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fbbe00af9f0 0x7fbbe00ca220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:55.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.084+0000 7fbbe7a12700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe006ac80 con 0x7fbbe00af5a0 2023-11-28T13:01:55.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.084+0000 7fbbe7a12700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe00cee00 con 0x7fbbe015fed0 2023-11-28T13:01:55.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.084+0000 7fbbe7a12700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe00cf9a0 con 0x7fbbe00af9f0 2023-11-28T13:01:55.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbe4fad700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbbe00af5a0 0x7fbbe00b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38022/0 (socket says 172.21.15.111:38022) 2023-11-28T13:01:55.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbe4fad700 1 -- 172.21.15.111:0/2785404360 learned_addr learned my addr 172.21.15.111:0/2785404360 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:55.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932769759 0 0) 0x7fbbe006ac80 con 0x7fbbe00af5a0 2023-11-28T13:01:55.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbe5faf700 1 --2- 172.21.15.111:0/2785404360 >> v2:172.21.15.149:3300/0 conn(0x7fbbe00af9f0 0x7fbbe00ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:55.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbcc0035b0 con 0x7fbbe00af5a0 2023-11-28T13:01:55.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 211267520 0 0) 0x7fbbcc0035b0 con 0x7fbbe00af5a0 2023-11-28T13:01:55.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 >> v2:172.21.15.149:3300/0 conn(0x7fbbe00af9f0 msgr2=0x7fbbe00ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:55.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 --2- 172.21.15.111:0/2785404360 >> v2:172.21.15.149:3300/0 conn(0x7fbbe00af9f0 0x7fbbe00ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:55.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 >> v1:172.21.15.149:6789/0 conn(0x7fbbe015fed0 legacy=0x7fbbe00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:55.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe006aed0 con 0x7fbbe00af5a0 2023-11-28T13:01:55.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fbbdc001670 con 0x7fbbe00af5a0 2023-11-28T13:01:55.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 >> v1:172.21.15.111:6789/0 conn(0x7fbbe00af5a0 legacy=0x7fbbe00b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:55.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 --2- 172.21.15.111:0/2785404360 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbcc003a00 0x7fbbcc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:55.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 --2- 172.21.15.111:0/2785404360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc004850 0x7fbbcc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:55.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 --2- 172.21.15.111:0/2785404360 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbcc005200 0x7fbbcc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:55.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe00cf9a0 con 0x7fbbcc004850 2023-11-28T13:01:55.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc00c080 con 0x7fbbcc003a00 2023-11-28T13:01:55.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc00c240 con 0x7fbbcc005200 2023-11-28T13:01:55.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbe4fad700 1 --2- 172.21.15.111:0/2785404360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc004850 0x7fbbcc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:55.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbe57ae700 1 --2- 172.21.15.111:0/2785404360 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbcc003a00 0x7fbbcc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:55.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbe5faf700 1 --2- 172.21.15.111:0/2785404360 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbcc005200 0x7fbbcc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:55.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbe4fad700 1 -- 172.21.15.111:0/2785404360 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbcc005200 msgr2=0x7fbbcc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:55.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbe4fad700 1 --2- 172.21.15.111:0/2785404360 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbcc005200 0x7fbbcc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:55.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbe4fad700 1 -- 172.21.15.111:0/2785404360 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbcc003a00 msgr2=0x7fbbcc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:55.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbe4fad700 1 --2- 172.21.15.111:0/2785404360 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbcc003a00 0x7fbbcc003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:55.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbe4fad700 1 -- 172.21.15.111:0/2785404360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbcc0035b0 con 0x7fbbcc004850 2023-11-28T13:01:55.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.085+0000 7fbbe5faf700 1 -- 172.21.15.111:0/2785404360 reap_dead start 2023-11-28T13:01:55.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.086+0000 7fbbe57ae700 1 --2- 172.21.15.111:0/2785404360 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbcc003a00 0x7fbbcc003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:55.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.086+0000 7fbbe4fad700 1 --2- 172.21.15.111:0/2785404360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc004850 0x7fbbcc004cc0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fbbe006cf90 tx=0x7fbbcc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2412872de88803f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:55.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.086+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc007c10 con 0x7fbbcc004850 2023-11-28T13:01:55.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.086+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbdc00cd60 con 0x7fbbcc004850 2023-11-28T13:01:55.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.086+0000 7fbbd7fff700 1 -- 172.21.15.111:0/2785404360 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc005ba0 con 0x7fbbcc004850 2023-11-28T13:01:55.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.086+0000 7fbbe7a12700 1 -- 172.21.15.111:0/2785404360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc004850 msgr2=0x7fbbcc004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:55.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.086+0000 7fbbe7a12700 1 --2- 172.21.15.111:0/2785404360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc004850 0x7fbbcc004cc0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fbbe006cf90 tx=0x7fbbcc005880 comp rx=0 tx=0).stop 2023-11-28T13:01:55.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.086+0000 7fbbe7a12700 1 -- 172.21.15.111:0/2785404360 shutdown_connections 2023-11-28T13:01:55.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.086+0000 7fbbe7a12700 1 --2- 172.21.15.111:0/2785404360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc004850 0x7fbbcc004cc0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:55.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.086+0000 7fbbe7a12700 1 -- 172.21.15.111:0/2785404360 >> 172.21.15.111:0/2785404360 conn(0x7fbbe006bca0 msgr2=0x7fbbe00cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:55.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.086+0000 7fbbe7a12700 1 -- 172.21.15.111:0/2785404360 shutdown_connections 2023-11-28T13:01:55.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.086+0000 7fbbe7a12700 1 -- 172.21.15.111:0/2785404360 wait complete. 2023-11-28T13:01:55.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.088+0000 7fbbe7a12700 1 Processor -- start 2023-11-28T13:01:55.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.088+0000 7fbbe7a12700 1 -- start start 2023-11-28T13:01:55.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.088+0000 7fbbe7a12700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc004850 0x7fbbe01f5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:55.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.088+0000 7fbbe7a12700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbe0154050 0x7fbbe01f5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:55.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.088+0000 7fbbe7a12700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbe00c4750 0x7fbbe00c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:55.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.088+0000 7fbbe7a12700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe00c7060 con 0x7fbbcc004850 2023-11-28T13:01:55.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.088+0000 7fbbe7a12700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe00c5fe0 con 0x7fbbe0154050 2023-11-28T13:01:55.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.088+0000 7fbbe7a12700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe00c5100 con 0x7fbbe00c4750 2023-11-28T13:01:55.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.088+0000 7fbbe57ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc004850 0x7fbbe01f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:55.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.088+0000 7fbbe57ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc004850 0x7fbbe01f5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37254/0 (socket says 172.21.15.111:37254) 2023-11-28T13:01:55.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.088+0000 7fbbe57ae700 1 -- 172.21.15.111:0/650206103 learned_addr learned my addr 172.21.15.111:0/650206103 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:55.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.089+0000 7fbbe57ae700 1 -- 172.21.15.111:0/650206103 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbe00c4750 msgr2=0x7fbbe00c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:55.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.089+0000 7fbbe57ae700 1 --2- 172.21.15.111:0/650206103 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbe00c4750 0x7fbbe00c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:55.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.089+0000 7fbbe57ae700 1 -- 172.21.15.111:0/650206103 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbe0154050 msgr2=0x7fbbe01f5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:55.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.089+0000 7fbbe57ae700 1 --2- 172.21.15.111:0/650206103 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbe0154050 0x7fbbe01f5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:55.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.089+0000 7fbbe57ae700 1 -- 172.21.15.111:0/650206103 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe01602a0 con 0x7fbbcc004850 2023-11-28T13:01:55.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.089+0000 7fbbe57ae700 1 --2- 172.21.15.111:0/650206103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc004850 0x7fbbe01f5700 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000e460 tx=0x7fbbd000e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:55.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.089+0000 7fbbd5ffb700 1 -- 172.21.15.111:0/650206103 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd00052e0 con 0x7fbbcc004850 2023-11-28T13:01:55.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.089+0000 7fbbd5ffb700 1 -- 172.21.15.111:0/650206103 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd0005440 con 0x7fbbcc004850 2023-11-28T13:01:55.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.089+0000 7fbbe7a12700 1 -- 172.21.15.111:0/650206103 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe00cea90 con 0x7fbbcc004850 2023-11-28T13:01:55.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.089+0000 7fbbd5ffb700 1 -- 172.21.15.111:0/650206103 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd00038a0 con 0x7fbbcc004850 2023-11-28T13:01:55.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.089+0000 7fbbe7a12700 1 -- 172.21.15.111:0/650206103 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe00ce6b0 con 0x7fbbcc004850 2023-11-28T13:01:55.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.090+0000 7fbbe7a12700 1 -- 172.21.15.111:0/650206103 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbe0067170 con 0x7fbbcc004850 2023-11-28T13:01:55.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.090+0000 7fbbd5ffb700 1 -- 172.21.15.111:0/650206103 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbbd001b030 con 0x7fbbcc004850 2023-11-28T13:01:55.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.091+0000 7fbbd5ffb700 1 --2- 172.21.15.111:0/650206103 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbbc40682f0 0x7fbbc406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:55.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.091+0000 7fbbd5ffb700 1 -- 172.21.15.111:0/650206103 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fbbd0085240 con 0x7fbbcc004850 2023-11-28T13:01:55.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.091+0000 7fbbe4fad700 1 --2- 172.21.15.111:0/650206103 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbbc40682f0 0x7fbbc406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:55.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.091+0000 7fbbe4fad700 1 --2- 172.21.15.111:0/650206103 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbbc40682f0 0x7fbbc406a7a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00c6490 tx=0x7fbbdc00a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:55.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.095+0000 7fbbd5ffb700 1 -- 172.21.15.111:0/650206103 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbbd004d5e0 con 0x7fbbcc004850 2023-11-28T13:01:55.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:55.325+0000 7fbbe7a12700 1 -- 172.21.15.111:0/650206103 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbbe00c53d0 con 0x7fbbcc004850 2023-11-28T13:01:56.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.269+0000 7fbbd5ffb700 1 -- 172.21.15.111:0/650206103 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbbd0051a10 con 0x7fbbcc004850 2023-11-28T13:01:56.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.272+0000 7fbbe7a12700 1 -- 172.21.15.111:0/650206103 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbbc40682f0 msgr2=0x7fbbc406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:56.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.272+0000 7fbbe7a12700 1 --2- 172.21.15.111:0/650206103 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbbc40682f0 0x7fbbc406a7a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00c6490 tx=0x7fbbdc00a000 comp rx=0 tx=0).stop 2023-11-28T13:01:56.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.272+0000 7fbbe7a12700 1 -- 172.21.15.111:0/650206103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc004850 msgr2=0x7fbbe01f5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:56.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.272+0000 7fbbe7a12700 1 --2- 172.21.15.111:0/650206103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc004850 0x7fbbe01f5700 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000e460 tx=0x7fbbd000e820 comp rx=0 tx=0).stop 2023-11-28T13:01:56.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.272+0000 7fbbe7a12700 1 -- 172.21.15.111:0/650206103 shutdown_connections 2023-11-28T13:01:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.272+0000 7fbbe7a12700 1 --2- 172.21.15.111:0/650206103 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbbe00c4750 0x7fbbe00c4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.272+0000 7fbbe7a12700 1 --2- 172.21.15.111:0/650206103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc004850 0x7fbbe01f5700 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.272+0000 7fbbe7a12700 1 --2- 172.21.15.111:0/650206103 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbbc40682f0 0x7fbbc406a7a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.272+0000 7fbbe7a12700 1 --2- 172.21.15.111:0/650206103 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbbe0154050 0x7fbbe01f5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.272+0000 7fbbe7a12700 1 -- 172.21.15.111:0/650206103 >> 172.21.15.111:0/650206103 conn(0x7fbbe006bca0 msgr2=0x7fbbe006a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.273+0000 7fbbe7a12700 1 -- 172.21.15.111:0/650206103 shutdown_connections 2023-11-28T13:01:56.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.273+0000 7fbbe7a12700 1 -- 172.21.15.111:0/650206103 wait complete. 2023-11-28T13:01:56.288 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T13:01:56.289 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T13:01:56.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.418+0000 7f485aa0f700 1 Processor -- start 2023-11-28T13:01:56.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.418+0000 7f485aa0f700 1 -- start start 2023-11-28T13:01:56.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.418+0000 7f485aa0f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f48540b0d70 0x7f48541604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:56.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.418+0000 7f485aa0f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f48540b1140 0x7f48540ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:56.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f485aa0f700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485415cd90 con 0x7f48540b15b0 2023-11-28T13:01:56.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f485aa0f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48541613e0 con 0x7f48540b0d70 2023-11-28T13:01:56.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f485aa0f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4854160b40 con 0x7f48540b1140 2023-11-28T13:01:56.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f4853fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f48540b0d70 0x7f48541604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:56.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f4853fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f48540b0d70 0x7f48541604b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37262/0 (socket says 172.21.15.111:37262) 2023-11-28T13:01:56.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f4853fff700 1 -- 172.21.15.111:0/799829038 learned_addr learned my addr 172.21.15.111:0/799829038 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:56.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f4853fff700 1 -- 172.21.15.111:0/799829038 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f48540b1140 msgr2=0x7f48540ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:56.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f48537fe700 1 --2- 172.21.15.111:0/799829038 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f48540b1140 0x7f48540ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:56.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f4853fff700 1 --2- 172.21.15.111:0/799829038 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f48540b1140 0x7f48540ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:56.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f4853fff700 1 -- 172.21.15.111:0/799829038 >> v1:172.21.15.149:6789/0 conn(0x7f48540b15b0 legacy=0x7f48540cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:56.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f4853fff700 1 -- 172.21.15.111:0/799829038 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4854160e20 con 0x7f48540b0d70 2023-11-28T13:01:56.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f48537fe700 1 --2- 172.21.15.111:0/799829038 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f48540b1140 0x7f48540ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:56.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f4853fff700 1 --2- 172.21.15.111:0/799829038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48540b0d70 0x7f48541604b0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4844006fd0 tx=0x7f4844009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d9ced85bd1acd8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:56.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f4852ffd700 1 -- 172.21.15.111:0/799829038 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844005e30 con 0x7f48540b0d70 2023-11-28T13:01:56.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f4852ffd700 1 -- 172.21.15.111:0/799829038 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4844004030 con 0x7f48540b0d70 2023-11-28T13:01:56.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.419+0000 7f4852ffd700 1 -- 172.21.15.111:0/799829038 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48440043e0 con 0x7f48540b0d70 2023-11-28T13:01:56.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.420+0000 7f485aa0f700 1 -- 172.21.15.111:0/799829038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48540b0d70 msgr2=0x7f48541604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:56.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.420+0000 7f485aa0f700 1 --2- 172.21.15.111:0/799829038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48540b0d70 0x7f48541604b0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4844006fd0 tx=0x7f4844009bd0 comp rx=0 tx=0).stop 2023-11-28T13:01:56.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.420+0000 7f485aa0f700 1 -- 172.21.15.111:0/799829038 shutdown_connections 2023-11-28T13:01:56.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.420+0000 7f485aa0f700 1 --2- 172.21.15.111:0/799829038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48540b0d70 0x7f48541604b0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:56.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.420+0000 7f485aa0f700 1 --2- 172.21.15.111:0/799829038 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f48540b1140 0x7f48540ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:56.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.420+0000 7f485aa0f700 1 -- 172.21.15.111:0/799829038 >> 172.21.15.111:0/799829038 conn(0x7f4854098ee0 msgr2=0x7f485406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:56.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.420+0000 7f485aa0f700 1 -- 172.21.15.111:0/799829038 shutdown_connections 2023-11-28T13:01:56.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.420+0000 7f485aa0f700 1 -- 172.21.15.111:0/799829038 wait complete. 2023-11-28T13:01:56.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.421+0000 7f485aa0f700 1 Processor -- start 2023-11-28T13:01:56.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.422+0000 7f485aa0f700 1 -- start start 2023-11-28T13:01:56.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.422+0000 7f485aa0f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48540b0d70 0x7f48541560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:56.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f485aa0f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f48540b1140 0x7f48540c4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:56.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f485aa0f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f48540b15b0 0x7f4854155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:56.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f485aa0f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4854156970 con 0x7f48540b0d70 2023-11-28T13:01:56.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f485aa0f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48540c5460 con 0x7f48540b15b0 2023-11-28T13:01:56.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f485aa0f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48540c57b0 con 0x7f48540b1140 2023-11-28T13:01:56.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f4853fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48540b0d70 0x7f48541560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:56.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f4853fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48540b0d70 0x7f48541560d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37268/0 (socket says 172.21.15.111:37268) 2023-11-28T13:01:56.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f4853fff700 1 -- 172.21.15.111:0/813877424 learned_addr learned my addr 172.21.15.111:0/813877424 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:56.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f4853fff700 1 -- 172.21.15.111:0/813877424 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f48540b1140 msgr2=0x7f48540c4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:56.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f4853fff700 1 --2- 172.21.15.111:0/813877424 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f48540b1140 0x7f48540c4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:56.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f4853fff700 1 -- 172.21.15.111:0/813877424 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f48540b15b0 msgr2=0x7f4854155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:56.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f4853fff700 1 --2- 172.21.15.111:0/813877424 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f48540b15b0 0x7f4854155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:56.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f4853fff700 1 -- 172.21.15.111:0/813877424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4844009500 con 0x7f48540b0d70 2023-11-28T13:01:56.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f4853fff700 1 --2- 172.21.15.111:0/813877424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48540b0d70 0x7f48541560d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4844009c10 tx=0x7f4844004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:56.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f4850ff9700 1 -- 172.21.15.111:0/813877424 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844011620 con 0x7f48540b0d70 2023-11-28T13:01:56.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f4850ff9700 1 -- 172.21.15.111:0/813877424 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48440117a0 con 0x7f48540b0d70 2023-11-28T13:01:56.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f4850ff9700 1 -- 172.21.15.111:0/813877424 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844011b50 con 0x7f48540b0d70 2023-11-28T13:01:56.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f485aa0f700 1 -- 172.21.15.111:0/813877424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48540af340 con 0x7f48540b0d70 2023-11-28T13:01:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.423+0000 7f485aa0f700 1 -- 172.21.15.111:0/813877424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4854161100 con 0x7f48540b0d70 2023-11-28T13:01:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.424+0000 7f485aa0f700 1 -- 172.21.15.111:0/813877424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48540cec00 con 0x7f48540b0d70 2023-11-28T13:01:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.425+0000 7f4850ff9700 1 -- 172.21.15.111:0/813877424 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f484402b070 con 0x7f48540b0d70 2023-11-28T13:01:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.425+0000 7f4850ff9700 1 --2- 172.21.15.111:0/813877424 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f48400682f0 0x7f484006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.425+0000 7f4850ff9700 1 -- 172.21.15.111:0/813877424 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4844086d50 con 0x7f48540b0d70 2023-11-28T13:01:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.425+0000 7f48537fe700 1 --2- 172.21.15.111:0/813877424 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f48400682f0 0x7f484006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.425+0000 7f48537fe700 1 --2- 172.21.15.111:0/813877424 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f48400682f0 0x7f484006a7a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f48540b52c0 tx=0x7f4848008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:56.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.429+0000 7f4850ff9700 1 -- 172.21.15.111:0/813877424 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4844053470 con 0x7f48540b0d70 2023-11-28T13:01:56.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:56.654+0000 7f485aa0f700 1 -- 172.21.15.111:0/813877424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f48540b0920 con 0x7f48540b0d70 2023-11-28T13:01:57.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.275+0000 7f4850ff9700 1 -- 172.21.15.111:0/813877424 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f48440290c0 con 0x7f48540b0d70 2023-11-28T13:01:57.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.277+0000 7f485aa0f700 1 -- 172.21.15.111:0/813877424 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f48400682f0 msgr2=0x7f484006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.277+0000 7f485aa0f700 1 --2- 172.21.15.111:0/813877424 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f48400682f0 0x7f484006a7a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f48540b52c0 tx=0x7f4848008040 comp rx=0 tx=0).stop 2023-11-28T13:01:57.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.277+0000 7f485aa0f700 1 -- 172.21.15.111:0/813877424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48540b0d70 msgr2=0x7f48541560d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.277+0000 7f485aa0f700 1 --2- 172.21.15.111:0/813877424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48540b0d70 0x7f48541560d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4844009c10 tx=0x7f4844004750 comp rx=0 tx=0).stop 2023-11-28T13:01:57.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.277+0000 7f485aa0f700 1 -- 172.21.15.111:0/813877424 shutdown_connections 2023-11-28T13:01:57.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.277+0000 7f485aa0f700 1 --2- 172.21.15.111:0/813877424 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f48400682f0 0x7f484006a7a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.277+0000 7f485aa0f700 1 --2- 172.21.15.111:0/813877424 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f48540b15b0 0x7f4854155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.277+0000 7f485aa0f700 1 --2- 172.21.15.111:0/813877424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48540b0d70 0x7f48541560d0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.277+0000 7f485aa0f700 1 --2- 172.21.15.111:0/813877424 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f48540b1140 0x7f48540c4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.277+0000 7f485aa0f700 1 -- 172.21.15.111:0/813877424 >> 172.21.15.111:0/813877424 conn(0x7f4854098ee0 msgr2=0x7f485406f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:57.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.278+0000 7f485aa0f700 1 -- 172.21.15.111:0/813877424 shutdown_connections 2023-11-28T13:01:57.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.278+0000 7f485aa0f700 1 -- 172.21.15.111:0/813877424 wait complete. 2023-11-28T13:01:57.292 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T13:01:57.292 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:01:57.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.413+0000 7f2834205700 1 Processor -- start 2023-11-28T13:01:57.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.413+0000 7f2834205700 1 -- start start 2023-11-28T13:01:57.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.414+0000 7f2834205700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f282c15d270 0x7f282c0cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.414+0000 7f2834205700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282c0cd540 con 0x7f282c0cdf10 2023-11-28T13:01:57.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.414+0000 7f2834205700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282c1597b0 con 0x7f282c0cb240 2023-11-28T13:01:57.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.414+0000 7f2834205700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f282c159a20 con 0x7f282c15d270 2023-11-28T13:01:57.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.414+0000 7f28327a2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f282c0cdf10 0x7f282c06e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38056/0 (socket says 172.21.15.111:38056) 2023-11-28T13:01:57.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.414+0000 7f28327a2700 1 -- 172.21.15.111:0/2608762439 learned_addr learned my addr 172.21.15.111:0/2608762439 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:57.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f28317a0700 1 --2- 172.21.15.111:0/2608762439 >> v2:172.21.15.149:3300/0 conn(0x7f282c15d270 0x7f282c0cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:57.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147930887 0 0) 0x7f282c0cd540 con 0x7f282c0cdf10 2023-11-28T13:01:57.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28180035b0 con 0x7f282c0cdf10 2023-11-28T13:01:57.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1415422995 0 0) 0x7f28180035b0 con 0x7f282c0cdf10 2023-11-28T13:01:57.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 >> v2:172.21.15.149:3300/0 conn(0x7f282c15d270 msgr2=0x7f282c0cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:57.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 --2- 172.21.15.111:0/2608762439 >> v2:172.21.15.149:3300/0 conn(0x7f282c15d270 0x7f282c0cd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 >> v1:172.21.15.149:6789/0 conn(0x7f282c0cb240 legacy=0x7f282c1590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c159d00 con 0x7f282c0cdf10 2023-11-28T13:01:57.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f28240019f0 con 0x7f282c0cdf10 2023-11-28T13:01:57.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 >> v1:172.21.15.111:6789/0 conn(0x7f282c0cdf10 legacy=0x7f282c06e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 --2- 172.21.15.111:0/2608762439 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2818003a00 0x7f2818003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 --2- 172.21.15.111:0/2608762439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28180043f0 0x7f28180048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 --2- 172.21.15.111:0/2608762439 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2818005150 0x7f281800cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282c159a20 con 0x7f28180043f0 2023-11-28T13:01:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281800d190 con 0x7f2818003a00 2023-11-28T13:01:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2818005a60 con 0x7f2818005150 2023-11-28T13:01:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f28317a0700 1 --2- 172.21.15.111:0/2608762439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28180043f0 0x7f28180048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f2831fa1700 1 --2- 172.21.15.111:0/2608762439 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2818003a00 0x7f2818003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f28327a2700 1 --2- 172.21.15.111:0/2608762439 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2818005150 0x7f281800cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f28317a0700 1 -- 172.21.15.111:0/2608762439 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2818005150 msgr2=0x7f281800cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f28317a0700 1 --2- 172.21.15.111:0/2608762439 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2818005150 0x7f281800cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f28317a0700 1 -- 172.21.15.111:0/2608762439 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2818003a00 msgr2=0x7f2818003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f28317a0700 1 --2- 172.21.15.111:0/2608762439 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2818003a00 0x7f2818003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f28327a2700 1 -- 172.21.15.111:0/2608762439 reap_dead start 2023-11-28T13:01:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f28317a0700 1 -- 172.21.15.111:0/2608762439 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28180035b0 con 0x7f28180043f0 2023-11-28T13:01:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.415+0000 7f28317a0700 1 --2- 172.21.15.111:0/2608762439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28180043f0 0x7f28180048b0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2828009b60 tx=0x7f2818005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da6b4cf40b6807d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:57.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.416+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282800e6c0 con 0x7f28180043f0 2023-11-28T13:01:57.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.416+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282800e820 con 0x7f28180043f0 2023-11-28T13:01:57.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.416+0000 7f2830f9f700 1 -- 172.21.15.111:0/2608762439 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282800ebb0 con 0x7f28180043f0 2023-11-28T13:01:57.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.416+0000 7f2834205700 1 -- 172.21.15.111:0/2608762439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28180043f0 msgr2=0x7f28180048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.416+0000 7f2834205700 1 --2- 172.21.15.111:0/2608762439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28180043f0 0x7f28180048b0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2828009b60 tx=0x7f2818005880 comp rx=0 tx=0).stop 2023-11-28T13:01:57.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.416+0000 7f2834205700 1 -- 172.21.15.111:0/2608762439 shutdown_connections 2023-11-28T13:01:57.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.416+0000 7f2834205700 1 --2- 172.21.15.111:0/2608762439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28180043f0 0x7f28180048b0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.416+0000 7f2834205700 1 -- 172.21.15.111:0/2608762439 >> 172.21.15.111:0/2608762439 conn(0x7f282c0b0db0 msgr2=0x7f282c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:57.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.416+0000 7f2834205700 1 -- 172.21.15.111:0/2608762439 shutdown_connections 2023-11-28T13:01:57.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.416+0000 7f2834205700 1 -- 172.21.15.111:0/2608762439 wait complete. 2023-11-28T13:01:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.418+0000 7f2834205700 1 Processor -- start 2023-11-28T13:01:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.418+0000 7f2834205700 1 -- start start 2023-11-28T13:01:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2834205700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28180043f0 0x7f282c15bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2834205700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f282c154e20 0x7f282c1f0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2834205700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f282c144510 0x7f282c144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2834205700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282c0c9010 con 0x7f28180043f0 2023-11-28T13:01:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2834205700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282c144ea0 con 0x7f282c144510 2023-11-28T13:01:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2834205700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282c0c6390 con 0x7f282c154e20 2023-11-28T13:01:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2831fa1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28180043f0 0x7f282c15bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2831fa1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28180043f0 0x7f282c15bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37286/0 (socket says 172.21.15.111:37286) 2023-11-28T13:01:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2831fa1700 1 -- 172.21.15.111:0/2065145832 learned_addr learned my addr 172.21.15.111:0/2065145832 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f28327a2700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f282c144510 0x7f282c144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f28317a0700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f282c154e20 0x7f282c1f0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2831fa1700 1 -- 172.21.15.111:0/2065145832 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f282c154e20 msgr2=0x7f282c1f0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2831fa1700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f282c154e20 0x7f282c1f0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2831fa1700 1 -- 172.21.15.111:0/2065145832 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f282c144510 msgr2=0x7f282c144960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2831fa1700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f282c144510 0x7f282c144960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2831fa1700 1 -- 172.21.15.111:0/2065145832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c0cd1d0 con 0x7f28180043f0 2023-11-28T13:01:57.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f28317a0700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f282c154e20 0x7f282c1f0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:57.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2831fa1700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28180043f0 0x7f282c15bfc0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f281c00f7d0 tx=0x7f281c00fb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:57.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f28227fc700 1 -- 172.21.15.111:0/2065145832 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c0052e0 con 0x7f28180043f0 2023-11-28T13:01:57.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f28227fc700 1 -- 172.21.15.111:0/2065145832 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281c005460 con 0x7f28180043f0 2023-11-28T13:01:57.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f28227fc700 1 -- 172.21.15.111:0/2065145832 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c0038c0 con 0x7f28180043f0 2023-11-28T13:01:57.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2834205700 1 -- 172.21.15.111:0/2065145832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282c159fe0 con 0x7f28180043f0 2023-11-28T13:01:57.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.419+0000 7f2834205700 1 -- 172.21.15.111:0/2065145832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282c0cd9d0 con 0x7f28180043f0 2023-11-28T13:01:57.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.420+0000 7f2834205700 1 -- 172.21.15.111:0/2065145832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f282c0aeb90 con 0x7f28180043f0 2023-11-28T13:01:57.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.421+0000 7f28227fc700 1 -- 172.21.15.111:0/2065145832 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f281c003a20 con 0x7f28180043f0 2023-11-28T13:01:57.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.421+0000 7f28227fc700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28100682f0 0x7f281006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.421+0000 7f28227fc700 1 -- 172.21.15.111:0/2065145832 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f281c085d10 con 0x7f28180043f0 2023-11-28T13:01:57.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.421+0000 7f28317a0700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28100682f0 0x7f281006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:57.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.421+0000 7f28317a0700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28100682f0 0x7f281006a7a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f282c153e50 tx=0x7f2828008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:57.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.425+0000 7f28227fc700 1 -- 172.21.15.111:0/2065145832 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f281c052430 con 0x7f28180043f0 2023-11-28T13:01:57.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.652+0000 7f2834205700 1 -- 172.21.15.111:0/2065145832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f282c0cb8c0 con 0x7f28180043f0 2023-11-28T13:01:57.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.653+0000 7f28227fc700 1 -- 172.21.15.111:0/2065145832 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4659 (secure 0 0 0) 0x7f281c055a50 con 0x7f28180043f0 2023-11-28T13:01:57.654 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:57.655 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/1668572397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1668572397},{"type":"v1","addr":"172.21.15.149:6834","nonce":1668572397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5755,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1129092621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1129092621},{"type":"v1","addr":"172.21.15.149:6835","nonce":1129092621}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/904588136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":904588136},{"type":"v1","addr":"172.21.15.111:6836","nonce":904588136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:01:54.257638+0000","modified":"2023-11-28T13:01:57.268879+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/1897250522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1897250522},{"type":"v1","addr":"172.21.15.111:6837","nonce":1897250522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T13:01:57.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.655+0000 7f2834205700 1 -- 172.21.15.111:0/2065145832 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28100682f0 msgr2=0x7f281006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.655+0000 7f2834205700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28100682f0 0x7f281006a7a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f282c153e50 tx=0x7f2828008030 comp rx=0 tx=0).stop 2023-11-28T13:01:57.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.655+0000 7f2834205700 1 -- 172.21.15.111:0/2065145832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28180043f0 msgr2=0x7f282c15bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.655+0000 7f2834205700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28180043f0 0x7f282c15bfc0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f281c00f7d0 tx=0x7f281c00fb90 comp rx=0 tx=0).stop 2023-11-28T13:01:57.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.655+0000 7f2834205700 1 -- 172.21.15.111:0/2065145832 shutdown_connections 2023-11-28T13:01:57.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.655+0000 7f2834205700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f28100682f0 0x7f281006a7a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.655+0000 7f2834205700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f282c144510 0x7f282c144960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.655+0000 7f2834205700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28180043f0 0x7f282c15bfc0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.655+0000 7f2834205700 1 --2- 172.21.15.111:0/2065145832 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f282c154e20 0x7f282c1f0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.655+0000 7f2834205700 1 -- 172.21.15.111:0/2065145832 >> 172.21.15.111:0/2065145832 conn(0x7f282c0b0db0 msgr2=0x7f282c06a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:57.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.655+0000 7f2834205700 1 -- 172.21.15.111:0/2065145832 shutdown_connections 2023-11-28T13:01:57.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.655+0000 7f2834205700 1 -- 172.21.15.111:0/2065145832 wait complete. 2023-11-28T13:01:57.658 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 32 2023-11-28T13:01:57.670 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:01:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.788+0000 7f43b98d5700 1 Processor -- start 2023-11-28T13:01:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.788+0000 7f43b98d5700 1 -- start start 2023-11-28T13:01:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.789+0000 7f43b98d5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43b415cad0 0x7f43b40af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.789+0000 7f43b98d5700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f43b40b0b10 0x7f43b40ca230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.789+0000 7f43b98d5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b40cc1a0 con 0x7f43b40ccf50 2023-11-28T13:01:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.789+0000 7f43b98d5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b40b0270 con 0x7f43b415cad0 2023-11-28T13:01:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.789+0000 7f43b98d5700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b40cf400 con 0x7f43b40b0b10 2023-11-28T13:01:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.789+0000 7f43b2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43b415cad0 0x7f43b40af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.789+0000 7f43b2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43b415cad0 0x7f43b40af8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37294/0 (socket says 172.21.15.111:37294) 2023-11-28T13:01:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.789+0000 7f43b2ffd700 1 -- 172.21.15.111:0/725335500 learned_addr learned my addr 172.21.15.111:0/725335500 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.789+0000 7f43b27fc700 1 --1- 172.21.15.111:0/725335500 >> v1:172.21.15.111:6789/0 conn(0x7f43b40ccf50 0x7f43b40b0400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38072/0 (socket says 172.21.15.111:38072) 2023-11-28T13:01:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b37fe700 1 --2- 172.21.15.111:0/725335500 >> v2:172.21.15.149:3300/0 conn(0x7f43b40b0b10 0x7f43b40ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b2ffd700 1 -- 172.21.15.111:0/725335500 >> v2:172.21.15.149:3300/0 conn(0x7f43b40b0b10 msgr2=0x7f43b40ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b2ffd700 1 --2- 172.21.15.111:0/725335500 >> v2:172.21.15.149:3300/0 conn(0x7f43b40b0b10 0x7f43b40ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b2ffd700 1 -- 172.21.15.111:0/725335500 >> v1:172.21.15.111:6789/0 conn(0x7f43b40ccf50 legacy=0x7f43b40b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b1ffb700 1 -- 172.21.15.111:0/725335500 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239813313 0 0) 0x7f43b40cc1a0 con 0x7f43b40ccf50 2023-11-28T13:01:57.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b2ffd700 1 -- 172.21.15.111:0/725335500 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b40ce860 con 0x7f43b415cad0 2023-11-28T13:01:57.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b37fe700 1 --2- 172.21.15.111:0/725335500 >> v2:172.21.15.149:3300/0 conn(0x7f43b40b0b10 0x7f43b40ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:57.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b2ffd700 1 --2- 172.21.15.111:0/725335500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43b415cad0 0x7f43b40af8f0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f43b408a320 tx=0x7f43a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe79d292e5fbeea8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:57.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b1ffb700 1 -- 172.21.15.111:0/725335500 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a400b870 con 0x7f43b415cad0 2023-11-28T13:01:57.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b1ffb700 1 -- 172.21.15.111:0/725335500 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a400b9d0 con 0x7f43b415cad0 2023-11-28T13:01:57.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b1ffb700 1 -- 172.21.15.111:0/725335500 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a400bd60 con 0x7f43b415cad0 2023-11-28T13:01:57.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b98d5700 1 -- 172.21.15.111:0/725335500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43b415cad0 msgr2=0x7f43b40af8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b98d5700 1 --2- 172.21.15.111:0/725335500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43b415cad0 0x7f43b40af8f0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f43b408a320 tx=0x7f43a4009bd0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b98d5700 1 -- 172.21.15.111:0/725335500 shutdown_connections 2023-11-28T13:01:57.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b98d5700 1 --2- 172.21.15.111:0/725335500 >> v2:172.21.15.149:3300/0 conn(0x7f43b40b0b10 0x7f43b40ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b98d5700 1 --2- 172.21.15.111:0/725335500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43b415cad0 0x7f43b40af8f0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.790+0000 7f43b98d5700 1 -- 172.21.15.111:0/725335500 >> 172.21.15.111:0/725335500 conn(0x7f43b406bca0 msgr2=0x7f43b40cfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:57.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.791+0000 7f43b98d5700 1 -- 172.21.15.111:0/725335500 shutdown_connections 2023-11-28T13:01:57.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.791+0000 7f43b98d5700 1 -- 172.21.15.111:0/725335500 wait complete. 2023-11-28T13:01:57.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.792+0000 7f43b98d5700 1 Processor -- start 2023-11-28T13:01:57.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.792+0000 7f43b98d5700 1 -- start start 2023-11-28T13:01:57.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b98d5700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f43b40b0b10 0x7f43b41f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b98d5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43b40ccf50 0x7f43b4157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b98d5700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f43b415cad0 0x7f43b40c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b98d5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b41f2170 con 0x7f43b40ccf50 2023-11-28T13:01:57.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b98d5700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40c3a40 con 0x7f43b415cad0 2023-11-28T13:01:57.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b98d5700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40c3d20 con 0x7f43b40b0b10 2023-11-28T13:01:57.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43b40ccf50 0x7f43b4157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:57.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43b40ccf50 0x7f43b4157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37300/0 (socket says 172.21.15.111:37300) 2023-11-28T13:01:57.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b27fc700 1 -- 172.21.15.111:0/1285314094 learned_addr learned my addr 172.21.15.111:0/1285314094 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:57.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b27fc700 1 -- 172.21.15.111:0/1285314094 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f43b40b0b10 msgr2=0x7f43b41f1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b27fc700 1 --2- 172.21.15.111:0/1285314094 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f43b40b0b10 0x7f43b41f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b27fc700 1 -- 172.21.15.111:0/1285314094 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f43b415cad0 msgr2=0x7f43b40c59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b27fc700 1 --2- 172.21.15.111:0/1285314094 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f43b415cad0 0x7f43b40c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b27fc700 1 -- 172.21.15.111:0/1285314094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a4009550 con 0x7f43b40ccf50 2023-11-28T13:01:57.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b27fc700 1 --2- 172.21.15.111:0/1285314094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43b40ccf50 0x7f43b4157f60 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f43a80011e0 tx=0x7f43a800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:57.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f439bfff700 1 -- 172.21.15.111:0/1285314094 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8004c80 con 0x7f43b40ccf50 2023-11-28T13:01:57.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f439bfff700 1 -- 172.21.15.111:0/1285314094 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a8004e00 con 0x7f43b40ccf50 2023-11-28T13:01:57.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b98d5700 1 -- 172.21.15.111:0/1285314094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b40cead0 con 0x7f43b40ccf50 2023-11-28T13:01:57.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f439bfff700 1 -- 172.21.15.111:0/1285314094 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8007310 con 0x7f43b40ccf50 2023-11-28T13:01:57.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.793+0000 7f43b98d5700 1 -- 172.21.15.111:0/1285314094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b415cea0 con 0x7f43b40ccf50 2023-11-28T13:01:57.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.794+0000 7f43b98d5700 1 -- 172.21.15.111:0/1285314094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b40cedb0 con 0x7f43b40ccf50 2023-11-28T13:01:57.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.795+0000 7f439bfff700 1 -- 172.21.15.111:0/1285314094 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f43a801a020 con 0x7f43b40ccf50 2023-11-28T13:01:57.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.795+0000 7f439bfff700 1 --2- 172.21.15.111:0/1285314094 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f43a00682f0 0x7f43a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:57.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.795+0000 7f439bfff700 1 -- 172.21.15.111:0/1285314094 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f43a8085a20 con 0x7f43b40ccf50 2023-11-28T13:01:57.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.795+0000 7f43b2ffd700 1 --2- 172.21.15.111:0/1285314094 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f43a00682f0 0x7f43a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:57.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.795+0000 7f43b2ffd700 1 --2- 172.21.15.111:0/1285314094 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f43a00682f0 0x7f43a006a7a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f43b41e7070 tx=0x7f43a4010000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:57.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.799+0000 7f439bfff700 1 -- 172.21.15.111:0/1285314094 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f43a8052140 con 0x7f43b40ccf50 2023-11-28T13:01:57.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.983+0000 7f43b98d5700 1 -- 172.21.15.111:0/1285314094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43b4067170 con 0x7f43b40ccf50 2023-11-28T13:01:57.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.984+0000 7f439bfff700 1 -- 172.21.15.111:0/1285314094 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+18231 (secure 0 0 0) 0x7f43a8055760 con 0x7f43b40ccf50 2023-11-28T13:01:57.985 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:57.986 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":67,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:01:55.259297+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:01:51.934388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-28T13:01:53.219817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:01:57.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.986+0000 7f43b98d5700 1 -- 172.21.15.111:0/1285314094 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f43a00682f0 msgr2=0x7f43a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.986+0000 7f43b98d5700 1 --2- 172.21.15.111:0/1285314094 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f43a00682f0 0x7f43a006a7a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f43b41e7070 tx=0x7f43a4010000 comp rx=0 tx=0).stop 2023-11-28T13:01:57.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.986+0000 7f43b98d5700 1 -- 172.21.15.111:0/1285314094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43b40ccf50 msgr2=0x7f43b4157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:57.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.986+0000 7f43b98d5700 1 --2- 172.21.15.111:0/1285314094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43b40ccf50 0x7f43b4157f60 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f43a80011e0 tx=0x7f43a800cae0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.986+0000 7f43b98d5700 1 -- 172.21.15.111:0/1285314094 shutdown_connections 2023-11-28T13:01:57.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.986+0000 7f43b98d5700 1 --2- 172.21.15.111:0/1285314094 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f43a00682f0 0x7f43a006a7a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.986+0000 7f43b98d5700 1 --2- 172.21.15.111:0/1285314094 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f43b415cad0 0x7f43b40c59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.986+0000 7f43b98d5700 1 --2- 172.21.15.111:0/1285314094 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f43b40b0b10 0x7f43b41f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.986+0000 7f43b98d5700 1 --2- 172.21.15.111:0/1285314094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43b40ccf50 0x7f43b4157f60 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:57.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.986+0000 7f43b98d5700 1 -- 172.21.15.111:0/1285314094 >> 172.21.15.111:0/1285314094 conn(0x7f43b406bca0 msgr2=0x7f43b406f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:57.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.986+0000 7f43b98d5700 1 -- 172.21.15.111:0/1285314094 shutdown_connections 2023-11-28T13:01:57.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:57.986+0000 7f43b98d5700 1 -- 172.21.15.111:0/1285314094 wait complete. 2023-11-28T13:01:58.000 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T13:01:58.000 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:01:58.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.118+0000 7f8b9cd09700 1 Processor -- start 2023-11-28T13:01:58.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.118+0000 7f8b9cd09700 1 -- start start 2023-11-28T13:01:58.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9cd09700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f8b9815d6e0 0x7f8b98164360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9cd09700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9815de70 con 0x7f8b981596e0 2023-11-28T13:01:58.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9cd09700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b980ced10 con 0x7f8b98159310 2023-11-28T13:01:58.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9cd09700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b980ceff0 con 0x7f8b9815d6e0 2023-11-28T13:01:58.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b95d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8b981596e0 0x7f8b9815cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38086/0 (socket says 172.21.15.111:38086) 2023-11-28T13:01:58.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b95d9b700 1 -- 172.21.15.111:0/541252781 learned_addr learned my addr 172.21.15.111:0/541252781 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b96d9d700 1 --2- 172.21.15.111:0/541252781 >> v2:172.21.15.149:3300/0 conn(0x7f8b9815d6e0 0x7f8b98164360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018004036 0 0) 0x7f8b9815de70 con 0x7f8b981596e0 2023-11-28T13:01:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b800035b0 con 0x7f8b981596e0 2023-11-28T13:01:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277540072 0 0) 0x7f8b980ced10 con 0x7f8b98159310 2023-11-28T13:01:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b9815de70 con 0x7f8b98159310 2023-11-28T13:01:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2397602035 0 0) 0x7f8b800035b0 con 0x7f8b981596e0 2023-11-28T13:01:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 >> v2:172.21.15.149:3300/0 conn(0x7f8b9815d6e0 msgr2=0x7f8b98164360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 --2- 172.21.15.111:0/541252781 >> v2:172.21.15.149:3300/0 conn(0x7f8b9815d6e0 0x7f8b98164360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 >> v1:172.21.15.149:6789/0 conn(0x7f8b98159310 legacy=0x7f8b980ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b980cf2d0 con 0x7f8b981596e0 2023-11-28T13:01:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f8b88001670 con 0x7f8b981596e0 2023-11-28T13:01:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 >> v1:172.21.15.111:6789/0 conn(0x7f8b981596e0 legacy=0x7f8b9815cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 --2- 172.21.15.111:0/541252781 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b80003be0 0x7f8b80003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 --2- 172.21.15.111:0/541252781 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b80004850 0x7f8b80004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 --2- 172.21.15.111:0/541252781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80005200 0x7f8b8000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980ceff0 con 0x7f8b80005200 2023-11-28T13:01:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8000c0e0 con 0x7f8b80004850 2023-11-28T13:01:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.120+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8000c270 con 0x7f8b80003be0 2023-11-28T13:01:58.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b96d9d700 1 --2- 172.21.15.111:0/541252781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80005200 0x7f8b8000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b95d9b700 1 --2- 172.21.15.111:0/541252781 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b80004850 0x7f8b80004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b9659c700 1 --2- 172.21.15.111:0/541252781 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b80003be0 0x7f8b80003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b96d9d700 1 -- 172.21.15.111:0/541252781 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b80003be0 msgr2=0x7f8b80003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b96d9d700 1 --2- 172.21.15.111:0/541252781 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b80003be0 0x7f8b80003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b96d9d700 1 -- 172.21.15.111:0/541252781 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b80004850 msgr2=0x7f8b80004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b96d9d700 1 --2- 172.21.15.111:0/541252781 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b80004850 0x7f8b80004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b96d9d700 1 -- 172.21.15.111:0/541252781 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b800035b0 con 0x7f8b80005200 2023-11-28T13:01:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b9659c700 1 --2- 172.21.15.111:0/541252781 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b80003be0 0x7f8b80003fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:01:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b96d9d700 1 -- 172.21.15.111:0/541252781 reap_dead start 2023-11-28T13:01:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b96d9d700 1 --2- 172.21.15.111:0/541252781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80005200 0x7f8b8000ba10 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8b9815dad0 tx=0x7f8b80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48901e7cb943d684 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:58.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b90003c20 con 0x7f8b80005200 2023-11-28T13:01:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9000f040 con 0x7f8b80005200 2023-11-28T13:01:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b9559a700 1 -- 172.21.15.111:0/541252781 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9000e9d0 con 0x7f8b80005200 2023-11-28T13:01:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b9cd09700 1 -- 172.21.15.111:0/541252781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80005200 msgr2=0x7f8b8000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.121+0000 7f8b9cd09700 1 --2- 172.21.15.111:0/541252781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80005200 0x7f8b8000ba10 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8b9815dad0 tx=0x7f8b80005880 comp rx=0 tx=0).stop 2023-11-28T13:01:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.122+0000 7f8b9cd09700 1 -- 172.21.15.111:0/541252781 shutdown_connections 2023-11-28T13:01:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.122+0000 7f8b9cd09700 1 --2- 172.21.15.111:0/541252781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80005200 0x7f8b8000ba10 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.122+0000 7f8b9cd09700 1 -- 172.21.15.111:0/541252781 >> 172.21.15.111:0/541252781 conn(0x7f8b9806db10 msgr2=0x7f8b9806df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.122+0000 7f8b9cd09700 1 -- 172.21.15.111:0/541252781 shutdown_connections 2023-11-28T13:01:58.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.122+0000 7f8b9cd09700 1 -- 172.21.15.111:0/541252781 wait complete. 2023-11-28T13:01:58.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.123+0000 7f8b9cd09700 1 Processor -- start 2023-11-28T13:01:58.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.123+0000 7f8b9cd09700 1 -- start start 2023-11-28T13:01:58.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.124+0000 7f8b9cd09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80005200 0x7f8b980c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.124+0000 7f8b9cd09700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b9814c020 0x7f8b981ee630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.124+0000 7f8b9cd09700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b980cb7f0 0x7f8b980c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.124+0000 7f8b9cd09700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980c4aa0 con 0x7f8b80005200 2023-11-28T13:01:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.124+0000 7f8b9cd09700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b981eeb70 con 0x7f8b980cb7f0 2023-11-28T13:01:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.124+0000 7f8b9cd09700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b980c98e0 con 0x7f8b9814c020 2023-11-28T13:01:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.124+0000 7f8b9659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80005200 0x7f8b980c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.124+0000 7f8b9659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80005200 0x7f8b980c9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37318/0 (socket says 172.21.15.111:37318) 2023-11-28T13:01:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.124+0000 7f8b9659c700 1 -- 172.21.15.111:0/1379506838 learned_addr learned my addr 172.21.15.111:0/1379506838 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.125+0000 7f8b9659c700 1 -- 172.21.15.111:0/1379506838 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b9814c020 msgr2=0x7f8b981ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.125+0000 7f8b9659c700 1 --2- 172.21.15.111:0/1379506838 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b9814c020 0x7f8b981ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.125+0000 7f8b9659c700 1 -- 172.21.15.111:0/1379506838 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b980cb7f0 msgr2=0x7f8b980c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.125+0000 7f8b96d9d700 1 --2- 172.21.15.111:0/1379506838 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b980cb7f0 0x7f8b980c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.125+0000 7f8b9659c700 1 --2- 172.21.15.111:0/1379506838 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b980cb7f0 0x7f8b980c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.125+0000 7f8b9659c700 1 -- 172.21.15.111:0/1379506838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b980cb3d0 con 0x7f8b80005200 2023-11-28T13:01:58.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.125+0000 7f8b9659c700 1 --2- 172.21.15.111:0/1379506838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80005200 0x7f8b980c9200 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00e7b0 tx=0x7f8b8c00eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.125+0000 7f8b86ffd700 1 -- 172.21.15.111:0/1379506838 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c017070 con 0x7f8b80005200 2023-11-28T13:01:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.125+0000 7f8b86ffd700 1 -- 172.21.15.111:0/1379506838 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8c0052e0 con 0x7f8b80005200 2023-11-28T13:01:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.125+0000 7f8b9cd09700 1 -- 172.21.15.111:0/1379506838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b980cb0f0 con 0x7f8b80005200 2023-11-28T13:01:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.125+0000 7f8b86ffd700 1 -- 172.21.15.111:0/1379506838 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c005450 con 0x7f8b80005200 2023-11-28T13:01:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.125+0000 7f8b9cd09700 1 -- 172.21.15.111:0/1379506838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9815cfd0 con 0x7f8b80005200 2023-11-28T13:01:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.126+0000 7f8b9cd09700 1 -- 172.21.15.111:0/1379506838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9806c4d0 con 0x7f8b80005200 2023-11-28T13:01:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.126+0000 7f8b86ffd700 1 -- 172.21.15.111:0/1379506838 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8b8c006860 con 0x7f8b80005200 2023-11-28T13:01:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.127+0000 7f8b86ffd700 1 --2- 172.21.15.111:0/1379506838 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8b780682f0 0x7f8b7806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.127+0000 7f8b95d9b700 1 --2- 172.21.15.111:0/1379506838 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8b780682f0 0x7f8b7806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.127+0000 7f8b86ffd700 1 -- 172.21.15.111:0/1379506838 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f8b8c085620 con 0x7f8b80005200 2023-11-28T13:01:58.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.127+0000 7f8b95d9b700 1 --2- 172.21.15.111:0/1379506838 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8b780682f0 0x7f8b7806a7a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8b88001050 tx=0x7f8b88001150 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:58.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.131+0000 7f8b86ffd700 1 -- 172.21.15.111:0/1379506838 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b8c051d40 con 0x7f8b80005200 2023-11-28T13:01:58.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.314+0000 7f8b9cd09700 1 -- 172.21.15.111:0/1379506838 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8b98067170 con 0x7f8b780682f0 2023-11-28T13:01:58.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.323+0000 7f8b86ffd700 1 -- 172.21.15.111:0/1379506838 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+308876 (secure 0 0 0) 0x7f8b98067170 con 0x7f8b780682f0 2023-11-28T13:01:58.328 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.326+0000 7f8b9cd09700 1 -- 172.21.15.111:0/1379506838 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8b780682f0 msgr2=0x7f8b7806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.326+0000 7f8b9cd09700 1 --2- 172.21.15.111:0/1379506838 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8b780682f0 0x7f8b7806a7a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8b88001050 tx=0x7f8b88001150 comp rx=0 tx=0).stop 2023-11-28T13:01:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.326+0000 7f8b9cd09700 1 -- 172.21.15.111:0/1379506838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80005200 msgr2=0x7f8b980c9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.326+0000 7f8b9cd09700 1 --2- 172.21.15.111:0/1379506838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80005200 0x7f8b980c9200 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00e7b0 tx=0x7f8b8c00eb70 comp rx=0 tx=0).stop 2023-11-28T13:01:58.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.326+0000 7f8b9cd09700 1 -- 172.21.15.111:0/1379506838 shutdown_connections 2023-11-28T13:01:58.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.326+0000 7f8b9cd09700 1 --2- 172.21.15.111:0/1379506838 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8b780682f0 0x7f8b7806a7a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.326+0000 7f8b9cd09700 1 --2- 172.21.15.111:0/1379506838 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b980cb7f0 0x7f8b980c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.326+0000 7f8b9cd09700 1 --2- 172.21.15.111:0/1379506838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80005200 0x7f8b980c9200 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.326+0000 7f8b9cd09700 1 --2- 172.21.15.111:0/1379506838 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b9814c020 0x7f8b981ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.326+0000 7f8b9cd09700 1 -- 172.21.15.111:0/1379506838 >> 172.21.15.111:0/1379506838 conn(0x7f8b9806db10 msgr2=0x7f8b980cde10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:58.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.326+0000 7f8b9cd09700 1 -- 172.21.15.111:0/1379506838 shutdown_connections 2023-11-28T13:01:58.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.326+0000 7f8b9cd09700 1 -- 172.21.15.111:0/1379506838 wait complete. 2023-11-28T13:01:58.331 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:01:58.340 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2023-11-28T13:01:56.373881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":170,"acting":170,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":102,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":80012,"kb_used_data":7436,"kb_used_omap":0,"kb_used_meta":72512,"kb_avail":754894708,"statfs":{"total":773094113280,"available":773012180992,"internally_reserved":0,"allocated":7614464,"data_stored":4161255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":74252288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000783"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956076+0000","last_change":"2023-11-28T13:01:54.280039+0000","last_active":"2023-11-28T13:01:55.956076+0000","last_peered":"2023-11-28T13:01:55.956076+0000","last_clean":"2023-11-28T13:01:55.956076+0000","last_became_active":"2023-11-28T13:01:54.279714+0000","last_became_peered":"2023-11-28T13:01:54.279714+0000","last_unstale":"2023-11-28T13:01:55.956076+0000","last_undegraded":"2023-11-28T13:01:55.956076+0000","last_fullsized":"2023-11-28T13:01:55.956076+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:32:15.867118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.275127+0000","last_change":"2023-11-28T13:01:54.286015+0000","last_active":"2023-11-28T13:01:55.275127+0000","last_peered":"2023-11-28T13:01:55.275127+0000","last_clean":"2023-11-28T13:01:55.275127+0000","last_became_active":"2023-11-28T13:01:54.285718+0000","last_became_peered":"2023-11-28T13:01:54.285718+0000","last_unstale":"2023-11-28T13:01:55.275127+0000","last_undegraded":"2023-11-28T13:01:55.275127+0000","last_fullsized":"2023-11-28T13:01:55.275127+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:56:02.698142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956065+0000","last_change":"2023-11-28T13:01:54.284664+0000","last_active":"2023-11-28T13:01:55.956065+0000","last_peered":"2023-11-28T13:01:55.956065+0000","last_clean":"2023-11-28T13:01:55.956065+0000","last_became_active":"2023-11-28T13:01:54.284054+0000","last_became_peered":"2023-11-28T13:01:54.284054+0000","last_unstale":"2023-11-28T13:01:55.956065+0000","last_undegraded":"2023-11-28T13:01:55.956065+0000","last_fullsized":"2023-11-28T13:01:55.956065+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:16.913565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268484+0000","last_change":"2023-11-28T13:01:54.280657+0000","last_active":"2023-11-28T13:01:55.268484+0000","last_peered":"2023-11-28T13:01:55.268484+0000","last_clean":"2023-11-28T13:01:55.268484+0000","last_became_active":"2023-11-28T13:01:54.280429+0000","last_became_peered":"2023-11-28T13:01:54.280429+0000","last_unstale":"2023-11-28T13:01:55.268484+0000","last_undegraded":"2023-11-28T13:01:55.268484+0000","last_fullsized":"2023-11-28T13:01:55.268484+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:29:16.982538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955888+0000","last_change":"2023-11-28T13:01:54.281825+0000","last_active":"2023-11-28T13:01:55.955888+0000","last_peered":"2023-11-28T13:01:55.955888+0000","last_clean":"2023-11-28T13:01:55.955888+0000","last_became_active":"2023-11-28T13:01:54.281590+0000","last_became_peered":"2023-11-28T13:01:54.281590+0000","last_unstale":"2023-11-28T13:01:55.955888+0000","last_undegraded":"2023-11-28T13:01:55.955888+0000","last_fullsized":"2023-11-28T13:01:55.955888+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:38:55.471135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274431+0000","last_change":"2023-11-28T13:01:54.280760+0000","last_active":"2023-11-28T13:01:55.274431+0000","last_peered":"2023-11-28T13:01:55.274431+0000","last_clean":"2023-11-28T13:01:55.274431+0000","last_became_active":"2023-11-28T13:01:54.280443+0000","last_became_peered":"2023-11-28T13:01:54.280443+0000","last_unstale":"2023-11-28T13:01:55.274431+0000","last_undegraded":"2023-11-28T13:01:55.274431+0000","last_fullsized":"2023-11-28T13:01:55.274431+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:31.867332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274290+0000","last_change":"2023-11-28T13:01:54.279041+0000","last_active":"2023-11-28T13:01:55.274290+0000","last_peered":"2023-11-28T13:01:55.274290+0000","last_clean":"2023-11-28T13:01:55.274290+0000","last_became_active":"2023-11-28T13:01:54.278770+0000","last_became_peered":"2023-11-28T13:01:54.278770+0000","last_unstale":"2023-11-28T13:01:55.274290+0000","last_undegraded":"2023-11-28T13:01:55.274290+0000","last_fullsized":"2023-11-28T13:01:55.274290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:57.250241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956188+0000","last_change":"2023-11-28T13:01:54.287203+0000","last_active":"2023-11-28T13:01:55.956188+0000","last_peered":"2023-11-28T13:01:55.956188+0000","last_clean":"2023-11-28T13:01:55.956188+0000","last_became_active":"2023-11-28T13:01:54.286706+0000","last_became_peered":"2023-11-28T13:01:54.286706+0000","last_unstale":"2023-11-28T13:01:55.956188+0000","last_undegraded":"2023-11-28T13:01:55.956188+0000","last_fullsized":"2023-11-28T13:01:55.956188+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:54:17.490750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273778+0000","last_change":"2023-11-28T13:01:54.275650+0000","last_active":"2023-11-28T13:01:55.273778+0000","last_peered":"2023-11-28T13:01:55.273778+0000","last_clean":"2023-11-28T13:01:55.273778+0000","last_became_active":"2023-11-28T13:01:54.275362+0000","last_became_peered":"2023-11-28T13:01:54.275362+0000","last_unstale":"2023-11-28T13:01:55.273778+0000","last_undegraded":"2023-11-28T13:01:55.273778+0000","last_fullsized":"2023-11-28T13:01:55.273778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:16:34.789034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956019+0000","last_change":"2023-11-28T13:01:54.286696+0000","last_active":"2023-11-28T13:01:55.956019+0000","last_peered":"2023-11-28T13:01:55.956019+0000","last_clean":"2023-11-28T13:01:55.956019+0000","last_became_active":"2023-11-28T13:01:54.286353+0000","last_became_peered":"2023-11-28T13:01:54.286353+0000","last_unstale":"2023-11-28T13:01:55.956019+0000","last_undegraded":"2023-11-28T13:01:55.956019+0000","last_fullsized":"2023-11-28T13:01:55.956019+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:01:09.680935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268559+0000","last_change":"2023-11-28T13:01:54.278686+0000","last_active":"2023-11-28T13:01:55.268559+0000","last_peered":"2023-11-28T13:01:55.268559+0000","last_clean":"2023-11-28T13:01:55.268559+0000","last_became_active":"2023-11-28T13:01:54.278095+0000","last_became_peered":"2023-11-28T13:01:54.278095+0000","last_unstale":"2023-11-28T13:01:55.268559+0000","last_undegraded":"2023-11-28T13:01:55.268559+0000","last_fullsized":"2023-11-28T13:01:55.268559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:02:20.978981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956529+0000","last_change":"2023-11-28T13:01:54.270310+0000","last_active":"2023-11-28T13:01:55.956529+0000","last_peered":"2023-11-28T13:01:55.956529+0000","last_clean":"2023-11-28T13:01:55.956529+0000","last_became_active":"2023-11-28T13:01:54.270020+0000","last_became_peered":"2023-11-28T13:01:54.270020+0000","last_unstale":"2023-11-28T13:01:55.956529+0000","last_undegraded":"2023-11-28T13:01:55.956529+0000","last_fullsized":"2023-11-28T13:01:55.956529+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:02.659205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273582+0000","last_change":"2023-11-28T13:01:54.281074+0000","last_active":"2023-11-28T13:01:55.273582+0000","last_peered":"2023-11-28T13:01:55.273582+0000","last_clean":"2023-11-28T13:01:55.273582+0000","last_became_active":"2023-11-28T13:01:54.280806+0000","last_became_peered":"2023-11-28T13:01:54.280806+0000","last_unstale":"2023-11-28T13:01:55.273582+0000","last_undegraded":"2023-11-28T13:01:55.273582+0000","last_fullsized":"2023-11-28T13:01:55.273582+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:33:48.936825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956324+0000","last_change":"2023-11-28T13:01:54.287083+0000","last_active":"2023-11-28T13:01:55.956324+0000","last_peered":"2023-11-28T13:01:55.956324+0000","last_clean":"2023-11-28T13:01:55.956324+0000","last_became_active":"2023-11-28T13:01:54.286355+0000","last_became_peered":"2023-11-28T13:01:54.286355+0000","last_unstale":"2023-11-28T13:01:55.956324+0000","last_undegraded":"2023-11-28T13:01:55.956324+0000","last_fullsized":"2023-11-28T13:01:55.956324+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:39:59.303634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268698+0000","last_change":"2023-11-28T13:01:54.280923+0000","last_active":"2023-11-28T13:01:55.268698+0000","last_peered":"2023-11-28T13:01:55.268698+0000","last_clean":"2023-11-28T13:01:55.268698+0000","last_became_active":"2023-11-28T13:01:54.280638+0000","last_became_peered":"2023-11-28T13:01:54.280638+0000","last_unstale":"2023-11-28T13:01:55.268698+0000","last_undegraded":"2023-11-28T13:01:55.268698+0000","last_fullsized":"2023-11-28T13:01:55.268698+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:22:44.391296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274574+0000","last_change":"2023-11-28T13:01:54.280414+0000","last_active":"2023-11-28T13:01:55.274574+0000","last_peered":"2023-11-28T13:01:55.274574+0000","last_clean":"2023-11-28T13:01:55.274574+0000","last_became_active":"2023-11-28T13:01:54.280146+0000","last_became_peered":"2023-11-28T13:01:54.280146+0000","last_unstale":"2023-11-28T13:01:55.274574+0000","last_undegraded":"2023-11-28T13:01:55.274574+0000","last_fullsized":"2023-11-28T13:01:55.274574+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:23:02.851709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273860+0000","last_change":"2023-11-28T13:01:54.275383+0000","last_active":"2023-11-28T13:01:55.273860+0000","last_peered":"2023-11-28T13:01:55.273860+0000","last_clean":"2023-11-28T13:01:55.273860+0000","last_became_active":"2023-11-28T13:01:54.275105+0000","last_became_peered":"2023-11-28T13:01:54.275105+0000","last_unstale":"2023-11-28T13:01:55.273860+0000","last_undegraded":"2023-11-28T13:01:55.273860+0000","last_fullsized":"2023-11-28T13:01:55.273860+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:37:43.785010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274009+0000","last_change":"2023-11-28T13:01:54.279015+0000","last_active":"2023-11-28T13:01:55.274009+0000","last_peered":"2023-11-28T13:01:55.274009+0000","last_clean":"2023-11-28T13:01:55.274009+0000","last_became_active":"2023-11-28T13:01:54.275856+0000","last_became_peered":"2023-11-28T13:01:54.275856+0000","last_unstale":"2023-11-28T13:01:55.274009+0000","last_undegraded":"2023-11-28T13:01:55.274009+0000","last_fullsized":"2023-11-28T13:01:55.274009+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:17:45.542317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955697+0000","last_change":"2023-11-28T13:01:54.280892+0000","last_active":"2023-11-28T13:01:55.955697+0000","last_peered":"2023-11-28T13:01:55.955697+0000","last_clean":"2023-11-28T13:01:55.955697+0000","last_became_active":"2023-11-28T13:01:54.280475+0000","last_became_peered":"2023-11-28T13:01:54.280475+0000","last_unstale":"2023-11-28T13:01:55.955697+0000","last_undegraded":"2023-11-28T13:01:55.955697+0000","last_fullsized":"2023-11-28T13:01:55.955697+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:56:02.698142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955665+0000","last_change":"2023-11-28T13:01:54.287592+0000","last_active":"2023-11-28T13:01:55.955665+0000","last_peered":"2023-11-28T13:01:55.955665+0000","last_clean":"2023-11-28T13:01:55.955665+0000","last_became_active":"2023-11-28T13:01:54.287272+0000","last_became_peered":"2023-11-28T13:01:54.287272+0000","last_unstale":"2023-11-28T13:01:55.955665+0000","last_undegraded":"2023-11-28T13:01:55.955665+0000","last_fullsized":"2023-11-28T13:01:55.955665+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:58:01.162914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273998+0000","last_change":"2023-11-28T13:01:54.282031+0000","last_active":"2023-11-28T13:01:55.273998+0000","last_peered":"2023-11-28T13:01:55.273998+0000","last_clean":"2023-11-28T13:01:55.273998+0000","last_became_active":"2023-11-28T13:01:54.281740+0000","last_became_peered":"2023-11-28T13:01:54.281740+0000","last_unstale":"2023-11-28T13:01:55.273998+0000","last_undegraded":"2023-11-28T13:01:55.273998+0000","last_fullsized":"2023-11-28T13:01:55.273998+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:58:58.810011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956154+0000","last_change":"2023-11-28T13:01:54.282120+0000","last_active":"2023-11-28T13:01:55.956154+0000","last_peered":"2023-11-28T13:01:55.956154+0000","last_clean":"2023-11-28T13:01:55.956154+0000","last_became_active":"2023-11-28T13:01:54.281881+0000","last_became_peered":"2023-11-28T13:01:54.281881+0000","last_unstale":"2023-11-28T13:01:55.956154+0000","last_undegraded":"2023-11-28T13:01:55.956154+0000","last_fullsized":"2023-11-28T13:01:55.956154+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:30:13.266788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273847+0000","last_change":"2023-11-28T13:01:54.275902+0000","last_active":"2023-11-28T13:01:55.273847+0000","last_peered":"2023-11-28T13:01:55.273847+0000","last_clean":"2023-11-28T13:01:55.273847+0000","last_became_active":"2023-11-28T13:01:54.275095+0000","last_became_peered":"2023-11-28T13:01:54.275095+0000","last_unstale":"2023-11-28T13:01:55.273847+0000","last_undegraded":"2023-11-28T13:01:55.273847+0000","last_fullsized":"2023-11-28T13:01:55.273847+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:52.775628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955680+0000","last_change":"2023-11-28T13:01:54.284584+0000","last_active":"2023-11-28T13:01:55.955680+0000","last_peered":"2023-11-28T13:01:55.955680+0000","last_clean":"2023-11-28T13:01:55.955680+0000","last_became_active":"2023-11-28T13:01:54.284265+0000","last_became_peered":"2023-11-28T13:01:54.284265+0000","last_unstale":"2023-11-28T13:01:55.955680+0000","last_undegraded":"2023-11-28T13:01:55.955680+0000","last_fullsized":"2023-11-28T13:01:55.955680+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:46:34.088559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274294+0000","last_change":"2023-11-28T13:01:54.285770+0000","last_active":"2023-11-28T13:01:55.274294+0000","last_peered":"2023-11-28T13:01:55.274294+0000","last_clean":"2023-11-28T13:01:55.274294+0000","last_became_active":"2023-11-28T13:01:54.285413+0000","last_became_peered":"2023-11-28T13:01:54.285413+0000","last_unstale":"2023-11-28T13:01:55.274294+0000","last_undegraded":"2023-11-28T13:01:55.274294+0000","last_fullsized":"2023-11-28T13:01:55.274294+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:01.236307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956061+0000","last_change":"2023-11-28T13:01:54.286136+0000","last_active":"2023-11-28T13:01:55.956061+0000","last_peered":"2023-11-28T13:01:55.956061+0000","last_clean":"2023-11-28T13:01:55.956061+0000","last_became_active":"2023-11-28T13:01:54.283990+0000","last_became_peered":"2023-11-28T13:01:54.283990+0000","last_unstale":"2023-11-28T13:01:55.956061+0000","last_undegraded":"2023-11-28T13:01:55.956061+0000","last_fullsized":"2023-11-28T13:01:55.956061+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:12:01.448285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"66'4","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956279+0000","last_change":"2023-11-28T13:01:53.884721+0000","last_active":"2023-11-28T13:01:55.956279+0000","last_peered":"2023-11-28T13:01:55.956279+0000","last_clean":"2023-11-28T13:01:55.956279+0000","last_became_active":"2023-11-28T13:01:53.884417+0000","last_became_peered":"2023-11-28T13:01:53.884417+0000","last_unstale":"2023-11-28T13:01:55.956279+0000","last_undegraded":"2023-11-28T13:01:55.956279+0000","last_fullsized":"2023-11-28T13:01:55.956279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:30:42.052145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956417+0000","last_change":"2023-11-28T13:01:54.286937+0000","last_active":"2023-11-28T13:01:55.956417+0000","last_peered":"2023-11-28T13:01:55.956417+0000","last_clean":"2023-11-28T13:01:55.956417+0000","last_became_active":"2023-11-28T13:01:54.285911+0000","last_became_peered":"2023-11-28T13:01:54.285911+0000","last_unstale":"2023-11-28T13:01:55.956417+0000","last_undegraded":"2023-11-28T13:01:55.956417+0000","last_fullsized":"2023-11-28T13:01:55.956417+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:38:22.871336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:52.849821+0000","last_change":"2023-11-28T13:01:52.849821+0000","last_active":"2023-11-28T13:01:52.849821+0000","last_peered":"2023-11-28T13:01:52.849821+0000","last_clean":"2023-11-28T13:01:52.849821+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:52.849821+0000","last_undegraded":"2023-11-28T13:01:52.849821+0000","last_fullsized":"2023-11-28T13:01:52.849821+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274751+0000","last_change":"2023-11-28T13:01:54.283009+0000","last_active":"2023-11-28T13:01:55.274751+0000","last_peered":"2023-11-28T13:01:55.274751+0000","last_clean":"2023-11-28T13:01:55.274751+0000","last_became_active":"2023-11-28T13:01:54.282754+0000","last_became_peered":"2023-11-28T13:01:54.282754+0000","last_unstale":"2023-11-28T13:01:55.274751+0000","last_undegraded":"2023-11-28T13:01:55.274751+0000","last_fullsized":"2023-11-28T13:01:55.274751+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:39:38.829935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274055+0000","last_change":"2023-11-28T13:01:53.881143+0000","last_active":"2023-11-28T13:01:55.274055+0000","last_peered":"2023-11-28T13:01:55.274055+0000","last_clean":"2023-11-28T13:01:55.274055+0000","last_became_active":"2023-11-28T13:01:53.880891+0000","last_became_peered":"2023-11-28T13:01:53.880891+0000","last_unstale":"2023-11-28T13:01:55.274055+0000","last_undegraded":"2023-11-28T13:01:55.274055+0000","last_fullsized":"2023-11-28T13:01:55.274055+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:58:00.157022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956008+0000","last_change":"2023-11-28T13:01:54.280565+0000","last_active":"2023-11-28T13:01:55.956008+0000","last_peered":"2023-11-28T13:01:55.956008+0000","last_clean":"2023-11-28T13:01:55.956008+0000","last_became_active":"2023-11-28T13:01:54.277320+0000","last_became_peered":"2023-11-28T13:01:54.277320+0000","last_unstale":"2023-11-28T13:01:55.956008+0000","last_undegraded":"2023-11-28T13:01:55.956008+0000","last_fullsized":"2023-11-28T13:01:55.956008+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:47:35.430928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274839+0000","last_change":"2023-11-28T13:01:53.878303+0000","last_active":"2023-11-28T13:01:55.274839+0000","last_peered":"2023-11-28T13:01:55.274839+0000","last_clean":"2023-11-28T13:01:55.274839+0000","last_became_active":"2023-11-28T13:01:53.878021+0000","last_became_peered":"2023-11-28T13:01:53.878021+0000","last_unstale":"2023-11-28T13:01:55.274839+0000","last_undegraded":"2023-11-28T13:01:55.274839+0000","last_fullsized":"2023-11-28T13:01:55.274839+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:29:15.976646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955778+0000","last_change":"2023-11-28T13:01:53.880904+0000","last_active":"2023-11-28T13:01:55.955778+0000","last_peered":"2023-11-28T13:01:55.955778+0000","last_clean":"2023-11-28T13:01:55.955778+0000","last_became_active":"2023-11-28T13:01:53.880663+0000","last_became_peered":"2023-11-28T13:01:53.880663+0000","last_unstale":"2023-11-28T13:01:55.955778+0000","last_undegraded":"2023-11-28T13:01:55.955778+0000","last_fullsized":"2023-11-28T13:01:55.955778+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:01.139320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273963+0000","last_change":"2023-11-28T13:01:54.282262+0000","last_active":"2023-11-28T13:01:55.273963+0000","last_peered":"2023-11-28T13:01:55.273963+0000","last_clean":"2023-11-28T13:01:55.273963+0000","last_became_active":"2023-11-28T13:01:54.281463+0000","last_became_peered":"2023-11-28T13:01:54.281463+0000","last_unstale":"2023-11-28T13:01:55.273963+0000","last_undegraded":"2023-11-28T13:01:55.273963+0000","last_fullsized":"2023-11-28T13:01:55.273963+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:40:00.568094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274457+0000","last_change":"2023-11-28T13:01:53.884348+0000","last_active":"2023-11-28T13:01:55.274457+0000","last_peered":"2023-11-28T13:01:55.274457+0000","last_clean":"2023-11-28T13:01:55.274457+0000","last_became_active":"2023-11-28T13:01:53.883985+0000","last_became_peered":"2023-11-28T13:01:53.883985+0000","last_unstale":"2023-11-28T13:01:55.274457+0000","last_undegraded":"2023-11-28T13:01:55.274457+0000","last_fullsized":"2023-11-28T13:01:55.274457+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:02:19.973090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955549+0000","last_change":"2023-11-28T13:01:54.275630+0000","last_active":"2023-11-28T13:01:55.955549+0000","last_peered":"2023-11-28T13:01:55.955549+0000","last_clean":"2023-11-28T13:01:55.955549+0000","last_became_active":"2023-11-28T13:01:54.275378+0000","last_became_peered":"2023-11-28T13:01:54.275378+0000","last_unstale":"2023-11-28T13:01:55.955549+0000","last_undegraded":"2023-11-28T13:01:55.955549+0000","last_fullsized":"2023-11-28T13:01:55.955549+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:37:20.084624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268309+0000","last_change":"2023-11-28T13:01:53.879675+0000","last_active":"2023-11-28T13:01:55.268309+0000","last_peered":"2023-11-28T13:01:55.268309+0000","last_clean":"2023-11-28T13:01:55.268309+0000","last_became_active":"2023-11-28T13:01:53.879394+0000","last_became_peered":"2023-11-28T13:01:53.879394+0000","last_unstale":"2023-11-28T13:01:55.268309+0000","last_undegraded":"2023-11-28T13:01:55.268309+0000","last_fullsized":"2023-11-28T13:01:55.268309+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:41:31.469900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955604+0000","last_change":"2023-11-28T13:01:53.875733+0000","last_active":"2023-11-28T13:01:55.955604+0000","last_peered":"2023-11-28T13:01:55.955604+0000","last_clean":"2023-11-28T13:01:55.955604+0000","last_became_active":"2023-11-28T13:01:53.873457+0000","last_became_peered":"2023-11-28T13:01:53.873457+0000","last_unstale":"2023-11-28T13:01:55.955604+0000","last_undegraded":"2023-11-28T13:01:55.955604+0000","last_fullsized":"2023-11-28T13:01:55.955604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:22:16.262115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274044+0000","last_change":"2023-11-28T13:01:53.878636+0000","last_active":"2023-11-28T13:01:55.274044+0000","last_peered":"2023-11-28T13:01:55.274044+0000","last_clean":"2023-11-28T13:01:55.274044+0000","last_became_active":"2023-11-28T13:01:53.878365+0000","last_became_peered":"2023-11-28T13:01:53.878365+0000","last_unstale":"2023-11-28T13:01:55.274044+0000","last_undegraded":"2023-11-28T13:01:55.274044+0000","last_fullsized":"2023-11-28T13:01:55.274044+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:12:00.442393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956196+0000","last_change":"2023-11-28T13:01:54.286882+0000","last_active":"2023-11-28T13:01:55.956196+0000","last_peered":"2023-11-28T13:01:55.956196+0000","last_clean":"2023-11-28T13:01:55.956196+0000","last_became_active":"2023-11-28T13:01:54.286551+0000","last_became_peered":"2023-11-28T13:01:54.286551+0000","last_unstale":"2023-11-28T13:01:55.956196+0000","last_undegraded":"2023-11-28T13:01:55.956196+0000","last_fullsized":"2023-11-28T13:01:55.956196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:51.983868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.275203+0000","last_change":"2023-11-28T13:01:53.882341+0000","last_active":"2023-11-28T13:01:55.275203+0000","last_peered":"2023-11-28T13:01:55.275203+0000","last_clean":"2023-11-28T13:01:55.275203+0000","last_became_active":"2023-11-28T13:01:53.881524+0000","last_became_peered":"2023-11-28T13:01:53.881524+0000","last_unstale":"2023-11-28T13:01:55.275203+0000","last_undegraded":"2023-11-28T13:01:55.275203+0000","last_fullsized":"2023-11-28T13:01:55.275203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:03.973023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274787+0000","last_change":"2023-11-28T13:01:54.279194+0000","last_active":"2023-11-28T13:01:55.274787+0000","last_peered":"2023-11-28T13:01:55.274787+0000","last_clean":"2023-11-28T13:01:55.274787+0000","last_became_active":"2023-11-28T13:01:54.278893+0000","last_became_peered":"2023-11-28T13:01:54.278893+0000","last_unstale":"2023-11-28T13:01:55.274787+0000","last_undegraded":"2023-11-28T13:01:55.274787+0000","last_fullsized":"2023-11-28T13:01:55.274787+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:51:38.510175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"66'3","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956247+0000","last_change":"2023-11-28T13:01:53.879482+0000","last_active":"2023-11-28T13:01:55.956247+0000","last_peered":"2023-11-28T13:01:55.956247+0000","last_clean":"2023-11-28T13:01:55.956247+0000","last_became_active":"2023-11-28T13:01:53.879223+0000","last_became_peered":"2023-11-28T13:01:53.879223+0000","last_unstale":"2023-11-28T13:01:55.956247+0000","last_undegraded":"2023-11-28T13:01:55.956247+0000","last_fullsized":"2023-11-28T13:01:55.956247+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:19:04.317034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956080+0000","last_change":"2023-11-28T13:01:54.274271+0000","last_active":"2023-11-28T13:01:55.956080+0000","last_peered":"2023-11-28T13:01:55.956080+0000","last_clean":"2023-11-28T13:01:55.956080+0000","last_became_active":"2023-11-28T13:01:54.274013+0000","last_became_peered":"2023-11-28T13:01:54.274013+0000","last_unstale":"2023-11-28T13:01:55.956080+0000","last_undegraded":"2023-11-28T13:01:55.956080+0000","last_fullsized":"2023-11-28T13:01:55.956080+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:27:13.629149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"66'6","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268610+0000","last_change":"2023-11-28T13:01:53.871891+0000","last_active":"2023-11-28T13:01:55.268610+0000","last_peered":"2023-11-28T13:01:55.268610+0000","last_clean":"2023-11-28T13:01:55.268610+0000","last_became_active":"2023-11-28T13:01:53.871640+0000","last_became_peered":"2023-11-28T13:01:53.871640+0000","last_unstale":"2023-11-28T13:01:55.268610+0000","last_undegraded":"2023-11-28T13:01:55.268610+0000","last_fullsized":"2023-11-28T13:01:55.268610+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:38:21.865445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955739+0000","last_change":"2023-11-28T13:01:54.284067+0000","last_active":"2023-11-28T13:01:55.955739+0000","last_peered":"2023-11-28T13:01:55.955739+0000","last_clean":"2023-11-28T13:01:55.955739+0000","last_became_active":"2023-11-28T13:01:54.281833+0000","last_became_peered":"2023-11-28T13:01:54.281833+0000","last_unstale":"2023-11-28T13:01:55.955739+0000","last_undegraded":"2023-11-28T13:01:55.955739+0000","last_fullsized":"2023-11-28T13:01:55.955739+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:01.236307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268528+0000","last_change":"2023-11-28T13:01:53.880138+0000","last_active":"2023-11-28T13:01:55.268528+0000","last_peered":"2023-11-28T13:01:55.268528+0000","last_clean":"2023-11-28T13:01:55.268528+0000","last_became_active":"2023-11-28T13:01:53.879852+0000","last_became_peered":"2023-11-28T13:01:53.879852+0000","last_unstale":"2023-11-28T13:01:55.268528+0000","last_undegraded":"2023-11-28T13:01:55.268528+0000","last_fullsized":"2023-11-28T13:01:55.268528+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:38:54.465243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274554+0000","last_change":"2023-11-28T13:01:53.883625+0000","last_active":"2023-11-28T13:01:55.274554+0000","last_peered":"2023-11-28T13:01:55.274554+0000","last_clean":"2023-11-28T13:01:55.274554+0000","last_became_active":"2023-11-28T13:01:53.883326+0000","last_became_peered":"2023-11-28T13:01:53.883326+0000","last_unstale":"2023-11-28T13:01:55.274554+0000","last_undegraded":"2023-11-28T13:01:55.274554+0000","last_fullsized":"2023-11-28T13:01:55.274554+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:30:56.435654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:52.849821+0000","last_change":"2023-11-28T13:01:52.849821+0000","last_active":"2023-11-28T13:01:52.849821+0000","last_peered":"2023-11-28T13:01:52.849821+0000","last_clean":"2023-11-28T13:01:52.849821+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:52.849821+0000","last_undegraded":"2023-11-28T13:01:52.849821+0000","last_fullsized":"2023-11-28T13:01:52.849821+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955921+0000","last_change":"2023-11-28T13:01:54.287297+0000","last_active":"2023-11-28T13:01:55.955921+0000","last_peered":"2023-11-28T13:01:55.955921+0000","last_clean":"2023-11-28T13:01:55.955921+0000","last_became_active":"2023-11-28T13:01:54.287054+0000","last_became_peered":"2023-11-28T13:01:54.287054+0000","last_unstale":"2023-11-28T13:01:55.955921+0000","last_undegraded":"2023-11-28T13:01:55.955921+0000","last_fullsized":"2023-11-28T13:01:55.955921+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:40:12.276019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"66'8","reported_seq":19,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273708+0000","last_change":"2023-11-28T13:01:53.873547+0000","last_active":"2023-11-28T13:01:55.273708+0000","last_peered":"2023-11-28T13:01:55.273708+0000","last_clean":"2023-11-28T13:01:55.273708+0000","last_became_active":"2023-11-28T13:01:53.873205+0000","last_became_peered":"2023-11-28T13:01:53.873205+0000","last_unstale":"2023-11-28T13:01:55.273708+0000","last_undegraded":"2023-11-28T13:01:55.273708+0000","last_fullsized":"2023-11-28T13:01:55.273708+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:50.977977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.7","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.275349+0000","last_change":"2023-11-28T13:01:53.881135+0000","last_active":"2023-11-28T13:01:55.275349+0000","last_peered":"2023-11-28T13:01:55.275349+0000","last_clean":"2023-11-28T13:01:55.275349+0000","last_became_active":"2023-11-28T13:01:53.880764+0000","last_became_peered":"2023-11-28T13:01:53.880764+0000","last_unstale":"2023-11-28T13:01:55.275349+0000","last_undegraded":"2023-11-28T13:01:55.275349+0000","last_fullsized":"2023-11-28T13:01:55.275349+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:39:58.297742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:53.855713+0000","last_change":"2023-11-28T13:01:53.855713+0000","last_active":"2023-11-28T13:01:53.855713+0000","last_peered":"2023-11-28T13:01:53.855713+0000","last_clean":"2023-11-28T13:01:53.855713+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:53.855713+0000","last_undegraded":"2023-11-28T13:01:53.855713+0000","last_fullsized":"2023-11-28T13:01:53.855713+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:52.849821+0000","last_change":"2023-11-28T13:01:52.849821+0000","last_active":"2023-11-28T13:01:52.849821+0000","last_peered":"2023-11-28T13:01:52.849821+0000","last_clean":"2023-11-28T13:01:52.849821+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:52.849821+0000","last_undegraded":"2023-11-28T13:01:52.849821+0000","last_fullsized":"2023-11-28T13:01:52.849821+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955937+0000","last_change":"2023-11-28T13:01:54.286766+0000","last_active":"2023-11-28T13:01:55.955937+0000","last_peered":"2023-11-28T13:01:55.955937+0000","last_clean":"2023-11-28T13:01:55.955937+0000","last_became_active":"2023-11-28T13:01:54.284229+0000","last_became_peered":"2023-11-28T13:01:54.284229+0000","last_unstale":"2023-11-28T13:01:55.955937+0000","last_undegraded":"2023-11-28T13:01:55.955937+0000","last_fullsized":"2023-11-28T13:01:55.955937+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:33:46.554110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956286+0000","last_change":"2023-11-28T13:01:53.882782+0000","last_active":"2023-11-28T13:01:55.956286+0000","last_peered":"2023-11-28T13:01:55.956286+0000","last_clean":"2023-11-28T13:01:55.956286+0000","last_became_active":"2023-11-28T13:01:53.882444+0000","last_became_peered":"2023-11-28T13:01:53.882444+0000","last_unstale":"2023-11-28T13:01:55.956286+0000","last_undegraded":"2023-11-28T13:01:55.956286+0000","last_fullsized":"2023-11-28T13:01:55.956286+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:24:40.479670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956129+0000","last_change":"2023-11-28T13:01:54.281150+0000","last_active":"2023-11-28T13:01:55.956129+0000","last_peered":"2023-11-28T13:01:55.956129+0000","last_clean":"2023-11-28T13:01:55.956129+0000","last_became_active":"2023-11-28T13:01:54.280841+0000","last_became_peered":"2023-11-28T13:01:54.280841+0000","last_unstale":"2023-11-28T13:01:55.956129+0000","last_undegraded":"2023-11-28T13:01:55.956129+0000","last_fullsized":"2023-11-28T13:01:55.956129+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:02.145212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:52.849821+0000","last_change":"2023-11-28T13:01:52.849821+0000","last_active":"2023-11-28T13:01:52.849821+0000","last_peered":"2023-11-28T13:01:52.849821+0000","last_clean":"2023-11-28T13:01:52.849821+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:52.849821+0000","last_undegraded":"2023-11-28T13:01:52.849821+0000","last_fullsized":"2023-11-28T13:01:52.849821+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956618+0000","last_change":"2023-11-28T13:01:54.284724+0000","last_active":"2023-11-28T13:01:55.956618+0000","last_peered":"2023-11-28T13:01:55.956618+0000","last_clean":"2023-11-28T13:01:55.956618+0000","last_became_active":"2023-11-28T13:01:54.284398+0000","last_became_peered":"2023-11-28T13:01:54.284398+0000","last_unstale":"2023-11-28T13:01:55.956618+0000","last_undegraded":"2023-11-28T13:01:55.956618+0000","last_fullsized":"2023-11-28T13:01:55.956618+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:41:32.475792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955980+0000","last_change":"2023-11-28T13:01:53.870766+0000","last_active":"2023-11-28T13:01:55.955980+0000","last_peered":"2023-11-28T13:01:55.955980+0000","last_clean":"2023-11-28T13:01:55.955980+0000","last_became_active":"2023-11-28T13:01:53.870523+0000","last_became_peered":"2023-11-28T13:01:53.870523+0000","last_unstale":"2023-11-28T13:01:55.955980+0000","last_undegraded":"2023-11-28T13:01:55.955980+0000","last_fullsized":"2023-11-28T13:01:55.955980+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:43:04.853464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274622+0000","last_change":"2023-11-28T13:01:54.284458+0000","last_active":"2023-11-28T13:01:55.274622+0000","last_peered":"2023-11-28T13:01:55.274622+0000","last_clean":"2023-11-28T13:01:55.274622+0000","last_became_active":"2023-11-28T13:01:54.284178+0000","last_became_peered":"2023-11-28T13:01:54.284178+0000","last_unstale":"2023-11-28T13:01:55.274622+0000","last_undegraded":"2023-11-28T13:01:55.274622+0000","last_fullsized":"2023-11-28T13:01:55.274622+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:59.890096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268336+0000","last_change":"2023-11-28T13:01:53.879719+0000","last_active":"2023-11-28T13:01:55.268336+0000","last_peered":"2023-11-28T13:01:55.268336+0000","last_clean":"2023-11-28T13:01:55.268336+0000","last_became_active":"2023-11-28T13:01:53.879492+0000","last_became_peered":"2023-11-28T13:01:53.879492+0000","last_unstale":"2023-11-28T13:01:55.268336+0000","last_undegraded":"2023-11-28T13:01:55.268336+0000","last_fullsized":"2023-11-28T13:01:55.268336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:40:11.270127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268296+0000","last_change":"2023-11-28T13:01:54.284967+0000","last_active":"2023-11-28T13:01:55.268296+0000","last_peered":"2023-11-28T13:01:55.268296+0000","last_clean":"2023-11-28T13:01:55.268296+0000","last_became_active":"2023-11-28T13:01:54.284692+0000","last_became_peered":"2023-11-28T13:01:54.284692+0000","last_unstale":"2023-11-28T13:01:55.268296+0000","last_undegraded":"2023-11-28T13:01:55.268296+0000","last_fullsized":"2023-11-28T13:01:55.268296+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:27:39.151509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273962+0000","last_change":"2023-11-28T13:01:53.884732+0000","last_active":"2023-11-28T13:01:55.273962+0000","last_peered":"2023-11-28T13:01:55.273962+0000","last_clean":"2023-11-28T13:01:55.273962+0000","last_became_active":"2023-11-28T13:01:53.884445+0000","last_became_peered":"2023-11-28T13:01:53.884445+0000","last_unstale":"2023-11-28T13:01:55.273962+0000","last_undegraded":"2023-11-28T13:01:55.273962+0000","last_fullsized":"2023-11-28T13:01:55.273962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:46:33.082668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274377+0000","last_change":"2023-11-28T13:01:54.284922+0000","last_active":"2023-11-28T13:01:55.274377+0000","last_peered":"2023-11-28T13:01:55.274377+0000","last_clean":"2023-11-28T13:01:55.274377+0000","last_became_active":"2023-11-28T13:01:54.284769+0000","last_became_peered":"2023-11-28T13:01:54.284769+0000","last_unstale":"2023-11-28T13:01:55.274377+0000","last_undegraded":"2023-11-28T13:01:55.274377+0000","last_fullsized":"2023-11-28T13:01:55.274377+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:46:40.116660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274410+0000","last_change":"2023-11-28T13:01:53.877803+0000","last_active":"2023-11-28T13:01:55.274410+0000","last_peered":"2023-11-28T13:01:55.274410+0000","last_clean":"2023-11-28T13:01:55.274410+0000","last_became_active":"2023-11-28T13:01:53.877499+0000","last_became_peered":"2023-11-28T13:01:53.877499+0000","last_unstale":"2023-11-28T13:01:55.274410+0000","last_undegraded":"2023-11-28T13:01:55.274410+0000","last_fullsized":"2023-11-28T13:01:55.274410+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:50:25.775486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956179+0000","last_change":"2023-11-28T13:01:54.271933+0000","last_active":"2023-11-28T13:01:55.956179+0000","last_peered":"2023-11-28T13:01:55.956179+0000","last_clean":"2023-11-28T13:01:55.956179+0000","last_became_active":"2023-11-28T13:01:54.271708+0000","last_became_peered":"2023-11-28T13:01:54.271708+0000","last_unstale":"2023-11-28T13:01:55.956179+0000","last_undegraded":"2023-11-28T13:01:55.956179+0000","last_fullsized":"2023-11-28T13:01:55.956179+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:16:05.380760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955936+0000","last_change":"2023-11-28T13:01:53.884480+0000","last_active":"2023-11-28T13:01:55.955936+0000","last_peered":"2023-11-28T13:01:55.955936+0000","last_clean":"2023-11-28T13:01:55.955936+0000","last_became_active":"2023-11-28T13:01:53.884116+0000","last_became_peered":"2023-11-28T13:01:53.884116+0000","last_unstale":"2023-11-28T13:01:55.955936+0000","last_undegraded":"2023-11-28T13:01:55.955936+0000","last_fullsized":"2023-11-28T13:01:55.955936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:37:12.617319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":54,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.265296+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T13:01:54.265296+0000","last_peered":"2023-11-28T13:01:54.265296+0000","last_clean":"2023-11-28T13:01:54.265296+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:01:54.265296+0000","last_undegraded":"2023-11-28T13:01:54.265296+0000","last_fullsized":"2023-11-28T13:01:54.265296+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955814+0000","last_change":"2023-11-28T13:01:54.283713+0000","last_active":"2023-11-28T13:01:55.955814+0000","last_peered":"2023-11-28T13:01:55.955814+0000","last_clean":"2023-11-28T13:01:55.955814+0000","last_became_active":"2023-11-28T13:01:54.283449+0000","last_became_peered":"2023-11-28T13:01:54.283449+0000","last_unstale":"2023-11-28T13:01:55.955814+0000","last_undegraded":"2023-11-28T13:01:55.955814+0000","last_fullsized":"2023-11-28T13:01:55.955814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:41:59.072903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274666+0000","last_change":"2023-11-28T13:01:53.883683+0000","last_active":"2023-11-28T13:01:55.274666+0000","last_peered":"2023-11-28T13:01:55.274666+0000","last_clean":"2023-11-28T13:01:55.274666+0000","last_became_active":"2023-11-28T13:01:53.883419+0000","last_became_peered":"2023-11-28T13:01:53.883419+0000","last_unstale":"2023-11-28T13:01:55.274666+0000","last_undegraded":"2023-11-28T13:01:55.274666+0000","last_fullsized":"2023-11-28T13:01:55.274666+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:27:38.145617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":47,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T13:01:48.849264+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T13:01:48.849264+0000","last_peered":"2023-11-28T13:01:48.849264+0000","last_clean":"2023-11-28T13:01:48.849264+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:01:48.849264+0000","last_undegraded":"2023-11-28T13:01:48.849264+0000","last_fullsized":"2023-11-28T13:01:48.849264+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955743+0000","last_change":"2023-11-28T13:01:54.288673+0000","last_active":"2023-11-28T13:01:55.955743+0000","last_peered":"2023-11-28T13:01:55.955743+0000","last_clean":"2023-11-28T13:01:55.955743+0000","last_became_active":"2023-11-28T13:01:54.288422+0000","last_became_peered":"2023-11-28T13:01:54.288422+0000","last_unstale":"2023-11-28T13:01:55.955743+0000","last_undegraded":"2023-11-28T13:01:55.955743+0000","last_fullsized":"2023-11-28T13:01:55.955743+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:10:48.275248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:52.849821+0000","last_change":"2023-11-28T13:01:52.849821+0000","last_active":"2023-11-28T13:01:52.849821+0000","last_peered":"2023-11-28T13:01:52.849821+0000","last_clean":"2023-11-28T13:01:52.849821+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:52.849821+0000","last_undegraded":"2023-11-28T13:01:52.849821+0000","last_fullsized":"2023-11-28T13:01:52.849821+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":50,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.266976+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T13:01:54.266976+0000","last_peered":"2023-11-28T13:01:54.266976+0000","last_clean":"2023-11-28T13:01:54.266976+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:01:54.266976+0000","last_undegraded":"2023-11-28T13:01:54.266976+0000","last_fullsized":"2023-11-28T13:01:54.266976+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955924+0000","last_change":"2023-11-28T13:01:54.288055+0000","last_active":"2023-11-28T13:01:55.955924+0000","last_peered":"2023-11-28T13:01:55.955924+0000","last_clean":"2023-11-28T13:01:55.955924+0000","last_became_active":"2023-11-28T13:01:54.287752+0000","last_became_peered":"2023-11-28T13:01:54.287752+0000","last_unstale":"2023-11-28T13:01:55.955924+0000","last_undegraded":"2023-11-28T13:01:55.955924+0000","last_fullsized":"2023-11-28T13:01:55.955924+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:24:13.908547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274714+0000","last_change":"2023-11-28T13:01:53.879480+0000","last_active":"2023-11-28T13:01:55.274714+0000","last_peered":"2023-11-28T13:01:55.274714+0000","last_clean":"2023-11-28T13:01:55.274714+0000","last_became_active":"2023-11-28T13:01:53.878450+0000","last_became_peered":"2023-11-28T13:01:53.878450+0000","last_unstale":"2023-11-28T13:01:55.274714+0000","last_undegraded":"2023-11-28T13:01:55.274714+0000","last_fullsized":"2023-11-28T13:01:55.274714+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:22:43.385404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":47,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-11-28T13:01:48.849270+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T13:01:48.849270+0000","last_peered":"2023-11-28T13:01:48.849270+0000","last_clean":"2023-11-28T13:01:48.849270+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:01:48.849270+0000","last_undegraded":"2023-11-28T13:01:48.849270+0000","last_fullsized":"2023-11-28T13:01:48.849270+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":204,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.272035+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T13:01:54.272035+0000","last_peered":"2023-11-28T13:01:54.272035+0000","last_clean":"2023-11-28T13:01:54.272035+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:01:54.272035+0000","last_undegraded":"2023-11-28T13:01:54.272035+0000","last_fullsized":"2023-11-28T13:01:54.272035+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955836+0000","last_change":"2023-11-28T13:01:54.288873+0000","last_active":"2023-11-28T13:01:55.955836+0000","last_peered":"2023-11-28T13:01:55.955836+0000","last_clean":"2023-11-28T13:01:55.955836+0000","last_became_active":"2023-11-28T13:01:54.288450+0000","last_became_peered":"2023-11-28T13:01:54.288450+0000","last_unstale":"2023-11-28T13:01:55.955836+0000","last_undegraded":"2023-11-28T13:01:55.955836+0000","last_fullsized":"2023-11-28T13:01:55.955836+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:16:36.681083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-28T13:01:52.849821+0000","last_change":"2023-11-28T13:01:52.849821+0000","last_active":"2023-11-28T13:01:52.849821+0000","last_peered":"2023-11-28T13:01:52.849821+0000","last_clean":"2023-11-28T13:01:52.849821+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T13:01:52.849821+0000","last_undegraded":"2023-11-28T13:01:52.849821+0000","last_fullsized":"2023-11-28T13:01:52.849821+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":52,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.272125+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T13:01:54.272125+0000","last_peered":"2023-11-28T13:01:54.272125+0000","last_clean":"2023-11-28T13:01:54.272125+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:01:54.272125+0000","last_undegraded":"2023-11-28T13:01:54.272125+0000","last_fullsized":"2023-11-28T13:01:54.272125+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268569+0000","last_change":"2023-11-28T13:01:54.278616+0000","last_active":"2023-11-28T13:01:55.268569+0000","last_peered":"2023-11-28T13:01:55.268569+0000","last_clean":"2023-11-28T13:01:55.268569+0000","last_became_active":"2023-11-28T13:01:54.277488+0000","last_became_peered":"2023-11-28T13:01:54.277488+0000","last_unstale":"2023-11-28T13:01:55.268569+0000","last_undegraded":"2023-11-28T13:01:55.268569+0000","last_fullsized":"2023-11-28T13:01:55.268569+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:04.978915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274669+0000","last_change":"2023-11-28T13:01:53.882980+0000","last_active":"2023-11-28T13:01:55.274669+0000","last_peered":"2023-11-28T13:01:55.274669+0000","last_clean":"2023-11-28T13:01:55.274669+0000","last_became_active":"2023-11-28T13:01:53.881456+0000","last_became_peered":"2023-11-28T13:01:53.881456+0000","last_unstale":"2023-11-28T13:01:55.274669+0000","last_undegraded":"2023-11-28T13:01:55.274669+0000","last_fullsized":"2023-11-28T13:01:55.274669+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:15:03.394013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":52,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.272898+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T13:01:54.272898+0000","last_peered":"2023-11-28T13:01:54.272898+0000","last_clean":"2023-11-28T13:01:54.272898+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:01:54.272898+0000","last_undegraded":"2023-11-28T13:01:54.272898+0000","last_fullsized":"2023-11-28T13:01:54.272898+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956178+0000","last_change":"2023-11-28T13:01:54.286797+0000","last_active":"2023-11-28T13:01:55.956178+0000","last_peered":"2023-11-28T13:01:55.956178+0000","last_clean":"2023-11-28T13:01:55.956178+0000","last_became_active":"2023-11-28T13:01:54.286220+0000","last_became_peered":"2023-11-28T13:01:54.286220+0000","last_unstale":"2023-11-28T13:01:55.956178+0000","last_undegraded":"2023-11-28T13:01:55.956178+0000","last_fullsized":"2023-11-28T13:01:55.956178+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:34.324996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274056+0000","last_change":"2023-11-28T13:01:53.880863+0000","last_active":"2023-11-28T13:01:55.274056+0000","last_peered":"2023-11-28T13:01:55.274056+0000","last_clean":"2023-11-28T13:01:55.274056+0000","last_became_active":"2023-11-28T13:01:53.880498+0000","last_became_peered":"2023-11-28T13:01:53.880498+0000","last_unstale":"2023-11-28T13:01:55.274056+0000","last_undegraded":"2023-11-28T13:01:55.274056+0000","last_fullsized":"2023-11-28T13:01:55.274056+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:01:08.675044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.265122+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T13:01:54.265122+0000","last_peered":"2023-11-28T13:01:54.265122+0000","last_clean":"2023-11-28T13:01:54.265122+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:01:54.265122+0000","last_undegraded":"2023-11-28T13:01:54.265122+0000","last_fullsized":"2023-11-28T13:01:54.265122+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273879+0000","last_change":"2023-11-28T13:01:54.279411+0000","last_active":"2023-11-28T13:01:55.273879+0000","last_peered":"2023-11-28T13:01:55.273879+0000","last_clean":"2023-11-28T13:01:55.273879+0000","last_became_active":"2023-11-28T13:01:54.275789+0000","last_became_peered":"2023-11-28T13:01:54.275789+0000","last_unstale":"2023-11-28T13:01:55.273879+0000","last_undegraded":"2023-11-28T13:01:55.273879+0000","last_fullsized":"2023-11-28T13:01:55.273879+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:32:27.408420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.275371+0000","last_change":"2023-11-28T13:01:53.880342+0000","last_active":"2023-11-28T13:01:55.275371+0000","last_peered":"2023-11-28T13:01:55.275371+0000","last_clean":"2023-11-28T13:01:55.275371+0000","last_became_active":"2023-11-28T13:01:53.880088+0000","last_became_peered":"2023-11-28T13:01:53.880088+0000","last_unstale":"2023-11-28T13:01:55.275371+0000","last_undegraded":"2023-11-28T13:01:55.275371+0000","last_fullsized":"2023-11-28T13:01:55.275371+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:38:21.865445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":60,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.267363+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T13:01:54.267363+0000","last_peered":"2023-11-28T13:01:54.267363+0000","last_clean":"2023-11-28T13:01:54.267363+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:01:54.267363+0000","last_undegraded":"2023-11-28T13:01:54.267363+0000","last_fullsized":"2023-11-28T13:01:54.267363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":100,"acting":100,"num_store_stats":6},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37,"ondisk_log_size":37,"up":52,"acting":52,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10860,"kb_used_data":1380,"kb_used_omap":0,"kb_used_meta":9472,"kb_avail":94360980,"statfs":{"total":96636764160,"available":96625643520,"internally_reserved":0,"allocated":1413120,"data_stored":971523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9699328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.629,"15min":0.66400000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82699999999999996,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.628,"15min":0.67500000000000004},"min":{"1min":0.40799999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65900000000000003,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.61599999999999999}]},{"osd":1,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52300000000000002,"15min":0.52700000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64600000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.56499999999999995}]},{"osd":2,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.54500000000000004,"15min":0.55100000000000005},"min":{"1min":0.379,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.86399999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.42599999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70199999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.626}]},{"osd":3,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54400000000000004,"15min":0.54200000000000004},"min":{"1min":0.45000000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.40699999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.59199999999999997}]},{"osd":4,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.52100000000000002},"min":{"1min":0.36799999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53200000000000003,"15min":0.53600000000000003},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.42399999999999999}]},{"osd":5,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.51200000000000001,"15min":0.52300000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.41399999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.57799999999999996}]},{"osd":6,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.56100000000000005,"15min":0.57499999999999996},"min":{"1min":0.39500000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.45300000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.64000000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9768,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":8960,"kb_avail":94362072,"statfs":{"total":96636764160,"available":96626761728,"internally_reserved":0,"allocated":819200,"data_stored":380647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9175040},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54700000000000004,"15min":0.54500000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70099999999999996,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.434},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.51300000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68500000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68500000000000005}]},{"osd":1,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52100000000000002},"min":{"1min":0.32400000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.33000000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.747,"5min":1.016,"15min":1.016},"last":0.49099999999999999}]},{"osd":2,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.52700000000000002},"min":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.63600000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.41699999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79000000000000004,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.52500000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58199999999999996,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.371,"15min":0.371},"max":{"1min":0.77700000000000002,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79200000000000004,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.39100000000000001}]},{"osd":4,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52800000000000002,"15min":0.53800000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.84199999999999997,"5min":1,"15min":1},"last":0.628},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53900000000000003,"15min":0.54300000000000004},"min":{"1min":0.378,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73099999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.61299999999999999}]},{"osd":5,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84499999999999997,"5min":1.052,"15min":1.052},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88900000000000001,"5min":1.258,"15min":1.258},"last":0.58199999999999996}]},{"osd":7,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76900000000000002,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.621,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90400000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.59799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9716,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":8896,"kb_avail":94362124,"statfs":{"total":96636764160,"available":96626814976,"internally_reserved":0,"allocated":831488,"data_stored":382703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9109504},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.32300000000000001,"15min":0.32600000000000001},"min":{"1min":0.224,"5min":0.219,"15min":0.219},"max":{"1min":0.51500000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.35699999999999998,"15min":0.35399999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.52500000000000002,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.33800000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33700000000000002,"15min":0.33300000000000002},"min":{"1min":0.254,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.48199999999999998,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.36499999999999999,"15min":0.36599999999999999},"min":{"1min":0.26600000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.48299999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.45800000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.54300000000000004,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38700000000000001,"15min":0.39200000000000002},"min":{"1min":0.26000000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.46600000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.39300000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.36199999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83599999999999997,"5min":1.028,"15min":1.028},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.45600000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85899999999999999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.56299999999999994}]},{"osd":5,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83199999999999996,"5min":1.042,"15min":1.042},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.57499999999999996,"15min":0.55900000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83699999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.69099999999999995}]},{"osd":6,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57599999999999996,"15min":0.57799999999999996},"min":{"1min":0.40600000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.76700000000000002,"5min":1.006,"15min":1.006},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.58299999999999996}]},{"osd":7,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77700000000000002,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.622},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85799999999999998,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.59399999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10804,"kb_used_data":1324,"kb_used_omap":0,"kb_used_meta":9472,"kb_avail":94361036,"statfs":{"total":96636764160,"available":96625700864,"internally_reserved":0,"allocated":1355776,"data_stored":943064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9699328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.32400000000000001,"15min":0.32200000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34200000000000003,"15min":0.34100000000000003},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.41199999999999998}]},{"osd":2,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.34999999999999998,"15min":0.34499999999999997},"min":{"1min":0.26100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36699999999999999,"15min":0.36399999999999999},"min":{"1min":0.27700000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.33600000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.37,"15min":0.372},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.55100000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.38900000000000001,"15min":0.38600000000000001},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.40000000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.69399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56699999999999995,"15min":0.56899999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.70999999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.59799999999999998}]},{"osd":5,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.432,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.754,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.5}]},{"osd":6,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.73399999999999999,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60899999999999999,"15min":0.61599999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72299999999999998,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.45200000000000001}]},{"osd":7,"last update":"Tue Nov 28 13:00:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.627,"15min":0.629},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.78600000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.621,"15min":0.628},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.71899999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.53200000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9580,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":8768,"kb_avail":94362260,"statfs":{"total":96636764160,"available":96626954240,"internally_reserved":0,"allocated":823296,"data_stored":382103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8978432},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.32600000000000001,"15min":0.316},"min":{"1min":0.22500000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.254},{"interface":"front","average":{"1min":0.379,"5min":0.34599999999999997,"15min":0.33600000000000002},"min":{"1min":0.21199999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.35499999999999998}]},{"osd":1,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38100000000000001,"15min":0.38200000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.59199999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.36699999999999999,"15min":0.35499999999999998},"min":{"1min":0.25700000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56299999999999994,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.38800000000000001}]},{"osd":3,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34499999999999997,"15min":0.33900000000000002},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.70599999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.39800000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.59899999999999998,"15min":0.59299999999999997},"min":{"1min":0.47999999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76200000000000001,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.57399999999999995,"15min":0.55800000000000005},"min":{"1min":0.42199999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.55600000000000005}]},{"osd":5,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.45700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.85499999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.60099999999999998,"15min":0.59499999999999997},"min":{"1min":0.41699999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.77100000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.59399999999999997}]},{"osd":6,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.59099999999999997,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83999999999999997,"5min":1.002,"15min":1.002},"last":0.627},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63100000000000001,"15min":0.624},"min":{"1min":0.499,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.52100000000000002}]},{"osd":7,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.628,"15min":0.63},"min":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.83599999999999997,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.441,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85899999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.57599999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9820,"kb_used_data":788,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94362020,"statfs":{"total":96636764160,"available":96626708480,"internally_reserved":0,"allocated":806912,"data_stored":373694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.33000000000000002,"15min":0.32200000000000001},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46100000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.33800000000000002,"15min":0.33000000000000002},"min":{"1min":0.24099999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.312}]},{"osd":1,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.34799999999999998,"15min":0.34000000000000002},"min":{"1min":0.25800000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.377,"15min":0.373},"min":{"1min":0.27200000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.33000000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46700000000000003,"15min":0.48699999999999999},"min":{"1min":0.252,"5min":0.251,"15min":0.251},"max":{"1min":0.59299999999999997,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.442,"5min":0.499,"15min":0.52900000000000003},"min":{"1min":0.313,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.61599999999999999,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":0.35299999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.83999999999999997,"15min":0.94199999999999995},"min":{"1min":0.439,"5min":0.437,"15min":0.437},"max":{"1min":0.92200000000000004,"5min":8.2889999999999997,"15min":8.2889999999999997},"last":0.58999999999999997}]},{"osd":5,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61599999999999999,"15min":0.61899999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.75700000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.65100000000000002,"15min":0.67100000000000004},"min":{"1min":0.47799999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88200000000000001,"5min":2.165,"15min":2.165},"last":0.53900000000000003}]},{"osd":6,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.625,"15min":0.628},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84199999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.627},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.91300000000000003,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.57399999999999995}]},{"osd":7,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61499999999999999,"15min":0.60999999999999999},"min":{"1min":0.46899999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.83799999999999997,"15min":0.93500000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89600000000000002,"5min":7.0899999999999999,"15min":7.0899999999999999},"last":0.56399999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10020,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":9216,"kb_avail":94361820,"statfs":{"total":96636764160,"available":96626503680,"internally_reserved":0,"allocated":815104,"data_stored":380625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9437184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.67700000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67100000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.58399999999999996}]},{"osd":1,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.504},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76600000000000001,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70999999999999996,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.66100000000000003}]},{"osd":2,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.44400000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71499999999999997,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.41499999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74299999999999999,"5min":1.069,"15min":1.069},"last":0.77700000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.43099999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.71299999999999997,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41099999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.74199999999999999,"5min":0.879,"15min":0.879},"last":0.68500000000000005}]},{"osd":5,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51200000000000001,"15min":0.51700000000000002},"min":{"1min":0.33400000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.67800000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.51700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70299999999999996,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.40100000000000002}]},{"osd":6,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.52300000000000002},"min":{"1min":0.35399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72999999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.371},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.57899999999999996,"15min":0.58699999999999997},"min":{"1min":0.38300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71899999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.46100000000000002}]},{"osd":7,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71699999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58999999999999997,"15min":0.59099999999999997},"min":{"1min":0.39800000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69599999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.42899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9444,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":8704,"kb_avail":94362396,"statfs":{"total":96636764160,"available":96627093504,"internally_reserved":0,"allocated":749568,"data_stored":346896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8912896},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.34599999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69899999999999995,"5min":1.819,"15min":1.819},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.52600000000000002,"15min":0.52100000000000002},"min":{"1min":0.40799999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.29099999999999998}]},{"osd":1,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73599999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.35199999999999998}]},{"osd":2,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.75800000000000001,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.63,"15min":0.64800000000000002},"min":{"1min":0.372,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.748,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.47399999999999998}]},{"osd":3,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72799999999999998,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.37},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.66300000000000003,"15min":0.68600000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71099999999999997,"5min":2.702,"15min":2.702},"last":0.48999999999999999}]},{"osd":4,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.60099999999999998,"15min":0.61899999999999999},"min":{"1min":0.30399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.66500000000000004,"5min":3.0739999999999998,"15min":3.0739999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76500000000000001,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.46000000000000002}]},{"osd":6,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.34499999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.70199999999999996,"15min":0.72699999999999998},"min":{"1min":0.39100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73099999999999998,"5min":3.7679999999999998,"15min":3.7679999999999998},"last":0.51200000000000001}]},{"osd":7,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78400000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.65500000000000003,"15min":0.66600000000000004},"min":{"1min":0.41399999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.751,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:01:58.345 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:01:58.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.498+0000 7f4e84c0c700 1 Processor -- start 2023-11-28T13:01:58.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.498+0000 7f4e84c0c700 1 -- start start 2023-11-28T13:01:58.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.499+0000 7f4e84c0c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4e800cfc90 0x7f4e800d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.499+0000 7f4e84c0c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e800d05a0 con 0x7f4e800cdef0 2023-11-28T13:01:58.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.499+0000 7f4e84c0c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e800ca260 con 0x7f4e800cc5a0 2023-11-28T13:01:58.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.499+0000 7f4e84c0c700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e800ccaa0 con 0x7f4e800cfc90 2023-11-28T13:01:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7ed9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4e800cdef0 0x7f4e800ca6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38108/0 (socket says 172.21.15.111:38108) 2023-11-28T13:01:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7ed9d700 1 -- 172.21.15.111:0/1819158131 learned_addr learned my addr 172.21.15.111:0/1819158131 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195986119 0 0) 0x7f4e78001490 con 0x7f4e800cdef0 2023-11-28T13:01:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e680035b0 con 0x7f4e800cdef0 2023-11-28T13:01:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7e59c700 1 --2- 172.21.15.111:0/1819158131 >> v2:172.21.15.149:3300/0 conn(0x7f4e800cfc90 0x7f4e800d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2581322922 0 0) 0x7f4e680035b0 con 0x7f4e800cdef0 2023-11-28T13:01:58.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 >> v2:172.21.15.149:3300/0 conn(0x7f4e800cfc90 msgr2=0x7f4e800d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:58.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 --2- 172.21.15.111:0/1819158131 >> v2:172.21.15.149:3300/0 conn(0x7f4e800cfc90 0x7f4e800d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 >> v1:172.21.15.149:6789/0 conn(0x7f4e800cc5a0 legacy=0x7f4e800cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e800ccd80 con 0x7f4e800cdef0 2023-11-28T13:01:58.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f4e78001b80 con 0x7f4e800cdef0 2023-11-28T13:01:58.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 >> v1:172.21.15.111:6789/0 conn(0x7f4e800cdef0 legacy=0x7f4e800ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 --2- 172.21.15.111:0/1819158131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e68003a00 0x7f4e68003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 --2- 172.21.15.111:0/1819158131 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e68004850 0x7f4e68004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 --2- 172.21.15.111:0/1819158131 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e68005200 0x7f4e6800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e800ccaa0 con 0x7f4e68003a00 2023-11-28T13:01:58.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6800c190 con 0x7f4e68005200 2023-11-28T13:01:58.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6800c350 con 0x7f4e68004850 2023-11-28T13:01:58.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7e59c700 1 --2- 172.21.15.111:0/1819158131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e68003a00 0x7f4e68003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7dd9b700 1 --2- 172.21.15.111:0/1819158131 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e68004850 0x7f4e68004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7ed9d700 1 --2- 172.21.15.111:0/1819158131 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e68005200 0x7f4e6800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7e59c700 1 -- 172.21.15.111:0/1819158131 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e68004850 msgr2=0x7f4e68004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.500+0000 7f4e7e59c700 1 --2- 172.21.15.111:0/1819158131 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e68004850 0x7f4e68004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e7e59c700 1 -- 172.21.15.111:0/1819158131 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e68005200 msgr2=0x7f4e6800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e7e59c700 1 --2- 172.21.15.111:0/1819158131 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e68005200 0x7f4e6800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e7e59c700 1 -- 172.21.15.111:0/1819158131 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e680035b0 con 0x7f4e68003a00 2023-11-28T13:01:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e7ed9d700 1 -- 172.21.15.111:0/1819158131 reap_dead start 2023-11-28T13:01:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e7e59c700 1 --2- 172.21.15.111:0/1819158131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e68003a00 0x7f4e68003e10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009860 tx=0x7f4e68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4fd2eade02ef40 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000c070 con 0x7f4e68003a00 2023-11-28T13:01:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e70013610 con 0x7f4e68003a00 2023-11-28T13:01:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e7d59a700 1 -- 172.21.15.111:0/1819158131 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e700139a0 con 0x7f4e68003a00 2023-11-28T13:01:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e84c0c700 1 -- 172.21.15.111:0/1819158131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e68003a00 msgr2=0x7f4e68003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e84c0c700 1 --2- 172.21.15.111:0/1819158131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e68003a00 0x7f4e68003e10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009860 tx=0x7f4e68005880 comp rx=0 tx=0).stop 2023-11-28T13:01:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e84c0c700 1 -- 172.21.15.111:0/1819158131 shutdown_connections 2023-11-28T13:01:58.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e84c0c700 1 --2- 172.21.15.111:0/1819158131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e68003a00 0x7f4e68003e10 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e84c0c700 1 -- 172.21.15.111:0/1819158131 >> 172.21.15.111:0/1819158131 conn(0x7f4e80098ee0 msgr2=0x7f4e8006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:58.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e84c0c700 1 -- 172.21.15.111:0/1819158131 shutdown_connections 2023-11-28T13:01:58.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.501+0000 7f4e84c0c700 1 -- 172.21.15.111:0/1819158131 wait complete. 2023-11-28T13:01:58.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.503+0000 7f4e84c0c700 1 Processor -- start 2023-11-28T13:01:58.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.503+0000 7f4e84c0c700 1 -- start start 2023-11-28T13:01:58.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.503+0000 7f4e84c0c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e68003a00 0x7f4e80157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.503+0000 7f4e84c0c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e800c8640 0x7f4e8006a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e84c0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e800c8ab0 0x7f4e8006a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e84c0c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e800c8fd0 con 0x7f4e800c8ab0 2023-11-28T13:01:58.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e84c0c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e800c92b0 con 0x7f4e68003a00 2023-11-28T13:01:58.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e84c0c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e800c7e40 con 0x7f4e800c8640 2023-11-28T13:01:58.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e7ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e800c8ab0 0x7f4e8006a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e7ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e800c8ab0 0x7f4e8006a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37342/0 (socket says 172.21.15.111:37342) 2023-11-28T13:01:58.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e7ed9d700 1 -- 172.21.15.111:0/352912159 learned_addr learned my addr 172.21.15.111:0/352912159 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:58.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e7e59c700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e68003a00 0x7f4e80157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e7dd9b700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e800c8640 0x7f4e8006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e7ed9d700 1 -- 172.21.15.111:0/352912159 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e800c8640 msgr2=0x7f4e8006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e7ed9d700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e800c8640 0x7f4e8006a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e7ed9d700 1 -- 172.21.15.111:0/352912159 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e68003a00 msgr2=0x7f4e80157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e7ed9d700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e68003a00 0x7f4e80157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e7ed9d700 1 -- 172.21.15.111:0/352912159 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e800cd340 con 0x7f4e800c8ab0 2023-11-28T13:01:58.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e7dd9b700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e800c8640 0x7f4e8006a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:01:58.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e7ed9d700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e800c8ab0 0x7f4e8006a7b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f4e780016f0 tx=0x7f4e78007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:58.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e6effd700 1 -- 172.21.15.111:0/352912159 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e780079d0 con 0x7f4e800c8ab0 2023-11-28T13:01:58.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e6effd700 1 -- 172.21.15.111:0/352912159 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e78007b30 con 0x7f4e800c8ab0 2023-11-28T13:01:58.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e84c0c700 1 -- 172.21.15.111:0/352912159 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e800cd060 con 0x7f4e800c8ab0 2023-11-28T13:01:58.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e6effd700 1 -- 172.21.15.111:0/352912159 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e780135f0 con 0x7f4e800c8ab0 2023-11-28T13:01:58.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.504+0000 7f4e84c0c700 1 -- 172.21.15.111:0/352912159 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e80143280 con 0x7f4e800c8ab0 2023-11-28T13:01:58.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.505+0000 7f4e84c0c700 1 -- 172.21.15.111:0/352912159 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e800c8360 con 0x7f4e800c8ab0 2023-11-28T13:01:58.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.506+0000 7f4e6effd700 1 -- 172.21.15.111:0/352912159 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4e78013750 con 0x7f4e800c8ab0 2023-11-28T13:01:58.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.506+0000 7f4e6effd700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4e600682f0 0x7f4e6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.506+0000 7f4e6effd700 1 -- 172.21.15.111:0/352912159 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4e780865d0 con 0x7f4e800c8ab0 2023-11-28T13:01:58.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.506+0000 7f4e7e59c700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4e600682f0 0x7f4e6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.507+0000 7f4e7e59c700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4e600682f0 0x7f4e6006a7a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4e70016040 tx=0x7f4e7001e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:58.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.510+0000 7f4e6effd700 1 -- 172.21.15.111:0/352912159 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4e800c8360 con 0x7f4e800c8ab0 2023-11-28T13:01:58.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.688+0000 7f4e84c0c700 1 -- 172.21.15.111:0/352912159 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4e80067a90 con 0x7f4e600682f0 2023-11-28T13:01:58.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.696+0000 7f4e6effd700 1 -- 172.21.15.111:0/352912159 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+312142 (secure 0 0 0) 0x7f4e80067a90 con 0x7f4e600682f0 2023-11-28T13:01:58.701 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:58.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.700+0000 7f4e84c0c700 1 -- 172.21.15.111:0/352912159 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4e600682f0 msgr2=0x7f4e6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.700+0000 7f4e84c0c700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4e600682f0 0x7f4e6006a7a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4e70016040 tx=0x7f4e7001e040 comp rx=0 tx=0).stop 2023-11-28T13:01:58.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.700+0000 7f4e84c0c700 1 -- 172.21.15.111:0/352912159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e800c8ab0 msgr2=0x7f4e8006a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.700+0000 7f4e84c0c700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e800c8ab0 0x7f4e8006a7b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f4e780016f0 tx=0x7f4e78007340 comp rx=0 tx=0).stop 2023-11-28T13:01:58.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.700+0000 7f4e84c0c700 1 -- 172.21.15.111:0/352912159 shutdown_connections 2023-11-28T13:01:58.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.700+0000 7f4e84c0c700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e800c8ab0 0x7f4e8006a7b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.700+0000 7f4e84c0c700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4e600682f0 0x7f4e6006a7a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.700+0000 7f4e84c0c700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4e68003a00 0x7f4e80157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.700+0000 7f4e84c0c700 1 --2- 172.21.15.111:0/352912159 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4e800c8640 0x7f4e8006a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.700+0000 7f4e84c0c700 1 -- 172.21.15.111:0/352912159 >> 172.21.15.111:0/352912159 conn(0x7f4e80098ee0 msgr2=0x7f4e8006f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:58.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.700+0000 7f4e84c0c700 1 -- 172.21.15.111:0/352912159 shutdown_connections 2023-11-28T13:01:58.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.700+0000 7f4e84c0c700 1 -- 172.21.15.111:0/352912159 wait complete. 2023-11-28T13:01:58.705 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:01:58.713 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2023-11-28T13:01:58.374425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":80368,"kb_used_data":7536,"kb_used_omap":0,"kb_used_meta":72768,"kb_avail":754894352,"statfs":{"total":773094113280,"available":773011816448,"internally_reserved":0,"allocated":7716864,"data_stored":4215890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":74514432},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.518627"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956076+0000","last_change":"2023-11-28T13:01:54.280039+0000","last_active":"2023-11-28T13:01:55.956076+0000","last_peered":"2023-11-28T13:01:55.956076+0000","last_clean":"2023-11-28T13:01:55.956076+0000","last_became_active":"2023-11-28T13:01:54.279714+0000","last_became_peered":"2023-11-28T13:01:54.279714+0000","last_unstale":"2023-11-28T13:01:55.956076+0000","last_undegraded":"2023-11-28T13:01:55.956076+0000","last_fullsized":"2023-11-28T13:01:55.956076+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:32:15.867118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.275127+0000","last_change":"2023-11-28T13:01:54.286015+0000","last_active":"2023-11-28T13:01:55.275127+0000","last_peered":"2023-11-28T13:01:55.275127+0000","last_clean":"2023-11-28T13:01:55.275127+0000","last_became_active":"2023-11-28T13:01:54.285718+0000","last_became_peered":"2023-11-28T13:01:54.285718+0000","last_unstale":"2023-11-28T13:01:55.275127+0000","last_undegraded":"2023-11-28T13:01:55.275127+0000","last_fullsized":"2023-11-28T13:01:55.275127+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:56:02.698142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956065+0000","last_change":"2023-11-28T13:01:54.284664+0000","last_active":"2023-11-28T13:01:55.956065+0000","last_peered":"2023-11-28T13:01:55.956065+0000","last_clean":"2023-11-28T13:01:55.956065+0000","last_became_active":"2023-11-28T13:01:54.284054+0000","last_became_peered":"2023-11-28T13:01:54.284054+0000","last_unstale":"2023-11-28T13:01:55.956065+0000","last_undegraded":"2023-11-28T13:01:55.956065+0000","last_fullsized":"2023-11-28T13:01:55.956065+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:16.913565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268484+0000","last_change":"2023-11-28T13:01:54.280657+0000","last_active":"2023-11-28T13:01:55.268484+0000","last_peered":"2023-11-28T13:01:55.268484+0000","last_clean":"2023-11-28T13:01:55.268484+0000","last_became_active":"2023-11-28T13:01:54.280429+0000","last_became_peered":"2023-11-28T13:01:54.280429+0000","last_unstale":"2023-11-28T13:01:55.268484+0000","last_undegraded":"2023-11-28T13:01:55.268484+0000","last_fullsized":"2023-11-28T13:01:55.268484+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:29:16.982538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955888+0000","last_change":"2023-11-28T13:01:54.281825+0000","last_active":"2023-11-28T13:01:55.955888+0000","last_peered":"2023-11-28T13:01:55.955888+0000","last_clean":"2023-11-28T13:01:55.955888+0000","last_became_active":"2023-11-28T13:01:54.281590+0000","last_became_peered":"2023-11-28T13:01:54.281590+0000","last_unstale":"2023-11-28T13:01:55.955888+0000","last_undegraded":"2023-11-28T13:01:55.955888+0000","last_fullsized":"2023-11-28T13:01:55.955888+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:38:55.471135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274431+0000","last_change":"2023-11-28T13:01:54.280760+0000","last_active":"2023-11-28T13:01:55.274431+0000","last_peered":"2023-11-28T13:01:55.274431+0000","last_clean":"2023-11-28T13:01:55.274431+0000","last_became_active":"2023-11-28T13:01:54.280443+0000","last_became_peered":"2023-11-28T13:01:54.280443+0000","last_unstale":"2023-11-28T13:01:55.274431+0000","last_undegraded":"2023-11-28T13:01:55.274431+0000","last_fullsized":"2023-11-28T13:01:55.274431+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:31.867332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.451465+0000","last_change":"2023-11-28T13:01:54.266759+0000","last_active":"2023-11-28T13:01:55.451465+0000","last_peered":"2023-11-28T13:01:55.451465+0000","last_clean":"2023-11-28T13:01:55.451465+0000","last_became_active":"2023-11-28T13:01:54.266423+0000","last_became_peered":"2023-11-28T13:01:54.266423+0000","last_unstale":"2023-11-28T13:01:55.451465+0000","last_undegraded":"2023-11-28T13:01:55.451465+0000","last_fullsized":"2023-11-28T13:01:55.451465+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:34.324996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274290+0000","last_change":"2023-11-28T13:01:54.279041+0000","last_active":"2023-11-28T13:01:55.274290+0000","last_peered":"2023-11-28T13:01:55.274290+0000","last_clean":"2023-11-28T13:01:55.274290+0000","last_became_active":"2023-11-28T13:01:54.278770+0000","last_became_peered":"2023-11-28T13:01:54.278770+0000","last_unstale":"2023-11-28T13:01:55.274290+0000","last_undegraded":"2023-11-28T13:01:55.274290+0000","last_fullsized":"2023-11-28T13:01:55.274290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:17:57.250241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956188+0000","last_change":"2023-11-28T13:01:54.287203+0000","last_active":"2023-11-28T13:01:55.956188+0000","last_peered":"2023-11-28T13:01:55.956188+0000","last_clean":"2023-11-28T13:01:55.956188+0000","last_became_active":"2023-11-28T13:01:54.286706+0000","last_became_peered":"2023-11-28T13:01:54.286706+0000","last_unstale":"2023-11-28T13:01:55.956188+0000","last_undegraded":"2023-11-28T13:01:55.956188+0000","last_fullsized":"2023-11-28T13:01:55.956188+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:54:17.490750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273778+0000","last_change":"2023-11-28T13:01:54.275650+0000","last_active":"2023-11-28T13:01:55.273778+0000","last_peered":"2023-11-28T13:01:55.273778+0000","last_clean":"2023-11-28T13:01:55.273778+0000","last_became_active":"2023-11-28T13:01:54.275362+0000","last_became_peered":"2023-11-28T13:01:54.275362+0000","last_unstale":"2023-11-28T13:01:55.273778+0000","last_undegraded":"2023-11-28T13:01:55.273778+0000","last_fullsized":"2023-11-28T13:01:55.273778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:16:34.789034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956019+0000","last_change":"2023-11-28T13:01:54.286696+0000","last_active":"2023-11-28T13:01:55.956019+0000","last_peered":"2023-11-28T13:01:55.956019+0000","last_clean":"2023-11-28T13:01:55.956019+0000","last_became_active":"2023-11-28T13:01:54.286353+0000","last_became_peered":"2023-11-28T13:01:54.286353+0000","last_unstale":"2023-11-28T13:01:55.956019+0000","last_undegraded":"2023-11-28T13:01:55.956019+0000","last_fullsized":"2023-11-28T13:01:55.956019+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:01:09.680935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268559+0000","last_change":"2023-11-28T13:01:54.278686+0000","last_active":"2023-11-28T13:01:55.268559+0000","last_peered":"2023-11-28T13:01:55.268559+0000","last_clean":"2023-11-28T13:01:55.268559+0000","last_became_active":"2023-11-28T13:01:54.278095+0000","last_became_peered":"2023-11-28T13:01:54.278095+0000","last_unstale":"2023-11-28T13:01:55.268559+0000","last_undegraded":"2023-11-28T13:01:55.268559+0000","last_fullsized":"2023-11-28T13:01:55.268559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:02:20.978981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956616+0000","last_change":"2023-11-28T13:01:54.286698+0000","last_active":"2023-11-28T13:01:55.956616+0000","last_peered":"2023-11-28T13:01:55.956616+0000","last_clean":"2023-11-28T13:01:55.956616+0000","last_became_active":"2023-11-28T13:01:54.286452+0000","last_became_peered":"2023-11-28T13:01:54.286452+0000","last_unstale":"2023-11-28T13:01:55.956616+0000","last_undegraded":"2023-11-28T13:01:55.956616+0000","last_fullsized":"2023-11-28T13:01:55.956616+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:16:36.681083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956529+0000","last_change":"2023-11-28T13:01:54.270310+0000","last_active":"2023-11-28T13:01:55.956529+0000","last_peered":"2023-11-28T13:01:55.956529+0000","last_clean":"2023-11-28T13:01:55.956529+0000","last_became_active":"2023-11-28T13:01:54.270020+0000","last_became_peered":"2023-11-28T13:01:54.270020+0000","last_unstale":"2023-11-28T13:01:55.956529+0000","last_undegraded":"2023-11-28T13:01:55.956529+0000","last_fullsized":"2023-11-28T13:01:55.956529+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:02.659205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956360+0000","last_change":"2023-11-28T13:01:54.274244+0000","last_active":"2023-11-28T13:01:55.956360+0000","last_peered":"2023-11-28T13:01:55.956360+0000","last_clean":"2023-11-28T13:01:55.956360+0000","last_became_active":"2023-11-28T13:01:54.273997+0000","last_became_peered":"2023-11-28T13:01:54.273997+0000","last_unstale":"2023-11-28T13:01:55.956360+0000","last_undegraded":"2023-11-28T13:01:55.956360+0000","last_fullsized":"2023-11-28T13:01:55.956360+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:10:48.275248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273582+0000","last_change":"2023-11-28T13:01:54.281074+0000","last_active":"2023-11-28T13:01:55.273582+0000","last_peered":"2023-11-28T13:01:55.273582+0000","last_clean":"2023-11-28T13:01:55.273582+0000","last_became_active":"2023-11-28T13:01:54.280806+0000","last_became_peered":"2023-11-28T13:01:54.280806+0000","last_unstale":"2023-11-28T13:01:55.273582+0000","last_undegraded":"2023-11-28T13:01:55.273582+0000","last_fullsized":"2023-11-28T13:01:55.273582+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:33:48.936825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956324+0000","last_change":"2023-11-28T13:01:54.287083+0000","last_active":"2023-11-28T13:01:55.956324+0000","last_peered":"2023-11-28T13:01:55.956324+0000","last_clean":"2023-11-28T13:01:55.956324+0000","last_became_active":"2023-11-28T13:01:54.286355+0000","last_became_peered":"2023-11-28T13:01:54.286355+0000","last_unstale":"2023-11-28T13:01:55.956324+0000","last_undegraded":"2023-11-28T13:01:55.956324+0000","last_fullsized":"2023-11-28T13:01:55.956324+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:39:59.303634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268698+0000","last_change":"2023-11-28T13:01:54.280923+0000","last_active":"2023-11-28T13:01:55.268698+0000","last_peered":"2023-11-28T13:01:55.268698+0000","last_clean":"2023-11-28T13:01:55.268698+0000","last_became_active":"2023-11-28T13:01:54.280638+0000","last_became_peered":"2023-11-28T13:01:54.280638+0000","last_unstale":"2023-11-28T13:01:55.268698+0000","last_undegraded":"2023-11-28T13:01:55.268698+0000","last_fullsized":"2023-11-28T13:01:55.268698+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:22:44.391296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274574+0000","last_change":"2023-11-28T13:01:54.280414+0000","last_active":"2023-11-28T13:01:55.274574+0000","last_peered":"2023-11-28T13:01:55.274574+0000","last_clean":"2023-11-28T13:01:55.274574+0000","last_became_active":"2023-11-28T13:01:54.280146+0000","last_became_peered":"2023-11-28T13:01:54.280146+0000","last_unstale":"2023-11-28T13:01:55.274574+0000","last_undegraded":"2023-11-28T13:01:55.274574+0000","last_fullsized":"2023-11-28T13:01:55.274574+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:23:02.851709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956490+0000","last_change":"2023-11-28T13:01:54.271812+0000","last_active":"2023-11-28T13:01:55.956490+0000","last_peered":"2023-11-28T13:01:55.956490+0000","last_clean":"2023-11-28T13:01:55.956490+0000","last_became_active":"2023-11-28T13:01:54.271574+0000","last_became_peered":"2023-11-28T13:01:54.271574+0000","last_unstale":"2023-11-28T13:01:55.956490+0000","last_undegraded":"2023-11-28T13:01:55.956490+0000","last_fullsized":"2023-11-28T13:01:55.956490+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:01:09.680935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273860+0000","last_change":"2023-11-28T13:01:54.275383+0000","last_active":"2023-11-28T13:01:55.273860+0000","last_peered":"2023-11-28T13:01:55.273860+0000","last_clean":"2023-11-28T13:01:55.273860+0000","last_became_active":"2023-11-28T13:01:54.275105+0000","last_became_peered":"2023-11-28T13:01:54.275105+0000","last_unstale":"2023-11-28T13:01:55.273860+0000","last_undegraded":"2023-11-28T13:01:55.273860+0000","last_fullsized":"2023-11-28T13:01:55.273860+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:37:43.785010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274009+0000","last_change":"2023-11-28T13:01:54.279015+0000","last_active":"2023-11-28T13:01:55.274009+0000","last_peered":"2023-11-28T13:01:55.274009+0000","last_clean":"2023-11-28T13:01:55.274009+0000","last_became_active":"2023-11-28T13:01:54.275856+0000","last_became_peered":"2023-11-28T13:01:54.275856+0000","last_unstale":"2023-11-28T13:01:55.274009+0000","last_undegraded":"2023-11-28T13:01:55.274009+0000","last_fullsized":"2023-11-28T13:01:55.274009+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:17:45.542317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955697+0000","last_change":"2023-11-28T13:01:54.280892+0000","last_active":"2023-11-28T13:01:55.955697+0000","last_peered":"2023-11-28T13:01:55.955697+0000","last_clean":"2023-11-28T13:01:55.955697+0000","last_became_active":"2023-11-28T13:01:54.280475+0000","last_became_peered":"2023-11-28T13:01:54.280475+0000","last_unstale":"2023-11-28T13:01:55.955697+0000","last_undegraded":"2023-11-28T13:01:55.955697+0000","last_fullsized":"2023-11-28T13:01:55.955697+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:56:02.698142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955665+0000","last_change":"2023-11-28T13:01:54.287592+0000","last_active":"2023-11-28T13:01:55.955665+0000","last_peered":"2023-11-28T13:01:55.955665+0000","last_clean":"2023-11-28T13:01:55.955665+0000","last_became_active":"2023-11-28T13:01:54.287272+0000","last_became_peered":"2023-11-28T13:01:54.287272+0000","last_unstale":"2023-11-28T13:01:55.955665+0000","last_undegraded":"2023-11-28T13:01:55.955665+0000","last_fullsized":"2023-11-28T13:01:55.955665+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:58:01.162914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273998+0000","last_change":"2023-11-28T13:01:54.282031+0000","last_active":"2023-11-28T13:01:55.273998+0000","last_peered":"2023-11-28T13:01:55.273998+0000","last_clean":"2023-11-28T13:01:55.273998+0000","last_became_active":"2023-11-28T13:01:54.281740+0000","last_became_peered":"2023-11-28T13:01:54.281740+0000","last_unstale":"2023-11-28T13:01:55.273998+0000","last_undegraded":"2023-11-28T13:01:55.273998+0000","last_fullsized":"2023-11-28T13:01:55.273998+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:58:58.810011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.451880+0000","last_change":"2023-11-28T13:01:54.278602+0000","last_active":"2023-11-28T13:01:55.451880+0000","last_peered":"2023-11-28T13:01:55.451880+0000","last_clean":"2023-11-28T13:01:55.451880+0000","last_became_active":"2023-11-28T13:01:54.278375+0000","last_became_peered":"2023-11-28T13:01:54.278375+0000","last_unstale":"2023-11-28T13:01:55.451880+0000","last_undegraded":"2023-11-28T13:01:55.451880+0000","last_fullsized":"2023-11-28T13:01:55.451880+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:12:01.448285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956154+0000","last_change":"2023-11-28T13:01:54.282120+0000","last_active":"2023-11-28T13:01:55.956154+0000","last_peered":"2023-11-28T13:01:55.956154+0000","last_clean":"2023-11-28T13:01:55.956154+0000","last_became_active":"2023-11-28T13:01:54.281881+0000","last_became_peered":"2023-11-28T13:01:54.281881+0000","last_unstale":"2023-11-28T13:01:55.956154+0000","last_undegraded":"2023-11-28T13:01:55.956154+0000","last_fullsized":"2023-11-28T13:01:55.956154+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:30:13.266788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273847+0000","last_change":"2023-11-28T13:01:54.275902+0000","last_active":"2023-11-28T13:01:55.273847+0000","last_peered":"2023-11-28T13:01:55.273847+0000","last_clean":"2023-11-28T13:01:55.273847+0000","last_became_active":"2023-11-28T13:01:54.275095+0000","last_became_peered":"2023-11-28T13:01:54.275095+0000","last_unstale":"2023-11-28T13:01:55.273847+0000","last_undegraded":"2023-11-28T13:01:55.273847+0000","last_fullsized":"2023-11-28T13:01:55.273847+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:52.775628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955680+0000","last_change":"2023-11-28T13:01:54.284584+0000","last_active":"2023-11-28T13:01:55.955680+0000","last_peered":"2023-11-28T13:01:55.955680+0000","last_clean":"2023-11-28T13:01:55.955680+0000","last_became_active":"2023-11-28T13:01:54.284265+0000","last_became_peered":"2023-11-28T13:01:54.284265+0000","last_unstale":"2023-11-28T13:01:55.955680+0000","last_undegraded":"2023-11-28T13:01:55.955680+0000","last_fullsized":"2023-11-28T13:01:55.955680+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:46:34.088559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.451598+0000","last_change":"2023-11-28T13:01:54.276706+0000","last_active":"2023-11-28T13:01:55.451598+0000","last_peered":"2023-11-28T13:01:55.451598+0000","last_clean":"2023-11-28T13:01:55.451598+0000","last_became_active":"2023-11-28T13:01:54.276400+0000","last_became_peered":"2023-11-28T13:01:54.276400+0000","last_unstale":"2023-11-28T13:01:55.451598+0000","last_undegraded":"2023-11-28T13:01:55.451598+0000","last_fullsized":"2023-11-28T13:01:55.451598+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:47:35.430928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.451513+0000","last_change":"2023-11-28T13:01:54.272845+0000","last_active":"2023-11-28T13:01:55.451513+0000","last_peered":"2023-11-28T13:01:55.451513+0000","last_clean":"2023-11-28T13:01:55.451513+0000","last_became_active":"2023-11-28T13:01:54.272528+0000","last_became_peered":"2023-11-28T13:01:54.272528+0000","last_unstale":"2023-11-28T13:01:55.451513+0000","last_undegraded":"2023-11-28T13:01:55.451513+0000","last_fullsized":"2023-11-28T13:01:55.451513+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:56:02.698142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274294+0000","last_change":"2023-11-28T13:01:54.285770+0000","last_active":"2023-11-28T13:01:55.274294+0000","last_peered":"2023-11-28T13:01:55.274294+0000","last_clean":"2023-11-28T13:01:55.274294+0000","last_became_active":"2023-11-28T13:01:54.285413+0000","last_became_peered":"2023-11-28T13:01:54.285413+0000","last_unstale":"2023-11-28T13:01:55.274294+0000","last_undegraded":"2023-11-28T13:01:55.274294+0000","last_fullsized":"2023-11-28T13:01:55.274294+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:01.236307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956061+0000","last_change":"2023-11-28T13:01:54.286136+0000","last_active":"2023-11-28T13:01:55.956061+0000","last_peered":"2023-11-28T13:01:55.956061+0000","last_clean":"2023-11-28T13:01:55.956061+0000","last_became_active":"2023-11-28T13:01:54.283990+0000","last_became_peered":"2023-11-28T13:01:54.283990+0000","last_unstale":"2023-11-28T13:01:55.956061+0000","last_undegraded":"2023-11-28T13:01:55.956061+0000","last_fullsized":"2023-11-28T13:01:55.956061+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:12:01.448285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"66'4","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956279+0000","last_change":"2023-11-28T13:01:53.884721+0000","last_active":"2023-11-28T13:01:55.956279+0000","last_peered":"2023-11-28T13:01:55.956279+0000","last_clean":"2023-11-28T13:01:55.956279+0000","last_became_active":"2023-11-28T13:01:53.884417+0000","last_became_peered":"2023-11-28T13:01:53.884417+0000","last_unstale":"2023-11-28T13:01:55.956279+0000","last_undegraded":"2023-11-28T13:01:55.956279+0000","last_fullsized":"2023-11-28T13:01:55.956279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:30:42.052145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956417+0000","last_change":"2023-11-28T13:01:54.286937+0000","last_active":"2023-11-28T13:01:55.956417+0000","last_peered":"2023-11-28T13:01:55.956417+0000","last_clean":"2023-11-28T13:01:55.956417+0000","last_became_active":"2023-11-28T13:01:54.285911+0000","last_became_peered":"2023-11-28T13:01:54.285911+0000","last_unstale":"2023-11-28T13:01:55.956417+0000","last_undegraded":"2023-11-28T13:01:55.956417+0000","last_fullsized":"2023-11-28T13:01:55.956417+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:38:22.871336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956851+0000","last_change":"2023-11-28T13:01:53.881784+0000","last_active":"2023-11-28T13:01:55.956851+0000","last_peered":"2023-11-28T13:01:55.956851+0000","last_clean":"2023-11-28T13:01:55.956851+0000","last_became_active":"2023-11-28T13:01:53.881444+0000","last_became_peered":"2023-11-28T13:01:53.881444+0000","last_unstale":"2023-11-28T13:01:55.956851+0000","last_undegraded":"2023-11-28T13:01:55.956851+0000","last_fullsized":"2023-11-28T13:01:55.956851+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:47:34.425036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274751+0000","last_change":"2023-11-28T13:01:54.283009+0000","last_active":"2023-11-28T13:01:55.274751+0000","last_peered":"2023-11-28T13:01:55.274751+0000","last_clean":"2023-11-28T13:01:55.274751+0000","last_became_active":"2023-11-28T13:01:54.282754+0000","last_became_peered":"2023-11-28T13:01:54.282754+0000","last_unstale":"2023-11-28T13:01:55.274751+0000","last_undegraded":"2023-11-28T13:01:55.274751+0000","last_fullsized":"2023-11-28T13:01:55.274751+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:39:38.829935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274055+0000","last_change":"2023-11-28T13:01:53.881143+0000","last_active":"2023-11-28T13:01:55.274055+0000","last_peered":"2023-11-28T13:01:55.274055+0000","last_clean":"2023-11-28T13:01:55.274055+0000","last_became_active":"2023-11-28T13:01:53.880891+0000","last_became_peered":"2023-11-28T13:01:53.880891+0000","last_unstale":"2023-11-28T13:01:55.274055+0000","last_undegraded":"2023-11-28T13:01:55.274055+0000","last_fullsized":"2023-11-28T13:01:55.274055+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:58:00.157022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956008+0000","last_change":"2023-11-28T13:01:54.280565+0000","last_active":"2023-11-28T13:01:55.956008+0000","last_peered":"2023-11-28T13:01:55.956008+0000","last_clean":"2023-11-28T13:01:55.956008+0000","last_became_active":"2023-11-28T13:01:54.277320+0000","last_became_peered":"2023-11-28T13:01:54.277320+0000","last_unstale":"2023-11-28T13:01:55.956008+0000","last_undegraded":"2023-11-28T13:01:55.956008+0000","last_fullsized":"2023-11-28T13:01:55.956008+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:47:35.430928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274839+0000","last_change":"2023-11-28T13:01:53.878303+0000","last_active":"2023-11-28T13:01:55.274839+0000","last_peered":"2023-11-28T13:01:55.274839+0000","last_clean":"2023-11-28T13:01:55.274839+0000","last_became_active":"2023-11-28T13:01:53.878021+0000","last_became_peered":"2023-11-28T13:01:53.878021+0000","last_unstale":"2023-11-28T13:01:55.274839+0000","last_undegraded":"2023-11-28T13:01:55.274839+0000","last_fullsized":"2023-11-28T13:01:55.274839+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:29:15.976646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.451672+0000","last_change":"2023-11-28T13:01:54.285640+0000","last_active":"2023-11-28T13:01:55.451672+0000","last_peered":"2023-11-28T13:01:55.451672+0000","last_clean":"2023-11-28T13:01:55.451672+0000","last_became_active":"2023-11-28T13:01:54.285324+0000","last_became_peered":"2023-11-28T13:01:54.285324+0000","last_unstale":"2023-11-28T13:01:55.451672+0000","last_undegraded":"2023-11-28T13:01:55.451672+0000","last_fullsized":"2023-11-28T13:01:55.451672+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:32:15.867118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955778+0000","last_change":"2023-11-28T13:01:53.880904+0000","last_active":"2023-11-28T13:01:55.955778+0000","last_peered":"2023-11-28T13:01:55.955778+0000","last_clean":"2023-11-28T13:01:55.955778+0000","last_became_active":"2023-11-28T13:01:53.880663+0000","last_became_peered":"2023-11-28T13:01:53.880663+0000","last_unstale":"2023-11-28T13:01:55.955778+0000","last_undegraded":"2023-11-28T13:01:55.955778+0000","last_fullsized":"2023-11-28T13:01:55.955778+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:01.139320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273963+0000","last_change":"2023-11-28T13:01:54.282262+0000","last_active":"2023-11-28T13:01:55.273963+0000","last_peered":"2023-11-28T13:01:55.273963+0000","last_clean":"2023-11-28T13:01:55.273963+0000","last_became_active":"2023-11-28T13:01:54.281463+0000","last_became_peered":"2023-11-28T13:01:54.281463+0000","last_unstale":"2023-11-28T13:01:55.273963+0000","last_undegraded":"2023-11-28T13:01:55.273963+0000","last_fullsized":"2023-11-28T13:01:55.273963+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:40:00.568094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274457+0000","last_change":"2023-11-28T13:01:53.884348+0000","last_active":"2023-11-28T13:01:55.274457+0000","last_peered":"2023-11-28T13:01:55.274457+0000","last_clean":"2023-11-28T13:01:55.274457+0000","last_became_active":"2023-11-28T13:01:53.883985+0000","last_became_peered":"2023-11-28T13:01:53.883985+0000","last_unstale":"2023-11-28T13:01:55.274457+0000","last_undegraded":"2023-11-28T13:01:55.274457+0000","last_fullsized":"2023-11-28T13:01:55.274457+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:02:19.973090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955549+0000","last_change":"2023-11-28T13:01:54.275630+0000","last_active":"2023-11-28T13:01:55.955549+0000","last_peered":"2023-11-28T13:01:55.955549+0000","last_clean":"2023-11-28T13:01:55.955549+0000","last_became_active":"2023-11-28T13:01:54.275378+0000","last_became_peered":"2023-11-28T13:01:54.275378+0000","last_unstale":"2023-11-28T13:01:55.955549+0000","last_undegraded":"2023-11-28T13:01:55.955549+0000","last_fullsized":"2023-11-28T13:01:55.955549+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:37:20.084624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268309+0000","last_change":"2023-11-28T13:01:53.879675+0000","last_active":"2023-11-28T13:01:55.268309+0000","last_peered":"2023-11-28T13:01:55.268309+0000","last_clean":"2023-11-28T13:01:55.268309+0000","last_became_active":"2023-11-28T13:01:53.879394+0000","last_became_peered":"2023-11-28T13:01:53.879394+0000","last_unstale":"2023-11-28T13:01:55.268309+0000","last_undegraded":"2023-11-28T13:01:55.268309+0000","last_fullsized":"2023-11-28T13:01:55.268309+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:41:31.469900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956467+0000","last_change":"2023-11-28T13:01:54.277992+0000","last_active":"2023-11-28T13:01:55.956467+0000","last_peered":"2023-11-28T13:01:55.956467+0000","last_clean":"2023-11-28T13:01:55.956467+0000","last_became_active":"2023-11-28T13:01:54.277723+0000","last_became_peered":"2023-11-28T13:01:54.277723+0000","last_unstale":"2023-11-28T13:01:55.956467+0000","last_undegraded":"2023-11-28T13:01:55.956467+0000","last_fullsized":"2023-11-28T13:01:55.956467+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:16.913565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955604+0000","last_change":"2023-11-28T13:01:53.875733+0000","last_active":"2023-11-28T13:01:55.955604+0000","last_peered":"2023-11-28T13:01:55.955604+0000","last_clean":"2023-11-28T13:01:55.955604+0000","last_became_active":"2023-11-28T13:01:53.873457+0000","last_became_peered":"2023-11-28T13:01:53.873457+0000","last_unstale":"2023-11-28T13:01:55.955604+0000","last_undegraded":"2023-11-28T13:01:55.955604+0000","last_fullsized":"2023-11-28T13:01:55.955604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:22:16.262115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.451829+0000","last_change":"2023-11-28T13:01:54.278092+0000","last_active":"2023-11-28T13:01:55.451829+0000","last_peered":"2023-11-28T13:01:55.451829+0000","last_clean":"2023-11-28T13:01:55.451829+0000","last_became_active":"2023-11-28T13:01:54.277741+0000","last_became_peered":"2023-11-28T13:01:54.277741+0000","last_unstale":"2023-11-28T13:01:55.451829+0000","last_undegraded":"2023-11-28T13:01:55.451829+0000","last_fullsized":"2023-11-28T13:01:55.451829+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:33:46.554110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274044+0000","last_change":"2023-11-28T13:01:53.878636+0000","last_active":"2023-11-28T13:01:55.274044+0000","last_peered":"2023-11-28T13:01:55.274044+0000","last_clean":"2023-11-28T13:01:55.274044+0000","last_became_active":"2023-11-28T13:01:53.878365+0000","last_became_peered":"2023-11-28T13:01:53.878365+0000","last_unstale":"2023-11-28T13:01:55.274044+0000","last_undegraded":"2023-11-28T13:01:55.274044+0000","last_fullsized":"2023-11-28T13:01:55.274044+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:12:00.442393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956196+0000","last_change":"2023-11-28T13:01:54.286882+0000","last_active":"2023-11-28T13:01:55.956196+0000","last_peered":"2023-11-28T13:01:55.956196+0000","last_clean":"2023-11-28T13:01:55.956196+0000","last_became_active":"2023-11-28T13:01:54.286551+0000","last_became_peered":"2023-11-28T13:01:54.286551+0000","last_unstale":"2023-11-28T13:01:55.956196+0000","last_undegraded":"2023-11-28T13:01:55.956196+0000","last_fullsized":"2023-11-28T13:01:55.956196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:51.983868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.275203+0000","last_change":"2023-11-28T13:01:53.882341+0000","last_active":"2023-11-28T13:01:55.275203+0000","last_peered":"2023-11-28T13:01:55.275203+0000","last_clean":"2023-11-28T13:01:55.275203+0000","last_became_active":"2023-11-28T13:01:53.881524+0000","last_became_peered":"2023-11-28T13:01:53.881524+0000","last_unstale":"2023-11-28T13:01:55.275203+0000","last_undegraded":"2023-11-28T13:01:55.275203+0000","last_fullsized":"2023-11-28T13:01:55.275203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:03.973023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274787+0000","last_change":"2023-11-28T13:01:54.279194+0000","last_active":"2023-11-28T13:01:55.274787+0000","last_peered":"2023-11-28T13:01:55.274787+0000","last_clean":"2023-11-28T13:01:55.274787+0000","last_became_active":"2023-11-28T13:01:54.278893+0000","last_became_peered":"2023-11-28T13:01:54.278893+0000","last_unstale":"2023-11-28T13:01:55.274787+0000","last_undegraded":"2023-11-28T13:01:55.274787+0000","last_fullsized":"2023-11-28T13:01:55.274787+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:51:38.510175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"66'3","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956247+0000","last_change":"2023-11-28T13:01:53.879482+0000","last_active":"2023-11-28T13:01:55.956247+0000","last_peered":"2023-11-28T13:01:55.956247+0000","last_clean":"2023-11-28T13:01:55.956247+0000","last_became_active":"2023-11-28T13:01:53.879223+0000","last_became_peered":"2023-11-28T13:01:53.879223+0000","last_unstale":"2023-11-28T13:01:55.956247+0000","last_undegraded":"2023-11-28T13:01:55.956247+0000","last_fullsized":"2023-11-28T13:01:55.956247+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:19:04.317034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956080+0000","last_change":"2023-11-28T13:01:54.274271+0000","last_active":"2023-11-28T13:01:55.956080+0000","last_peered":"2023-11-28T13:01:55.956080+0000","last_clean":"2023-11-28T13:01:55.956080+0000","last_became_active":"2023-11-28T13:01:54.274013+0000","last_became_peered":"2023-11-28T13:01:54.274013+0000","last_unstale":"2023-11-28T13:01:55.956080+0000","last_undegraded":"2023-11-28T13:01:55.956080+0000","last_fullsized":"2023-11-28T13:01:55.956080+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:27:13.629149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"66'6","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268610+0000","last_change":"2023-11-28T13:01:53.871891+0000","last_active":"2023-11-28T13:01:55.268610+0000","last_peered":"2023-11-28T13:01:55.268610+0000","last_clean":"2023-11-28T13:01:55.268610+0000","last_became_active":"2023-11-28T13:01:53.871640+0000","last_became_peered":"2023-11-28T13:01:53.871640+0000","last_unstale":"2023-11-28T13:01:55.268610+0000","last_undegraded":"2023-11-28T13:01:55.268610+0000","last_fullsized":"2023-11-28T13:01:55.268610+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:38:21.865445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955739+0000","last_change":"2023-11-28T13:01:54.284067+0000","last_active":"2023-11-28T13:01:55.955739+0000","last_peered":"2023-11-28T13:01:55.955739+0000","last_clean":"2023-11-28T13:01:55.955739+0000","last_became_active":"2023-11-28T13:01:54.281833+0000","last_became_peered":"2023-11-28T13:01:54.281833+0000","last_unstale":"2023-11-28T13:01:55.955739+0000","last_undegraded":"2023-11-28T13:01:55.955739+0000","last_fullsized":"2023-11-28T13:01:55.955739+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:46:01.236307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268528+0000","last_change":"2023-11-28T13:01:53.880138+0000","last_active":"2023-11-28T13:01:55.268528+0000","last_peered":"2023-11-28T13:01:55.268528+0000","last_clean":"2023-11-28T13:01:55.268528+0000","last_became_active":"2023-11-28T13:01:53.879852+0000","last_became_peered":"2023-11-28T13:01:53.879852+0000","last_unstale":"2023-11-28T13:01:55.268528+0000","last_undegraded":"2023-11-28T13:01:55.268528+0000","last_fullsized":"2023-11-28T13:01:55.268528+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:38:54.465243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.451781+0000","last_change":"2023-11-28T13:01:54.275101+0000","last_active":"2023-11-28T13:01:55.451781+0000","last_peered":"2023-11-28T13:01:55.451781+0000","last_clean":"2023-11-28T13:01:55.451781+0000","last_became_active":"2023-11-28T13:01:54.274843+0000","last_became_peered":"2023-11-28T13:01:54.274843+0000","last_unstale":"2023-11-28T13:01:55.451781+0000","last_undegraded":"2023-11-28T13:01:55.451781+0000","last_fullsized":"2023-11-28T13:01:55.451781+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:19:05.322926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274554+0000","last_change":"2023-11-28T13:01:53.883625+0000","last_active":"2023-11-28T13:01:55.274554+0000","last_peered":"2023-11-28T13:01:55.274554+0000","last_clean":"2023-11-28T13:01:55.274554+0000","last_became_active":"2023-11-28T13:01:53.883326+0000","last_became_peered":"2023-11-28T13:01:53.883326+0000","last_unstale":"2023-11-28T13:01:55.274554+0000","last_undegraded":"2023-11-28T13:01:55.274554+0000","last_fullsized":"2023-11-28T13:01:55.274554+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:30:56.435654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955550+0000","last_change":"2023-11-28T13:01:54.280697+0000","last_active":"2023-11-28T13:01:55.955550+0000","last_peered":"2023-11-28T13:01:55.955550+0000","last_clean":"2023-11-28T13:01:55.955550+0000","last_became_active":"2023-11-28T13:01:54.280440+0000","last_became_peered":"2023-11-28T13:01:54.280440+0000","last_unstale":"2023-11-28T13:01:55.955550+0000","last_undegraded":"2023-11-28T13:01:55.955550+0000","last_fullsized":"2023-11-28T13:01:55.955550+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:24:13.908547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.451861+0000","last_change":"2023-11-28T13:01:53.883495+0000","last_active":"2023-11-28T13:01:55.451861+0000","last_peered":"2023-11-28T13:01:55.451861+0000","last_clean":"2023-11-28T13:01:55.451861+0000","last_became_active":"2023-11-28T13:01:53.883232+0000","last_became_peered":"2023-11-28T13:01:53.883232+0000","last_unstale":"2023-11-28T13:01:55.451861+0000","last_undegraded":"2023-11-28T13:01:55.451861+0000","last_fullsized":"2023-11-28T13:01:55.451861+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:01:25.198031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955921+0000","last_change":"2023-11-28T13:01:54.287297+0000","last_active":"2023-11-28T13:01:55.955921+0000","last_peered":"2023-11-28T13:01:55.955921+0000","last_clean":"2023-11-28T13:01:55.955921+0000","last_became_active":"2023-11-28T13:01:54.287054+0000","last_became_peered":"2023-11-28T13:01:54.287054+0000","last_unstale":"2023-11-28T13:01:55.955921+0000","last_undegraded":"2023-11-28T13:01:55.955921+0000","last_fullsized":"2023-11-28T13:01:55.955921+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:40:12.276019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"66'8","reported_seq":19,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273708+0000","last_change":"2023-11-28T13:01:53.873547+0000","last_active":"2023-11-28T13:01:55.273708+0000","last_peered":"2023-11-28T13:01:55.273708+0000","last_clean":"2023-11-28T13:01:55.273708+0000","last_became_active":"2023-11-28T13:01:53.873205+0000","last_became_peered":"2023-11-28T13:01:53.873205+0000","last_unstale":"2023-11-28T13:01:55.273708+0000","last_undegraded":"2023-11-28T13:01:55.273708+0000","last_fullsized":"2023-11-28T13:01:55.273708+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:32:50.977977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956802+0000","last_change":"2023-11-28T13:01:54.274471+0000","last_active":"2023-11-28T13:01:55.956802+0000","last_peered":"2023-11-28T13:01:55.956802+0000","last_clean":"2023-11-28T13:01:55.956802+0000","last_became_active":"2023-11-28T13:01:54.274298+0000","last_became_peered":"2023-11-28T13:01:54.274298+0000","last_unstale":"2023-11-28T13:01:55.956802+0000","last_undegraded":"2023-11-28T13:01:55.956802+0000","last_fullsized":"2023-11-28T13:01:55.956802+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:41:59.072903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.275349+0000","last_change":"2023-11-28T13:01:53.881135+0000","last_active":"2023-11-28T13:01:55.275349+0000","last_peered":"2023-11-28T13:01:55.275349+0000","last_clean":"2023-11-28T13:01:55.275349+0000","last_became_active":"2023-11-28T13:01:53.880764+0000","last_became_peered":"2023-11-28T13:01:53.880764+0000","last_unstale":"2023-11-28T13:01:55.275349+0000","last_undegraded":"2023-11-28T13:01:55.275349+0000","last_fullsized":"2023-11-28T13:01:55.275349+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:39:58.297742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.451856+0000","last_change":"2023-11-28T13:01:54.266813+0000","last_active":"2023-11-28T13:01:55.451856+0000","last_peered":"2023-11-28T13:01:55.451856+0000","last_clean":"2023-11-28T13:01:55.451856+0000","last_became_active":"2023-11-28T13:01:54.266339+0000","last_became_peered":"2023-11-28T13:01:54.266339+0000","last_unstale":"2023-11-28T13:01:55.451856+0000","last_undegraded":"2023-11-28T13:01:55.451856+0000","last_fullsized":"2023-11-28T13:01:55.451856+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:02.145212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.451919+0000","last_change":"2023-11-28T13:01:53.882582+0000","last_active":"2023-11-28T13:01:55.451919+0000","last_peered":"2023-11-28T13:01:55.451919+0000","last_clean":"2023-11-28T13:01:55.451919+0000","last_became_active":"2023-11-28T13:01:53.882371+0000","last_became_peered":"2023-11-28T13:01:53.882371+0000","last_unstale":"2023-11-28T13:01:55.451919+0000","last_undegraded":"2023-11-28T13:01:55.451919+0000","last_fullsized":"2023-11-28T13:01:55.451919+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:07:31.693512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955937+0000","last_change":"2023-11-28T13:01:54.286766+0000","last_active":"2023-11-28T13:01:55.955937+0000","last_peered":"2023-11-28T13:01:55.955937+0000","last_clean":"2023-11-28T13:01:55.955937+0000","last_became_active":"2023-11-28T13:01:54.284229+0000","last_became_peered":"2023-11-28T13:01:54.284229+0000","last_unstale":"2023-11-28T13:01:55.955937+0000","last_undegraded":"2023-11-28T13:01:55.955937+0000","last_fullsized":"2023-11-28T13:01:55.955937+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:33:46.554110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956286+0000","last_change":"2023-11-28T13:01:53.882782+0000","last_active":"2023-11-28T13:01:55.956286+0000","last_peered":"2023-11-28T13:01:55.956286+0000","last_clean":"2023-11-28T13:01:55.956286+0000","last_became_active":"2023-11-28T13:01:53.882444+0000","last_became_peered":"2023-11-28T13:01:53.882444+0000","last_unstale":"2023-11-28T13:01:55.956286+0000","last_undegraded":"2023-11-28T13:01:55.956286+0000","last_fullsized":"2023-11-28T13:01:55.956286+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:24:40.479670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956129+0000","last_change":"2023-11-28T13:01:54.281150+0000","last_active":"2023-11-28T13:01:55.956129+0000","last_peered":"2023-11-28T13:01:55.956129+0000","last_clean":"2023-11-28T13:01:55.956129+0000","last_became_active":"2023-11-28T13:01:54.280841+0000","last_became_peered":"2023-11-28T13:01:54.280841+0000","last_unstale":"2023-11-28T13:01:55.956129+0000","last_undegraded":"2023-11-28T13:01:55.956129+0000","last_fullsized":"2023-11-28T13:01:55.956129+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:44:02.145212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956491+0000","last_change":"2023-11-28T13:01:53.881718+0000","last_active":"2023-11-28T13:01:55.956491+0000","last_peered":"2023-11-28T13:01:55.956491+0000","last_clean":"2023-11-28T13:01:55.956491+0000","last_became_active":"2023-11-28T13:01:53.881460+0000","last_became_peered":"2023-11-28T13:01:53.881460+0000","last_unstale":"2023-11-28T13:01:55.956491+0000","last_undegraded":"2023-11-28T13:01:55.956491+0000","last_fullsized":"2023-11-28T13:01:55.956491+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:19:04.317034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956618+0000","last_change":"2023-11-28T13:01:54.284724+0000","last_active":"2023-11-28T13:01:55.956618+0000","last_peered":"2023-11-28T13:01:55.956618+0000","last_clean":"2023-11-28T13:01:55.956618+0000","last_became_active":"2023-11-28T13:01:54.284398+0000","last_became_peered":"2023-11-28T13:01:54.284398+0000","last_unstale":"2023-11-28T13:01:55.956618+0000","last_undegraded":"2023-11-28T13:01:55.956618+0000","last_fullsized":"2023-11-28T13:01:55.956618+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:41:32.475792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955980+0000","last_change":"2023-11-28T13:01:53.870766+0000","last_active":"2023-11-28T13:01:55.955980+0000","last_peered":"2023-11-28T13:01:55.955980+0000","last_clean":"2023-11-28T13:01:55.955980+0000","last_became_active":"2023-11-28T13:01:53.870523+0000","last_became_peered":"2023-11-28T13:01:53.870523+0000","last_unstale":"2023-11-28T13:01:55.955980+0000","last_undegraded":"2023-11-28T13:01:55.955980+0000","last_fullsized":"2023-11-28T13:01:55.955980+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:43:04.853464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274622+0000","last_change":"2023-11-28T13:01:54.284458+0000","last_active":"2023-11-28T13:01:55.274622+0000","last_peered":"2023-11-28T13:01:55.274622+0000","last_clean":"2023-11-28T13:01:55.274622+0000","last_became_active":"2023-11-28T13:01:54.284178+0000","last_became_peered":"2023-11-28T13:01:54.284178+0000","last_unstale":"2023-11-28T13:01:55.274622+0000","last_undegraded":"2023-11-28T13:01:55.274622+0000","last_fullsized":"2023-11-28T13:01:55.274622+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:59.890096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268336+0000","last_change":"2023-11-28T13:01:53.879719+0000","last_active":"2023-11-28T13:01:55.268336+0000","last_peered":"2023-11-28T13:01:55.268336+0000","last_clean":"2023-11-28T13:01:55.268336+0000","last_became_active":"2023-11-28T13:01:53.879492+0000","last_became_peered":"2023-11-28T13:01:53.879492+0000","last_unstale":"2023-11-28T13:01:55.268336+0000","last_undegraded":"2023-11-28T13:01:55.268336+0000","last_fullsized":"2023-11-28T13:01:55.268336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:40:11.270127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268296+0000","last_change":"2023-11-28T13:01:54.284967+0000","last_active":"2023-11-28T13:01:55.268296+0000","last_peered":"2023-11-28T13:01:55.268296+0000","last_clean":"2023-11-28T13:01:55.268296+0000","last_became_active":"2023-11-28T13:01:54.284692+0000","last_became_peered":"2023-11-28T13:01:54.284692+0000","last_unstale":"2023-11-28T13:01:55.268296+0000","last_undegraded":"2023-11-28T13:01:55.268296+0000","last_fullsized":"2023-11-28T13:01:55.268296+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:27:39.151509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273962+0000","last_change":"2023-11-28T13:01:53.884732+0000","last_active":"2023-11-28T13:01:55.273962+0000","last_peered":"2023-11-28T13:01:55.273962+0000","last_clean":"2023-11-28T13:01:55.273962+0000","last_became_active":"2023-11-28T13:01:53.884445+0000","last_became_peered":"2023-11-28T13:01:53.884445+0000","last_unstale":"2023-11-28T13:01:55.273962+0000","last_undegraded":"2023-11-28T13:01:55.273962+0000","last_fullsized":"2023-11-28T13:01:55.273962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:46:33.082668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274377+0000","last_change":"2023-11-28T13:01:54.284922+0000","last_active":"2023-11-28T13:01:55.274377+0000","last_peered":"2023-11-28T13:01:55.274377+0000","last_clean":"2023-11-28T13:01:55.274377+0000","last_became_active":"2023-11-28T13:01:54.284769+0000","last_became_peered":"2023-11-28T13:01:54.284769+0000","last_unstale":"2023-11-28T13:01:55.274377+0000","last_undegraded":"2023-11-28T13:01:55.274377+0000","last_fullsized":"2023-11-28T13:01:55.274377+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:46:40.116660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274410+0000","last_change":"2023-11-28T13:01:53.877803+0000","last_active":"2023-11-28T13:01:55.274410+0000","last_peered":"2023-11-28T13:01:55.274410+0000","last_clean":"2023-11-28T13:01:55.274410+0000","last_became_active":"2023-11-28T13:01:53.877499+0000","last_became_peered":"2023-11-28T13:01:53.877499+0000","last_unstale":"2023-11-28T13:01:55.274410+0000","last_undegraded":"2023-11-28T13:01:55.274410+0000","last_fullsized":"2023-11-28T13:01:55.274410+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:50:25.775486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956179+0000","last_change":"2023-11-28T13:01:54.271933+0000","last_active":"2023-11-28T13:01:55.956179+0000","last_peered":"2023-11-28T13:01:55.956179+0000","last_clean":"2023-11-28T13:01:55.956179+0000","last_became_active":"2023-11-28T13:01:54.271708+0000","last_became_peered":"2023-11-28T13:01:54.271708+0000","last_unstale":"2023-11-28T13:01:55.956179+0000","last_undegraded":"2023-11-28T13:01:55.956179+0000","last_fullsized":"2023-11-28T13:01:55.956179+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:16:05.380760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955936+0000","last_change":"2023-11-28T13:01:53.884480+0000","last_active":"2023-11-28T13:01:55.955936+0000","last_peered":"2023-11-28T13:01:55.955936+0000","last_clean":"2023-11-28T13:01:55.955936+0000","last_became_active":"2023-11-28T13:01:53.884116+0000","last_became_peered":"2023-11-28T13:01:53.884116+0000","last_unstale":"2023-11-28T13:01:55.955936+0000","last_undegraded":"2023-11-28T13:01:55.955936+0000","last_fullsized":"2023-11-28T13:01:55.955936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:37:12.617319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":54,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.265296+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T13:01:54.265296+0000","last_peered":"2023-11-28T13:01:54.265296+0000","last_clean":"2023-11-28T13:01:54.265296+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:01:54.265296+0000","last_undegraded":"2023-11-28T13:01:54.265296+0000","last_fullsized":"2023-11-28T13:01:54.265296+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955814+0000","last_change":"2023-11-28T13:01:54.283713+0000","last_active":"2023-11-28T13:01:55.955814+0000","last_peered":"2023-11-28T13:01:55.955814+0000","last_clean":"2023-11-28T13:01:55.955814+0000","last_became_active":"2023-11-28T13:01:54.283449+0000","last_became_peered":"2023-11-28T13:01:54.283449+0000","last_unstale":"2023-11-28T13:01:55.955814+0000","last_undegraded":"2023-11-28T13:01:55.955814+0000","last_fullsized":"2023-11-28T13:01:55.955814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:41:59.072903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274666+0000","last_change":"2023-11-28T13:01:53.883683+0000","last_active":"2023-11-28T13:01:55.274666+0000","last_peered":"2023-11-28T13:01:55.274666+0000","last_clean":"2023-11-28T13:01:55.274666+0000","last_became_active":"2023-11-28T13:01:53.883419+0000","last_became_peered":"2023-11-28T13:01:53.883419+0000","last_unstale":"2023-11-28T13:01:55.274666+0000","last_undegraded":"2023-11-28T13:01:55.274666+0000","last_fullsized":"2023-11-28T13:01:55.274666+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:27:38.145617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.262345+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T13:01:54.262345+0000","last_peered":"2023-11-28T13:01:54.262345+0000","last_clean":"2023-11-28T13:01:54.262345+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:01:54.262345+0000","last_undegraded":"2023-11-28T13:01:54.262345+0000","last_fullsized":"2023-11-28T13:01:54.262345+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955743+0000","last_change":"2023-11-28T13:01:54.288673+0000","last_active":"2023-11-28T13:01:55.955743+0000","last_peered":"2023-11-28T13:01:55.955743+0000","last_clean":"2023-11-28T13:01:55.955743+0000","last_became_active":"2023-11-28T13:01:54.288422+0000","last_became_peered":"2023-11-28T13:01:54.288422+0000","last_unstale":"2023-11-28T13:01:55.955743+0000","last_undegraded":"2023-11-28T13:01:55.955743+0000","last_fullsized":"2023-11-28T13:01:55.955743+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:10:48.275248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.451836+0000","last_change":"2023-11-28T13:01:53.871456+0000","last_active":"2023-11-28T13:01:55.451836+0000","last_peered":"2023-11-28T13:01:55.451836+0000","last_clean":"2023-11-28T13:01:55.451836+0000","last_became_active":"2023-11-28T13:01:53.871164+0000","last_became_peered":"2023-11-28T13:01:53.871164+0000","last_unstale":"2023-11-28T13:01:55.451836+0000","last_undegraded":"2023-11-28T13:01:55.451836+0000","last_fullsized":"2023-11-28T13:01:55.451836+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:49:22.683439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":50,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.266976+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T13:01:54.266976+0000","last_peered":"2023-11-28T13:01:54.266976+0000","last_clean":"2023-11-28T13:01:54.266976+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:01:54.266976+0000","last_undegraded":"2023-11-28T13:01:54.266976+0000","last_fullsized":"2023-11-28T13:01:54.266976+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955924+0000","last_change":"2023-11-28T13:01:54.288055+0000","last_active":"2023-11-28T13:01:55.955924+0000","last_peered":"2023-11-28T13:01:55.955924+0000","last_clean":"2023-11-28T13:01:55.955924+0000","last_became_active":"2023-11-28T13:01:54.287752+0000","last_became_peered":"2023-11-28T13:01:54.287752+0000","last_unstale":"2023-11-28T13:01:55.955924+0000","last_undegraded":"2023-11-28T13:01:55.955924+0000","last_fullsized":"2023-11-28T13:01:55.955924+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:24:13.908547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274714+0000","last_change":"2023-11-28T13:01:53.879480+0000","last_active":"2023-11-28T13:01:55.274714+0000","last_peered":"2023-11-28T13:01:55.274714+0000","last_clean":"2023-11-28T13:01:55.274714+0000","last_became_active":"2023-11-28T13:01:53.878450+0000","last_became_peered":"2023-11-28T13:01:53.878450+0000","last_unstale":"2023-11-28T13:01:55.274714+0000","last_undegraded":"2023-11-28T13:01:55.274714+0000","last_fullsized":"2023-11-28T13:01:55.274714+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:22:43.385404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.262295+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T13:01:54.262295+0000","last_peered":"2023-11-28T13:01:54.262295+0000","last_clean":"2023-11-28T13:01:54.262295+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:01:54.262295+0000","last_undegraded":"2023-11-28T13:01:54.262295+0000","last_fullsized":"2023-11-28T13:01:54.262295+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":204,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.272035+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T13:01:54.272035+0000","last_peered":"2023-11-28T13:01:54.272035+0000","last_clean":"2023-11-28T13:01:54.272035+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:01:54.272035+0000","last_undegraded":"2023-11-28T13:01:54.272035+0000","last_fullsized":"2023-11-28T13:01:54.272035+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.955836+0000","last_change":"2023-11-28T13:01:54.288873+0000","last_active":"2023-11-28T13:01:55.955836+0000","last_peered":"2023-11-28T13:01:55.955836+0000","last_clean":"2023-11-28T13:01:55.955836+0000","last_became_active":"2023-11-28T13:01:54.288450+0000","last_became_peered":"2023-11-28T13:01:54.288450+0000","last_unstale":"2023-11-28T13:01:55.955836+0000","last_undegraded":"2023-11-28T13:01:55.955836+0000","last_fullsized":"2023-11-28T13:01:55.955836+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:16:36.681083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956612+0000","last_change":"2023-11-28T13:01:53.882489+0000","last_active":"2023-11-28T13:01:55.956612+0000","last_peered":"2023-11-28T13:01:55.956612+0000","last_clean":"2023-11-28T13:01:55.956612+0000","last_became_active":"2023-11-28T13:01:53.882199+0000","last_became_peered":"2023-11-28T13:01:53.882199+0000","last_unstale":"2023-11-28T13:01:55.956612+0000","last_undegraded":"2023-11-28T13:01:55.956612+0000","last_fullsized":"2023-11-28T13:01:55.956612+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:33.319104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":52,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.272125+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T13:01:54.272125+0000","last_peered":"2023-11-28T13:01:54.272125+0000","last_clean":"2023-11-28T13:01:54.272125+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:01:54.272125+0000","last_undegraded":"2023-11-28T13:01:54.272125+0000","last_fullsized":"2023-11-28T13:01:54.272125+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.268569+0000","last_change":"2023-11-28T13:01:54.278616+0000","last_active":"2023-11-28T13:01:55.268569+0000","last_peered":"2023-11-28T13:01:55.268569+0000","last_clean":"2023-11-28T13:01:55.268569+0000","last_became_active":"2023-11-28T13:01:54.277488+0000","last_became_peered":"2023-11-28T13:01:54.277488+0000","last_unstale":"2023-11-28T13:01:55.268569+0000","last_undegraded":"2023-11-28T13:01:55.268569+0000","last_fullsized":"2023-11-28T13:01:55.268569+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:04.978915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274669+0000","last_change":"2023-11-28T13:01:53.882980+0000","last_active":"2023-11-28T13:01:55.274669+0000","last_peered":"2023-11-28T13:01:55.274669+0000","last_clean":"2023-11-28T13:01:55.274669+0000","last_became_active":"2023-11-28T13:01:53.881456+0000","last_became_peered":"2023-11-28T13:01:53.881456+0000","last_unstale":"2023-11-28T13:01:55.274669+0000","last_undegraded":"2023-11-28T13:01:55.274669+0000","last_fullsized":"2023-11-28T13:01:55.274669+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:15:03.394013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":52,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.272898+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T13:01:54.272898+0000","last_peered":"2023-11-28T13:01:54.272898+0000","last_clean":"2023-11-28T13:01:54.272898+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:01:54.272898+0000","last_undegraded":"2023-11-28T13:01:54.272898+0000","last_fullsized":"2023-11-28T13:01:54.272898+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.956178+0000","last_change":"2023-11-28T13:01:54.286797+0000","last_active":"2023-11-28T13:01:55.956178+0000","last_peered":"2023-11-28T13:01:55.956178+0000","last_clean":"2023-11-28T13:01:55.956178+0000","last_became_active":"2023-11-28T13:01:54.286220+0000","last_became_peered":"2023-11-28T13:01:54.286220+0000","last_unstale":"2023-11-28T13:01:55.956178+0000","last_undegraded":"2023-11-28T13:01:55.956178+0000","last_fullsized":"2023-11-28T13:01:55.956178+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:11:34.324996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.274056+0000","last_change":"2023-11-28T13:01:53.880863+0000","last_active":"2023-11-28T13:01:55.274056+0000","last_peered":"2023-11-28T13:01:55.274056+0000","last_clean":"2023-11-28T13:01:55.274056+0000","last_became_active":"2023-11-28T13:01:53.880498+0000","last_became_peered":"2023-11-28T13:01:53.880498+0000","last_unstale":"2023-11-28T13:01:55.274056+0000","last_undegraded":"2023-11-28T13:01:55.274056+0000","last_fullsized":"2023-11-28T13:01:55.274056+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:01:08.675044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":53,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.265122+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T13:01:54.265122+0000","last_peered":"2023-11-28T13:01:54.265122+0000","last_clean":"2023-11-28T13:01:54.265122+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:01:54.265122+0000","last_undegraded":"2023-11-28T13:01:54.265122+0000","last_fullsized":"2023-11-28T13:01:54.265122+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.273879+0000","last_change":"2023-11-28T13:01:54.279411+0000","last_active":"2023-11-28T13:01:55.273879+0000","last_peered":"2023-11-28T13:01:55.273879+0000","last_clean":"2023-11-28T13:01:55.273879+0000","last_became_active":"2023-11-28T13:01:54.275789+0000","last_became_peered":"2023-11-28T13:01:54.275789+0000","last_unstale":"2023-11-28T13:01:55.273879+0000","last_undegraded":"2023-11-28T13:01:55.273879+0000","last_fullsized":"2023-11-28T13:01:55.273879+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:53.855713+0000","last_clean_scrub_stamp":"2023-11-28T13:01:53.855713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T14:32:27.408420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-11-28T13:01:55.275371+0000","last_change":"2023-11-28T13:01:53.880342+0000","last_active":"2023-11-28T13:01:55.275371+0000","last_peered":"2023-11-28T13:01:55.275371+0000","last_clean":"2023-11-28T13:01:55.275371+0000","last_became_active":"2023-11-28T13:01:53.880088+0000","last_became_peered":"2023-11-28T13:01:53.880088+0000","last_unstale":"2023-11-28T13:01:55.275371+0000","last_undegraded":"2023-11-28T13:01:55.275371+0000","last_fullsized":"2023-11-28T13:01:55.275371+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:01:52.849821+0000","last_clean_scrub_stamp":"2023-11-28T13:01:52.849821+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T15:38:21.865445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":60,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-11-28T13:01:54.267363+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T13:01:54.267363+0000","last_peered":"2023-11-28T13:01:54.267363+0000","last_clean":"2023-11-28T13:01:54.267363+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:01:54.267363+0000","last_undegraded":"2023-11-28T13:01:54.267363+0000","last_fullsized":"2023-11-28T13:01:54.267363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10860,"kb_used_data":1380,"kb_used_omap":0,"kb_used_meta":9472,"kb_avail":94360980,"statfs":{"total":96636764160,"available":96625643520,"internally_reserved":0,"allocated":1413120,"data_stored":971523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9699328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.629,"15min":0.66400000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82699999999999996,"5min":3.3279999999999998,"15min":3.3279999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.628,"15min":0.67500000000000004},"min":{"1min":0.40799999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65900000000000003,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.61599999999999999}]},{"osd":1,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52300000000000002,"15min":0.52700000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64600000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.56499999999999995}]},{"osd":2,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.54500000000000004,"15min":0.55100000000000005},"min":{"1min":0.379,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.86399999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.42599999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70199999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.626}]},{"osd":3,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54400000000000004,"15min":0.54200000000000004},"min":{"1min":0.45000000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.40699999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.59199999999999997}]},{"osd":4,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.52100000000000002},"min":{"1min":0.36799999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53200000000000003,"15min":0.53600000000000003},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.42399999999999999}]},{"osd":5,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.51200000000000001,"15min":0.52300000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.41399999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.57799999999999996}]},{"osd":6,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.56100000000000005,"15min":0.57499999999999996},"min":{"1min":0.39500000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.45300000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.64000000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9768,"kb_used_data":800,"kb_used_omap":0,"kb_used_meta":8960,"kb_avail":94362072,"statfs":{"total":96636764160,"available":96626761728,"internally_reserved":0,"allocated":819200,"data_stored":380647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9175040},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54700000000000004,"15min":0.54500000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70099999999999996,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.434},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.51300000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68500000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68500000000000005}]},{"osd":1,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52100000000000002},"min":{"1min":0.32400000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.33000000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.747,"5min":1.016,"15min":1.016},"last":0.49099999999999999}]},{"osd":2,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.52700000000000002},"min":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.63600000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57299999999999995,"15min":0.58199999999999996},"min":{"1min":0.41699999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79000000000000004,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.52500000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58199999999999996,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.371,"15min":0.371},"max":{"1min":0.77700000000000002,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79200000000000004,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.39100000000000001}]},{"osd":4,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52800000000000002,"15min":0.53800000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.84199999999999997,"5min":1,"15min":1},"last":0.628},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53900000000000003,"15min":0.54300000000000004},"min":{"1min":0.378,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73099999999999998,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.61299999999999999}]},{"osd":5,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84499999999999997,"5min":1.052,"15min":1.052},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88900000000000001,"5min":1.258,"15min":1.258},"last":0.58199999999999996}]},{"osd":7,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76900000000000002,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.621,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90400000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.59799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9716,"kb_used_data":812,"kb_used_omap":0,"kb_used_meta":8896,"kb_avail":94362124,"statfs":{"total":96636764160,"available":96626814976,"internally_reserved":0,"allocated":831488,"data_stored":382703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9109504},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.32300000000000001,"15min":0.32600000000000001},"min":{"1min":0.224,"5min":0.219,"15min":0.219},"max":{"1min":0.51500000000000001,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.35699999999999998,"15min":0.35399999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.52500000000000002,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.33800000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33700000000000002,"15min":0.33300000000000002},"min":{"1min":0.254,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.48199999999999998,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.36499999999999999,"15min":0.36599999999999999},"min":{"1min":0.26600000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.48299999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.45800000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.54300000000000004,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38700000000000001,"15min":0.39200000000000002},"min":{"1min":0.26000000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.46600000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.39300000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.36199999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83599999999999997,"5min":1.028,"15min":1.028},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.45600000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85899999999999999,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.56299999999999994}]},{"osd":5,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83199999999999996,"5min":1.042,"15min":1.042},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.57499999999999996,"15min":0.55900000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83699999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.69099999999999995}]},{"osd":6,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57599999999999996,"15min":0.57799999999999996},"min":{"1min":0.40600000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.76700000000000002,"5min":1.006,"15min":1.006},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.58299999999999996}]},{"osd":7,"last update":"Tue Nov 28 13:01:49 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77700000000000002,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.622},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85799999999999998,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.59399999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10968,"kb_used_data":1360,"kb_used_omap":0,"kb_used_meta":9600,"kb_avail":94360872,"statfs":{"total":96636764160,"available":96625532928,"internally_reserved":0,"allocated":1392640,"data_stored":963950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9830400},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32400000000000001,"15min":0.32200000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.42499999999999999,"5min":0.5,"15min":0.5},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34399999999999997,"15min":0.34200000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.53400000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.33900000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34999999999999998,"15min":0.34499999999999997},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.47699999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.37,"15min":0.36499999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50600000000000001,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.29699999999999999}]},{"osd":3,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36799999999999999,"15min":0.372},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.499,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.39300000000000002,"15min":0.38700000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59499999999999997,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.35699999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59099999999999997,"15min":0.60099999999999998},"min":{"1min":0.309,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.80000000000000004,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76100000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.68500000000000005}]},{"osd":5,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56899999999999995,"15min":0.56699999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70499999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.61099999999999999,"15min":0.61899999999999999},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.77400000000000002,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57699999999999996}]},{"osd":6,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59699999999999998,"15min":0.60299999999999998},"min":{"1min":0.46999999999999997,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81499999999999995,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.60399999999999998,"15min":0.61399999999999999},"min":{"1min":0.41499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.751,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.73299999999999998}]},{"osd":7,"last update":"Tue Nov 28 13:01:54 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.62,"15min":0.627},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71899999999999997,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.626},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78900000000000003,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.59799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9580,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":8768,"kb_avail":94362260,"statfs":{"total":96636764160,"available":96626954240,"internally_reserved":0,"allocated":823296,"data_stored":382103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8978432},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.32600000000000001,"15min":0.316},"min":{"1min":0.22500000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.254},{"interface":"front","average":{"1min":0.379,"5min":0.34599999999999997,"15min":0.33600000000000002},"min":{"1min":0.21199999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.35499999999999998}]},{"osd":1,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38100000000000001,"15min":0.38200000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.59199999999999997,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.36699999999999999,"15min":0.35499999999999998},"min":{"1min":0.25700000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56299999999999994,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.38800000000000001}]},{"osd":3,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.34499999999999997,"15min":0.33900000000000002},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.70599999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.39800000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.59899999999999998,"15min":0.59299999999999997},"min":{"1min":0.47999999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76200000000000001,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.57399999999999995,"15min":0.55800000000000005},"min":{"1min":0.42199999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.55600000000000005}]},{"osd":5,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.45700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.85499999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.60099999999999998,"15min":0.59499999999999997},"min":{"1min":0.41699999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.77100000000000002,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.59399999999999997}]},{"osd":6,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.59099999999999997,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83999999999999997,"5min":1.002,"15min":1.002},"last":0.627},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63100000000000001,"15min":0.624},"min":{"1min":0.499,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.52100000000000002}]},{"osd":7,"last update":"Tue Nov 28 13:01:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.628,"15min":0.63},"min":{"1min":0.48999999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.83599999999999997,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.441,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85899999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.57599999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9820,"kb_used_data":788,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94362020,"statfs":{"total":96636764160,"available":96626708480,"internally_reserved":0,"allocated":806912,"data_stored":373694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.33000000000000002,"15min":0.32200000000000001},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46100000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.379,"5min":0.33800000000000002,"15min":0.33000000000000002},"min":{"1min":0.24099999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.312}]},{"osd":1,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.34799999999999998,"15min":0.34000000000000002},"min":{"1min":0.25800000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.377,"15min":0.373},"min":{"1min":0.27200000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.33000000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46700000000000003,"15min":0.48699999999999999},"min":{"1min":0.252,"5min":0.251,"15min":0.251},"max":{"1min":0.59299999999999997,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.442,"5min":0.499,"15min":0.52900000000000003},"min":{"1min":0.313,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.61599999999999999,"5min":2.7160000000000002,"15min":2.7160000000000002},"last":0.35299999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.83999999999999997,"15min":0.94199999999999995},"min":{"1min":0.439,"5min":0.437,"15min":0.437},"max":{"1min":0.92200000000000004,"5min":8.2889999999999997,"15min":8.2889999999999997},"last":0.58999999999999997}]},{"osd":5,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61599999999999999,"15min":0.61899999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.75700000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.65100000000000002,"15min":0.67100000000000004},"min":{"1min":0.47799999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88200000000000001,"5min":2.165,"15min":2.165},"last":0.53900000000000003}]},{"osd":6,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.625,"15min":0.628},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84199999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.627},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.91300000000000003,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.57399999999999995}]},{"osd":7,"last update":"Tue Nov 28 13:01:52 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61499999999999999,"15min":0.60999999999999999},"min":{"1min":0.46899999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.83799999999999997,"15min":0.93500000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.89600000000000002,"5min":7.0899999999999999,"15min":7.0899999999999999},"last":0.56399999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10020,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":9216,"kb_avail":94361820,"statfs":{"total":96636764160,"available":96626503680,"internally_reserved":0,"allocated":815104,"data_stored":380625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9437184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.67700000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67100000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.58399999999999996}]},{"osd":1,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.504},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76600000000000001,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70999999999999996,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.66100000000000003}]},{"osd":2,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.44400000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71499999999999997,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.41499999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74299999999999999,"5min":1.069,"15min":1.069},"last":0.77700000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.43099999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.71299999999999997,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41099999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.74199999999999999,"5min":0.879,"15min":0.879},"last":0.68500000000000005}]},{"osd":5,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51200000000000001,"15min":0.51700000000000002},"min":{"1min":0.33400000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.67800000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.51700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70299999999999996,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.40100000000000002}]},{"osd":6,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.52300000000000002},"min":{"1min":0.35399999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72999999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.371},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.57899999999999996,"15min":0.58699999999999997},"min":{"1min":0.38300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71899999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.46100000000000002}]},{"osd":7,"last update":"Tue Nov 28 13:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71699999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58999999999999997,"15min":0.59099999999999997},"min":{"1min":0.39800000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69599999999999995,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.42899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9636,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362204,"statfs":{"total":96636764160,"available":96626896896,"internally_reserved":0,"allocated":815104,"data_stored":380645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.34599999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69899999999999995,"5min":1.819,"15min":1.819},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.52600000000000002,"15min":0.52100000000000002},"min":{"1min":0.40799999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.52200000000000002}]},{"osd":1,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73599999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.67500000000000004}]},{"osd":2,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.75800000000000001,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.63,"15min":0.64800000000000002},"min":{"1min":0.372,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.748,"5min":2.4020000000000001,"15min":2.4020000000000001},"last":0.69199999999999995}]},{"osd":3,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72799999999999998,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.66300000000000003,"15min":0.68600000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71099999999999997,"5min":2.702,"15min":2.702},"last":0.60199999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.60099999999999998,"15min":0.61899999999999999},"min":{"1min":0.30399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.66500000000000004,"5min":3.0739999999999998,"15min":3.0739999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76500000000000001,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.626}]},{"osd":6,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.34499999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68999999999999995,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.70199999999999996,"15min":0.72699999999999998},"min":{"1min":0.39100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73099999999999998,"5min":3.7679999999999998,"15min":3.7679999999999998},"last":0.61199999999999999}]},{"osd":7,"last update":"Tue Nov 28 13:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78400000000000003,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.65500000000000003,"15min":0.66600000000000004},"min":{"1min":0.41399999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.751,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.55100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:01:58.719 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T13:01:58.719 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw pool=cephfs_data' mds allow 2023-11-28T13:01:58.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.880+0000 7f975ead0700 1 Processor -- start 2023-11-28T13:01:58.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.880+0000 7f975ead0700 1 -- start start 2023-11-28T13:01:58.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.881+0000 7f975ead0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f975806bff0 0x7f975815d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.881+0000 7f975ead0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9758161110 con 0x7f97580cde60 2023-11-28T13:01:58.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.881+0000 7f975ead0700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97580b1d50 con 0x7f9758160ca0 2023-11-28T13:01:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.881+0000 7f975ead0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975815cd40 con 0x7f975806bff0 2023-11-28T13:01:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.881+0000 7f975d06d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f97580cde60 0x7f975806a700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38130/0 (socket says 172.21.15.111:38130) 2023-11-28T13:01:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.881+0000 7f975d06d700 1 -- 172.21.15.111:0/3595761121 learned_addr learned my addr 172.21.15.111:0/3595761121 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.881+0000 7f975c86c700 1 --2- 172.21.15.111:0/3595761121 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f975806bff0 0x7f975815d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.882+0000 7f97577fe700 1 -- 172.21.15.111:0/3595761121 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750932828 0 0) 0x7f9758161110 con 0x7f97580cde60 2023-11-28T13:01:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.882+0000 7f97577fe700 1 -- 172.21.15.111:0/3595761121 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97480035b0 con 0x7f97580cde60 2023-11-28T13:01:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.882+0000 7f97577fe700 1 -- 172.21.15.111:0/3595761121 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462770211 0 0) 0x7f97580b1d50 con 0x7f9758160ca0 2023-11-28T13:01:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.882+0000 7f97577fe700 1 -- 172.21.15.111:0/3595761121 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9758161110 con 0x7f9758160ca0 2023-11-28T13:01:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.882+0000 7f975c86c700 1 -- 172.21.15.111:0/3595761121 >> v1:172.21.15.149:6789/0 conn(0x7f9758160ca0 legacy=0x7f97580cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.882+0000 7f975c86c700 1 -- 172.21.15.111:0/3595761121 >> v1:172.21.15.111:6789/0 conn(0x7f97580cde60 legacy=0x7f975806a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.882+0000 7f975c86c700 1 -- 172.21.15.111:0/3595761121 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975815d020 con 0x7f975806bff0 2023-11-28T13:01:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.882+0000 7f975c86c700 1 --2- 172.21.15.111:0/3595761121 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f975806bff0 0x7f975815d820 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f974c009870 tx=0x7f974c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8607847896f8495b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:58.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.882+0000 7f97577fe700 1 -- 172.21.15.111:0/3595761121 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c011830 con 0x7f975806bff0 2023-11-28T13:01:58.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.882+0000 7f97577fe700 1 -- 172.21.15.111:0/3595761121 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974c011990 con 0x7f975806bff0 2023-11-28T13:01:58.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.882+0000 7f97577fe700 1 -- 172.21.15.111:0/3595761121 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c011d00 con 0x7f975806bff0 2023-11-28T13:01:58.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.883+0000 7f975ead0700 1 -- 172.21.15.111:0/3595761121 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f975806bff0 msgr2=0x7f975815d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.883+0000 7f975ead0700 1 --2- 172.21.15.111:0/3595761121 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f975806bff0 0x7f975815d820 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f974c009870 tx=0x7f974c009b80 comp rx=0 tx=0).stop 2023-11-28T13:01:58.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.883+0000 7f975ead0700 1 -- 172.21.15.111:0/3595761121 shutdown_connections 2023-11-28T13:01:58.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.883+0000 7f975ead0700 1 --2- 172.21.15.111:0/3595761121 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f975806bff0 0x7f975815d820 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.883+0000 7f975ead0700 1 -- 172.21.15.111:0/3595761121 >> 172.21.15.111:0/3595761121 conn(0x7f975806e790 msgr2=0x7f97580cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:58.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.883+0000 7f975ead0700 1 -- 172.21.15.111:0/3595761121 shutdown_connections 2023-11-28T13:01:58.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.883+0000 7f975ead0700 1 -- 172.21.15.111:0/3595761121 wait complete. 2023-11-28T13:01:58.887 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:01:58.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.884+0000 7f975ead0700 1 Processor -- start 2023-11-28T13:01:58.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.884+0000 7f975ead0700 1 -- start start 2023-11-28T13:01:58.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975ead0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f975806bff0 0x7f97580b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975ead0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97580cde60 0x7f97580c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975ead0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9758160ca0 0x7f9758159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975ead0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97580c9f90 con 0x7f975806bff0 2023-11-28T13:01:58.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975ead0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975815a140 con 0x7f9758160ca0 2023-11-28T13:01:58.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975ead0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97580c6df0 con 0x7f97580cde60 2023-11-28T13:01:58.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975c86c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f975806bff0 0x7f97580b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975c86c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f975806bff0 0x7f97580b33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37354/0 (socket says 172.21.15.111:37354) 2023-11-28T13:01:58.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975c86c700 1 -- 172.21.15.111:0/220681630 learned_addr learned my addr 172.21.15.111:0/220681630 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:58.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f9757fff700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97580cde60 0x7f97580c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975d06d700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9758160ca0 0x7f9758159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975c86c700 1 -- 172.21.15.111:0/220681630 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97580cde60 msgr2=0x7f97580c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975c86c700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97580cde60 0x7f97580c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975c86c700 1 -- 172.21.15.111:0/220681630 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9758160ca0 msgr2=0x7f9758159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975c86c700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9758160ca0 0x7f9758159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.885+0000 7f975c86c700 1 -- 172.21.15.111:0/220681630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c0094a0 con 0x7f975806bff0 2023-11-28T13:01:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.886+0000 7f975c86c700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f975806bff0 0x7f97580b33c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f974c0097c0 tx=0x7f974c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.886+0000 7f97557fa700 1 -- 172.21.15.111:0/220681630 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c0044d0 con 0x7f975806bff0 2023-11-28T13:01:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.886+0000 7f97557fa700 1 -- 172.21.15.111:0/220681630 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974c01f070 con 0x7f975806bff0 2023-11-28T13:01:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.886+0000 7f975ead0700 1 -- 172.21.15.111:0/220681630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97580b2030 con 0x7f975806bff0 2023-11-28T13:01:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.886+0000 7f97557fa700 1 -- 172.21.15.111:0/220681630 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c0038e0 con 0x7f975806bff0 2023-11-28T13:01:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.886+0000 7f975ead0700 1 -- 172.21.15.111:0/220681630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f975815d300 con 0x7f975806bff0 2023-11-28T13:01:58.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.887+0000 7f975ead0700 1 -- 172.21.15.111:0/220681630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97580b1600 con 0x7f975806bff0 2023-11-28T13:01:58.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.887+0000 7f97557fa700 1 -- 172.21.15.111:0/220681630 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f974c00f050 con 0x7f975806bff0 2023-11-28T13:01:58.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.887+0000 7f97557fa700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f973c0682f0 0x7f973c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:58.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.887+0000 7f97557fa700 1 -- 172.21.15.111:0/220681630 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f974c021070 con 0x7f975806bff0 2023-11-28T13:01:58.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.888+0000 7f9757fff700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f973c0682f0 0x7f973c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:58.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.888+0000 7f9757fff700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f973c0682f0 0x7f973c06a7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f97580cec40 tx=0x7f9744009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:58.893 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:01:58.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:58.892+0000 7f97557fa700 1 -- 172.21.15.111:0/220681630 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f974c053c90 con 0x7f975806bff0 2023-11-28T13:01:59.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.149+0000 7f975ead0700 1 -- 172.21.15.111:0/220681630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]} v 0) v1 -- 0x7f9758159660 con 0x7f975806bff0 2023-11-28T13:01:59.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.153+0000 7f97557fa700 1 -- 172.21.15.111:0/220681630 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw pool=cephfs_data", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 184+0+0 (secure 0 0 0) 0x7f974c0572b0 con 0x7f975806bff0 2023-11-28T13:01:59.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.155+0000 7f975ead0700 1 -- 172.21.15.111:0/220681630 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f973c0682f0 msgr2=0x7f973c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:59.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.155+0000 7f975ead0700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f973c0682f0 0x7f973c06a7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f97580cec40 tx=0x7f9744009040 comp rx=0 tx=0).stop 2023-11-28T13:01:59.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.155+0000 7f975ead0700 1 -- 172.21.15.111:0/220681630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f975806bff0 msgr2=0x7f97580b33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:59.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.155+0000 7f975ead0700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f975806bff0 0x7f97580b33c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f974c0097c0 tx=0x7f974c004750 comp rx=0 tx=0).stop 2023-11-28T13:01:59.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.156+0000 7f975ead0700 1 -- 172.21.15.111:0/220681630 shutdown_connections 2023-11-28T13:01:59.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.156+0000 7f975ead0700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f973c0682f0 0x7f973c06a7a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:59.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.156+0000 7f975ead0700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9758160ca0 0x7f9758159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:59.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.156+0000 7f975ead0700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f975806bff0 0x7f97580b33c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:59.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.156+0000 7f975ead0700 1 --2- 172.21.15.111:0/220681630 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97580cde60 0x7f97580c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:59.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.156+0000 7f975ead0700 1 -- 172.21.15.111:0/220681630 >> 172.21.15.111:0/220681630 conn(0x7f975806e790 msgr2=0x7f975815b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:59.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.156+0000 7f975ead0700 1 -- 172.21.15.111:0/220681630 shutdown_connections 2023-11-28T13:01:59.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.156+0000 7f975ead0700 1 -- 172.21.15.111:0/220681630 wait complete. 2023-11-28T13:01:59.159 INFO:teuthology.orchestra.run.smithi111.stderr:updated caps for client.0 2023-11-28T13:01:59.169 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:01:59.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.287+0000 7f720d3ce700 1 Processor -- start 2023-11-28T13:01:59.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.288+0000 7f720d3ce700 1 -- start start 2023-11-28T13:01:59.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.288+0000 7f720d3ce700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72080cdd90 0x7f72080b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:59.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.288+0000 7f720d3ce700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f72080aecc0 0x7f72080ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:59.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.288+0000 7f720d3ce700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72080ad420 con 0x7f7208159960 2023-11-28T13:01:59.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.288+0000 7f720d3ce700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7208159d30 con 0x7f72080aecc0 2023-11-28T13:01:59.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.288+0000 7f720d3ce700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72080ac810 con 0x7f72080cdd90 2023-11-28T13:01:59.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.289+0000 7f72067fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72080cdd90 0x7f72080b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:59.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.289+0000 7f72077fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f72080aecc0 0x7f72080ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:59.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.289+0000 7f7206ffd700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f7208159960 0x7f72080cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:46682/0 (socket says 172.21.15.111:46682) 2023-11-28T13:01:59.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.289+0000 7f7206ffd700 1 -- 172.21.15.111:0/1454272545 learned_addr learned my addr 172.21.15.111:0/1454272545 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:01:59.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.289+0000 7f72067fc700 1 -- 172.21.15.111:0/1454272545 >> v2:172.21.15.149:3300/0 conn(0x7f72080aecc0 msgr2=0x7f72080ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:01:59.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.289+0000 7f72067fc700 1 --2- 172.21.15.111:0/1454272545 >> v2:172.21.15.149:3300/0 conn(0x7f72080aecc0 0x7f72080ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:59.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.289+0000 7f72067fc700 1 -- 172.21.15.111:0/1454272545 >> v1:172.21.15.149:6789/0 conn(0x7f7208159960 legacy=0x7f72080cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:59.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.289+0000 7f72067fc700 1 -- 172.21.15.111:0/1454272545 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72080acaf0 con 0x7f72080cdd90 2023-11-28T13:01:59.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.289+0000 7f72067fc700 1 --2- 172.21.15.111:0/1454272545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72080cdd90 0x7f72080b1fd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009840 tx=0x7f71f8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d81cc6c2320a69e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:59.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.289+0000 7f7205ffb700 1 -- 172.21.15.111:0/1454272545 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8011830 con 0x7f72080cdd90 2023-11-28T13:01:59.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.289+0000 7f7205ffb700 1 -- 172.21.15.111:0/1454272545 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f8011990 con 0x7f72080cdd90 2023-11-28T13:01:59.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.289+0000 7f7205ffb700 1 -- 172.21.15.111:0/1454272545 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8011d20 con 0x7f72080cdd90 2023-11-28T13:01:59.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.290+0000 7f720d3ce700 1 -- 172.21.15.111:0/1454272545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72080cdd90 msgr2=0x7f72080b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:59.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.290+0000 7f720d3ce700 1 --2- 172.21.15.111:0/1454272545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72080cdd90 0x7f72080b1fd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009840 tx=0x7f71f8009b50 comp rx=0 tx=0).stop 2023-11-28T13:01:59.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.290+0000 7f720d3ce700 1 -- 172.21.15.111:0/1454272545 shutdown_connections 2023-11-28T13:01:59.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.290+0000 7f720d3ce700 1 --2- 172.21.15.111:0/1454272545 >> v2:172.21.15.149:3300/0 conn(0x7f72080aecc0 0x7f72080ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:59.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.290+0000 7f720d3ce700 1 --2- 172.21.15.111:0/1454272545 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72080cdd90 0x7f72080b1fd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:59.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.290+0000 7f720d3ce700 1 -- 172.21.15.111:0/1454272545 >> 172.21.15.111:0/1454272545 conn(0x7f7208098ee0 msgr2=0x7f720806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:59.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.290+0000 7f720d3ce700 1 -- 172.21.15.111:0/1454272545 shutdown_connections 2023-11-28T13:01:59.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.290+0000 7f720d3ce700 1 -- 172.21.15.111:0/1454272545 wait complete. 2023-11-28T13:01:59.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.291+0000 7f720d3ce700 1 Processor -- start 2023-11-28T13:01:59.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.291+0000 7f720d3ce700 1 -- start start 2023-11-28T13:01:59.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f720d3ce700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72080aecc0 0x7f7208156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f720d3ce700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72080cdd90 0x7f720809a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f720d3ce700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7208159960 0x7f720806df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f720d3ce700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720809c6e0 con 0x7f72080aecc0 2023-11-28T13:01:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f720d3ce700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7208156640 con 0x7f7208159960 2023-11-28T13:01:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f720d3ce700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72080c6780 con 0x7f72080cdd90 2023-11-28T13:01:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f7206ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72080aecc0 0x7f7208156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f7206ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72080aecc0 0x7f7208156100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37368/0 (socket says 172.21.15.111:37368) 2023-11-28T13:01:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f7206ffd700 1 -- 172.21.15.111:0/742259366 learned_addr learned my addr 172.21.15.111:0/742259366 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:01:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f72067fc700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72080cdd90 0x7f720809a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f72077fe700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7208159960 0x7f720806df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f7206ffd700 1 -- 172.21.15.111:0/742259366 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72080cdd90 msgr2=0x7f720809a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:59.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f7206ffd700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72080cdd90 0x7f720809a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:59.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f7206ffd700 1 -- 172.21.15.111:0/742259366 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7208159960 msgr2=0x7f720806df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:59.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.292+0000 7f7206ffd700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7208159960 0x7f720806df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:59.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.293+0000 7f7206ffd700 1 -- 172.21.15.111:0/742259366 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f80094a0 con 0x7f72080aecc0 2023-11-28T13:01:59.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.293+0000 7f7206ffd700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72080aecc0 0x7f7208156100 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00c500 tx=0x7f71fc00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:59.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.293+0000 7f71ef7fe700 1 -- 172.21.15.111:0/742259366 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc005dc0 con 0x7f72080aecc0 2023-11-28T13:01:59.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.293+0000 7f71ef7fe700 1 -- 172.21.15.111:0/742259366 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71fc0070d0 con 0x7f72080aecc0 2023-11-28T13:01:59.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.293+0000 7f71ef7fe700 1 -- 172.21.15.111:0/742259366 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc007460 con 0x7f72080aecc0 2023-11-28T13:01:59.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.293+0000 7f720d3ce700 1 -- 172.21.15.111:0/742259366 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72080ad0b0 con 0x7f72080aecc0 2023-11-28T13:01:59.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.293+0000 7f720d3ce700 1 -- 172.21.15.111:0/742259366 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72080acdd0 con 0x7f72080aecc0 2023-11-28T13:01:59.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.294+0000 7f720d3ce700 1 -- 172.21.15.111:0/742259366 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72080addf0 con 0x7f72080aecc0 2023-11-28T13:01:59.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.294+0000 7f71ef7fe700 1 -- 172.21.15.111:0/742259366 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f71fc019030 con 0x7f72080aecc0 2023-11-28T13:01:59.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.295+0000 7f71ef7fe700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f71f40682f0 0x7f71f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:01:59.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.295+0000 7f71ef7fe700 1 -- 172.21.15.111:0/742259366 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f71fc084fd0 con 0x7f72080aecc0 2023-11-28T13:01:59.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.295+0000 7f72067fc700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f71f40682f0 0x7f71f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:01:59.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.295+0000 7f72067fc700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f71f40682f0 0x7f71f406a7a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f72080ce180 tx=0x7f71f8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:01:59.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.299+0000 7f71ef7fe700 1 -- 172.21.15.111:0/742259366 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f71fc0516f0 con 0x7f72080aecc0 2023-11-28T13:01:59.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.523+0000 7f720d3ce700 1 -- 172.21.15.111:0/742259366 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7208067170 con 0x7f72080aecc0 2023-11-28T13:01:59.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.524+0000 7f71ef7fe700 1 -- 172.21.15.111:0/742259366 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4659 (secure 0 0 0) 0x7f71fc017090 con 0x7f72080aecc0 2023-11-28T13:01:59.525 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:01:59.525 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/1668572397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1668572397},{"type":"v1","addr":"172.21.15.149:6834","nonce":1668572397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5755,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1129092621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1129092621},{"type":"v1","addr":"172.21.15.149:6835","nonce":1129092621}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/904588136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":904588136},{"type":"v1","addr":"172.21.15.111:6836","nonce":904588136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:01:54.257638+0000","modified":"2023-11-28T13:01:57.268879+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/1897250522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1897250522},{"type":"v1","addr":"172.21.15.111:6837","nonce":1897250522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T13:01:59.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.525+0000 7f720d3ce700 1 -- 172.21.15.111:0/742259366 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f71f40682f0 msgr2=0x7f71f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:59.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.525+0000 7f720d3ce700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f71f40682f0 0x7f71f406a7a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f72080ce180 tx=0x7f71f8008030 comp rx=0 tx=0).stop 2023-11-28T13:01:59.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.526+0000 7f720d3ce700 1 -- 172.21.15.111:0/742259366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72080aecc0 msgr2=0x7f7208156100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:01:59.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.526+0000 7f720d3ce700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72080aecc0 0x7f7208156100 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00c500 tx=0x7f71fc00c810 comp rx=0 tx=0).stop 2023-11-28T13:01:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.526+0000 7f720d3ce700 1 -- 172.21.15.111:0/742259366 shutdown_connections 2023-11-28T13:01:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.526+0000 7f720d3ce700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f71f40682f0 0x7f71f406a7a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.526+0000 7f720d3ce700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7208159960 0x7f720806df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.526+0000 7f720d3ce700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72080aecc0 0x7f7208156100 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.526+0000 7f720d3ce700 1 --2- 172.21.15.111:0/742259366 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72080cdd90 0x7f720809a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:01:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.526+0000 7f720d3ce700 1 -- 172.21.15.111:0/742259366 >> 172.21.15.111:0/742259366 conn(0x7f7208098ee0 msgr2=0x7f720806efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:01:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.526+0000 7f720d3ce700 1 -- 172.21.15.111:0/742259366 shutdown_connections 2023-11-28T13:01:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:01:59.526+0000 7f720d3ce700 1 -- 172.21.15.111:0/742259366 wait complete. 2023-11-28T13:01:59.528 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 32 2023-11-28T13:01:59.941 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 32, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T13:01:54.257638+0000', 'modified': '2023-11-28T13:01:57.268879+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6252}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6252': {'gid': 6252, 'name': 'c', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6837/1897250522', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6835', 'nonce': 1897250522}, {'type': 'v1', 'addr': '172.21.15.111:6837', 'nonce': 1897250522}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T13:01:59.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T13:01:59.942 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T13:02:00.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.062+0000 7f243939d700 1 Processor -- start 2023-11-28T13:02:00.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.062+0000 7f243939d700 1 -- start start 2023-11-28T13:02:00.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.062+0000 7f243939d700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f243415a210 0x7f24340c9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.062+0000 7f243939d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24340caf40 0x7f2434071140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.062+0000 7f243939d700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f243415a5e0 con 0x7f24340cb390 2023-11-28T13:02:00.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.062+0000 7f243939d700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f243415e6c0 con 0x7f243415a210 2023-11-28T13:02:00.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.062+0000 7f243939d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24340ce200 con 0x7f24340caf40 2023-11-28T13:02:00.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.063+0000 7f2433fff700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f243415a210 0x7f24340c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.063+0000 7f24337fe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24340caf40 0x7f2434071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.063+0000 7f2438b9c700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f24340cb390 0x7f24340ccce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:46698/0 (socket says 172.21.15.111:46698) 2023-11-28T13:02:00.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.063+0000 7f2438b9c700 1 -- 172.21.15.111:0/2770722929 learned_addr learned my addr 172.21.15.111:0/2770722929 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:02:00.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.063+0000 7f24337fe700 1 -- 172.21.15.111:0/2770722929 >> v2:172.21.15.149:3300/0 conn(0x7f243415a210 msgr2=0x7f24340c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:00.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.063+0000 7f24337fe700 1 --2- 172.21.15.111:0/2770722929 >> v2:172.21.15.149:3300/0 conn(0x7f243415a210 0x7f24340c9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.063+0000 7f24337fe700 1 -- 172.21.15.111:0/2770722929 >> v1:172.21.15.149:6789/0 conn(0x7f24340cb390 legacy=0x7f24340ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.063+0000 7f24337fe700 1 -- 172.21.15.111:0/2770722929 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243406b280 con 0x7f24340caf40 2023-11-28T13:02:00.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.063+0000 7f2433fff700 1 --2- 172.21.15.111:0/2770722929 >> v2:172.21.15.149:3300/0 conn(0x7f243415a210 0x7f24340c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:02:00.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.063+0000 7f24337fe700 1 --2- 172.21.15.111:0/2770722929 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24340caf40 0x7f2434071140 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2428009840 tx=0x7f2428009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=420c59fc699f4130 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:00.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.063+0000 7f2432ffd700 1 -- 172.21.15.111:0/2770722929 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2428011830 con 0x7f24340caf40 2023-11-28T13:02:00.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.064+0000 7f2432ffd700 1 -- 172.21.15.111:0/2770722929 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2428011990 con 0x7f24340caf40 2023-11-28T13:02:00.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.064+0000 7f2432ffd700 1 -- 172.21.15.111:0/2770722929 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2428011d20 con 0x7f24340caf40 2023-11-28T13:02:00.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.064+0000 7f243939d700 1 -- 172.21.15.111:0/2770722929 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24340caf40 msgr2=0x7f2434071140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.064+0000 7f243939d700 1 --2- 172.21.15.111:0/2770722929 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24340caf40 0x7f2434071140 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2428009840 tx=0x7f2428009b50 comp rx=0 tx=0).stop 2023-11-28T13:02:00.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.064+0000 7f243939d700 1 -- 172.21.15.111:0/2770722929 shutdown_connections 2023-11-28T13:02:00.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.064+0000 7f243939d700 1 --2- 172.21.15.111:0/2770722929 >> v2:172.21.15.149:3300/0 conn(0x7f243415a210 0x7f24340c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.064+0000 7f243939d700 1 --2- 172.21.15.111:0/2770722929 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f24340caf40 0x7f2434071140 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.064+0000 7f243939d700 1 -- 172.21.15.111:0/2770722929 >> 172.21.15.111:0/2770722929 conn(0x7f24340cf110 msgr2=0x7f24340cf4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:00.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.064+0000 7f243939d700 1 -- 172.21.15.111:0/2770722929 shutdown_connections 2023-11-28T13:02:00.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.064+0000 7f243939d700 1 -- 172.21.15.111:0/2770722929 wait complete. 2023-11-28T13:02:00.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.065+0000 7f243939d700 1 Processor -- start 2023-11-28T13:02:00.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.065+0000 7f243939d700 1 -- start start 2023-11-28T13:02:00.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.066+0000 7f243939d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24340caf40 0x7f243414f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.066+0000 7f243939d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24340cb390 0x7f24340c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.066+0000 7f243939d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f243415a210 0x7f243414eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.066+0000 7f243939d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24340c9140 con 0x7f24340cb390 2023-11-28T13:02:00.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.066+0000 7f243939d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24340c7b60 con 0x7f24340caf40 2023-11-28T13:02:00.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.066+0000 7f243939d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24340c7e40 con 0x7f243415a210 2023-11-28T13:02:00.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.066+0000 7f24337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24340cb390 0x7f24340c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.066+0000 7f24337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24340cb390 0x7f24340c8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37384/0 (socket says 172.21.15.111:37384) 2023-11-28T13:02:00.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.066+0000 7f24337fe700 1 -- 172.21.15.111:0/2461298312 learned_addr learned my addr 172.21.15.111:0/2461298312 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:00.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f2438b9c700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f243415a210 0x7f243414eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f2433fff700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24340caf40 0x7f243414f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f24337fe700 1 -- 172.21.15.111:0/2461298312 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f243415a210 msgr2=0x7f243414eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f24337fe700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f243415a210 0x7f243414eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f24337fe700 1 -- 172.21.15.111:0/2461298312 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24340caf40 msgr2=0x7f243414f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f24337fe700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24340caf40 0x7f243414f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f24337fe700 1 -- 172.21.15.111:0/2461298312 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24240094a0 con 0x7f24340cb390 2023-11-28T13:02:00.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f2433fff700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24340caf40 0x7f243414f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:02:00.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f2438b9c700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f243415a210 0x7f243414eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:02:00.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f24337fe700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24340cb390 0x7f24340c8c00 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2428009fd0 tx=0x7f2428004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:00.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f2430ff9700 1 -- 172.21.15.111:0/2461298312 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24280036a0 con 0x7f24340cb390 2023-11-28T13:02:00.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f2430ff9700 1 -- 172.21.15.111:0/2461298312 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242801f070 con 0x7f24340cb390 2023-11-28T13:02:00.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f243939d700 1 -- 172.21.15.111:0/2461298312 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24280094a0 con 0x7f24340cb390 2023-11-28T13:02:00.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f2430ff9700 1 -- 172.21.15.111:0/2461298312 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2428003a20 con 0x7f24340cb390 2023-11-28T13:02:00.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.067+0000 7f243939d700 1 -- 172.21.15.111:0/2461298312 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243415e070 con 0x7f24340cb390 2023-11-28T13:02:00.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.068+0000 7f2430ff9700 1 -- 172.21.15.111:0/2461298312 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f242800f050 con 0x7f24340cb390 2023-11-28T13:02:00.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.069+0000 7f2430ff9700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24200682f0 0x7f242006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.069+0000 7f2430ff9700 1 -- 172.21.15.111:0/2461298312 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2428021070 con 0x7f24340cb390 2023-11-28T13:02:00.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.069+0000 7f2433fff700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24200682f0 0x7f242006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.069+0000 7f2433fff700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24200682f0 0x7f242006a7a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f24341550f0 tx=0x7f2424008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:00.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.071+0000 7f2446788b80 1 Processor -- start 2023-11-28T13:02:00.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.071+0000 7f2446788b80 1 -- start start 2023-11-28T13:02:00.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.071+0000 7f2446788b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d22a70 0x55d257cf8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f2446788b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257cf28c0 0x55d257d22e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f2446788b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257d23380 0x55d257d1fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f2446788b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d257be2bc0 con 0x55d257cf28c0 2023-11-28T13:02:00.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f2446788b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d257c51350 con 0x55d257d23380 2023-11-28T13:02:00.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f2446788b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d257c4a880 con 0x55d257d22a70 2023-11-28T13:02:00.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f24337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257cf28c0 0x55d257d22e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f24337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257cf28c0 0x55d257d22e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37392/0 (socket says 172.21.15.111:37392) 2023-11-28T13:02:00.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f24337fe700 1 -- 172.21.15.111:0/292336482 learned_addr learned my addr 172.21.15.111:0/292336482 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:00.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f2433fff700 1 --2- 172.21.15.111:0/292336482 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d22a70 0x55d257cf8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f2438b9c700 1 --2- 172.21.15.111:0/292336482 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257d23380 0x55d257d1fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f24337fe700 1 -- 172.21.15.111:0/292336482 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d22a70 msgr2=0x55d257cf8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:00.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f24337fe700 1 --2- 172.21.15.111:0/292336482 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d22a70 0x55d257cf8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f24337fe700 1 -- 172.21.15.111:0/292336482 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257d23380 msgr2=0x55d257d1fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:00.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f24337fe700 1 --2- 172.21.15.111:0/292336482 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257d23380 0x55d257d1fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f24337fe700 1 -- 172.21.15.111:0/292336482 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d257acc5e0 con 0x55d257cf28c0 2023-11-28T13:02:00.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f24337fe700 1 --2- 172.21.15.111:0/292336482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257cf28c0 0x55d257d22e40 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f242805f6a0 tx=0x7f242805fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3feca21e115d3d30 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:00.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f2419ffb700 1 -- 172.21.15.111:0/292336482 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2428085ea0 con 0x55d257cf28c0 2023-11-28T13:02:00.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f2419ffb700 1 -- 172.21.15.111:0/292336482 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242805f940 con 0x55d257cf28c0 2023-11-28T13:02:00.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.072+0000 7f2419ffb700 1 -- 172.21.15.111:0/292336482 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2428003dc0 con 0x55d257cf28c0 2023-11-28T13:02:00.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.073+0000 7f2446788b80 1 -- 172.21.15.111:0/292336482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257cf28c0 msgr2=0x55d257d22e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.073+0000 7f2446788b80 1 --2- 172.21.15.111:0/292336482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257cf28c0 0x55d257d22e40 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f242805f6a0 tx=0x7f242805fbd0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.073+0000 7f2446788b80 1 -- 172.21.15.111:0/292336482 shutdown_connections 2023-11-28T13:02:00.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.073+0000 7f2446788b80 1 --2- 172.21.15.111:0/292336482 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257d23380 0x55d257d1fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.073+0000 7f2446788b80 1 --2- 172.21.15.111:0/292336482 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d22a70 0x55d257cf8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.073+0000 7f2446788b80 1 --2- 172.21.15.111:0/292336482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257cf28c0 0x55d257d22e40 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.073+0000 7f2446788b80 1 -- 172.21.15.111:0/292336482 >> 172.21.15.111:0/292336482 conn(0x55d257cf5fc0 msgr2=0x55d257cf6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:00.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.073+0000 7f2446788b80 1 -- 172.21.15.111:0/292336482 shutdown_connections 2023-11-28T13:02:00.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.073+0000 7f2446788b80 1 -- 172.21.15.111:0/292336482 wait complete. 2023-11-28T13:02:00.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.073+0000 7f2446788b80 1 -- start start 2023-11-28T13:02:00.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.074+0000 7f2446788b80 1 Processor -- start 2023-11-28T13:02:00.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.074+0000 7f2446788b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257cf28c0 0x55d257d0d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.074+0000 7f2446788b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d22a70 0x55d257d40680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.074+0000 7f2446788b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 0x55d257d3e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.074+0000 7f2446788b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d257ace8d0 con 0x55d257d23380 2023-11-28T13:02:00.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.074+0000 7f2446788b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d257d0ea70 con 0x55d257cf28c0 2023-11-28T13:02:00.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.074+0000 7f2446788b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d257d3f390 con 0x55d257d22a70 2023-11-28T13:02:00.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.074+0000 7f2446788b80 10 client.0 _ll_register_callbacks cb 0x55d257af0240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:02:00.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2438b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 0x55d257d3e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2438b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 0x55d257d3e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37400/0 (socket says 172.21.15.111:37400) 2023-11-28T13:02:00.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2438b9c700 1 -- 172.21.15.111:0/2490569699 learned_addr learned my addr 172.21.15.111:0/2490569699 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:00.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2433fff700 1 --2- 172.21.15.111:0/2490569699 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257cf28c0 0x55d257d0d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f24337fe700 1 --2- 172.21.15.111:0/2490569699 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d22a70 0x55d257d40680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2438b9c700 1 -- 172.21.15.111:0/2490569699 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d22a70 msgr2=0x55d257d40680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:00.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2438b9c700 1 --2- 172.21.15.111:0/2490569699 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d22a70 0x55d257d40680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2438b9c700 1 -- 172.21.15.111:0/2490569699 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257cf28c0 msgr2=0x55d257d0d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:00.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2438b9c700 1 --2- 172.21.15.111:0/2490569699 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257cf28c0 0x55d257d0d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2438b9c700 1 -- 172.21.15.111:0/2490569699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243415e070 con 0x55d257d23380 2023-11-28T13:02:00.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2438b9c700 1 --2- 172.21.15.111:0/2490569699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 0x55d257d3e9b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f242c00f820 tx=0x7f242c00fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e19381e9b3998f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:00.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2412ffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:02:00.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2412ffd700 1 -- 172.21.15.111:0/2490569699 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c013600 con 0x55d257d23380 2023-11-28T13:02:00.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2412ffd700 1 -- 172.21.15.111:0/2490569699 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f242c013760 con 0x55d257d23380 2023-11-28T13:02:00.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2446788b80 1 -- 172.21.15.111:0/2490569699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d257c18140 con 0x55d257d23380 2023-11-28T13:02:00.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2412ffd700 1 -- 172.21.15.111:0/2490569699 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f242c013920 con 0x55d257d23380 2023-11-28T13:02:00.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2446788b80 10 client.6405 fetch_fsmap learned FSMap version 32 2023-11-28T13:02:00.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2446788b80 1 -- 172.21.15.111:0/2490569699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55d257a78d90 con 0x55d257d23380 2023-11-28T13:02:00.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.075+0000 7f2412ffd700 1 -- 172.21.15.111:0/2490569699 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d257c18140 con 0x55d257d23380 2023-11-28T13:02:00.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.076+0000 7f2412ffd700 1 -- 172.21.15.111:0/2490569699 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f242c012070 con 0x55d257d23380 2023-11-28T13:02:00.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.076+0000 7f2446788b80 10 client.6405 fetch_fsmap finished waiting for FSMap version 32 2023-11-28T13:02:00.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.076+0000 7f2446788b80 10 client.6405 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T13:02:00.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.076+0000 7f2446788b80 20 client.6405 populate_metadata read hostname 'smithi111' 2023-11-28T13:02:00.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.076+0000 7f2446788b80 1 --2- 172.21.15.111:0/2490569699 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55d257d3eef0 0x55d257d22010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.076+0000 7f2446788b80 4 client.6405 mds_command: new command op to 6252 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-28T13:02:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.076+0000 7f2446788b80 1 -- 172.21.15.111:0/2490569699 --> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d257acef10 con 0x55d257d3eef0 2023-11-28T13:02:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.076+0000 7f2433fff700 1 --2- 172.21.15.111:0/2490569699 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55d257d3eef0 0x55d257d22010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.076+0000 7f2412ffd700 10 client.6405 ms_handle_connect on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.076+0000 7f2433fff700 1 --2- 172.21.15.111:0/2490569699 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55d257d3eef0 0x55d257d22010 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=894a1bc91ba63b78 server_cookie=250d97722a6dc642 in_seq=0 out_seq=0 2023-11-28T13:02:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2412ffd700 1 -- 172.21.15.111:0/2490569699 <== mds.0 v2:172.21.15.111:6835/1897250522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55d257acef10 con 0x55d257d3eef0 2023-11-28T13:02:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2412ffd700 10 client.6405 handle_command_reply: tid=0 2023-11-28T13:02:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2446788b80 1 client.6405 shutdown 2023-11-28T13:02:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2446788b80 1 -- 172.21.15.111:0/2490569699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 msgr2=0x55d257d3e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2446788b80 1 --2- 172.21.15.111:0/2490569699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 0x55d257d3e9b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f242c00f820 tx=0x7f242c00fb30 comp rx=0 tx=0).stop 2023-11-28T13:02:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2446788b80 1 -- 172.21.15.111:0/2490569699 shutdown_connections 2023-11-28T13:02:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2446788b80 1 --2- 172.21.15.111:0/2490569699 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55d257d3eef0 0x55d257d22010 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2446788b80 1 --2- 172.21.15.111:0/2490569699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 0x55d257d3e9b0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2446788b80 1 --2- 172.21.15.111:0/2490569699 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257cf28c0 0x55d257d0d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2412ffd700 0 client.6405 ms_handle_reset on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:00.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2446788b80 1 --2- 172.21.15.111:0/2490569699 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d22a70 0x55d257d40680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2446788b80 1 -- 172.21.15.111:0/2490569699 >> 172.21.15.111:0/2490569699 conn(0x55d257cf5fc0 msgr2=0x55d257d3d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:00.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2446788b80 1 -- 172.21.15.111:0/2490569699 shutdown_connections 2023-11-28T13:02:00.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2446788b80 1 -- 172.21.15.111:0/2490569699 wait complete. 2023-11-28T13:02:00.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.077+0000 7f2446788b80 20 client.6405 trim_cache size 0 max 16384 2023-11-28T13:02:00.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.094+0000 7f2446788b80 1 Processor -- start 2023-11-28T13:02:00.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.094+0000 7f2446788b80 1 -- start start 2023-11-28T13:02:00.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2446788b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257cf28c0 0x55d257ceef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2446788b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257d22a70 0x55d257d3d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2446788b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 0x55d257cf3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2446788b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d257d5f740 con 0x55d257d23380 2023-11-28T13:02:00.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2446788b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d257d344a0 con 0x55d257d22a70 2023-11-28T13:02:00.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2446788b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d257cf2cb0 con 0x55d257cf28c0 2023-11-28T13:02:00.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2438b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 0x55d257cf3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2438b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 0x55d257cf3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37406/0 (socket says 172.21.15.111:37406) 2023-11-28T13:02:00.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2438b9c700 1 -- 172.21.15.111:0/3854342431 learned_addr learned my addr 172.21.15.111:0/3854342431 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:00.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2438b9c700 1 -- 172.21.15.111:0/3854342431 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257cf28c0 msgr2=0x55d257ceef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:00.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2438b9c700 1 --2- 172.21.15.111:0/3854342431 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257cf28c0 0x55d257ceef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2438b9c700 1 -- 172.21.15.111:0/3854342431 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257d22a70 msgr2=0x55d257d3d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:00.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2438b9c700 1 --2- 172.21.15.111:0/3854342431 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257d22a70 0x55d257d3d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2438b9c700 1 -- 172.21.15.111:0/3854342431 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d257c11140 con 0x55d257d23380 2023-11-28T13:02:00.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f2438b9c700 1 --2- 172.21.15.111:0/3854342431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 0x55d257cf3290 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f242c005c90 tx=0x7f242c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8bb3ab1b52bdeba server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:00.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f241a7fc700 1 -- 172.21.15.111:0/3854342431 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c013db0 con 0x55d257d23380 2023-11-28T13:02:00.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f241a7fc700 1 -- 172.21.15.111:0/3854342431 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242c00d040 con 0x55d257d23380 2023-11-28T13:02:00.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.095+0000 7f241a7fc700 1 -- 172.21.15.111:0/3854342431 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c01cb30 con 0x55d257d23380 2023-11-28T13:02:00.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.096+0000 7f2446788b80 1 -- 172.21.15.111:0/3854342431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 msgr2=0x55d257cf3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.096+0000 7f2446788b80 1 --2- 172.21.15.111:0/3854342431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 0x55d257cf3290 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f242c005c90 tx=0x7f242c013000 comp rx=0 tx=0).stop 2023-11-28T13:02:00.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.096+0000 7f2446788b80 1 -- 172.21.15.111:0/3854342431 shutdown_connections 2023-11-28T13:02:00.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.096+0000 7f2446788b80 1 --2- 172.21.15.111:0/3854342431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d23380 0x55d257cf3290 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.096+0000 7f2446788b80 1 --2- 172.21.15.111:0/3854342431 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257cf28c0 0x55d257ceef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.096+0000 7f2446788b80 1 --2- 172.21.15.111:0/3854342431 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257d22a70 0x55d257d3d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.096+0000 7f2446788b80 1 -- 172.21.15.111:0/3854342431 >> 172.21.15.111:0/3854342431 conn(0x55d257cf5fc0 msgr2=0x55d257d22010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:00.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.096+0000 7f2446788b80 1 -- 172.21.15.111:0/3854342431 shutdown_connections 2023-11-28T13:02:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.096+0000 7f2446788b80 1 -- 172.21.15.111:0/3854342431 wait complete. 2023-11-28T13:02:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.096+0000 7f2446788b80 1 -- start start 2023-11-28T13:02:00.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.097+0000 7f2446788b80 1 Processor -- start 2023-11-28T13:02:00.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.097+0000 7f2446788b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257cf28c0 0x55d257cf6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.097+0000 7f2446788b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d22a70 0x55d257d1eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.097+0000 7f2446788b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d23380 0x55d257d1f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.097+0000 7f2446788b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d257cf7280 con 0x55d257d22a70 2023-11-28T13:02:00.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.097+0000 7f2446788b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d257d56980 con 0x55d257cf28c0 2023-11-28T13:02:00.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.097+0000 7f2446788b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d257d56090 con 0x55d257d23380 2023-11-28T13:02:00.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.097+0000 7f2446788b80 10 client.0 _ll_register_callbacks cb 0x55d257af0240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:02:00.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.097+0000 7f24337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d22a70 0x55d257d1eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f24337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d22a70 0x55d257d1eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37412/0 (socket says 172.21.15.111:37412) 2023-11-28T13:02:00.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f24337fe700 1 -- 172.21.15.111:0/872034592 learned_addr learned my addr 172.21.15.111:0/872034592 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:00.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2433fff700 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257cf28c0 0x55d257cf6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f24337fe700 1 -- 172.21.15.111:0/872034592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d23380 msgr2=0x55d257d1f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:00.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2438b9c700 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d23380 0x55d257d1f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f24337fe700 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d23380 0x55d257d1f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f24337fe700 1 -- 172.21.15.111:0/872034592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257cf28c0 msgr2=0x55d257cf6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:00.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f24337fe700 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257cf28c0 0x55d257cf6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f24337fe700 1 -- 172.21.15.111:0/872034592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d257a78d90 con 0x55d257d22a70 2023-11-28T13:02:00.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2433fff700 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257cf28c0 0x55d257cf6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:02:00.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2438b9c700 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d23380 0x55d257d1f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:02:00.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f24337fe700 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d22a70 0x55d257d1eb10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f24280577f0 tx=0x7f2428004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110526818e31b90b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:00.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2412ffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:02:00.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2412ffd700 1 -- 172.21.15.111:0/872034592 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2428053e80 con 0x55d257d22a70 2023-11-28T13:02:00.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2412ffd700 1 -- 172.21.15.111:0/872034592 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f2428053930 con 0x55d257d22a70 2023-11-28T13:02:00.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2446788b80 1 -- 172.21.15.111:0/872034592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d257c08160 con 0x55d257d22a70 2023-11-28T13:02:00.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2412ffd700 1 -- 172.21.15.111:0/872034592 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2428053af0 con 0x55d257d22a70 2023-11-28T13:02:00.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2446788b80 10 client.6411 fetch_fsmap learned FSMap version 32 2023-11-28T13:02:00.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2446788b80 1 -- 172.21.15.111:0/872034592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55d257a9f200 con 0x55d257d22a70 2023-11-28T13:02:00.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2412ffd700 1 -- 172.21.15.111:0/872034592 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d257c08160 con 0x55d257d22a70 2023-11-28T13:02:00.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2412ffd700 1 -- 172.21.15.111:0/872034592 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f2428056aa0 con 0x55d257d22a70 2023-11-28T13:02:00.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2446788b80 10 client.6411 fetch_fsmap finished waiting for FSMap version 32 2023-11-28T13:02:00.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2446788b80 10 client.6411 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T13:02:00.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2446788b80 20 client.6411 populate_metadata read hostname 'smithi111' 2023-11-28T13:02:00.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2446788b80 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55d257d8c650 0x55d257d8ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2446788b80 4 client.6411 mds_command: new command op to 6252 tid=0[{"prefix": "status"}] 2023-11-28T13:02:00.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.098+0000 7f2446788b80 1 -- 172.21.15.111:0/872034592 --> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d257bf2170 con 0x55d257d8c650 2023-11-28T13:02:00.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2433fff700 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55d257d8c650 0x55d257d8ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2412ffd700 10 client.6411 ms_handle_connect on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:00.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2433fff700 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55d257d8c650 0x55d257d8ca20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a16b0c85f05ca1da server_cookie=97b0188af579655 in_seq=0 out_seq=0 2023-11-28T13:02:00.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2412ffd700 1 -- 172.21.15.111:0/872034592 <== mds.0 v2:172.21.15.111:6835/1897250522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55d257bf2170 con 0x55d257d8c650 2023-11-28T13:02:00.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2412ffd700 10 client.6411 handle_command_reply: tid=0 2023-11-28T13:02:00.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2446788b80 1 client.6411 shutdown 2023-11-28T13:02:00.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2446788b80 1 -- 172.21.15.111:0/872034592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d22a70 msgr2=0x55d257d1eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2446788b80 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d22a70 0x55d257d1eb10 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f24280577f0 tx=0x7f2428004680 comp rx=0 tx=0).stop 2023-11-28T13:02:00.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2446788b80 1 -- 172.21.15.111:0/872034592 shutdown_connections 2023-11-28T13:02:00.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2446788b80 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55d257d8c650 0x55d257d8ca20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2446788b80 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55d257d23380 0x55d257d1f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2446788b80 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55d257cf28c0 0x55d257cf6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2412ffd700 0 client.6411 ms_handle_reset on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:00.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2446788b80 1 --2- 172.21.15.111:0/872034592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d257d22a70 0x55d257d1eb10 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2446788b80 1 -- 172.21.15.111:0/872034592 >> 172.21.15.111:0/872034592 conn(0x55d257cf5fc0 msgr2=0x55d257d21f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:00.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2446788b80 1 -- 172.21.15.111:0/872034592 shutdown_connections 2023-11-28T13:02:00.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2446788b80 1 -- 172.21.15.111:0/872034592 wait complete. 2023-11-28T13:02:00.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.099+0000 7f2446788b80 20 client.6411 trim_cache size 0 max 16384 2023-11-28T13:02:00.110 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:02:00.110 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "a1b2d00c-aeec-4b5d-93e3-41820c4189ba", 2023-11-28T13:02:00.110 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-11-28T13:02:00.111 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 6252, 2023-11-28T13:02:00.111 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-11-28T13:02:00.111 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-11-28T13:02:00.111 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-11-28T13:02:00.111 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 5.8413319179999998, 2023-11-28T13:02:00.111 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 32, 2023-11-28T13:02:00.111 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 67, 2023-11-28T13:02:00.111 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 67, 2023-11-28T13:02:00.112 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 11.29945762 2023-11-28T13:02:00.112 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:02:00.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.100+0000 7f243939d700 1 -- 172.21.15.111:0/2461298312 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24200682f0 msgr2=0x7f242006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.100+0000 7f243939d700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24200682f0 0x7f242006a7a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f24341550f0 tx=0x7f2424008040 comp rx=0 tx=0).stop 2023-11-28T13:02:00.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.100+0000 7f243939d700 1 -- 172.21.15.111:0/2461298312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24340cb390 msgr2=0x7f24340c8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.100+0000 7f243939d700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24340cb390 0x7f24340c8c00 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2428009fd0 tx=0x7f2428004750 comp rx=0 tx=0).stop 2023-11-28T13:02:00.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.100+0000 7f243939d700 1 -- 172.21.15.111:0/2461298312 shutdown_connections 2023-11-28T13:02:00.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.100+0000 7f243939d700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f243415a210 0x7f243414eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.100+0000 7f243939d700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f24200682f0 0x7f242006a7a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.100+0000 7f243939d700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f24340caf40 0x7f243414f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.100+0000 7f243939d700 1 --2- 172.21.15.111:0/2461298312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24340cb390 0x7f24340c8c00 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.100+0000 7f243939d700 1 -- 172.21.15.111:0/2461298312 >> 172.21.15.111:0/2461298312 conn(0x7f24340cf110 msgr2=0x7f2434065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:00.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.100+0000 7f243939d700 1 -- 172.21.15.111:0/2461298312 shutdown_connections 2023-11-28T13:02:00.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.100+0000 7f243939d700 1 -- 172.21.15.111:0/2461298312 wait complete. 2023-11-28T13:02:00.115 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T13:02:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.233+0000 7fb650ef3700 1 Processor -- start 2023-11-28T13:02:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.234+0000 7fb650ef3700 1 -- start start 2023-11-28T13:02:00.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.234+0000 7fb650ef3700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb64c0cfc90 0x7fb64c0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.234+0000 7fb650ef3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c0d05a0 con 0x7fb64c0cc5a0 2023-11-28T13:02:00.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.234+0000 7fb650ef3700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c0ca260 con 0x7fb64c0cdef0 2023-11-28T13:02:00.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.234+0000 7fb650ef3700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb64c0ccaa0 con 0x7fb64c0cfc90 2023-11-28T13:02:00.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb649d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb64c0cc5a0 0x7fb64c0cd7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38198/0 (socket says 172.21.15.111:38198) 2023-11-28T13:02:00.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb649d9b700 1 -- 172.21.15.111:0/1206313756 learned_addr learned my addr 172.21.15.111:0/1206313756 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:02:00.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64a59c700 1 --2- 172.21.15.111:0/1206313756 >> v2:172.21.15.149:3300/0 conn(0x7fb64c0cfc90 0x7fb64c0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933923110 0 0) 0x7fb634001110 con 0x7fb64c0cc5a0 2023-11-28T13:02:00.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6380035b0 con 0x7fb64c0cc5a0 2023-11-28T13:02:00.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 295416756 0 0) 0x7fb6380035b0 con 0x7fb64c0cc5a0 2023-11-28T13:02:00.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 >> v2:172.21.15.149:3300/0 conn(0x7fb64c0cfc90 msgr2=0x7fb64c0d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:00.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 --2- 172.21.15.111:0/1206313756 >> v2:172.21.15.149:3300/0 conn(0x7fb64c0cfc90 0x7fb64c0d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 >> v1:172.21.15.149:6789/0 conn(0x7fb64c0cdef0 legacy=0x7fb64c0ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64c0ccd80 con 0x7fb64c0cc5a0 2023-11-28T13:02:00.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fb634001800 con 0x7fb64c0cc5a0 2023-11-28T13:02:00.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 >> v1:172.21.15.111:6789/0 conn(0x7fb64c0cc5a0 legacy=0x7fb64c0cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 --2- 172.21.15.111:0/1206313756 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb638003a00 0x7fb638003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 --2- 172.21.15.111:0/1206313756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6380043f0 0x7fb6380048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 --2- 172.21.15.111:0/1206313756 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6380050b0 0x7fb63800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c0ccaa0 con 0x7fb6380043f0 2023-11-28T13:02:00.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63800c000 con 0x7fb6380050b0 2023-11-28T13:02:00.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63800c1c0 con 0x7fb638003a00 2023-11-28T13:02:00.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb649d9b700 1 --2- 172.21.15.111:0/1206313756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6380043f0 0x7fb6380048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64a59c700 1 --2- 172.21.15.111:0/1206313756 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb638003a00 0x7fb638003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.235+0000 7fb64ad9d700 1 --2- 172.21.15.111:0/1206313756 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6380050b0 0x7fb63800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb649d9b700 1 -- 172.21.15.111:0/1206313756 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb638003a00 msgr2=0x7fb638003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:00.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb649d9b700 1 --2- 172.21.15.111:0/1206313756 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb638003a00 0x7fb638003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb649d9b700 1 -- 172.21.15.111:0/1206313756 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6380050b0 msgr2=0x7fb63800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:00.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb649d9b700 1 --2- 172.21.15.111:0/1206313756 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb6380050b0 0x7fb63800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb64ad9d700 1 -- 172.21.15.111:0/1206313756 reap_dead start 2023-11-28T13:02:00.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb649d9b700 1 -- 172.21.15.111:0/1206313756 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6380035b0 con 0x7fb6380043f0 2023-11-28T13:02:00.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb649d9b700 1 --2- 172.21.15.111:0/1206313756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6380043f0 0x7fb6380048b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb63400c210 tx=0x7fb638005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f299e034e1722e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:00.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63400cd50 con 0x7fb6380043f0 2023-11-28T13:02:00.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb634005960 con 0x7fb6380043f0 2023-11-28T13:02:00.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb64959a700 1 -- 172.21.15.111:0/1206313756 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb634005d10 con 0x7fb6380043f0 2023-11-28T13:02:00.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb650ef3700 1 -- 172.21.15.111:0/1206313756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6380043f0 msgr2=0x7fb6380048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb650ef3700 1 --2- 172.21.15.111:0/1206313756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6380043f0 0x7fb6380048b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb63400c210 tx=0x7fb638005880 comp rx=0 tx=0).stop 2023-11-28T13:02:00.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb650ef3700 1 -- 172.21.15.111:0/1206313756 shutdown_connections 2023-11-28T13:02:00.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb650ef3700 1 --2- 172.21.15.111:0/1206313756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6380043f0 0x7fb6380048b0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb650ef3700 1 -- 172.21.15.111:0/1206313756 >> 172.21.15.111:0/1206313756 conn(0x7fb64c098ee0 msgr2=0x7fb64c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:00.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.236+0000 7fb650ef3700 1 -- 172.21.15.111:0/1206313756 shutdown_connections 2023-11-28T13:02:00.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.237+0000 7fb650ef3700 1 -- 172.21.15.111:0/1206313756 wait complete. 2023-11-28T13:02:00.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.238+0000 7fb650ef3700 1 Processor -- start 2023-11-28T13:02:00.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.238+0000 7fb650ef3700 1 -- start start 2023-11-28T13:02:00.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.239+0000 7fb650ef3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6380043f0 0x7fb64c1efed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.239+0000 7fb650ef3700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb64c0b1530 0x7fb64c0b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.239+0000 7fb650ef3700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb64c0c3740 0x7fb64c0c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.239+0000 7fb650ef3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c0b1ee0 con 0x7fb6380043f0 2023-11-28T13:02:00.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.239+0000 7fb650ef3700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c0c4430 con 0x7fb64c0c3740 2023-11-28T13:02:00.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.239+0000 7fb650ef3700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c0c4710 con 0x7fb64c0b1530 2023-11-28T13:02:00.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.239+0000 7fb64a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6380043f0 0x7fb64c1efed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.239+0000 7fb64a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6380043f0 0x7fb64c1efed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37430/0 (socket says 172.21.15.111:37430) 2023-11-28T13:02:00.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.239+0000 7fb64a59c700 1 -- 172.21.15.111:0/2011335041 learned_addr learned my addr 172.21.15.111:0/2011335041 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:00.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.239+0000 7fb64a59c700 1 -- 172.21.15.111:0/2011335041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb64c0b1530 msgr2=0x7fb64c0b19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.239+0000 7fb649d9b700 1 --2- 172.21.15.111:0/2011335041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb64c0b1530 0x7fb64c0b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.240+0000 7fb64a59c700 1 --2- 172.21.15.111:0/2011335041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb64c0b1530 0x7fb64c0b19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.240+0000 7fb64a59c700 1 -- 172.21.15.111:0/2011335041 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb64c0c3740 msgr2=0x7fb64c0c3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.240+0000 7fb64a59c700 1 --2- 172.21.15.111:0/2011335041 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb64c0c3740 0x7fb64c0c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.240+0000 7fb64a59c700 1 -- 172.21.15.111:0/2011335041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64c0cd340 con 0x7fb6380043f0 2023-11-28T13:02:00.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.240+0000 7fb64a59c700 1 --2- 172.21.15.111:0/2011335041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6380043f0 0x7fb64c1efed0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009f90 tx=0x7fb63c00f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:00.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.240+0000 7fb642ffd700 1 -- 172.21.15.111:0/2011335041 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c00e040 con 0x7fb6380043f0 2023-11-28T13:02:00.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.240+0000 7fb642ffd700 1 -- 172.21.15.111:0/2011335041 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63c00f040 con 0x7fb6380043f0 2023-11-28T13:02:00.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.240+0000 7fb650ef3700 1 -- 172.21.15.111:0/2011335041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64c0cd060 con 0x7fb6380043f0 2023-11-28T13:02:00.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.240+0000 7fb642ffd700 1 -- 172.21.15.111:0/2011335041 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c015430 con 0x7fb6380043f0 2023-11-28T13:02:00.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.240+0000 7fb650ef3700 1 -- 172.21.15.111:0/2011335041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64c143280 con 0x7fb6380043f0 2023-11-28T13:02:00.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.241+0000 7fb650ef3700 1 -- 172.21.15.111:0/2011335041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64c0c5320 con 0x7fb6380043f0 2023-11-28T13:02:00.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.241+0000 7fb642ffd700 1 -- 172.21.15.111:0/2011335041 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb63c015590 con 0x7fb6380043f0 2023-11-28T13:02:00.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.241+0000 7fb642ffd700 1 --2- 172.21.15.111:0/2011335041 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb62c0682f0 0x7fb62c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:00.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.242+0000 7fb642ffd700 1 -- 172.21.15.111:0/2011335041 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fb63c086ad0 con 0x7fb6380043f0 2023-11-28T13:02:00.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.242+0000 7fb649d9b700 1 --2- 172.21.15.111:0/2011335041 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb62c0682f0 0x7fb62c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:00.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.242+0000 7fb649d9b700 1 --2- 172.21.15.111:0/2011335041 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb62c0682f0 0x7fb62c06a7a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb64c0b2540 tx=0x7fb634007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:00.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.246+0000 7fb642ffd700 1 -- 172.21.15.111:0/2011335041 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb63c0531f0 con 0x7fb6380043f0 2023-11-28T13:02:00.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.473+0000 7fb650ef3700 1 -- 172.21.15.111:0/2011335041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb64c067170 con 0x7fb6380043f0 2023-11-28T13:02:00.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.473+0000 7fb642ffd700 1 -- 172.21.15.111:0/2011335041 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v32) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb63c056810 con 0x7fb6380043f0 2023-11-28T13:02:00.475 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T13:02:00.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.475+0000 7fb650ef3700 1 -- 172.21.15.111:0/2011335041 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb62c0682f0 msgr2=0x7fb62c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.475+0000 7fb650ef3700 1 --2- 172.21.15.111:0/2011335041 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb62c0682f0 0x7fb62c06a7a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb64c0b2540 tx=0x7fb634007000 comp rx=0 tx=0).stop 2023-11-28T13:02:00.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.475+0000 7fb650ef3700 1 -- 172.21.15.111:0/2011335041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6380043f0 msgr2=0x7fb64c1efed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:00.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.475+0000 7fb650ef3700 1 --2- 172.21.15.111:0/2011335041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6380043f0 0x7fb64c1efed0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009f90 tx=0x7fb63c00f490 comp rx=0 tx=0).stop 2023-11-28T13:02:00.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.475+0000 7fb650ef3700 1 -- 172.21.15.111:0/2011335041 shutdown_connections 2023-11-28T13:02:00.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.475+0000 7fb650ef3700 1 --2- 172.21.15.111:0/2011335041 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb62c0682f0 0x7fb62c06a7a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.475+0000 7fb650ef3700 1 --2- 172.21.15.111:0/2011335041 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb64c0c3740 0x7fb64c0c3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.475+0000 7fb650ef3700 1 --2- 172.21.15.111:0/2011335041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6380043f0 0x7fb64c1efed0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.475+0000 7fb650ef3700 1 --2- 172.21.15.111:0/2011335041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb64c0b1530 0x7fb64c0b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:00.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.475+0000 7fb650ef3700 1 -- 172.21.15.111:0/2011335041 >> 172.21.15.111:0/2011335041 conn(0x7fb64c098ee0 msgr2=0x7fb64c0cd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:00.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.475+0000 7fb650ef3700 1 -- 172.21.15.111:0/2011335041 shutdown_connections 2023-11-28T13:02:00.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:00.475+0000 7fb650ef3700 1 -- 172.21.15.111:0/2011335041 wait complete. 2023-11-28T13:02:00.488 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-28T13:02:00.489 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T13:02:00.489 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-11-28T13:02:00.489 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T13:02:00.489 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T13:02:00.489 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-28T13:02:00.489 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-28T13:02:00.489 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T13:02:00.489 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T13:02:00.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:02:00.490 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-11-28T13:02:00.506 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T13:02:00.507 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T13:02:00.507 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T13:02:00.507 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:02:00.507 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-11-28T13:02:00.507 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:02:00.507 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T13:02:00.507 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-11-28T13:02:00.507 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T13:02:00.507 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-11-28T13:02:00.508 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T13:02:00.508 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-11-28T13:02:00.508 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T13:02:00.508 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-11-28T13:02:00.508 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T13:02:00.508 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 42206sec preferred_lft 42206sec 2023-11-28T13:02:00.508 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::9195:85a0:8e3c:2da4/64 scope link noprefixroute 2023-11-28T13:02:00.508 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:02:00.508 INFO:teuthology.orchestra.run.smithi111.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T13:02:00.508 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 96:9a:a2:c9:67:8f brd ff:ff:ff:ff:ff:ff 2023-11-28T13:02:00.509 INFO:teuthology.orchestra.run.smithi111.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T13:02:00.509 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:02:00.509 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::f893:44ff:fe17:3abf/64 scope link 2023-11-28T13:02:00.509 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:02:00.509 INFO:teuthology.orchestra.run.smithi111.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T13:02:00.509 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 96:9a:a2:c9:67:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T13:02:00.509 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::949a:a2ff:fec9:678f/64 scope link 2023-11-28T13:02:00.509 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:02:00.509 INFO:teuthology.orchestra.run.smithi111.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T13:02:00.509 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether ee:e6:45:35:2f:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T13:02:00.510 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ece6:45ff:fe35:2fc2/64 scope link 2023-11-28T13:02:00.510 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-11-28T13:02:00.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:02:00.510 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T13:02:00.572 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-11-28T13:02:00.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:02:00.573 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-11-28T13:02:00.632 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-11-28T13:02:00.632 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T13:02:00.632 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T13:02:00.632 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T13:02:00.632 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T13:02:00.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:02:00.633 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-11-28T13:02:00.633 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-11-28T13:02:00.633 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T13:02:00.633 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T13:02:00.633 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T13:02:00.633 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-11-28T13:02:00.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:02:00.737 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-11-28T13:02:00.754 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T13:02:00.754 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T13:02:00.754 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T13:02:00.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:02:00.755 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-11-28T13:02:00.820 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T13:02:00.821 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T13:02:00.878 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T13:02:00.878 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T13:02:00.878 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T13:02:00.934 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-11-28T13:02:00.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:02:00.935 DEBUG:teuthology.orchestra.run.smithi111:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T13:02:01.020 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T13:02:01.021 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T13:02:01.021 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T13:02:01.029 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T13:02:01.029 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T13:02:01.029 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T13:02:01.030 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T13:02:01.030 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T13:02:01.103 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-11-28T13:02:01.103 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-11-28T13:02:01.103 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-11-28T13:02:01.103 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.149:3300,172.21.15.149:3301" 2023-11-28T13:02:01.103 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-28T13:02:01.103 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790" 2023-11-28T13:02:01.103 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.111:6789,172.21.15.149:6789,172.21.15.149:6790:/ 2023-11-28T13:02:01.103 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=a1b2d00c-aeec-4b5d-93e3-41820c4189ba" will pass to kernel 2023-11-28T13:02:01.105 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-28T13:02:01.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:02:01.105 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T13:02:01.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:02:01.132 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-28T13:02:01.212 INFO:teuthology.orchestra.run.smithi111.stdout:client5444 2023-11-28T13:02:01.212 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:02:01.212 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:02:01.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:02:01.270 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_test) 2023-11-28T13:02:01.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:02:01.331 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir_test/file) 2023-11-28T13:02:01.389 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits daemon-helper kill python3 -c ' 2023-11-28T13:02:01.389 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-11-28T13:02:01.389 DEBUG:teuthology.orchestra.run.smithi111:> import stat 2023-11-28T13:02:01.389 DEBUG:teuthology.orchestra.run.smithi111:> 2023-11-28T13:02:01.389 DEBUG:teuthology.orchestra.run.smithi111:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_test/file").st_ino) 2023-11-28T13:02:01.389 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-11-28T13:02:01.495 INFO:teuthology.orchestra.run.smithi111.stdout:1099511627777 2023-11-28T13:02:01.684 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:02:01.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.844+0000 7f47273a0700 1 Processor -- start 2023-11-28T13:02:01.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.844+0000 7f47273a0700 1 -- start start 2023-11-28T13:02:01.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.845+0000 7f47273a0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4720159520 0x7f47200cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:01.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.845+0000 7f47273a0700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f472006c0a0 0x7f47200cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:01.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.845+0000 7f47273a0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47200b1510 con 0x7f47200aeb10 2023-11-28T13:02:01.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.845+0000 7f47273a0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47201598f0 con 0x7f4720159520 2023-11-28T13:02:01.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.845+0000 7f47273a0700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47200b0f50 con 0x7f472006c0a0 2023-11-28T13:02:01.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.845+0000 7f472513c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4720159520 0x7f47200cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:01.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.845+0000 7f472593d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f47200aeb10 0x7f47200af530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38224/0 (socket says 172.21.15.111:38224) 2023-11-28T13:02:01.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.845+0000 7f472593d700 1 -- 172.21.15.111:0/3063279114 learned_addr learned my addr 172.21.15.111:0/3063279114 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:02:01.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.846+0000 7f472493b700 1 --2- 172.21.15.111:0/3063279114 >> v2:172.21.15.149:3300/0 conn(0x7f472006c0a0 0x7f47200cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:01.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.846+0000 7f472513c700 1 -- 172.21.15.111:0/3063279114 >> v2:172.21.15.149:3300/0 conn(0x7f472006c0a0 msgr2=0x7f47200cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:01.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.846+0000 7f472513c700 1 --2- 172.21.15.111:0/3063279114 >> v2:172.21.15.149:3300/0 conn(0x7f472006c0a0 0x7f47200cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:01.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.846+0000 7f472513c700 1 -- 172.21.15.111:0/3063279114 >> v1:172.21.15.111:6789/0 conn(0x7f47200aeb10 legacy=0x7f47200af530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:01.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.846+0000 7f472513c700 1 -- 172.21.15.111:0/3063279114 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47200b1230 con 0x7f4720159520 2023-11-28T13:02:01.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.846+0000 7f4717fff700 1 -- 172.21.15.111:0/3063279114 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611307798 0 0) 0x7f47200b1510 con 0x7f47200aeb10 2023-11-28T13:02:01.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.846+0000 7f472513c700 1 --2- 172.21.15.111:0/3063279114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720159520 0x7f47200cd7a0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f472006ced0 tx=0x7f4710009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5f7a94e3578521 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:01.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.846+0000 7f4717fff700 1 -- 172.21.15.111:0/3063279114 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4710011830 con 0x7f4720159520 2023-11-28T13:02:01.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.846+0000 7f4717fff700 1 -- 172.21.15.111:0/3063279114 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4710011990 con 0x7f4720159520 2023-11-28T13:02:01.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.846+0000 7f4717fff700 1 -- 172.21.15.111:0/3063279114 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4710011d20 con 0x7f4720159520 2023-11-28T13:02:01.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.846+0000 7f47273a0700 1 -- 172.21.15.111:0/3063279114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720159520 msgr2=0x7f47200cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:01.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.846+0000 7f47273a0700 1 --2- 172.21.15.111:0/3063279114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720159520 0x7f47200cd7a0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f472006ced0 tx=0x7f4710009b20 comp rx=0 tx=0).stop 2023-11-28T13:02:01.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.847+0000 7f47273a0700 1 -- 172.21.15.111:0/3063279114 shutdown_connections 2023-11-28T13:02:01.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.847+0000 7f47273a0700 1 --2- 172.21.15.111:0/3063279114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720159520 0x7f47200cd7a0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:01.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.847+0000 7f47273a0700 1 --2- 172.21.15.111:0/3063279114 >> v2:172.21.15.149:3300/0 conn(0x7f472006c0a0 0x7f47200cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:01.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.847+0000 7f47273a0700 1 -- 172.21.15.111:0/3063279114 >> 172.21.15.111:0/3063279114 conn(0x7f472006bca0 msgr2=0x7f47200cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:01.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.847+0000 7f47273a0700 1 -- 172.21.15.111:0/3063279114 shutdown_connections 2023-11-28T13:02:01.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.847+0000 7f47273a0700 1 -- 172.21.15.111:0/3063279114 wait complete. 2023-11-28T13:02:01.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.848+0000 7f47273a0700 1 Processor -- start 2023-11-28T13:02:01.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.848+0000 7f47273a0700 1 -- start start 2023-11-28T13:02:01.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f47273a0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f472006c0a0 0x7f4720165030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:01.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f47273a0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47200aeb10 0x7f47200c28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:01.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f47273a0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4720159520 0x7f4720143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:01.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f47273a0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4720165570 con 0x7f472006c0a0 2023-11-28T13:02:01.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f47273a0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47200c2de0 con 0x7f4720159520 2023-11-28T13:02:01.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f47273a0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4720143b40 con 0x7f47200aeb10 2023-11-28T13:02:01.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f472513c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f472006c0a0 0x7f4720165030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:01.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f472493b700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47200aeb10 0x7f47200c28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:01.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f472513c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f472006c0a0 0x7f4720165030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37448/0 (socket says 172.21.15.111:37448) 2023-11-28T13:02:01.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f472513c700 1 -- 172.21.15.111:0/850565190 learned_addr learned my addr 172.21.15.111:0/850565190 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:01.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f472593d700 1 --2- 172.21.15.111:0/850565190 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4720159520 0x7f4720143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:01.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f472513c700 1 -- 172.21.15.111:0/850565190 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47200aeb10 msgr2=0x7f47200c28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:01.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f472513c700 1 --2- 172.21.15.111:0/850565190 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47200aeb10 0x7f47200c28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:01.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f472513c700 1 -- 172.21.15.111:0/850565190 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4720159520 msgr2=0x7f4720143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:01.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f472513c700 1 --2- 172.21.15.111:0/850565190 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4720159520 0x7f4720143600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:01.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f472513c700 1 -- 172.21.15.111:0/850565190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47100094a0 con 0x7f472006c0a0 2023-11-28T13:02:01.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f472513c700 1 --2- 172.21.15.111:0/850565190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f472006c0a0 0x7f4720165030 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f4710009fd0 tx=0x7f4710004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:01.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f4715ffb700 1 -- 172.21.15.111:0/850565190 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47100036a0 con 0x7f472006c0a0 2023-11-28T13:02:01.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f4715ffb700 1 -- 172.21.15.111:0/850565190 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471001e070 con 0x7f472006c0a0 2023-11-28T13:02:01.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f47273a0700 1 -- 172.21.15.111:0/850565190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47200acc90 con 0x7f472006c0a0 2023-11-28T13:02:01.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.849+0000 7f4715ffb700 1 -- 172.21.15.111:0/850565190 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4710003a20 con 0x7f472006c0a0 2023-11-28T13:02:01.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.850+0000 7f47273a0700 1 -- 172.21.15.111:0/850565190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47200ac9b0 con 0x7f472006c0a0 2023-11-28T13:02:01.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.851+0000 7f47273a0700 1 -- 172.21.15.111:0/850565190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47200acf70 con 0x7f472006c0a0 2023-11-28T13:02:01.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.851+0000 7f4715ffb700 1 -- 172.21.15.111:0/850565190 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f471000f050 con 0x7f472006c0a0 2023-11-28T13:02:01.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.851+0000 7f4715ffb700 1 --2- 172.21.15.111:0/850565190 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f47040682f0 0x7f470406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:01.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.852+0000 7f472493b700 1 --2- 172.21.15.111:0/850565190 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f47040682f0 0x7f470406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:01.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.852+0000 7f4715ffb700 1 -- 172.21.15.111:0/850565190 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f4710086dd0 con 0x7f472006c0a0 2023-11-28T13:02:01.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.852+0000 7f472493b700 1 --2- 172.21.15.111:0/850565190 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f47040682f0 0x7f470406a7a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f47201428f0 tx=0x7f471c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:01.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:01.856+0000 7f4715ffb700 1 -- 172.21.15.111:0/850565190 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f47100534f0 con 0x7f472006c0a0 2023-11-28T13:02:02.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.084+0000 7f47273a0700 1 -- 172.21.15.111:0/850565190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4720067170 con 0x7f472006c0a0 2023-11-28T13:02:02.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.085+0000 7f4715ffb700 1 -- 172.21.15.111:0/850565190 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4659 (secure 0 0 0) 0x7f4710020020 con 0x7f472006c0a0 2023-11-28T13:02:02.086 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:02:02.086 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/1668572397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1668572397},{"type":"v1","addr":"172.21.15.149:6834","nonce":1668572397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5755,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1129092621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1129092621},{"type":"v1","addr":"172.21.15.149:6835","nonce":1129092621}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/904588136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":904588136},{"type":"v1","addr":"172.21.15.111:6836","nonce":904588136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:01:54.257638+0000","modified":"2023-11-28T13:01:57.268879+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6837/1897250522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1897250522},{"type":"v1","addr":"172.21.15.111:6837","nonce":1897250522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T13:02:02.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.087+0000 7f47273a0700 1 -- 172.21.15.111:0/850565190 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f47040682f0 msgr2=0x7f470406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.087+0000 7f47273a0700 1 --2- 172.21.15.111:0/850565190 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f47040682f0 0x7f470406a7a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f47201428f0 tx=0x7f471c008040 comp rx=0 tx=0).stop 2023-11-28T13:02:02.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.087+0000 7f47273a0700 1 -- 172.21.15.111:0/850565190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f472006c0a0 msgr2=0x7f4720165030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.087+0000 7f47273a0700 1 --2- 172.21.15.111:0/850565190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f472006c0a0 0x7f4720165030 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f4710009fd0 tx=0x7f4710004750 comp rx=0 tx=0).stop 2023-11-28T13:02:02.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.087+0000 7f47273a0700 1 -- 172.21.15.111:0/850565190 shutdown_connections 2023-11-28T13:02:02.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.087+0000 7f47273a0700 1 --2- 172.21.15.111:0/850565190 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f47040682f0 0x7f470406a7a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.087+0000 7f47273a0700 1 --2- 172.21.15.111:0/850565190 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4720159520 0x7f4720143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.087+0000 7f47273a0700 1 --2- 172.21.15.111:0/850565190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f472006c0a0 0x7f4720165030 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.087+0000 7f47273a0700 1 --2- 172.21.15.111:0/850565190 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f47200aeb10 0x7f47200c28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.087+0000 7f47273a0700 1 -- 172.21.15.111:0/850565190 >> 172.21.15.111:0/850565190 conn(0x7f472006bca0 msgr2=0x7f472006dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.087+0000 7f47273a0700 1 -- 172.21.15.111:0/850565190 shutdown_connections 2023-11-28T13:02:02.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.087+0000 7f47273a0700 1 -- 172.21.15.111:0/850565190 wait complete. 2023-11-28T13:02:02.090 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 32 2023-11-28T13:02:02.100 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2023-11-28T13:02:02.253 INFO:teuthology.orchestra.run.smithi111.stdout:{"message":"","return_code":0} 2023-11-28T13:02:02.254 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-11-28T13:02:02.254 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2023-11-28T13:02:02.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.299+0000 7f7e2900c080 1 Processor -- start 2023-11-28T13:02:02.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.299+0000 7f7e2900c080 1 -- start start 2023-11-28T13:02:02.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e2900c080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56308c06a9c0 0x56308c115640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e2900c080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56308c06ad90 0x56308c089b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e2900c080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56308c06f5a0 0x56308c069170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e2900c080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56308bf7afe0 con 0x56308c06f5a0 2023-11-28T13:02:02.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e2900c080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x56308c06f170 con 0x56308c06a9c0 2023-11-28T13:02:02.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e2900c080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x56308c068750 con 0x56308c06ad90 2023-11-28T13:02:02.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1f910700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56308c06f5a0 0x56308c069170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1f910700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56308c06f5a0 0x56308c069170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37466/0 (socket says 172.21.15.111:37466) 2023-11-28T13:02:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1f910700 1 -- 172.21.15.111:0/4292331070 learned_addr learned my addr 172.21.15.111:0/4292331070 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1e90e700 1 --2- 172.21.15.111:0/4292331070 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56308c06ad90 0x56308c089b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1f10f700 1 --2- 172.21.15.111:0/4292331070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56308c06a9c0 0x56308c115640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1f910700 1 -- 172.21.15.111:0/4292331070 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56308c06ad90 msgr2=0x56308c089b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1f910700 1 --2- 172.21.15.111:0/4292331070 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56308c06ad90 0x56308c089b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1f910700 1 -- 172.21.15.111:0/4292331070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56308c06a9c0 msgr2=0x56308c115640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1f910700 1 --2- 172.21.15.111:0/4292331070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56308c06a9c0 0x56308c115640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1f910700 1 -- 172.21.15.111:0/4292331070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e14005de0 con 0x56308c06f5a0 2023-11-28T13:02:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1f910700 1 --2- 172.21.15.111:0/4292331070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56308c06f5a0 0x56308c069170 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800c4f0 tx=0x7f7e1800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40593e844503517 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1d90c700 1 -- 172.21.15.111:0/4292331070 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1800ce80 con 0x56308c06f5a0 2023-11-28T13:02:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1d90c700 1 -- 172.21.15.111:0/4292331070 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e18010590 con 0x56308c06f5a0 2023-11-28T13:02:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.300+0000 7f7e1d90c700 1 -- 172.21.15.111:0/4292331070 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18010900 con 0x56308c06f5a0 2023-11-28T13:02:02.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.301+0000 7f7e2900c080 1 -- 172.21.15.111:0/4292331070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56308c06f5a0 msgr2=0x56308c069170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.301+0000 7f7e2900c080 1 --2- 172.21.15.111:0/4292331070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56308c06f5a0 0x56308c069170 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800c4f0 tx=0x7f7e1800c800 comp rx=0 tx=0).stop 2023-11-28T13:02:02.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.301+0000 7f7e2900c080 1 -- 172.21.15.111:0/4292331070 shutdown_connections 2023-11-28T13:02:02.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.301+0000 7f7e2900c080 1 --2- 172.21.15.111:0/4292331070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56308c06f5a0 0x56308c069170 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.301+0000 7f7e2900c080 1 --2- 172.21.15.111:0/4292331070 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56308c06a9c0 0x56308c115640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.301+0000 7f7e2900c080 1 --2- 172.21.15.111:0/4292331070 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56308c06ad90 0x56308c089b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.301+0000 7f7e2900c080 1 -- 172.21.15.111:0/4292331070 >> 172.21.15.111:0/4292331070 conn(0x56308c088e20 msgr2=0x56308c06a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.301+0000 7f7e2900c080 1 -- 172.21.15.111:0/4292331070 shutdown_connections 2023-11-28T13:02:02.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.301+0000 7f7e2900c080 1 -- 172.21.15.111:0/4292331070 wait complete. 2023-11-28T13:02:02.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.301+0000 7f7e2900c080 1 Processor -- start 2023-11-28T13:02:02.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.302+0000 7f7e2900c080 1 -- start start 2023-11-28T13:02:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.302+0000 7f7e2900c080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56308c06a9c0 0x56308c114ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.302+0000 7f7e2900c080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56308c06ad90 0x56308c115640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.302+0000 7f7e2900c080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56308c06f5a0 0x56308c115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.302+0000 7f7e2900c080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56308c1177c0 con 0x56308c06ad90 2023-11-28T13:02:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.302+0000 7f7e2900c080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x56308c07ede0 con 0x56308c06f5a0 2023-11-28T13:02:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.302+0000 7f7e2900c080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x56308c069560 con 0x56308c06a9c0 2023-11-28T13:02:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.302+0000 7f7e1e90e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56308c06ad90 0x56308c115640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.302+0000 7f7e1e90e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56308c06ad90 0x56308c115640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37470/0 (socket says 172.21.15.111:37470) 2023-11-28T13:02:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.302+0000 7f7e1e90e700 1 -- 172.21.15.111:0/2250214667 learned_addr learned my addr 172.21.15.111:0/2250214667 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.302+0000 7f7e1f910700 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56308c06f5a0 0x56308c115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.302+0000 7f7e1f10f700 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56308c06a9c0 0x56308c114ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.303+0000 7f7e1e90e700 1 -- 172.21.15.111:0/2250214667 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56308c06a9c0 msgr2=0x56308c114ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.303+0000 7f7e1e90e700 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56308c06a9c0 0x56308c114ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.303+0000 7f7e1e90e700 1 -- 172.21.15.111:0/2250214667 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56308c06f5a0 msgr2=0x56308c115b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.303+0000 7f7e1e90e700 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x56308c06f5a0 0x56308c115b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.303+0000 7f7e1e90e700 1 -- 172.21.15.111:0/2250214667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1800d040 con 0x56308c06ad90 2023-11-28T13:02:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.303+0000 7f7e1f10f700 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x56308c06a9c0 0x56308c114ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:02:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.303+0000 7f7e1e90e700 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56308c06ad90 0x56308c115640 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400ac80 tx=0x7f7e14002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.303+0000 7f7e1c90a700 1 -- 172.21.15.111:0/2250214667 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14008670 con 0x56308c06ad90 2023-11-28T13:02:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.303+0000 7f7e1c90a700 1 -- 172.21.15.111:0/2250214667 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e14012820 con 0x56308c06ad90 2023-11-28T13:02:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.303+0000 7f7e1c90a700 1 -- 172.21.15.111:0/2250214667 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14012bd0 con 0x56308c06ad90 2023-11-28T13:02:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.303+0000 7f7e2900c080 1 -- 172.21.15.111:0/2250214667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56308bf93120 con 0x56308c06ad90 2023-11-28T13:02:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.303+0000 7f7e2900c080 1 -- 172.21.15.111:0/2250214667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56308bf86830 con 0x56308c06ad90 2023-11-28T13:02:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.304+0000 7f7e1c90a700 1 -- 172.21.15.111:0/2250214667 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7e14012d30 con 0x56308c06ad90 2023-11-28T13:02:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.305+0000 7f7e1c90a700 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7e0c066c10 0x7f7e0c0690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.305+0000 7f7e1c90a700 1 -- 172.21.15.111:0/2250214667 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f7e140864a0 con 0x56308c06ad90 2023-11-28T13:02:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.305+0000 7f7e1f10f700 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7e0c066c10 0x7f7e0c0690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.305+0000 7f7e2900c080 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x56308c104c80 0x56308c1b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.305+0000 7f7e2900c080 1 -- 172.21.15.111:0/2250214667 --> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] -- osd_op(unknown.0.0:1 11.7 11:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67) v8 -- 0x56308c1b77a0 con 0x56308c104c80 2023-11-28T13:02:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.305+0000 7f7e1f10f700 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7e0c066c10 0x7f7e0c0690c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x56308c0820b0 tx=0x7f7e10001810 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.305+0000 7f7e1f910700 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x56308c104c80 0x56308c1b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.305+0000 7f7e1f910700 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x56308c104c80 0x56308c1b7260 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.306+0000 7f7e1f910700 1 -- 172.21.15.111:0/2250214667 <== osd.5 v2:172.21.15.149:6808/103795 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (94) out=94b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+94 (crc 0 0 0) 0x7f7e18017020 con 0x56308c104c80 2023-11-28T13:02:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.306+0000 7f7e2900c080 1 -- 172.21.15.111:0/2250214667 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x56308c104c80 msgr2=0x56308c1b7260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.306+0000 7f7e2900c080 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x56308c104c80 0x56308c1b7260 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.306+0000 7f7e2900c080 1 -- 172.21.15.111:0/2250214667 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7e0c066c10 msgr2=0x7f7e0c0690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.306+0000 7f7e2900c080 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7e0c066c10 0x7f7e0c0690c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x56308c0820b0 tx=0x7f7e10001810 comp rx=0 tx=0).stop 2023-11-28T13:02:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.306+0000 7f7e2900c080 1 -- 172.21.15.111:0/2250214667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56308c06ad90 msgr2=0x56308c115640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.306+0000 7f7e2900c080 1 --2- 172.21.15.111:0/2250214667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56308c06ad90 0x56308c115640 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400ac80 tx=0x7f7e14002fb0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.306+0000 7f7e1f910700 1 -- 172.21.15.111:0/2250214667 reap_dead start 2023-11-28T13:02:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.306+0000 7f7e2900c080 1 -- 172.21.15.111:0/2250214667 shutdown_connections 2023-11-28T13:02:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.306+0000 7f7e2900c080 1 -- 172.21.15.111:0/2250214667 >> 172.21.15.111:0/2250214667 conn(0x56308c088e20 msgr2=0x56308c06e570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.306+0000 7f7e2900c080 1 -- 172.21.15.111:0/2250214667 shutdown_connections 2023-11-28T13:02:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.306+0000 7f7e2900c080 1 -- 172.21.15.111:0/2250214667 wait complete. 2023-11-28T13:02:02.313 INFO:teuthology.orchestra.run.smithi111.stdout:Xfiledir_test 2023-11-28T13:02:02.313 DEBUG:teuthology.orchestra.run.smithi111:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2023-11-28T13:02:02.367 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:02:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 1099511627777, 2023-11-28T13:02:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: "ancestors": [ 2023-11-28T13:02:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:02:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: "dirino": 1099511627776, 2023-11-28T13:02:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: "dname": "file", 2023-11-28T13:02:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: "version": 2 2023-11-28T13:02:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-11-28T13:02:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:02:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: "dirino": 1, 2023-11-28T13:02:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: "dname": "dir_test", 2023-11-28T13:02:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: "version": 4 2023-11-28T13:02:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:02:02.369 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-11-28T13:02:02.369 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 11, 2023-11-28T13:02:02.369 INFO:teuthology.orchestra.run.smithi111.stdout: "old_pools": [] 2023-11-28T13:02:02.369 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:02:02.369 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:02:02.370 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'The backtrace is corrupted' 2023-11-28T13:02:02.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.454+0000 7f043088a080 1 Processor -- start 2023-11-28T13:02:02.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.454+0000 7f043088a080 1 -- start start 2023-11-28T13:02:02.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.454+0000 7f043088a080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x559c79339bd0 0x559c7940c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.454+0000 7f043088a080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559c79339fa0 0x559c79362bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.454+0000 7f043088a080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x559c79352410 0x559c79364fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.454+0000 7f043088a080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c79274fe0 con 0x559c79339fa0 2023-11-28T13:02:02.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.454+0000 7f043088a080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c7933e470 con 0x559c79352410 2023-11-28T13:02:02.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f043088a080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c7940bad0 con 0x559c79339bd0 2023-11-28T13:02:02.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f042618c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559c79339fa0 0x559c79362bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f042618c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559c79339fa0 0x559c79362bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37486/0 (socket says 172.21.15.111:37486) 2023-11-28T13:02:02.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f042618c700 1 -- 172.21.15.111:0/1212611310 learned_addr learned my addr 172.21.15.111:0/1212611310 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f042618c700 1 -- 172.21.15.111:0/1212611310 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x559c79339bd0 msgr2=0x559c7940c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f042618c700 1 --2- 172.21.15.111:0/1212611310 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x559c79339bd0 0x559c7940c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f042618c700 1 -- 172.21.15.111:0/1212611310 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x559c79352410 msgr2=0x559c79364fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f042718e700 1 --2- 172.21.15.111:0/1212611310 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x559c79352410 0x559c79364fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f042618c700 1 --2- 172.21.15.111:0/1212611310 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x559c79352410 0x559c79364fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f042618c700 1 -- 172.21.15.111:0/1212611310 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0418005d20 con 0x559c79339fa0 2023-11-28T13:02:02.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f042618c700 1 --2- 172.21.15.111:0/1212611310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559c79339fa0 0x559c79362bb0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f041800ac80 tx=0x7f041800d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77446ded7e0a11bc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f042518a700 1 -- 172.21.15.111:0/1212611310 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0418008b60 con 0x559c79339fa0 2023-11-28T13:02:02.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f042518a700 1 -- 172.21.15.111:0/1212611310 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0418008cc0 con 0x559c79339fa0 2023-11-28T13:02:02.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.455+0000 7f042518a700 1 -- 172.21.15.111:0/1212611310 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041801e690 con 0x559c79339fa0 2023-11-28T13:02:02.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.456+0000 7f043088a080 1 -- 172.21.15.111:0/1212611310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559c79339fa0 msgr2=0x559c79362bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.456+0000 7f043088a080 1 --2- 172.21.15.111:0/1212611310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559c79339fa0 0x559c79362bb0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f041800ac80 tx=0x7f041800d000 comp rx=0 tx=0).stop 2023-11-28T13:02:02.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.456+0000 7f043088a080 1 -- 172.21.15.111:0/1212611310 shutdown_connections 2023-11-28T13:02:02.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.456+0000 7f043088a080 1 --2- 172.21.15.111:0/1212611310 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x559c79352410 0x559c79364fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.456+0000 7f043088a080 1 --2- 172.21.15.111:0/1212611310 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x559c79339bd0 0x559c7940c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.456+0000 7f043088a080 1 --2- 172.21.15.111:0/1212611310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559c79339fa0 0x559c79362bb0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.456+0000 7f043088a080 1 -- 172.21.15.111:0/1212611310 >> 172.21.15.111:0/1212611310 conn(0x559c793825b0 msgr2=0x559c79367130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.456+0000 7f043088a080 1 -- 172.21.15.111:0/1212611310 shutdown_connections 2023-11-28T13:02:02.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.456+0000 7f043088a080 1 -- 172.21.15.111:0/1212611310 wait complete. 2023-11-28T13:02:02.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.456+0000 7f043088a080 1 Processor -- start 2023-11-28T13:02:02.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.456+0000 7f043088a080 1 -- start start 2023-11-28T13:02:02.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f043088a080 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559c79339bd0 0x559c79327f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f043088a080 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x559c79339fa0 0x559c7940a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f043088a080 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x559c79352410 0x559c79366c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f043088a080 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c7940edf0 con 0x559c79339bd0 2023-11-28T13:02:02.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f043088a080 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c79338210 con 0x559c79339fa0 2023-11-28T13:02:02.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f043088a080 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c7940ac90 con 0x559c79352410 2023-11-28T13:02:02.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f042698d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559c79339bd0 0x559c79327f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f042698d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559c79339bd0 0x559c79327f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37490/0 (socket says 172.21.15.111:37490) 2023-11-28T13:02:02.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f042698d700 1 -- 172.21.15.111:0/1326966770 learned_addr learned my addr 172.21.15.111:0/1326966770 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f042618c700 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x559c79339fa0 0x559c7940a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f042718e700 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x559c79352410 0x559c79366c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f042698d700 1 -- 172.21.15.111:0/1326966770 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x559c79352410 msgr2=0x559c79366c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f042698d700 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x559c79352410 0x559c79366c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f042698d700 1 -- 172.21.15.111:0/1326966770 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x559c79339fa0 msgr2=0x559c7940a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f042698d700 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x559c79339fa0 0x559c7940a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f042698d700 1 -- 172.21.15.111:0/1326966770 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c792a74f0 con 0x559c79339bd0 2023-11-28T13:02:02.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.457+0000 7f042698d700 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559c79339bd0 0x559c79327f30 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f041c00bd80 tx=0x7f041c009470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.458+0000 7f040ffff700 1 -- 172.21.15.111:0/1326966770 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c014030 con 0x559c79339bd0 2023-11-28T13:02:02.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.458+0000 7f040ffff700 1 -- 172.21.15.111:0/1326966770 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041c009e80 con 0x559c79339bd0 2023-11-28T13:02:02.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.458+0000 7f043088a080 1 -- 172.21.15.111:0/1326966770 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559c7928d120 con 0x559c79339bd0 2023-11-28T13:02:02.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.458+0000 7f040ffff700 1 -- 172.21.15.111:0/1326966770 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c00f670 con 0x559c79339bd0 2023-11-28T13:02:02.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.458+0000 7f043088a080 1 -- 172.21.15.111:0/1326966770 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c79280830 con 0x559c79339bd0 2023-11-28T13:02:02.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.459+0000 7f040ffff700 1 -- 172.21.15.111:0/1326966770 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f041c00f7d0 con 0x559c79339bd0 2023-11-28T13:02:02.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.459+0000 7f040ffff700 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0414066c10 0x7f04140690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.459+0000 7f040ffff700 1 -- 172.21.15.111:0/1326966770 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f041c015070 con 0x559c79339bd0 2023-11-28T13:02:02.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.459+0000 7f042618c700 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0414066c10 0x7f04140690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.460+0000 7f043088a080 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x559c793644d0 0x559c794b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.460+0000 7f043088a080 1 -- 172.21.15.111:0/1326966770 --> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] -- osd_op(unknown.0.0:1 11.7 11:e2e07930:::10000000001.00000000:head [setxattr parent (26) in=32b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67) v8 -- 0x559c794110e0 con 0x559c793644d0 2023-11-28T13:02:02.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.460+0000 7f042618c700 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0414066c10 0x7f04140690c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f041801d000 tx=0x7f0418008000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.460+0000 7f042718e700 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x559c793644d0 0x559c794b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.460+0000 7f042718e700 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x559c793644d0 0x559c794b4700 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.461+0000 7f042718e700 1 -- 172.21.15.111:0/1326966770 <== osd.5 v2:172.21.15.149:6808/103795 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (26)] v67'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f042000d7a0 con 0x559c793644d0 2023-11-28T13:02:02.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.461+0000 7f043088a080 1 -- 172.21.15.111:0/1326966770 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x559c793644d0 msgr2=0x559c794b4700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.461+0000 7f043088a080 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x559c793644d0 0x559c794b4700 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.461+0000 7f043088a080 1 -- 172.21.15.111:0/1326966770 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0414066c10 msgr2=0x7f04140690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.461+0000 7f043088a080 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f0414066c10 0x7f04140690c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f041801d000 tx=0x7f0418008000 comp rx=0 tx=0).stop 2023-11-28T13:02:02.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.461+0000 7f043088a080 1 -- 172.21.15.111:0/1326966770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559c79339bd0 msgr2=0x559c79327f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.461+0000 7f043088a080 1 --2- 172.21.15.111:0/1326966770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559c79339bd0 0x559c79327f30 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f041c00bd80 tx=0x7f041c009470 comp rx=0 tx=0).stop 2023-11-28T13:02:02.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.461+0000 7f042718e700 1 -- 172.21.15.111:0/1326966770 reap_dead start 2023-11-28T13:02:02.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.462+0000 7f043088a080 1 -- 172.21.15.111:0/1326966770 shutdown_connections 2023-11-28T13:02:02.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.462+0000 7f043088a080 1 -- 172.21.15.111:0/1326966770 >> 172.21.15.111:0/1326966770 conn(0x559c793825b0 msgr2=0x559c79384ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.462+0000 7f043088a080 1 -- 172.21.15.111:0/1326966770 shutdown_connections 2023-11-28T13:02:02.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.462+0000 7f043088a080 1 -- 172.21.15.111:0/1326966770 wait complete. 2023-11-28T13:02:02.468 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub start / recursive 2023-11-28T13:02:02.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.589+0000 7fe3bae7a700 1 Processor -- start 2023-11-28T13:02:02.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.590+0000 7fe3bae7a700 1 -- start start 2023-11-28T13:02:02.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.590+0000 7fe3bae7a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe3b40b2360 0x7fe3b40c6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.590+0000 7fe3bae7a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3b40ce900 0x7fe3b40ced70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.590+0000 7fe3bae7a700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe3b40b4240 0x7fe3b40b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.590+0000 7fe3bae7a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b40c6cc0 con 0x7fe3b40b2360 2023-11-28T13:02:02.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.590+0000 7fe3bae7a700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b40c7750 con 0x7fe3b40b4240 2023-11-28T13:02:02.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.590+0000 7fe3bae7a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b40cf2b0 con 0x7fe3b40ce900 2023-11-28T13:02:02.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.590+0000 7fe3b8c16700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe3b40b2360 0x7fe3b40c6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.591+0000 7fe3b8c16700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe3b40b2360 0x7fe3b40c6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37500/0 (socket says 172.21.15.111:37500) 2023-11-28T13:02:02.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.591+0000 7fe3b8c16700 1 -- 172.21.15.111:0/3444072154 learned_addr learned my addr 172.21.15.111:0/3444072154 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.591+0000 7fe3b3fff700 1 --2- 172.21.15.111:0/3444072154 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3b40ce900 0x7fe3b40ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.591+0000 7fe3b9417700 1 --2- 172.21.15.111:0/3444072154 >> v2:172.21.15.149:3300/0 conn(0x7fe3b40b4240 0x7fe3b40b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.591+0000 7fe3b8c16700 1 -- 172.21.15.111:0/3444072154 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3b40ce900 msgr2=0x7fe3b40ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.591+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/3444072154 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3b40ce900 0x7fe3b40ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.591+0000 7fe3b8c16700 1 -- 172.21.15.111:0/3444072154 >> v2:172.21.15.149:3300/0 conn(0x7fe3b40b4240 msgr2=0x7fe3b40b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.591+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/3444072154 >> v2:172.21.15.149:3300/0 conn(0x7fe3b40b4240 0x7fe3b40b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.591+0000 7fe3b8c16700 1 -- 172.21.15.111:0/3444072154 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b40d0b90 con 0x7fe3b40b2360 2023-11-28T13:02:02.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.591+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/3444072154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3b40b2360 0x7fe3b40c6780 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe3b40d15d0 tx=0x7fe3a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccaf7f5178e4e481 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.591+0000 7fe3b37fe700 1 -- 172.21.15.111:0/3444072154 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8005e30 con 0x7fe3b40b2360 2023-11-28T13:02:02.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.591+0000 7fe3b37fe700 1 -- 172.21.15.111:0/3444072154 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a8004030 con 0x7fe3b40b2360 2023-11-28T13:02:02.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.591+0000 7fe3b37fe700 1 -- 172.21.15.111:0/3444072154 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a80043e0 con 0x7fe3b40b2360 2023-11-28T13:02:02.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.592+0000 7fe3bae7a700 1 -- 172.21.15.111:0/3444072154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3b40b2360 msgr2=0x7fe3b40c6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.592+0000 7fe3bae7a700 1 --2- 172.21.15.111:0/3444072154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3b40b2360 0x7fe3b40c6780 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe3b40d15d0 tx=0x7fe3a8009c00 comp rx=0 tx=0).stop 2023-11-28T13:02:02.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.592+0000 7fe3bae7a700 1 -- 172.21.15.111:0/3444072154 shutdown_connections 2023-11-28T13:02:02.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.592+0000 7fe3bae7a700 1 --2- 172.21.15.111:0/3444072154 >> v2:172.21.15.149:3300/0 conn(0x7fe3b40b4240 0x7fe3b40b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.592+0000 7fe3bae7a700 1 --2- 172.21.15.111:0/3444072154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3b40b2360 0x7fe3b40c6780 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.592+0000 7fe3bae7a700 1 --2- 172.21.15.111:0/3444072154 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3b40ce900 0x7fe3b40ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.592+0000 7fe3bae7a700 1 -- 172.21.15.111:0/3444072154 >> 172.21.15.111:0/3444072154 conn(0x7fe3b409a040 msgr2=0x7fe3b40d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.592+0000 7fe3bae7a700 1 -- 172.21.15.111:0/3444072154 shutdown_connections 2023-11-28T13:02:02.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.592+0000 7fe3bae7a700 1 -- 172.21.15.111:0/3444072154 wait complete. 2023-11-28T13:02:02.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.593+0000 7fe3bae7a700 1 Processor -- start 2023-11-28T13:02:02.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.593+0000 7fe3bae7a700 1 -- start start 2023-11-28T13:02:02.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3bae7a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3b40b2360 0x7fe3b41f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3bae7a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3b40b4240 0x7fe3b4159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3bae7a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3b40ce900 0x7fe3b41595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3bae7a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b40adf90 con 0x7fe3b40ce900 2023-11-28T13:02:02.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3bae7a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b41f4430 con 0x7fe3b40b2360 2023-11-28T13:02:02.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3bae7a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b40ad940 con 0x7fe3b40b4240 2023-11-28T13:02:02.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3b9417700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3b40ce900 0x7fe3b41595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3b9417700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3b40ce900 0x7fe3b41595b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37510/0 (socket says 172.21.15.111:37510) 2023-11-28T13:02:02.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3b9417700 1 -- 172.21.15.111:0/2212034587 learned_addr learned my addr 172.21.15.111:0/2212034587 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3b3fff700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3b40b4240 0x7fe3b4159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/2212034587 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3b40b2360 0x7fe3b41f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3b9417700 1 -- 172.21.15.111:0/2212034587 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3b40b4240 msgr2=0x7fe3b4159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3b9417700 1 --2- 172.21.15.111:0/2212034587 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3b40b4240 0x7fe3b4159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3b9417700 1 -- 172.21.15.111:0/2212034587 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3b40b2360 msgr2=0x7fe3b41f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3b9417700 1 --2- 172.21.15.111:0/2212034587 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3b40b2360 0x7fe3b41f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3b9417700 1 -- 172.21.15.111:0/2212034587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a8009500 con 0x7fe3b40ce900 2023-11-28T13:02:02.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3b9417700 1 --2- 172.21.15.111:0/2212034587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3b40ce900 0x7fe3b41595b0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac007e60 tx=0x7fe3ac00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3b17fa700 1 -- 172.21.15.111:0/2212034587 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac00ed30 con 0x7fe3b40ce900 2023-11-28T13:02:02.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.594+0000 7fe3b17fa700 1 -- 172.21.15.111:0/2212034587 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ac014420 con 0x7fe3b40ce900 2023-11-28T13:02:02.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.595+0000 7fe3bae7a700 1 -- 172.21.15.111:0/2212034587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b40c6eb0 con 0x7fe3b40ce900 2023-11-28T13:02:02.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.595+0000 7fe3b17fa700 1 -- 172.21.15.111:0/2212034587 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac0147d0 con 0x7fe3b40ce900 2023-11-28T13:02:02.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.595+0000 7fe3bae7a700 1 -- 172.21.15.111:0/2212034587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b40b3db0 con 0x7fe3b40ce900 2023-11-28T13:02:02.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.596+0000 7fe3b17fa700 1 -- 172.21.15.111:0/2212034587 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe3ac012050 con 0x7fe3b40ce900 2023-11-28T13:02:02.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.596+0000 7fe3b17fa700 1 --2- 172.21.15.111:0/2212034587 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe3980682f0 0x7fe39806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.596+0000 7fe3b17fa700 1 -- 172.21.15.111:0/2212034587 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe3ac086260 con 0x7fe3b40ce900 2023-11-28T13:02:02.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.596+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/2212034587 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe3980682f0 0x7fe39806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.597+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/2212034587 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe3980682f0 0x7fe39806a7a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8009130 tx=0x7fe3a801e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.598+0000 7fe3c7003b80 1 Processor -- start 2023-11-28T13:02:02.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.598+0000 7fe3c7003b80 1 -- start start 2023-11-28T13:02:02.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.599+0000 7fe3c7003b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 0x55bc868a9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.599+0000 7fe3c7003b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d11e0 0x55bc868d3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.599+0000 7fe3c7003b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 0x55bc868ef820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.599+0000 7fe3c7003b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc86793c70 con 0x55bc868d38c0 2023-11-28T13:02:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.599+0000 7fe3c7003b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc86802400 con 0x55bc868d41d0 2023-11-28T13:02:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.599+0000 7fe3c7003b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc867fb930 con 0x55bc868d11e0 2023-11-28T13:02:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.599+0000 7fe3b8c16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 0x55bc868a9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.599+0000 7fe3b8c16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 0x55bc868a9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37516/0 (socket says 172.21.15.111:37516) 2023-11-28T13:02:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.599+0000 7fe3b8c16700 1 -- 172.21.15.111:0/2118055556 learned_addr learned my addr 172.21.15.111:0/2118055556 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.599+0000 7fe3b3fff700 1 --2- 172.21.15.111:0/2118055556 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d11e0 0x55bc868d3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.599+0000 7fe3b9417700 1 --2- 172.21.15.111:0/2118055556 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 0x55bc868ef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.599+0000 7fe3b8c16700 1 -- 172.21.15.111:0/2118055556 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d11e0 msgr2=0x55bc868d3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.599+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/2118055556 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d11e0 0x55bc868d3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe3b8c16700 1 -- 172.21.15.111:0/2118055556 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 msgr2=0x55bc868ef820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/2118055556 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 0x55bc868ef820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe3b8c16700 1 -- 172.21.15.111:0/2118055556 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc8667d690 con 0x55bc868d38c0 2023-11-28T13:02:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/2118055556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 0x55bc868a9ad0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4067d30 tx=0x7fe3a8010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3757f71a02e3082 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe39e7fc700 1 -- 172.21.15.111:0/2118055556 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a801b8b0 con 0x55bc868d38c0 2023-11-28T13:02:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe39e7fc700 1 -- 172.21.15.111:0/2118055556 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a8022040 con 0x55bc868d38c0 2023-11-28T13:02:02.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe39e7fc700 1 -- 172.21.15.111:0/2118055556 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a801bc50 con 0x55bc868d38c0 2023-11-28T13:02:02.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe3c7003b80 1 -- 172.21.15.111:0/2118055556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 msgr2=0x55bc868a9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/2118055556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 0x55bc868a9ad0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4067d30 tx=0x7fe3a8010000 comp rx=0 tx=0).stop 2023-11-28T13:02:02.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe3c7003b80 1 -- 172.21.15.111:0/2118055556 shutdown_connections 2023-11-28T13:02:02.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/2118055556 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 0x55bc868ef820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/2118055556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 0x55bc868a9ad0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/2118055556 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d11e0 0x55bc868d3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe3c7003b80 1 -- 172.21.15.111:0/2118055556 >> 172.21.15.111:0/2118055556 conn(0x55bc868a6e10 msgr2=0x55bc868a77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe3c7003b80 1 -- 172.21.15.111:0/2118055556 shutdown_connections 2023-11-28T13:02:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.600+0000 7fe3c7003b80 1 -- 172.21.15.111:0/2118055556 wait complete. 2023-11-28T13:02:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.601+0000 7fe3c7003b80 1 -- start start 2023-11-28T13:02:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.601+0000 7fe3c7003b80 1 Processor -- start 2023-11-28T13:02:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3c7003b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 0x55bc868e7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3c7003b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d38c0 0x55bc868c9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3c7003b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 0x55bc868c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3c7003b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc8667f980 con 0x55bc868d11e0 2023-11-28T13:02:02.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3c7003b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc868d7490 con 0x55bc868d41d0 2023-11-28T13:02:02.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3c7003b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc868e7a10 con 0x55bc868d38c0 2023-11-28T13:02:02.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3c7003b80 10 client.0 _ll_register_callbacks cb 0x55bc866a12f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:02:02.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3b8c16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 0x55bc868e7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3b8c16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 0x55bc868e7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37520/0 (socket says 172.21.15.111:37520) 2023-11-28T13:02:02.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3b8c16700 1 -- 172.21.15.111:0/1039984269 learned_addr learned my addr 172.21.15.111:0/1039984269 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3b3fff700 1 --2- 172.21.15.111:0/1039984269 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d38c0 0x55bc868c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3b9417700 1 --2- 172.21.15.111:0/1039984269 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 0x55bc868c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3b8c16700 1 -- 172.21.15.111:0/1039984269 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d38c0 msgr2=0x55bc868c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/1039984269 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d38c0 0x55bc868c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3b8c16700 1 -- 172.21.15.111:0/1039984269 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 msgr2=0x55bc868c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/1039984269 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 0x55bc868c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.602+0000 7fe3b8c16700 1 -- 172.21.15.111:0/1039984269 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a80036a0 con 0x55bc868d11e0 2023-11-28T13:02:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/1039984269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 0x55bc868e7310 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800b0e0 tx=0x7fe3a80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c08854a36eb7d7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3937fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:02:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3937fe700 1 -- 172.21.15.111:0/1039984269 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a801d070 con 0x55bc868d11e0 2023-11-28T13:02:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3c7003b80 1 -- 172.21.15.111:0/1039984269 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc867c91f0 con 0x55bc868d11e0 2023-11-28T13:02:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3937fe700 1 -- 172.21.15.111:0/1039984269 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe3a801d1d0 con 0x55bc868d11e0 2023-11-28T13:02:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3937fe700 1 -- 172.21.15.111:0/1039984269 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3a801d390 con 0x55bc868d11e0 2023-11-28T13:02:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3c7003b80 10 client.6459 fetch_fsmap learned FSMap version 32 2023-11-28T13:02:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3c7003b80 1 -- 172.21.15.111:0/1039984269 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55bc86629e40 con 0x55bc868d11e0 2023-11-28T13:02:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3937fe700 1 -- 172.21.15.111:0/1039984269 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bc867c91f0 con 0x55bc868d11e0 2023-11-28T13:02:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3937fe700 1 -- 172.21.15.111:0/1039984269 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fe3a8044350 con 0x55bc868d11e0 2023-11-28T13:02:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3c7003b80 10 client.6459 fetch_fsmap finished waiting for FSMap version 32 2023-11-28T13:02:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3c7003b80 10 client.6459 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2023-11-28T13:02:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3c7003b80 20 client.6459 populate_metadata read hostname 'smithi111' 2023-11-28T13:02:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/1039984269 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc868f27f0 0x55bc868c7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3c7003b80 4 client.6459 mds_command: new command op to 6252 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-28T13:02:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3c7003b80 1 -- 172.21.15.111:0/1039984269 --> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc8667ffc0 con 0x55bc868f27f0 2023-11-28T13:02:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.603+0000 7fe3b3fff700 1 --2- 172.21.15.111:0/1039984269 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc868f27f0 0x55bc868c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3937fe700 10 client.6459 ms_handle_connect on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3b3fff700 1 --2- 172.21.15.111:0/1039984269 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc868f27f0 0x55bc868c7410 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cf5e191918304b1 server_cookie=bac2bd89234af302 in_seq=0 out_seq=0 2023-11-28T13:02:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3937fe700 1 -- 172.21.15.111:0/1039984269 <== mds.0 v2:172.21.15.111:6835/1897250522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55bc8667ffc0 con 0x55bc868f27f0 2023-11-28T13:02:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3937fe700 10 client.6459 handle_command_reply: tid=0 2023-11-28T13:02:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3c7003b80 1 client.6459 shutdown 2023-11-28T13:02:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3c7003b80 1 -- 172.21.15.111:0/1039984269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 msgr2=0x55bc868e7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/1039984269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 0x55bc868e7310 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800b0e0 tx=0x7fe3a80047c0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3c7003b80 1 -- 172.21.15.111:0/1039984269 shutdown_connections 2023-11-28T13:02:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/1039984269 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc868f27f0 0x55bc868c7410 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/1039984269 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 0x55bc868c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3937fe700 0 client.6459 ms_handle_reset on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/1039984269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 0x55bc868e7310 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/1039984269 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d38c0 0x55bc868c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3c7003b80 1 -- 172.21.15.111:0/1039984269 >> 172.21.15.111:0/1039984269 conn(0x55bc868a6e10 msgr2=0x55bc868a9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3c7003b80 1 -- 172.21.15.111:0/1039984269 shutdown_connections 2023-11-28T13:02:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.604+0000 7fe3c7003b80 1 -- 172.21.15.111:0/1039984269 wait complete. 2023-11-28T13:02:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.605+0000 7fe3c7003b80 20 client.6459 trim_cache size 0 max 16384 2023-11-28T13:02:02.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.621+0000 7fe3c7003b80 1 Processor -- start 2023-11-28T13:02:02.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.621+0000 7fe3c7003b80 1 -- start start 2023-11-28T13:02:02.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3c7003b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 0x55bc868ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3c7003b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d38c0 0x55bc868c9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3c7003b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 0x55bc868c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3c7003b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc868c6060 con 0x55bc868d11e0 2023-11-28T13:02:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3c7003b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc868aa120 con 0x55bc868d41d0 2023-11-28T13:02:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3c7003b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc868c7d80 con 0x55bc868d38c0 2023-11-28T13:02:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3b8c16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 0x55bc868ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3b8c16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 0x55bc868ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37528/0 (socket says 172.21.15.111:37528) 2023-11-28T13:02:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3b8c16700 1 -- 172.21.15.111:0/3983717903 learned_addr learned my addr 172.21.15.111:0/3983717903 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3b3fff700 1 --2- 172.21.15.111:0/3983717903 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d38c0 0x55bc868c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3b9417700 1 --2- 172.21.15.111:0/3983717903 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 0x55bc868c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3b8c16700 1 -- 172.21.15.111:0/3983717903 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d38c0 msgr2=0x55bc868c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/3983717903 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d38c0 0x55bc868c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3b8c16700 1 -- 172.21.15.111:0/3983717903 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 msgr2=0x55bc868c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/3983717903 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 0x55bc868c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3b8c16700 1 -- 172.21.15.111:0/3983717903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc867c21f0 con 0x55bc868d11e0 2023-11-28T13:02:02.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3b3fff700 1 --2- 172.21.15.111:0/3983717903 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d38c0 0x55bc868c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:02:02.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3b9417700 1 --2- 172.21.15.111:0/3983717903 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 0x55bc868c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:02:02.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/3983717903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 0x55bc868ca8f0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe3a80045c0 tx=0x7fe3a802d050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e5fa03116b2772 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe39effd700 1 -- 172.21.15.111:0/3983717903 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a801d980 con 0x55bc868d11e0 2023-11-28T13:02:02.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe39effd700 1 -- 172.21.15.111:0/3983717903 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a8022040 con 0x55bc868d11e0 2023-11-28T13:02:02.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.622+0000 7fe39effd700 1 -- 172.21.15.111:0/3983717903 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8044860 con 0x55bc868d11e0 2023-11-28T13:02:02.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.623+0000 7fe3c7003b80 1 -- 172.21.15.111:0/3983717903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 msgr2=0x55bc868ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.623+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/3983717903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 0x55bc868ca8f0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe3a80045c0 tx=0x7fe3a802d050 comp rx=0 tx=0).stop 2023-11-28T13:02:02.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.623+0000 7fe3c7003b80 1 -- 172.21.15.111:0/3983717903 shutdown_connections 2023-11-28T13:02:02.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.623+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/3983717903 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d41d0 0x55bc868c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.623+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/3983717903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d11e0 0x55bc868ca8f0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.623+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/3983717903 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d38c0 0x55bc868c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.623+0000 7fe3c7003b80 1 -- 172.21.15.111:0/3983717903 >> 172.21.15.111:0/3983717903 conn(0x55bc868a6e10 msgr2=0x55bc868c8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.623+0000 7fe3c7003b80 1 -- 172.21.15.111:0/3983717903 shutdown_connections 2023-11-28T13:02:02.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.623+0000 7fe3c7003b80 1 -- 172.21.15.111:0/3983717903 wait complete. 2023-11-28T13:02:02.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.624+0000 7fe3c7003b80 1 -- start start 2023-11-28T13:02:02.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.624+0000 7fe3c7003b80 1 Processor -- start 2023-11-28T13:02:02.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.624+0000 7fe3c7003b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d11e0 0x55bc868a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.624+0000 7fe3c7003b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 0x55bc8694e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.624+0000 7fe3c7003b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d41d0 0x55bc86954a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.624+0000 7fe3c7003b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc8694e840 con 0x55bc868d38c0 2023-11-28T13:02:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.624+0000 7fe3c7003b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc868a9510 con 0x55bc868d11e0 2023-11-28T13:02:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.624+0000 7fe3c7003b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc868e62a0 con 0x55bc868d41d0 2023-11-28T13:02:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3c7003b80 10 client.0 _ll_register_callbacks cb 0x55bc866a12f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:02:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3b3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 0x55bc8694e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3b3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 0x55bc8694e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37536/0 (socket says 172.21.15.111:37536) 2023-11-28T13:02:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3b3fff700 1 -- 172.21.15.111:0/780870492 learned_addr learned my addr 172.21.15.111:0/780870492 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/780870492 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d11e0 0x55bc868a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3b3fff700 1 -- 172.21.15.111:0/780870492 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d41d0 msgr2=0x55bc86954a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3b3fff700 1 --2- 172.21.15.111:0/780870492 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d41d0 0x55bc86954a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3b3fff700 1 -- 172.21.15.111:0/780870492 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d11e0 msgr2=0x55bc868a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3b3fff700 1 --2- 172.21.15.111:0/780870492 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d11e0 0x55bc868a8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3b3fff700 1 -- 172.21.15.111:0/780870492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc86629e40 con 0x55bc868d38c0 2023-11-28T13:02:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/780870492 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d11e0 0x55bc868a8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:02:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3b3fff700 1 --2- 172.21.15.111:0/780870492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 0x55bc8694e300 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0000c00 tx=0x7fe3a0007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68c74312411719a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3937fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:02:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3937fe700 1 -- 172.21.15.111:0/780870492 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a001b070 con 0x55bc868d38c0 2023-11-28T13:02:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3937fe700 1 -- 172.21.15.111:0/780870492 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe3a001d6d0 con 0x55bc868d38c0 2023-11-28T13:02:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3c7003b80 1 -- 172.21.15.111:0/780870492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc867b9210 con 0x55bc868d38c0 2023-11-28T13:02:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3937fe700 1 -- 172.21.15.111:0/780870492 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3a001d890 con 0x55bc868d38c0 2023-11-28T13:02:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3c7003b80 10 client.6465 fetch_fsmap learned FSMap version 32 2023-11-28T13:02:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3c7003b80 1 -- 172.21.15.111:0/780870492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55bc866502b0 con 0x55bc868d38c0 2023-11-28T13:02:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.625+0000 7fe3937fe700 1 -- 172.21.15.111:0/780870492 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bc867b9210 con 0x55bc868d38c0 2023-11-28T13:02:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.626+0000 7fe3937fe700 1 -- 172.21.15.111:0/780870492 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fe3a001d9f0 con 0x55bc868d38c0 2023-11-28T13:02:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.626+0000 7fe3c7003b80 10 client.6465 fetch_fsmap finished waiting for FSMap version 32 2023-11-28T13:02:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.626+0000 7fe3c7003b80 10 client.6465 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2023-11-28T13:02:02.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.626+0000 7fe3c7003b80 20 client.6465 populate_metadata read hostname 'smithi111' 2023-11-28T13:02:02.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.626+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/780870492 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc868aaa10 0x55bc868a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.626+0000 7fe3c7003b80 4 client.6465 mds_command: new command op to 6252 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}] 2023-11-28T13:02:02.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.626+0000 7fe3c7003b80 1 -- 172.21.15.111:0/780870492 --> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}) v1 -- 0x55bc867a3220 con 0x55bc868aaa10 2023-11-28T13:02:02.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.626+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/780870492 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc868aaa10 0x55bc868a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.626+0000 7fe3937fe700 10 client.6465 ms_handle_connect on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:02.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.626+0000 7fe3b8c16700 1 --2- 172.21.15.111:0/780870492 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc868aaa10 0x55bc868a3940 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad9fdbc57dae56e0 server_cookie=43b24bb7ce4a1594 in_seq=0 out_seq=0 2023-11-28T13:02:02.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3937fe700 1 -- 172.21.15.111:0/780870492 <== mds.0 v2:172.21.15.111:6835/1897250522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55bc867a3220 con 0x55bc868aaa10 2023-11-28T13:02:02.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3937fe700 10 client.6465 handle_command_reply: tid=0 2023-11-28T13:02:02.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3c7003b80 1 client.6465 shutdown 2023-11-28T13:02:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3c7003b80 1 -- 172.21.15.111:0/780870492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 msgr2=0x55bc8694e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/780870492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 0x55bc8694e300 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0000c00 tx=0x7fe3a0007f90 comp rx=0 tx=0).stop 2023-11-28T13:02:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3c7003b80 1 -- 172.21.15.111:0/780870492 shutdown_connections 2023-11-28T13:02:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/780870492 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc868aaa10 0x55bc868a3940 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/780870492 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc868d41d0 0x55bc86954a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3937fe700 0 client.6465 ms_handle_reset on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/780870492 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc868d11e0 0x55bc868a8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3c7003b80 1 --2- 172.21.15.111:0/780870492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc868d38c0 0x55bc8694e300 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3c7003b80 1 -- 172.21.15.111:0/780870492 >> 172.21.15.111:0/780870492 conn(0x55bc868a6e10 msgr2=0x55bc868c9310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.637 INFO:tasks.ceph.mds.c.smithi111.stderr:2023-11-28T13:02:02.627+0000 7f60fb868700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5444=0-4194304@1} caps={5444=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5444 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x55d68453bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(11)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-11-28T13:02:02.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3c7003b80 1 -- 172.21.15.111:0/780870492 shutdown_connections 2023-11-28T13:02:02.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3c7003b80 1 -- 172.21.15.111:0/780870492 wait complete. 2023-11-28T13:02:02.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.627+0000 7fe3c7003b80 20 client.6465 trim_cache size 0 max 16384 2023-11-28T13:02:02.638 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:02:02.638 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-11-28T13:02:02.638 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "d109bae6-6ec1-4362-a860-8d89c9cd9b56", 2023-11-28T13:02:02.638 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-11-28T13:02:02.638 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:02:02.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.628+0000 7fe3bae7a700 1 -- 172.21.15.111:0/2212034587 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe3980682f0 msgr2=0x7fe39806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.628+0000 7fe3bae7a700 1 --2- 172.21.15.111:0/2212034587 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe3980682f0 0x7fe39806a7a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8009130 tx=0x7fe3a801e040 comp rx=0 tx=0).stop 2023-11-28T13:02:02.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.628+0000 7fe3bae7a700 1 -- 172.21.15.111:0/2212034587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3b40ce900 msgr2=0x7fe3b41595b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.628+0000 7fe3bae7a700 1 --2- 172.21.15.111:0/2212034587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3b40ce900 0x7fe3b41595b0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac007e60 tx=0x7fe3ac00e3c0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.628+0000 7fe3bae7a700 1 -- 172.21.15.111:0/2212034587 shutdown_connections 2023-11-28T13:02:02.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.628+0000 7fe3bae7a700 1 --2- 172.21.15.111:0/2212034587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3b40ce900 0x7fe3b41595b0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.628+0000 7fe3bae7a700 1 --2- 172.21.15.111:0/2212034587 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe3980682f0 0x7fe39806a7a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.628+0000 7fe3bae7a700 1 --2- 172.21.15.111:0/2212034587 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe3b40b2360 0x7fe3b41f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.628+0000 7fe3bae7a700 1 --2- 172.21.15.111:0/2212034587 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe3b40b4240 0x7fe3b4159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.628+0000 7fe3bae7a700 1 -- 172.21.15.111:0/2212034587 >> 172.21.15.111:0/2212034587 conn(0x7fe3b409a040 msgr2=0x7fe3b415a150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.628+0000 7fe3bae7a700 1 -- 172.21.15.111:0/2212034587 shutdown_connections 2023-11-28T13:02:02.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.628+0000 7fe3bae7a700 1 -- 172.21.15.111:0/2212034587 wait complete. 2023-11-28T13:02:02.641 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub status 2023-11-28T13:02:02.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.761+0000 7f91b478f700 1 Processor -- start 2023-11-28T13:02:02.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.761+0000 7f91b478f700 1 -- start start 2023-11-28T13:02:02.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b478f700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f91ac159960 0x7f91ac0b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b478f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f91ac0aecc0 0x7f91ac0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b478f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac0ce020 con 0x7f91ac0cdbc0 2023-11-28T13:02:02.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b478f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ac159d30 con 0x7f91ac0aecc0 2023-11-28T13:02:02.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b478f700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ac0ad420 con 0x7f91ac159960 2023-11-28T13:02:02.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b1d2a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f91ac0aecc0 0x7f91ac0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b2d2c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f91ac0cdbc0 0x7f91ac0afa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38324/0 (socket says 172.21.15.111:38324) 2023-11-28T13:02:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b1d2a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f91ac0aecc0 0x7f91ac0cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37544/0 (socket says 172.21.15.111:37544) 2023-11-28T13:02:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b2d2c700 1 -- 172.21.15.111:0/142916941 learned_addr learned my addr 172.21.15.111:0/142916941 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:02:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b252b700 1 --2- 172.21.15.111:0/142916941 >> v2:172.21.15.149:3300/0 conn(0x7f91ac159960 0x7f91ac0b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b1d2a700 1 -- 172.21.15.111:0/142916941 >> v2:172.21.15.149:3300/0 conn(0x7f91ac159960 msgr2=0x7f91ac0b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b1d2a700 1 --2- 172.21.15.111:0/142916941 >> v2:172.21.15.149:3300/0 conn(0x7f91ac159960 0x7f91ac0b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b1d2a700 1 -- 172.21.15.111:0/142916941 >> v1:172.21.15.111:6789/0 conn(0x7f91ac0cdbc0 legacy=0x7f91ac0afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b1d2a700 1 -- 172.21.15.111:0/142916941 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ac0ac810 con 0x7f91ac0aecc0 2023-11-28T13:02:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b1529700 1 -- 172.21.15.111:0/142916941 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190911456 0 0) 0x7f91ac0ce020 con 0x7f91ac0cdbc0 2023-11-28T13:02:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.762+0000 7f91b1d2a700 1 --2- 172.21.15.111:0/142916941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ac0aecc0 0x7f91ac0cd680 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f91ac06c6a0 tx=0x7f919c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7068cdbd1f94eb2e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.763+0000 7f91b1529700 1 -- 172.21.15.111:0/142916941 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919c011830 con 0x7f91ac0aecc0 2023-11-28T13:02:02.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.763+0000 7f91b1529700 1 -- 172.21.15.111:0/142916941 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919c011990 con 0x7f91ac0aecc0 2023-11-28T13:02:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.763+0000 7f91b1529700 1 -- 172.21.15.111:0/142916941 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919c011d20 con 0x7f91ac0aecc0 2023-11-28T13:02:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.763+0000 7f91b478f700 1 -- 172.21.15.111:0/142916941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ac0aecc0 msgr2=0x7f91ac0cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.763+0000 7f91b478f700 1 --2- 172.21.15.111:0/142916941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ac0aecc0 0x7f91ac0cd680 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f91ac06c6a0 tx=0x7f919c009b20 comp rx=0 tx=0).stop 2023-11-28T13:02:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.763+0000 7f91b478f700 1 -- 172.21.15.111:0/142916941 shutdown_connections 2023-11-28T13:02:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.763+0000 7f91b478f700 1 --2- 172.21.15.111:0/142916941 >> v2:172.21.15.149:3300/0 conn(0x7f91ac159960 0x7f91ac0b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.763+0000 7f91b478f700 1 --2- 172.21.15.111:0/142916941 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ac0aecc0 0x7f91ac0cd680 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.763+0000 7f91b478f700 1 -- 172.21.15.111:0/142916941 >> 172.21.15.111:0/142916941 conn(0x7f91ac098ee0 msgr2=0x7f91ac06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.763+0000 7f91b478f700 1 -- 172.21.15.111:0/142916941 shutdown_connections 2023-11-28T13:02:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.763+0000 7f91b478f700 1 -- 172.21.15.111:0/142916941 wait complete. 2023-11-28T13:02:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.764+0000 7f91b478f700 1 Processor -- start 2023-11-28T13:02:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.764+0000 7f91b478f700 1 -- start start 2023-11-28T13:02:02.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b478f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91ac0aecc0 0x7f91ac0caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b478f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91ac0cdbc0 0x7f91ac15bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b478f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ac159960 0x7f91ac15c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b478f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac15a090 con 0x7f91ac159960 2023-11-28T13:02:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b478f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac147480 con 0x7f91ac0aecc0 2023-11-28T13:02:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b478f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac15bb60 con 0x7f91ac0cdbc0 2023-11-28T13:02:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b2d2c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ac159960 0x7f91ac15c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b2d2c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ac159960 0x7f91ac15c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37552/0 (socket says 172.21.15.111:37552) 2023-11-28T13:02:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b2d2c700 1 -- 172.21.15.111:0/1445168544 learned_addr learned my addr 172.21.15.111:0/1445168544 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b252b700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91ac0aecc0 0x7f91ac0caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b1d2a700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91ac0cdbc0 0x7f91ac15bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b2d2c700 1 -- 172.21.15.111:0/1445168544 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91ac0cdbc0 msgr2=0x7f91ac15bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b2d2c700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91ac0cdbc0 0x7f91ac15bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b2d2c700 1 -- 172.21.15.111:0/1445168544 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91ac0aecc0 msgr2=0x7f91ac0caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b2d2c700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91ac0aecc0 0x7f91ac0caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b2d2c700 1 -- 172.21.15.111:0/1445168544 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919c0094a0 con 0x7f91ac159960 2023-11-28T13:02:02.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.765+0000 7f91b1d2a700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91ac0cdbc0 0x7f91ac15bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:02:02.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.766+0000 7f91b2d2c700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ac159960 0x7f91ac15c530 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f91a400cca0 tx=0x7f91a400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.766+0000 7f91a2ffd700 1 -- 172.21.15.111:0/1445168544 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a400cea0 con 0x7f91ac159960 2023-11-28T13:02:02.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.766+0000 7f91a2ffd700 1 -- 172.21.15.111:0/1445168544 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a400ade0 con 0x7f91ac159960 2023-11-28T13:02:02.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.766+0000 7f91b478f700 1 -- 172.21.15.111:0/1445168544 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ac0acdd0 con 0x7f91ac159960 2023-11-28T13:02:02.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.766+0000 7f91a2ffd700 1 -- 172.21.15.111:0/1445168544 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a4007310 con 0x7f91ac159960 2023-11-28T13:02:02.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.766+0000 7f91b478f700 1 -- 172.21.15.111:0/1445168544 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ac0acaf0 con 0x7f91ac159960 2023-11-28T13:02:02.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.767+0000 7f91a2ffd700 1 -- 172.21.15.111:0/1445168544 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f91a401a030 con 0x7f91ac159960 2023-11-28T13:02:02.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.768+0000 7f91a2ffd700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f91900682f0 0x7f919006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.768+0000 7f91b252b700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f91900682f0 0x7f919006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.768+0000 7f91a2ffd700 1 -- 172.21.15.111:0/1445168544 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f91a4085c60 con 0x7f91ac159960 2023-11-28T13:02:02.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.768+0000 7f91b252b700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f91900682f0 0x7f919006a7a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f91ac150380 tx=0x7f91a8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.770+0000 7f91c0918b80 1 Processor -- start 2023-11-28T13:02:02.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.770+0000 7f91c0918b80 1 -- start start 2023-11-28T13:02:02.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91c0918b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 0x5618477a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91c0918b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 0x5618477cac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91c0918b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477cb170 0x5618477e67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91c0918b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56184768ac10 con 0x5618477c8180 2023-11-28T13:02:02.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91c0918b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618476f93a0 con 0x5618477cb170 2023-11-28T13:02:02.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91c0918b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5618476f28d0 con 0x5618477ca860 2023-11-28T13:02:02.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91b1d2a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 0x5618477cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91b1d2a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 0x5618477cac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37558/0 (socket says 172.21.15.111:37558) 2023-11-28T13:02:02.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91b1d2a700 1 -- 172.21.15.111:0/3515158317 learned_addr learned my addr 172.21.15.111:0/3515158317 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91b252b700 1 --2- 172.21.15.111:0/3515158317 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 0x5618477a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91b2d2c700 1 --2- 172.21.15.111:0/3515158317 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477cb170 0x5618477e67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91b1d2a700 1 -- 172.21.15.111:0/3515158317 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 msgr2=0x5618477a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91b1d2a700 1 --2- 172.21.15.111:0/3515158317 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 0x5618477a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91b1d2a700 1 -- 172.21.15.111:0/3515158317 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477cb170 msgr2=0x5618477e67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91b1d2a700 1 --2- 172.21.15.111:0/3515158317 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477cb170 0x5618477e67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91b1d2a700 1 -- 172.21.15.111:0/3515158317 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561847574630 con 0x5618477c8180 2023-11-28T13:02:02.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91b2d2c700 1 --2- 172.21.15.111:0/3515158317 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477cb170 0x5618477e67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:02:02.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f91b1d2a700 1 --2- 172.21.15.111:0/3515158317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 0x5618477cac30 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f919c013000 tx=0x7f919c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d4ee039b594097 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f918ffff700 1 -- 172.21.15.111:0/3515158317 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919c01f070 con 0x5618477c8180 2023-11-28T13:02:02.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f918ffff700 1 -- 172.21.15.111:0/3515158317 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919c0117b0 con 0x5618477c8180 2023-11-28T13:02:02.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.771+0000 7f918ffff700 1 -- 172.21.15.111:0/3515158317 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919c011d00 con 0x5618477c8180 2023-11-28T13:02:02.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.772+0000 7f91c0918b80 1 -- 172.21.15.111:0/3515158317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 msgr2=0x5618477cac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.772+0000 7f91c0918b80 1 --2- 172.21.15.111:0/3515158317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 0x5618477cac30 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f919c013000 tx=0x7f919c010000 comp rx=0 tx=0).stop 2023-11-28T13:02:02.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.772+0000 7f91c0918b80 1 -- 172.21.15.111:0/3515158317 shutdown_connections 2023-11-28T13:02:02.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.772+0000 7f91c0918b80 1 --2- 172.21.15.111:0/3515158317 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477cb170 0x5618477e67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.772+0000 7f91c0918b80 1 --2- 172.21.15.111:0/3515158317 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 0x5618477a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.772+0000 7f91c0918b80 1 --2- 172.21.15.111:0/3515158317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 0x5618477cac30 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.772+0000 7f91c0918b80 1 -- 172.21.15.111:0/3515158317 >> 172.21.15.111:0/3515158317 conn(0x56184779ddb0 msgr2=0x56184779e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.772+0000 7f91c0918b80 1 -- 172.21.15.111:0/3515158317 shutdown_connections 2023-11-28T13:02:02.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.772+0000 7f91c0918b80 1 -- 172.21.15.111:0/3515158317 wait complete. 2023-11-28T13:02:02.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.773+0000 7f91c0918b80 1 -- start start 2023-11-28T13:02:02.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.773+0000 7f91c0918b80 1 Processor -- start 2023-11-28T13:02:02.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91c0918b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477c8180 0x5618477de890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91c0918b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477ca860 0x5618477e8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91c0918b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477cb170 0x5618477e90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91c0918b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x561847576920 con 0x5618477ca860 2023-11-28T13:02:02.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91c0918b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618477df740 con 0x5618477c8180 2023-11-28T13:02:02.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91c0918b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5618477ea7e0 con 0x5618477cb170 2023-11-28T13:02:02.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91c0918b80 10 client.0 _ll_register_callbacks cb 0x561847598290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:02:02.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91b1d2a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477ca860 0x5618477e8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91b1d2a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477ca860 0x5618477e8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37564/0 (socket says 172.21.15.111:37564) 2023-11-28T13:02:02.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91b1d2a700 1 -- 172.21.15.111:0/2372671446 learned_addr learned my addr 172.21.15.111:0/2372671446 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91b252b700 1 --2- 172.21.15.111:0/2372671446 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477c8180 0x5618477de890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91b2d2c700 1 --2- 172.21.15.111:0/2372671446 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477cb170 0x5618477e90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91b1d2a700 1 -- 172.21.15.111:0/2372671446 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477cb170 msgr2=0x5618477e90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91b1d2a700 1 --2- 172.21.15.111:0/2372671446 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477cb170 0x5618477e90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91b1d2a700 1 -- 172.21.15.111:0/2372671446 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477c8180 msgr2=0x5618477de890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91b1d2a700 1 --2- 172.21.15.111:0/2372671446 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477c8180 0x5618477de890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91b1d2a700 1 -- 172.21.15.111:0/2372671446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ac0ac810 con 0x5618477ca860 2023-11-28T13:02:02.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f91b1d2a700 1 --2- 172.21.15.111:0/2372671446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477ca860 0x5618477e8b80 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f919c026040 tx=0x7f919c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda9c8ad54c82e1a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f918d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:02:02.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.774+0000 7f918d7fa700 1 -- 172.21.15.111:0/2372671446 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919c004450 con 0x5618477ca860 2023-11-28T13:02:02.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f918d7fa700 1 -- 172.21.15.111:0/2372671446 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f919c0036a0 con 0x5618477ca860 2023-11-28T13:02:02.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f91c0918b80 1 -- 172.21.15.111:0/2372671446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5618476c0190 con 0x5618477ca860 2023-11-28T13:02:02.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f918d7fa700 1 -- 172.21.15.111:0/2372671446 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f919c0045b0 con 0x5618477ca860 2023-11-28T13:02:02.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f91c0918b80 10 client.6480 fetch_fsmap learned FSMap version 32 2023-11-28T13:02:02.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f91c0918b80 1 -- 172.21.15.111:0/2372671446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x561847520de0 con 0x5618477ca860 2023-11-28T13:02:02.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f918d7fa700 1 -- 172.21.15.111:0/2372671446 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5618476c0190 con 0x5618477ca860 2023-11-28T13:02:02.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f918d7fa700 1 -- 172.21.15.111:0/2372671446 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f919c02a070 con 0x5618477ca860 2023-11-28T13:02:02.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f91c0918b80 10 client.6480 fetch_fsmap finished waiting for FSMap version 32 2023-11-28T13:02:02.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f91c0918b80 10 client.6480 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2023-11-28T13:02:02.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f91c0918b80 20 client.6480 populate_metadata read hostname 'smithi111' 2023-11-28T13:02:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f91c0918b80 1 --2- 172.21.15.111:0/2372671446 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x5618477c28e0 0x5618477ddd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f91c0918b80 4 client.6480 mds_command: new command op to 6252 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-28T13:02:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f91c0918b80 1 -- 172.21.15.111:0/2372671446 --> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561847576f60 con 0x5618477c28e0 2023-11-28T13:02:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f91b252b700 1 --2- 172.21.15.111:0/2372671446 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x5618477c28e0 0x5618477ddd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f918d7fa700 10 client.6480 ms_handle_connect on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.775+0000 7f91b252b700 1 --2- 172.21.15.111:0/2372671446 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x5618477c28e0 0x5618477ddd30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f63a50eb044d599 server_cookie=51c42f5906f19532 in_seq=0 out_seq=0 2023-11-28T13:02:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f918d7fa700 1 -- 172.21.15.111:0/2372671446 <== mds.0 v2:172.21.15.111:6835/1897250522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561847576f60 con 0x5618477c28e0 2023-11-28T13:02:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f918d7fa700 10 client.6480 handle_command_reply: tid=0 2023-11-28T13:02:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f91c0918b80 1 client.6480 shutdown 2023-11-28T13:02:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f91c0918b80 1 -- 172.21.15.111:0/2372671446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477ca860 msgr2=0x5618477e8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f91c0918b80 1 --2- 172.21.15.111:0/2372671446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477ca860 0x5618477e8b80 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f919c026040 tx=0x7f919c0047c0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f91c0918b80 1 -- 172.21.15.111:0/2372671446 shutdown_connections 2023-11-28T13:02:02.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f91c0918b80 1 --2- 172.21.15.111:0/2372671446 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x5618477c28e0 0x5618477ddd30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f91c0918b80 1 --2- 172.21.15.111:0/2372671446 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477cb170 0x5618477e90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f91c0918b80 1 --2- 172.21.15.111:0/2372671446 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477c8180 0x5618477de890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f918d7fa700 0 client.6480 ms_handle_reset on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:02.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f91c0918b80 1 --2- 172.21.15.111:0/2372671446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477ca860 0x5618477e8b80 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f91c0918b80 1 -- 172.21.15.111:0/2372671446 >> 172.21.15.111:0/2372671446 conn(0x56184779ddb0 msgr2=0x5618477a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f91c0918b80 1 -- 172.21.15.111:0/2372671446 shutdown_connections 2023-11-28T13:02:02.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f91c0918b80 1 -- 172.21.15.111:0/2372671446 wait complete. 2023-11-28T13:02:02.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.776+0000 7f91c0918b80 20 client.6480 trim_cache size 0 max 16384 2023-11-28T13:02:02.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.793+0000 7f91c0918b80 1 Processor -- start 2023-11-28T13:02:02.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.793+0000 7f91c0918b80 1 -- start start 2023-11-28T13:02:02.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.793+0000 7f91c0918b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 0x5618477e9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.793+0000 7f91c0918b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 0x5618477ea1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.793+0000 7f91c0918b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477cb170 0x5618477994a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.793+0000 7f91c0918b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618477ed750 con 0x5618477c8180 2023-11-28T13:02:02.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.793+0000 7f91c0918b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x56184779be90 con 0x5618477cb170 2023-11-28T13:02:02.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.793+0000 7f91c0918b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5618477c7ff0 con 0x5618477ca860 2023-11-28T13:02:02.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91b252b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 0x5618477e9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91b252b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 0x5618477e9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37570/0 (socket says 172.21.15.111:37570) 2023-11-28T13:02:02.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91b252b700 1 -- 172.21.15.111:0/1802406371 learned_addr learned my addr 172.21.15.111:0/1802406371 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91b1d2a700 1 --2- 172.21.15.111:0/1802406371 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 0x5618477ea1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91b2d2c700 1 --2- 172.21.15.111:0/1802406371 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477cb170 0x5618477994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91b252b700 1 -- 172.21.15.111:0/1802406371 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 msgr2=0x5618477ea1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91b252b700 1 --2- 172.21.15.111:0/1802406371 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 0x5618477ea1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91b252b700 1 -- 172.21.15.111:0/1802406371 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477cb170 msgr2=0x5618477994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91b252b700 1 --2- 172.21.15.111:0/1802406371 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477cb170 0x5618477994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91b252b700 1 -- 172.21.15.111:0/1802406371 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618477c3360 con 0x5618477c8180 2023-11-28T13:02:02.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91b252b700 1 --2- 172.21.15.111:0/1802406371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 0x5618477e9c60 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f91a8006fd0 tx=0x7f91a8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb257cdd47be3daf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91a0ff9700 1 -- 172.21.15.111:0/1802406371 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a8028070 con 0x5618477c8180 2023-11-28T13:02:02.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91a0ff9700 1 -- 172.21.15.111:0/1802406371 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a8029250 con 0x5618477c8180 2023-11-28T13:02:02.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.794+0000 7f91a0ff9700 1 -- 172.21.15.111:0/1802406371 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a8005a20 con 0x5618477c8180 2023-11-28T13:02:02.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.795+0000 7f91c0918b80 1 -- 172.21.15.111:0/1802406371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 msgr2=0x5618477e9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.795+0000 7f91c0918b80 1 --2- 172.21.15.111:0/1802406371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 0x5618477e9c60 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f91a8006fd0 tx=0x7f91a8015000 comp rx=0 tx=0).stop 2023-11-28T13:02:02.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.795+0000 7f91c0918b80 1 -- 172.21.15.111:0/1802406371 shutdown_connections 2023-11-28T13:02:02.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.795+0000 7f91c0918b80 1 --2- 172.21.15.111:0/1802406371 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477cb170 0x5618477994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.795+0000 7f91c0918b80 1 --2- 172.21.15.111:0/1802406371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477c8180 0x5618477e9c60 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.795+0000 7f91c0918b80 1 --2- 172.21.15.111:0/1802406371 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 0x5618477ea1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.795+0000 7f91c0918b80 1 -- 172.21.15.111:0/1802406371 >> 172.21.15.111:0/1802406371 conn(0x56184779ddb0 msgr2=0x56184779e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.795+0000 7f91c0918b80 1 -- 172.21.15.111:0/1802406371 shutdown_connections 2023-11-28T13:02:02.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.795+0000 7f91c0918b80 1 -- 172.21.15.111:0/1802406371 wait complete. 2023-11-28T13:02:02.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.795+0000 7f91c0918b80 1 -- start start 2023-11-28T13:02:02.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.795+0000 7f91c0918b80 1 Processor -- start 2023-11-28T13:02:02.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91c0918b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477c8180 0x5618477dc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91c0918b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 0x5618477c0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91c0918b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477cb170 0x5618477ddd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91c0918b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618477dce90 con 0x5618477cb170 2023-11-28T13:02:02.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91c0918b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618477de6c0 con 0x5618477c8180 2023-11-28T13:02:02.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91c0918b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x5618477c0510 con 0x5618477ca860 2023-11-28T13:02:02.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91c0918b80 10 client.0 _ll_register_callbacks cb 0x561847598290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:02:02.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91b2d2c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477cb170 0x5618477ddd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91b2d2c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477cb170 0x5618477ddd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37578/0 (socket says 172.21.15.111:37578) 2023-11-28T13:02:02.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91b2d2c700 1 -- 172.21.15.111:0/1946111704 learned_addr learned my addr 172.21.15.111:0/1946111704 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:02.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91b1d2a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 0x5618477c0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91b252b700 1 --2- 172.21.15.111:0/1946111704 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477c8180 0x5618477dc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91b2d2c700 1 -- 172.21.15.111:0/1946111704 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 msgr2=0x5618477c0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91b2d2c700 1 --2- 172.21.15.111:0/1946111704 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 0x5618477c0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91b2d2c700 1 -- 172.21.15.111:0/1946111704 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477c8180 msgr2=0x5618477dc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:02.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91b2d2c700 1 --2- 172.21.15.111:0/1946111704 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477c8180 0x5618477dc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91b2d2c700 1 -- 172.21.15.111:0/1946111704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a800e040 con 0x5618477cb170 2023-11-28T13:02:02.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91b1d2a700 1 --2- 172.21.15.111:0/1946111704 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 0x5618477c0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:02:02.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91b2d2c700 1 --2- 172.21.15.111:0/1946111704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477cb170 0x5618477ddd30 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f91a405daf0 tx=0x7f91a4055760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac8336a70312911 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:02.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f918d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:02:02.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f918d7fa700 1 -- 172.21.15.111:0/1946111704 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a4051f80 con 0x5618477cb170 2023-11-28T13:02:02.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f918d7fa700 1 -- 172.21.15.111:0/1946111704 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f91a40520e0 con 0x5618477cb170 2023-11-28T13:02:02.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.797+0000 7f91c0918b80 1 -- 172.21.15.111:0/1946111704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5618476b01b0 con 0x5618477cb170 2023-11-28T13:02:02.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f918d7fa700 1 -- 172.21.15.111:0/1946111704 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91a4054ef0 con 0x5618477cb170 2023-11-28T13:02:02.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f91c0918b80 10 client.6486 fetch_fsmap learned FSMap version 32 2023-11-28T13:02:02.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f91c0918b80 1 -- 172.21.15.111:0/1946111704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x561847509c60 con 0x5618477cb170 2023-11-28T13:02:02.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f918d7fa700 1 -- 172.21.15.111:0/1946111704 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5618476b01b0 con 0x5618477cb170 2023-11-28T13:02:02.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f918d7fa700 1 -- 172.21.15.111:0/1946111704 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 32) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f91a4085050 con 0x5618477cb170 2023-11-28T13:02:02.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f91c0918b80 10 client.6486 fetch_fsmap finished waiting for FSMap version 32 2023-11-28T13:02:02.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f91c0918b80 10 client.6486 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2023-11-28T13:02:02.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f91c0918b80 20 client.6486 populate_metadata read hostname 'smithi111' 2023-11-28T13:02:02.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f91c0918b80 1 --2- 172.21.15.111:0/1946111704 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x56184784a1a0 0x5618477e9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:02.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f91c0918b80 4 client.6486 mds_command: new command op to 6252 tid=0[{"prefix": "scrub status"}] 2023-11-28T13:02:02.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f91c0918b80 1 -- 172.21.15.111:0/1946111704 --> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56184769a1c0 con 0x56184784a1a0 2023-11-28T13:02:02.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f91b252b700 1 --2- 172.21.15.111:0/1946111704 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x56184784a1a0 0x5618477e9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:02.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f918d7fa700 10 client.6486 ms_handle_connect on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:02.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f91b252b700 1 --2- 172.21.15.111:0/1946111704 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x56184784a1a0 0x5618477e9c60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21e4d01ee2a46a61 server_cookie=ab7514acf96f3e12 in_seq=0 out_seq=0 2023-11-28T13:02:02.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f918d7fa700 1 -- 172.21.15.111:0/1946111704 <== mds.0 v2:172.21.15.111:6835/1897250522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+253 (crc 0 0 0) 0x56184769a1c0 con 0x56184784a1a0 2023-11-28T13:02:02.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f918d7fa700 10 client.6486 handle_command_reply: tid=0 2023-11-28T13:02:02.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f91c0918b80 1 client.6486 shutdown 2023-11-28T13:02:02.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f91c0918b80 1 -- 172.21.15.111:0/1946111704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477cb170 msgr2=0x5618477ddd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.798+0000 7f91c0918b80 1 --2- 172.21.15.111:0/1946111704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477cb170 0x5618477ddd30 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f91a405daf0 tx=0x7f91a4055760 comp rx=0 tx=0).stop 2023-11-28T13:02:02.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91c0918b80 1 -- 172.21.15.111:0/1946111704 shutdown_connections 2023-11-28T13:02:02.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91c0918b80 1 --2- 172.21.15.111:0/1946111704 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x56184784a1a0 0x5618477e9c60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91c0918b80 1 --2- 172.21.15.111:0/1946111704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618477cb170 0x5618477ddd30 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91c0918b80 1 --2- 172.21.15.111:0/1946111704 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x5618477c8180 0x5618477dc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91c0918b80 1 --2- 172.21.15.111:0/1946111704 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x5618477ca860 0x5618477c0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f918d7fa700 0 client.6486 ms_handle_reset on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:02.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91c0918b80 1 -- 172.21.15.111:0/1946111704 >> 172.21.15.111:0/1946111704 conn(0x56184779ddb0 msgr2=0x56184779e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91c0918b80 1 -- 172.21.15.111:0/1946111704 shutdown_connections 2023-11-28T13:02:02.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91c0918b80 1 -- 172.21.15.111:0/1946111704 wait complete. 2023-11-28T13:02:02.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91c0918b80 20 client.6486 trim_cache size 0 max 16384 2023-11-28T13:02:02.810 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:02:02.810 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-11-28T13:02:02.810 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-11-28T13:02:02.810 INFO:teuthology.orchestra.run.smithi111.stdout: "d109bae6-6ec1-4362-a860-8d89c9cd9b56": { 2023-11-28T13:02:02.810 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-11-28T13:02:02.810 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "d109bae6-6ec1-4362-a860-8d89c9cd9b56", 2023-11-28T13:02:02.811 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive" 2023-11-28T13:02:02.811 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:02:02.811 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:02:02.811 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:02:02.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91b478f700 1 -- 172.21.15.111:0/1445168544 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f91900682f0 msgr2=0x7f919006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91b478f700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f91900682f0 0x7f919006a7a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f91ac150380 tx=0x7f91a8008040 comp rx=0 tx=0).stop 2023-11-28T13:02:02.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91b478f700 1 -- 172.21.15.111:0/1445168544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ac159960 msgr2=0x7f91ac15c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:02.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91b478f700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ac159960 0x7f91ac15c530 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f91a400cca0 tx=0x7f91a400a340 comp rx=0 tx=0).stop 2023-11-28T13:02:02.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91b478f700 1 -- 172.21.15.111:0/1445168544 shutdown_connections 2023-11-28T13:02:02.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91b478f700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91ac159960 0x7f91ac15c530 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91b478f700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f91900682f0 0x7f919006a7a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91b478f700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f91ac0aecc0 0x7f91ac0caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91b478f700 1 --2- 172.21.15.111:0/1445168544 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f91ac0cdbc0 0x7f91ac15bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:02.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91b478f700 1 -- 172.21.15.111:0/1445168544 >> 172.21.15.111:0/1445168544 conn(0x7f91ac098ee0 msgr2=0x7f91ac06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:02.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91b478f700 1 -- 172.21.15.111:0/1445168544 shutdown_connections 2023-11-28T13:02:02.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:02.799+0000 7f91b478f700 1 -- 172.21.15.111:0/1445168544 wait complete. 2023-11-28T13:02:02.814 INFO:tasks.cephfs.filesystem:scrub status for tag:d109bae6-6ec1-4362-a860-8d89c9cd9b56 - {'path': '/', 'tag': 'd109bae6-6ec1-4362-a860-8d89c9cd9b56', 'options': 'recursive'} 2023-11-28T13:02:06.634 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-11-28T13:02:06.633+0000 7fb379352700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2023-11-28T13:02:29.068 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:02:29.071 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:02:32.815 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub status 2023-11-28T13:02:32.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.947+0000 7f922fbda700 1 Processor -- start 2023-11-28T13:02:32.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.948+0000 7f922fbda700 1 -- start start 2023-11-28T13:02:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.948+0000 7f922fbda700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9228159960 0x7f92280b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.948+0000 7f922fbda700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92280ad420 con 0x7f92280cdd90 2023-11-28T13:02:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.948+0000 7f922fbda700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92280ac810 con 0x7f92280aecc0 2023-11-28T13:02:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.948+0000 7f922fbda700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9228159d30 con 0x7f9228159960 2023-11-28T13:02:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922e177700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f92280cdd90 0x7f92280ad8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38364/0 (socket says 172.21.15.111:38364) 2023-11-28T13:02:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922e177700 1 -- 172.21.15.111:0/1513081462 learned_addr learned my addr 172.21.15.111:0/1513081462 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:02:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922d976700 1 --2- 172.21.15.111:0/1513081462 >> v2:172.21.15.149:3300/0 conn(0x7f9228159960 0x7f92280b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141769088 0 0) 0x7f92280ad420 con 0x7f92280cdd90 2023-11-28T13:02:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92140035b0 con 0x7f92280cdd90 2023-11-28T13:02:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288362744 0 0) 0x7f92280ac810 con 0x7f92280aecc0 2023-11-28T13:02:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92280ad420 con 0x7f92280aecc0 2023-11-28T13:02:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3048864849 0 0) 0x7f92140035b0 con 0x7f92280cdd90 2023-11-28T13:02:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 >> v2:172.21.15.149:3300/0 conn(0x7f9228159960 msgr2=0x7f92280b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:32.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 --2- 172.21.15.111:0/1513081462 >> v2:172.21.15.149:3300/0 conn(0x7f9228159960 0x7f92280b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 >> v1:172.21.15.149:6789/0 conn(0x7f92280aecc0 legacy=0x7f92280cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:32.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92280acaf0 con 0x7f92280cdd90 2023-11-28T13:02:32.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f92200019f0 con 0x7f92280cdd90 2023-11-28T13:02:32.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 >> v1:172.21.15.111:6789/0 conn(0x7f92280cdd90 legacy=0x7f92280ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:32.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 --2- 172.21.15.111:0/1513081462 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9214003be0 0x7f9214003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 --2- 172.21.15.111:0/1513081462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9214004850 0x7f9214004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 --2- 172.21.15.111:0/1513081462 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9214005200 0x7f921400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9228159d30 con 0x7f9214004850 2023-11-28T13:02:32.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921400c0e0 con 0x7f9214005200 2023-11-28T13:02:32.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f921400c270 con 0x7f9214003be0 2023-11-28T13:02:32.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922d175700 1 --2- 172.21.15.111:0/1513081462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9214004850 0x7f9214004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.949+0000 7f922d976700 1 --2- 172.21.15.111:0/1513081462 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9214003be0 0x7f9214003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922e177700 1 --2- 172.21.15.111:0/1513081462 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9214005200 0x7f921400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922d175700 1 -- 172.21.15.111:0/1513081462 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9214003be0 msgr2=0x7f9214003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:32.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922d175700 1 --2- 172.21.15.111:0/1513081462 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9214003be0 0x7f9214003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922d175700 1 -- 172.21.15.111:0/1513081462 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9214005200 msgr2=0x7f921400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:32.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922d175700 1 --2- 172.21.15.111:0/1513081462 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9214005200 0x7f921400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922d175700 1 -- 172.21.15.111:0/1513081462 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92140035b0 con 0x7f9214004850 2023-11-28T13:02:32.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922e177700 1 --2- 172.21.15.111:0/1513081462 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9214005200 0x7f921400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:02:32.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922e177700 1 -- 172.21.15.111:0/1513081462 reap_dead start 2023-11-28T13:02:32.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922d175700 1 --2- 172.21.15.111:0/1513081462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9214004850 0x7f9214004cc0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f9224007fd0 tx=0x7f9214005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6197eba3f380741c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:32.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9224001430 con 0x7f9214004850 2023-11-28T13:02:32.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92240052f0 con 0x7f9214004850 2023-11-28T13:02:32.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922c974700 1 -- 172.21.15.111:0/1513081462 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9224001d40 con 0x7f9214004850 2023-11-28T13:02:32.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922fbda700 1 -- 172.21.15.111:0/1513081462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9214004850 msgr2=0x7f9214004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:32.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922fbda700 1 --2- 172.21.15.111:0/1513081462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9214004850 0x7f9214004cc0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f9224007fd0 tx=0x7f9214005880 comp rx=0 tx=0).stop 2023-11-28T13:02:32.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922fbda700 1 -- 172.21.15.111:0/1513081462 shutdown_connections 2023-11-28T13:02:32.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922fbda700 1 --2- 172.21.15.111:0/1513081462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9214004850 0x7f9214004cc0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.950+0000 7f922fbda700 1 -- 172.21.15.111:0/1513081462 >> 172.21.15.111:0/1513081462 conn(0x7f9228098ee0 msgr2=0x7f922806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:32.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.951+0000 7f922fbda700 1 -- 172.21.15.111:0/1513081462 shutdown_connections 2023-11-28T13:02:32.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.951+0000 7f922fbda700 1 -- 172.21.15.111:0/1513081462 wait complete. 2023-11-28T13:02:32.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.952+0000 7f922fbda700 1 Processor -- start 2023-11-28T13:02:32.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.952+0000 7f922fbda700 1 -- start start 2023-11-28T13:02:32.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922fbda700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9214004850 0x7f92280c2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922fbda700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f922815b8d0 0x7f922815bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922fbda700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9228142ad0 0x7f9228142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922fbda700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92281434b0 con 0x7f9214004850 2023-11-28T13:02:32.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922fbda700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922806e480 con 0x7f9228142ad0 2023-11-28T13:02:32.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922fbda700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92280c4860 con 0x7f922815b8d0 2023-11-28T13:02:32.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922d976700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9214004850 0x7f92280c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922d976700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9214004850 0x7f92280c2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37594/0 (socket says 172.21.15.111:37594) 2023-11-28T13:02:32.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922d976700 1 -- 172.21.15.111:0/2866946423 learned_addr learned my addr 172.21.15.111:0/2866946423 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:32.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922d175700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f922815b8d0 0x7f922815bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922e177700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9228142ad0 0x7f9228142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922d976700 1 -- 172.21.15.111:0/2866946423 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f922815b8d0 msgr2=0x7f922815bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:32.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922d976700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f922815b8d0 0x7f922815bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922d976700 1 -- 172.21.15.111:0/2866946423 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9228142ad0 msgr2=0x7f9228142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:32.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922d976700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9228142ad0 0x7f9228142f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922d976700 1 -- 172.21.15.111:0/2866946423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92280ad0b0 con 0x7f9214004850 2023-11-28T13:02:32.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922e177700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9228142ad0 0x7f9228142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:02:32.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922d976700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9214004850 0x7f92280c2770 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f9218011a70 tx=0x7f9218011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:32.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f921e7fc700 1 -- 172.21.15.111:0/2866946423 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9218005a20 con 0x7f9214004850 2023-11-28T13:02:32.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f921e7fc700 1 -- 172.21.15.111:0/2866946423 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9218005b80 con 0x7f9214004850 2023-11-28T13:02:32.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922fbda700 1 -- 172.21.15.111:0/2866946423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92280acdd0 con 0x7f9214004850 2023-11-28T13:02:32.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f921e7fc700 1 -- 172.21.15.111:0/2866946423 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921800fa00 con 0x7f9214004850 2023-11-28T13:02:32.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.953+0000 7f922fbda700 1 -- 172.21.15.111:0/2866946423 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92280b1fd0 con 0x7f9214004850 2023-11-28T13:02:32.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.955+0000 7f921e7fc700 1 -- 172.21.15.111:0/2866946423 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9218024070 con 0x7f9214004850 2023-11-28T13:02:32.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.955+0000 7f921e7fc700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f920c0682f0 0x7f920c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.955+0000 7f921e7fc700 1 -- 172.21.15.111:0/2866946423 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f92180860f0 con 0x7f9214004850 2023-11-28T13:02:32.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.955+0000 7f922d175700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f920c0682f0 0x7f920c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.956+0000 7f922d175700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f920c0682f0 0x7f920c06a7a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9224005040 tx=0x7f922400a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:32.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.957+0000 7f923bd63b80 1 Processor -- start 2023-11-28T13:02:32.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.957+0000 7f923bd63b80 1 -- start start 2023-11-28T13:02:32.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f923bd63b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088d860 0x55f530863a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f923bd63b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088b180 0x55f53088dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f923bd63b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088e170 0x55f5308a97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f923bd63b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f53074dc10 con 0x55f53088d860 2023-11-28T13:02:32.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f923bd63b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5307bc3a0 con 0x55f53088b180 2023-11-28T13:02:32.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f923bd63b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f5307b58d0 con 0x55f53088e170 2023-11-28T13:02:32.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f922d976700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088d860 0x55f530863a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f922d976700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088d860 0x55f530863a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37602/0 (socket says 172.21.15.111:37602) 2023-11-28T13:02:32.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f922d976700 1 -- 172.21.15.111:0/2436980364 learned_addr learned my addr 172.21.15.111:0/2436980364 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:32.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f922d175700 1 --2- 172.21.15.111:0/2436980364 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088b180 0x55f53088dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f922e177700 1 --2- 172.21.15.111:0/2436980364 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088e170 0x55f5308a97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f922d976700 1 -- 172.21.15.111:0/2436980364 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088e170 msgr2=0x55f5308a97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:32.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f922d976700 1 --2- 172.21.15.111:0/2436980364 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088e170 0x55f5308a97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f922d976700 1 -- 172.21.15.111:0/2436980364 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088b180 msgr2=0x55f53088dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:32.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f922d976700 1 --2- 172.21.15.111:0/2436980364 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088b180 0x55f53088dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.958+0000 7f922d976700 1 -- 172.21.15.111:0/2436980364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f530637630 con 0x55f53088d860 2023-11-28T13:02:32.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.959+0000 7f922d976700 1 --2- 172.21.15.111:0/2436980364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088d860 0x55f530863a70 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f921805e030 tx=0x7f9218055e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90bc56ce37199b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:32.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.959+0000 7f920b7fe700 1 -- 172.21.15.111:0/2436980364 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92180546d0 con 0x55f53088d860 2023-11-28T13:02:32.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.959+0000 7f920b7fe700 1 -- 172.21.15.111:0/2436980364 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921805c070 con 0x55f53088d860 2023-11-28T13:02:32.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.959+0000 7f920b7fe700 1 -- 172.21.15.111:0/2436980364 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9218055bc0 con 0x55f53088d860 2023-11-28T13:02:32.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.959+0000 7f923bd63b80 1 -- 172.21.15.111:0/2436980364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088d860 msgr2=0x55f530863a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:32.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.959+0000 7f923bd63b80 1 --2- 172.21.15.111:0/2436980364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088d860 0x55f530863a70 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f921805e030 tx=0x7f9218055e30 comp rx=0 tx=0).stop 2023-11-28T13:02:32.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.959+0000 7f923bd63b80 1 -- 172.21.15.111:0/2436980364 shutdown_connections 2023-11-28T13:02:32.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.959+0000 7f923bd63b80 1 --2- 172.21.15.111:0/2436980364 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088e170 0x55f5308a97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.959+0000 7f923bd63b80 1 --2- 172.21.15.111:0/2436980364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088d860 0x55f530863a70 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.959+0000 7f923bd63b80 1 --2- 172.21.15.111:0/2436980364 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088b180 0x55f53088dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.959+0000 7f923bd63b80 1 -- 172.21.15.111:0/2436980364 >> 172.21.15.111:0/2436980364 conn(0x55f530860db0 msgr2=0x55f530861740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:32.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.959+0000 7f923bd63b80 1 -- 172.21.15.111:0/2436980364 shutdown_connections 2023-11-28T13:02:32.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.959+0000 7f923bd63b80 1 -- 172.21.15.111:0/2436980364 wait complete. 2023-11-28T13:02:32.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.960+0000 7f923bd63b80 1 -- start start 2023-11-28T13:02:32.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.960+0000 7f923bd63b80 1 Processor -- start 2023-11-28T13:02:32.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f923bd63b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 0x55f530880360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f923bd63b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088d860 0x55f530883850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f923bd63b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088e170 0x55f5308acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f923bd63b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f530639920 con 0x55f53088b180 2023-11-28T13:02:32.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f923bd63b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5308808a0 con 0x55f53088e170 2023-11-28T13:02:32.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f923bd63b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f5308ad500 con 0x55f53088d860 2023-11-28T13:02:32.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f923bd63b80 10 client.0 _ll_register_callbacks cb 0x55f53065b290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:02:32.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f922d976700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 0x55f530880360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f922d976700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 0x55f530880360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37608/0 (socket says 172.21.15.111:37608) 2023-11-28T13:02:32.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f922d976700 1 -- 172.21.15.111:0/3317249877 learned_addr learned my addr 172.21.15.111:0/3317249877 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:32.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f922d175700 1 --2- 172.21.15.111:0/3317249877 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088d860 0x55f530883850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f922d976700 1 -- 172.21.15.111:0/3317249877 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088d860 msgr2=0x55f530883850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:32.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f922d976700 1 --2- 172.21.15.111:0/3317249877 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088d860 0x55f530883850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f922e177700 1 --2- 172.21.15.111:0/3317249877 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088e170 0x55f5308acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f922d976700 1 -- 172.21.15.111:0/3317249877 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088e170 msgr2=0x55f5308acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:32.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f922d976700 1 --2- 172.21.15.111:0/3317249877 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088e170 0x55f5308acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.961+0000 7f922d976700 1 -- 172.21.15.111:0/3317249877 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92280b1fd0 con 0x55f53088b180 2023-11-28T13:02:32.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f922d976700 1 --2- 172.21.15.111:0/3317249877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 0x55f530880360 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f9218052ef0 tx=0x7f9218005f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10bd07fb91f4cbe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:32.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f9208ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:02:32.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f9208ff9700 1 -- 172.21.15.111:0/3317249877 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921805c070 con 0x55f53088b180 2023-11-28T13:02:32.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f9208ff9700 1 -- 172.21.15.111:0/3317249877 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9218084180 con 0x55f53088b180 2023-11-28T13:02:32.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f923bd63b80 1 -- 172.21.15.111:0/3317249877 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f530783190 con 0x55f53088b180 2023-11-28T13:02:32.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f9208ff9700 1 -- 172.21.15.111:0/3317249877 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9218084340 con 0x55f53088b180 2023-11-28T13:02:32.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f923bd63b80 10 client.6498 fetch_fsmap learned FSMap version 33 2023-11-28T13:02:32.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f923bd63b80 1 -- 172.21.15.111:0/3317249877 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55f5305e3de0 con 0x55f53088b180 2023-11-28T13:02:32.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f9208ff9700 1 -- 172.21.15.111:0/3317249877 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f530783190 con 0x55f53088b180 2023-11-28T13:02:32.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f9208ff9700 1 -- 172.21.15.111:0/3317249877 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f921800fb60 con 0x55f53088b180 2023-11-28T13:02:32.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f923bd63b80 10 client.6498 fetch_fsmap finished waiting for FSMap version 33 2023-11-28T13:02:32.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f923bd63b80 10 client.6498 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2023-11-28T13:02:32.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f923bd63b80 20 client.6498 populate_metadata read hostname 'smithi111' 2023-11-28T13:02:32.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f923bd63b80 1 --2- 172.21.15.111:0/3317249877 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55f53088cb50 0x55f530890530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f923bd63b80 4 client.6498 mds_command: new command op to 6252 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-28T13:02:32.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f923bd63b80 1 -- 172.21.15.111:0/3317249877 --> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f530639f60 con 0x55f53088cb50 2023-11-28T13:02:32.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f922d175700 1 --2- 172.21.15.111:0/3317249877 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55f53088cb50 0x55f530890530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f9208ff9700 10 client.6498 ms_handle_connect on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:32.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.962+0000 7f922d175700 1 --2- 172.21.15.111:0/3317249877 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55f53088cb50 0x55f530890530 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdc3586c50b9e4b4 server_cookie=ac3a4e751f034aac in_seq=0 out_seq=0 2023-11-28T13:02:32.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f9208ff9700 1 -- 172.21.15.111:0/3317249877 <== mds.0 v2:172.21.15.111:6835/1897250522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f530639f60 con 0x55f53088cb50 2023-11-28T13:02:32.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f9208ff9700 10 client.6498 handle_command_reply: tid=0 2023-11-28T13:02:32.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f923bd63b80 1 client.6498 shutdown 2023-11-28T13:02:32.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f923bd63b80 1 -- 172.21.15.111:0/3317249877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 msgr2=0x55f530880360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:32.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f923bd63b80 1 --2- 172.21.15.111:0/3317249877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 0x55f530880360 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f9218052ef0 tx=0x7f9218005f50 comp rx=0 tx=0).stop 2023-11-28T13:02:32.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f923bd63b80 1 -- 172.21.15.111:0/3317249877 shutdown_connections 2023-11-28T13:02:32.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f923bd63b80 1 --2- 172.21.15.111:0/3317249877 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55f53088cb50 0x55f530890530 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f923bd63b80 1 --2- 172.21.15.111:0/3317249877 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088e170 0x55f5308acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f923bd63b80 1 --2- 172.21.15.111:0/3317249877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 0x55f530880360 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f9208ff9700 0 client.6498 ms_handle_reset on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:32.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f923bd63b80 1 --2- 172.21.15.111:0/3317249877 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088d860 0x55f530883850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f923bd63b80 1 -- 172.21.15.111:0/3317249877 >> 172.21.15.111:0/3317249877 conn(0x55f530860db0 msgr2=0x55f5308861c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:32.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f923bd63b80 1 -- 172.21.15.111:0/3317249877 shutdown_connections 2023-11-28T13:02:32.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f923bd63b80 1 -- 172.21.15.111:0/3317249877 wait complete. 2023-11-28T13:02:32.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.963+0000 7f923bd63b80 20 client.6498 trim_cache size 0 max 16384 2023-11-28T13:02:32.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.980+0000 7f923bd63b80 1 Processor -- start 2023-11-28T13:02:32.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.980+0000 7f923bd63b80 1 -- start start 2023-11-28T13:02:32.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.980+0000 7f923bd63b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 0x55f5308615b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.980+0000 7f923bd63b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088d860 0x55f53088ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.980+0000 7f923bd63b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088e170 0x55f53088ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.980+0000 7f923bd63b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f53085e240 con 0x55f53088b180 2023-11-28T13:02:32.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.980+0000 7f923bd63b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5308633c0 con 0x55f53088e170 2023-11-28T13:02:32.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.980+0000 7f923bd63b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f5308830a0 con 0x55f53088d860 2023-11-28T13:02:32.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f922d976700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 0x55f5308615b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f922d976700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 0x55f5308615b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37616/0 (socket says 172.21.15.111:37616) 2023-11-28T13:02:32.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f922d976700 1 -- 172.21.15.111:0/679478485 learned_addr learned my addr 172.21.15.111:0/679478485 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:32.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f922e177700 1 --2- 172.21.15.111:0/679478485 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088e170 0x55f53088ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f922d175700 1 --2- 172.21.15.111:0/679478485 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088d860 0x55f53088ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f922d976700 1 -- 172.21.15.111:0/679478485 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088d860 msgr2=0x55f53088ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:32.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f922d976700 1 --2- 172.21.15.111:0/679478485 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088d860 0x55f53088ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f922d976700 1 -- 172.21.15.111:0/679478485 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088e170 msgr2=0x55f53088ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:32.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f922d976700 1 --2- 172.21.15.111:0/679478485 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088e170 0x55f53088ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f922d976700 1 -- 172.21.15.111:0/679478485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f53085f560 con 0x55f53088b180 2023-11-28T13:02:32.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f922d175700 1 --2- 172.21.15.111:0/679478485 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088d860 0x55f53088ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:02:32.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f922e177700 1 --2- 172.21.15.111:0/679478485 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088e170 0x55f53088ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:02:32.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f922d976700 1 --2- 172.21.15.111:0/679478485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 0x55f5308615b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f92180529d0 tx=0x7f921806b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b1a811c45cd69e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:32.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f920bfff700 1 -- 172.21.15.111:0/679478485 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921805c070 con 0x55f53088b180 2023-11-28T13:02:32.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f920bfff700 1 -- 172.21.15.111:0/679478485 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9218084180 con 0x55f53088b180 2023-11-28T13:02:32.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.981+0000 7f920bfff700 1 -- 172.21.15.111:0/679478485 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921800fb60 con 0x55f53088b180 2023-11-28T13:02:32.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.982+0000 7f923bd63b80 1 -- 172.21.15.111:0/679478485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 msgr2=0x55f5308615b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:32.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.982+0000 7f923bd63b80 1 --2- 172.21.15.111:0/679478485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 0x55f5308615b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f92180529d0 tx=0x7f921806b070 comp rx=0 tx=0).stop 2023-11-28T13:02:32.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.982+0000 7f923bd63b80 1 -- 172.21.15.111:0/679478485 shutdown_connections 2023-11-28T13:02:32.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.982+0000 7f923bd63b80 1 --2- 172.21.15.111:0/679478485 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088e170 0x55f53088ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.982+0000 7f923bd63b80 1 --2- 172.21.15.111:0/679478485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088b180 0x55f5308615b0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.982+0000 7f923bd63b80 1 --2- 172.21.15.111:0/679478485 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088d860 0x55f53088ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.982+0000 7f923bd63b80 1 -- 172.21.15.111:0/679478485 >> 172.21.15.111:0/679478485 conn(0x55f530860db0 msgr2=0x55f53088cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:32.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.982+0000 7f923bd63b80 1 -- 172.21.15.111:0/679478485 shutdown_connections 2023-11-28T13:02:32.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.982+0000 7f923bd63b80 1 -- 172.21.15.111:0/679478485 wait complete. 2023-11-28T13:02:32.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.982+0000 7f923bd63b80 1 -- start start 2023-11-28T13:02:32.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.982+0000 7f923bd63b80 1 Processor -- start 2023-11-28T13:02:32.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f923bd63b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088b180 0x55f5308f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f923bd63b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088d860 0x55f5308c1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f923bd63b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088e170 0x55f5308af650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f923bd63b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f53087f860 con 0x55f53088e170 2023-11-28T13:02:32.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f923bd63b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5308f47e0 con 0x55f53088d860 2023-11-28T13:02:32.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f923bd63b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f53087ec00 con 0x55f53088b180 2023-11-28T13:02:32.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f923bd63b80 10 client.0 _ll_register_callbacks cb 0x55f53065b290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:02:32.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f922e177700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088e170 0x55f5308af650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f922e177700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088e170 0x55f5308af650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37626/0 (socket says 172.21.15.111:37626) 2023-11-28T13:02:32.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f922e177700 1 -- 172.21.15.111:0/57906536 learned_addr learned my addr 172.21.15.111:0/57906536 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:32.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f922d175700 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088d860 0x55f5308c1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f922d976700 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088b180 0x55f5308f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f922e177700 1 -- 172.21.15.111:0/57906536 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088b180 msgr2=0x55f5308f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:32.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f922e177700 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088b180 0x55f5308f62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f922e177700 1 -- 172.21.15.111:0/57906536 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088d860 msgr2=0x55f5308c1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:32.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f922e177700 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088d860 0x55f5308c1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:32.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f922e177700 1 -- 172.21.15.111:0/57906536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5305e3de0 con 0x55f53088e170 2023-11-28T13:02:32.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.983+0000 7f922d976700 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088b180 0x55f5308f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:02:32.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f922e177700 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088e170 0x55f5308af650 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9228099590 tx=0x7f9220007b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a7bc738fee46e5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:32.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f9208ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:02:32.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f9208ff9700 1 -- 172.21.15.111:0/57906536 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92200158c0 con 0x55f53088e170 2023-11-28T13:02:32.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f9208ff9700 1 -- 172.21.15.111:0/57906536 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f9220015a20 con 0x55f53088e170 2023-11-28T13:02:32.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f923bd63b80 1 -- 172.21.15.111:0/57906536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f5307731b0 con 0x55f53088e170 2023-11-28T13:02:32.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f9208ff9700 1 -- 172.21.15.111:0/57906536 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9220015be0 con 0x55f53088e170 2023-11-28T13:02:32.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f923bd63b80 10 client.6504 fetch_fsmap learned FSMap version 33 2023-11-28T13:02:32.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f923bd63b80 1 -- 172.21.15.111:0/57906536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55f5305ccc60 con 0x55f53088e170 2023-11-28T13:02:32.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f9208ff9700 1 -- 172.21.15.111:0/57906536 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f5307731b0 con 0x55f53088e170 2023-11-28T13:02:32.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f9208ff9700 1 -- 172.21.15.111:0/57906536 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 3106+0+0 (secure 0 0 0) 0x7f9220013070 con 0x55f53088e170 2023-11-28T13:02:32.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f923bd63b80 10 client.6504 fetch_fsmap finished waiting for FSMap version 33 2023-11-28T13:02:32.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f923bd63b80 10 client.6504 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2023-11-28T13:02:32.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f923bd63b80 20 client.6504 populate_metadata read hostname 'smithi111' 2023-11-28T13:02:32.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f923bd63b80 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55f5308afb90 0x55f5308f36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:32.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f923bd63b80 4 client.6504 mds_command: new command op to 6252 tid=0[{"prefix": "scrub status"}] 2023-11-28T13:02:32.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f923bd63b80 1 -- 172.21.15.111:0/57906536 --> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f53075d1c0 con 0x55f5308afb90 2023-11-28T13:02:32.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.984+0000 7f922d976700 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55f5308afb90 0x55f5308f36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:32.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f922d976700 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55f5308afb90 0x55f5308f36d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c15ce351ed2fc0e3 server_cookie=64c7af585074670e in_seq=0 out_seq=0 2023-11-28T13:02:32.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f9208ff9700 10 client.6504 ms_handle_connect on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:32.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f9208ff9700 1 -- 172.21.15.111:0/57906536 <== mds.0 v2:172.21.15.111:6835/1897250522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f53075d1c0 con 0x55f5308afb90 2023-11-28T13:02:32.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f9208ff9700 10 client.6504 handle_command_reply: tid=0 2023-11-28T13:02:32.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f923bd63b80 1 client.6504 shutdown 2023-11-28T13:02:32.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f923bd63b80 1 -- 172.21.15.111:0/57906536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088e170 msgr2=0x55f5308af650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:32.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f923bd63b80 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088e170 0x55f5308af650 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9228099590 tx=0x7f9220007b40 comp rx=0 tx=0).stop 2023-11-28T13:02:32.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f923bd63b80 1 -- 172.21.15.111:0/57906536 shutdown_connections 2023-11-28T13:02:33.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f923bd63b80 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55f5308afb90 0x55f5308f36d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f923bd63b80 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f53088e170 0x55f5308af650 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f923bd63b80 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55f53088b180 0x55f5308f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f9208ff9700 0 client.6504 ms_handle_reset on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:33.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f923bd63b80 1 --2- 172.21.15.111:0/57906536 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55f53088d860 0x55f5308c1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f923bd63b80 1 -- 172.21.15.111:0/57906536 >> 172.21.15.111:0/57906536 conn(0x55f530860db0 msgr2=0x55f530862d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:33.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f923bd63b80 1 -- 172.21.15.111:0/57906536 shutdown_connections 2023-11-28T13:02:33.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f923bd63b80 1 -- 172.21.15.111:0/57906536 wait complete. 2023-11-28T13:02:33.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f923bd63b80 20 client.6504 trim_cache size 0 max 16384 2023-11-28T13:02:33.001 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:02:33.001 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-11-28T13:02:33.001 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-11-28T13:02:33.001 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:02:33.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f922fbda700 1 -- 172.21.15.111:0/2866946423 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f920c0682f0 msgr2=0x7f920c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f922fbda700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f920c0682f0 0x7f920c06a7a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9224005040 tx=0x7f922400a000 comp rx=0 tx=0).stop 2023-11-28T13:02:33.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f922fbda700 1 -- 172.21.15.111:0/2866946423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9214004850 msgr2=0x7f92280c2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.985+0000 7f922fbda700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9214004850 0x7f92280c2770 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f9218011a70 tx=0x7f9218011e30 comp rx=0 tx=0).stop 2023-11-28T13:02:33.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.986+0000 7f922fbda700 1 -- 172.21.15.111:0/2866946423 shutdown_connections 2023-11-28T13:02:33.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.986+0000 7f922fbda700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f920c0682f0 0x7f920c06a7a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.986+0000 7f922fbda700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9228142ad0 0x7f9228142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.986+0000 7f922fbda700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9214004850 0x7f92280c2770 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.986+0000 7f922fbda700 1 --2- 172.21.15.111:0/2866946423 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f922815b8d0 0x7f922815bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.986+0000 7f922fbda700 1 -- 172.21.15.111:0/2866946423 >> 172.21.15.111:0/2866946423 conn(0x7f9228098ee0 msgr2=0x7f92280ae410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:33.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.986+0000 7f922fbda700 1 -- 172.21.15.111:0/2866946423 shutdown_connections 2023-11-28T13:02:33.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:32.986+0000 7f922fbda700 1 -- 172.21.15.111:0/2866946423 wait complete. 2023-11-28T13:02:33.003 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-11-28T13:02:33.004 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:02:33.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.122+0000 7ffb1e792700 1 Processor -- start 2023-11-28T13:02:33.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.122+0000 7ffb1e792700 1 -- start start 2023-11-28T13:02:33.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb1e792700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ffb180cf010 0x7ffb180cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb1e792700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffb180cb380 0x7ffb180b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb1e792700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180cc7b0 con 0x7ffb180b2220 2023-11-28T13:02:33.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb1e792700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1808ae90 con 0x7ffb180cb380 2023-11-28T13:02:33.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb1e792700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb180b25f0 con 0x7ffb180cf010 2023-11-28T13:02:33.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb1cd2f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffb180cb380 0x7ffb180b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb1cd2f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffb180cb380 0x7ffb180b07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37634/0 (socket says 172.21.15.111:37634) 2023-11-28T13:02:33.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb1cd2f700 1 -- 172.21.15.111:0/3250809851 learned_addr learned my addr 172.21.15.111:0/3250809851 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:33.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb177fe700 1 --2- 172.21.15.111:0/3250809851 >> v2:172.21.15.149:3300/0 conn(0x7ffb180cf010 0x7ffb180cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb1cd2f700 1 -- 172.21.15.111:0/3250809851 >> v2:172.21.15.149:3300/0 conn(0x7ffb180cf010 msgr2=0x7ffb180cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb1cd2f700 1 --2- 172.21.15.111:0/3250809851 >> v2:172.21.15.149:3300/0 conn(0x7ffb180cf010 0x7ffb180cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb1cd2f700 1 -- 172.21.15.111:0/3250809851 >> v1:172.21.15.149:6789/0 conn(0x7ffb180b2220 legacy=0x7ffb180ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb1cd2f700 1 -- 172.21.15.111:0/3250809851 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb180cca10 con 0x7ffb180cb380 2023-11-28T13:02:33.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb1cd2f700 1 --2- 172.21.15.111:0/3250809851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb180cb380 0x7ffb180b07b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ffb1808a070 tx=0x7ffb1000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e5298cec832eb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:33.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb16ffd700 1 -- 172.21.15.111:0/3250809851 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1000e070 con 0x7ffb180cb380 2023-11-28T13:02:33.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb16ffd700 1 -- 172.21.15.111:0/3250809851 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb10003a20 con 0x7ffb180cb380 2023-11-28T13:02:33.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.123+0000 7ffb16ffd700 1 -- 172.21.15.111:0/3250809851 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10003dd0 con 0x7ffb180cb380 2023-11-28T13:02:33.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.124+0000 7ffb1e792700 1 -- 172.21.15.111:0/3250809851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb180cb380 msgr2=0x7ffb180b07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.124+0000 7ffb1e792700 1 --2- 172.21.15.111:0/3250809851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb180cb380 0x7ffb180b07b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ffb1808a070 tx=0x7ffb1000b330 comp rx=0 tx=0).stop 2023-11-28T13:02:33.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.124+0000 7ffb1e792700 1 -- 172.21.15.111:0/3250809851 shutdown_connections 2023-11-28T13:02:33.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.124+0000 7ffb1e792700 1 --2- 172.21.15.111:0/3250809851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb180cb380 0x7ffb180b07b0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.124+0000 7ffb1e792700 1 --2- 172.21.15.111:0/3250809851 >> v2:172.21.15.149:3300/0 conn(0x7ffb180cf010 0x7ffb180cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.124+0000 7ffb1e792700 1 -- 172.21.15.111:0/3250809851 >> 172.21.15.111:0/3250809851 conn(0x7ffb1809a040 msgr2=0x7ffb180d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:33.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.124+0000 7ffb1e792700 1 -- 172.21.15.111:0/3250809851 shutdown_connections 2023-11-28T13:02:33.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.124+0000 7ffb1e792700 1 -- 172.21.15.111:0/3250809851 wait complete. 2023-11-28T13:02:33.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.125+0000 7ffb1e792700 1 Processor -- start 2023-11-28T13:02:33.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.125+0000 7ffb1e792700 1 -- start start 2023-11-28T13:02:33.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb1e792700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb180b2220 0x7ffb180c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb1e792700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb180cb380 0x7ffb1806bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb1e792700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb180cf010 0x7ffb181f1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb1e792700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1806c2d0 con 0x7ffb180cb380 2023-11-28T13:02:33.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb1e792700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1806d0f0 con 0x7ffb180b2220 2023-11-28T13:02:33.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb1e792700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb181f1c60 con 0x7ffb180cf010 2023-11-28T13:02:33.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb180cb380 0x7ffb1806bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb180cb380 0x7ffb1806bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37638/0 (socket says 172.21.15.111:37638) 2023-11-28T13:02:33.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb177fe700 1 -- 172.21.15.111:0/3897254156 learned_addr learned my addr 172.21.15.111:0/3897254156 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:33.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb17fff700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb180b2220 0x7ffb180c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb1cd2f700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb180cf010 0x7ffb181f1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb177fe700 1 -- 172.21.15.111:0/3897254156 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb180cf010 msgr2=0x7ffb181f1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb177fe700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb180cf010 0x7ffb181f1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb177fe700 1 -- 172.21.15.111:0/3897254156 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb180b2220 msgr2=0x7ffb180c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb177fe700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb180b2220 0x7ffb180c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb177fe700 1 -- 172.21.15.111:0/3897254156 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb10009b30 con 0x7ffb180cb380 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb17fff700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb180b2220 0x7ffb180c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.126+0000 7ffb1cd2f700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb180cf010 0x7ffb181f1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.127+0000 7ffb177fe700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb180cb380 0x7ffb1806bd90 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c00c7f0 tx=0x7ffb0c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.127+0000 7ffb14ff9700 1 -- 172.21.15.111:0/3897254156 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c011800 con 0x7ffb180cb380 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.127+0000 7ffb14ff9700 1 -- 172.21.15.111:0/3897254156 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c011960 con 0x7ffb180cb380 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.127+0000 7ffb14ff9700 1 -- 172.21.15.111:0/3897254156 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c011cf0 con 0x7ffb180cb380 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.127+0000 7ffb1e792700 1 -- 172.21.15.111:0/3897254156 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb180ccfd0 con 0x7ffb180cb380 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.127+0000 7ffb1e792700 1 -- 172.21.15.111:0/3897254156 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb180cccf0 con 0x7ffb180cb380 2023-11-28T13:02:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.128+0000 7ffb1e792700 1 -- 172.21.15.111:0/3897254156 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb180cd2b0 con 0x7ffb180cb380 2023-11-28T13:02:33.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.128+0000 7ffb14ff9700 1 -- 172.21.15.111:0/3897254156 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffb0c021070 con 0x7ffb180cb380 2023-11-28T13:02:33.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.128+0000 7ffb14ff9700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffb040682f0 0x7ffb0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.129+0000 7ffb14ff9700 1 -- 172.21.15.111:0/3897254156 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ffb0c085be0 con 0x7ffb180cb380 2023-11-28T13:02:33.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.129+0000 7ffb17fff700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffb040682f0 0x7ffb0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.129+0000 7ffb17fff700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffb040682f0 0x7ffb0406a7a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ffb08001750 tx=0x7ffb08009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:33.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.133+0000 7ffb14ff9700 1 -- 172.21.15.111:0/3897254156 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffb0c052300 con 0x7ffb180cb380 2023-11-28T13:02:33.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.357+0000 7ffb1e792700 1 -- 172.21.15.111:0/3897254156 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb18067170 con 0x7ffb180cb380 2023-11-28T13:02:33.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.357+0000 7ffb14ff9700 1 -- 172.21.15.111:0/3897254156 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4659 (secure 0 0 0) 0x7ffb0c01f020 con 0x7ffb180cb380 2023-11-28T13:02:33.359 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:02:33.360 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/1668572397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1668572397},{"type":"v1","addr":"172.21.15.149:6834","nonce":1668572397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5755,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1129092621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1129092621},{"type":"v1","addr":"172.21.15.149:6835","nonce":1129092621}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/904588136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":904588136},{"type":"v1","addr":"172.21.15.111:6836","nonce":904588136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:01:54.257638+0000","modified":"2023-11-28T13:02:06.634463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.111:6837/1897250522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1897250522},{"type":"v1","addr":"172.21.15.111:6837","nonce":1897250522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T13:02:33.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.359+0000 7ffb1e792700 1 -- 172.21.15.111:0/3897254156 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffb040682f0 msgr2=0x7ffb0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.359+0000 7ffb1e792700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffb040682f0 0x7ffb0406a7a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ffb08001750 tx=0x7ffb08009040 comp rx=0 tx=0).stop 2023-11-28T13:02:33.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.359+0000 7ffb1e792700 1 -- 172.21.15.111:0/3897254156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb180cb380 msgr2=0x7ffb1806bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.359+0000 7ffb1e792700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb180cb380 0x7ffb1806bd90 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c00c7f0 tx=0x7ffb0c00cbb0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.360+0000 7ffb1e792700 1 -- 172.21.15.111:0/3897254156 shutdown_connections 2023-11-28T13:02:33.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.360+0000 7ffb1e792700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb180cf010 0x7ffb181f1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.360+0000 7ffb1e792700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffb040682f0 0x7ffb0406a7a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.360+0000 7ffb1e792700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb180b2220 0x7ffb180c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.360+0000 7ffb1e792700 1 --2- 172.21.15.111:0/3897254156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb180cb380 0x7ffb1806bd90 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.360+0000 7ffb1e792700 1 -- 172.21.15.111:0/3897254156 >> 172.21.15.111:0/3897254156 conn(0x7ffb1809a040 msgr2=0x7ffb180b2730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:33.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.360+0000 7ffb1e792700 1 -- 172.21.15.111:0/3897254156 shutdown_connections 2023-11-28T13:02:33.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.360+0000 7ffb1e792700 1 -- 172.21.15.111:0/3897254156 wait complete. 2023-11-28T13:02:33.364 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2023-11-28T13:02:33.373 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2023-11-28T13:02:33.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.493+0000 7fe247aa8700 1 Processor -- start 2023-11-28T13:02:33.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.493+0000 7fe247aa8700 1 -- start start 2023-11-28T13:02:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe247aa8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe240159960 0x7fe2400b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe247aa8700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe2400aecc0 0x7fe2400cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe247aa8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2400cdbc0 0x7fe2400afa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe247aa8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2400ce040 con 0x7fe2400cdbc0 2023-11-28T13:02:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe247aa8700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe240159d30 con 0x7fe2400aecc0 2023-11-28T13:02:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe247aa8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2400ad420 con 0x7fe240159960 2023-11-28T13:02:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe246045700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2400cdbc0 0x7fe2400afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe246045700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe2400cdbc0 0x7fe2400afa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37648/0 (socket says 172.21.15.111:37648) 2023-11-28T13:02:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe246045700 1 -- 172.21.15.111:0/2761738267 learned_addr learned my addr 172.21.15.111:0/2761738267 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:33.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe245844700 1 --2- 172.21.15.111:0/2761738267 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe240159960 0x7fe2400b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe245043700 1 --2- 172.21.15.111:0/2761738267 >> v2:172.21.15.149:3300/0 conn(0x7fe2400aecc0 0x7fe2400cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe246045700 1 -- 172.21.15.111:0/2761738267 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe240159960 msgr2=0x7fe2400b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe246045700 1 --2- 172.21.15.111:0/2761738267 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe240159960 0x7fe2400b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe246045700 1 -- 172.21.15.111:0/2761738267 >> v2:172.21.15.149:3300/0 conn(0x7fe2400aecc0 msgr2=0x7fe2400cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe246045700 1 --2- 172.21.15.111:0/2761738267 >> v2:172.21.15.149:3300/0 conn(0x7fe2400aecc0 0x7fe2400cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.494+0000 7fe246045700 1 -- 172.21.15.111:0/2761738267 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2400ac810 con 0x7fe2400cdbc0 2023-11-28T13:02:33.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.495+0000 7fe246045700 1 --2- 172.21.15.111:0/2761738267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cdbc0 0x7fe2400afa70 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe240071570 tx=0x7fe23800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c44a7ba6dad9c1e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:33.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.495+0000 7fe244842700 1 -- 172.21.15.111:0/2761738267 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23800e070 con 0x7fe2400cdbc0 2023-11-28T13:02:33.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.495+0000 7fe244842700 1 -- 172.21.15.111:0/2761738267 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23800b9c0 con 0x7fe2400cdbc0 2023-11-28T13:02:33.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.495+0000 7fe244842700 1 -- 172.21.15.111:0/2761738267 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23800bd70 con 0x7fe2400cdbc0 2023-11-28T13:02:33.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.495+0000 7fe247aa8700 1 -- 172.21.15.111:0/2761738267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cdbc0 msgr2=0x7fe2400afa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.495+0000 7fe247aa8700 1 --2- 172.21.15.111:0/2761738267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cdbc0 0x7fe2400afa70 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe240071570 tx=0x7fe23800b330 comp rx=0 tx=0).stop 2023-11-28T13:02:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.495+0000 7fe247aa8700 1 -- 172.21.15.111:0/2761738267 shutdown_connections 2023-11-28T13:02:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.495+0000 7fe247aa8700 1 --2- 172.21.15.111:0/2761738267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400cdbc0 0x7fe2400afa70 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.495+0000 7fe247aa8700 1 --2- 172.21.15.111:0/2761738267 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe240159960 0x7fe2400b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.495+0000 7fe247aa8700 1 --2- 172.21.15.111:0/2761738267 >> v2:172.21.15.149:3300/0 conn(0x7fe2400aecc0 0x7fe2400cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.495+0000 7fe247aa8700 1 -- 172.21.15.111:0/2761738267 >> 172.21.15.111:0/2761738267 conn(0x7fe240098ee0 msgr2=0x7fe24006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.495+0000 7fe247aa8700 1 -- 172.21.15.111:0/2761738267 shutdown_connections 2023-11-28T13:02:33.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.496+0000 7fe247aa8700 1 -- 172.21.15.111:0/2761738267 wait complete. 2023-11-28T13:02:33.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.497+0000 7fe247aa8700 1 Processor -- start 2023-11-28T13:02:33.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.497+0000 7fe247aa8700 1 -- start start 2023-11-28T13:02:33.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.497+0000 7fe247aa8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400aecc0 0x7fe240159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.497+0000 7fe247aa8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe2400cdbc0 0x7fe2400b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.497+0000 7fe247aa8700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe240159960 0x7fe2400b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.497+0000 7fe247aa8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe240154450 con 0x7fe2400aecc0 2023-11-28T13:02:33.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.497+0000 7fe247aa8700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe240153fc0 con 0x7fe240159960 2023-11-28T13:02:33.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.497+0000 7fe247aa8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24015a450 con 0x7fe2400cdbc0 2023-11-28T13:02:33.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.497+0000 7fe245844700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400aecc0 0x7fe240159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.497+0000 7fe245844700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400aecc0 0x7fe240159e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37650/0 (socket says 172.21.15.111:37650) 2023-11-28T13:02:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.497+0000 7fe245844700 1 -- 172.21.15.111:0/1071238891 learned_addr learned my addr 172.21.15.111:0/1071238891 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe245043700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe2400cdbc0 0x7fe2400b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe246045700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe240159960 0x7fe2400b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe245844700 1 -- 172.21.15.111:0/1071238891 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe2400cdbc0 msgr2=0x7fe2400b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe245844700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe2400cdbc0 0x7fe2400b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe245844700 1 -- 172.21.15.111:0/1071238891 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe240159960 msgr2=0x7fe2400b4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe245844700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe240159960 0x7fe2400b4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe245844700 1 -- 172.21.15.111:0/1071238891 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe238009b30 con 0x7fe2400aecc0 2023-11-28T13:02:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe246045700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe240159960 0x7fe2400b4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:02:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe245844700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400aecc0 0x7fe240159e70 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe23000d800 tx=0x7fe23000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe2367fc700 1 -- 172.21.15.111:0/1071238891 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2300125f0 con 0x7fe2400aecc0 2023-11-28T13:02:33.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe2367fc700 1 -- 172.21.15.111:0/1071238891 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe230012750 con 0x7fe2400aecc0 2023-11-28T13:02:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe2367fc700 1 -- 172.21.15.111:0/1071238891 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe230012b00 con 0x7fe2400aecc0 2023-11-28T13:02:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe247aa8700 1 -- 172.21.15.111:0/1071238891 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2400acdd0 con 0x7fe2400aecc0 2023-11-28T13:02:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.498+0000 7fe247aa8700 1 -- 172.21.15.111:0/1071238891 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2400acaf0 con 0x7fe2400aecc0 2023-11-28T13:02:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.499+0000 7fe2367fc700 1 -- 172.21.15.111:0/1071238891 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe230012c60 con 0x7fe2400aecc0 2023-11-28T13:02:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.500+0000 7fe2367fc700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2240682f0 0x7fe22406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.500+0000 7fe2367fc700 1 -- 172.21.15.111:0/1071238891 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe230086040 con 0x7fe2400aecc0 2023-11-28T13:02:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.500+0000 7fe245043700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2240682f0 0x7fe22406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.500+0000 7fe245043700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2240682f0 0x7fe22406a7a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe2401f2890 tx=0x7fe23c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:33.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.502+0000 7fe253c31b80 1 Processor -- start 2023-11-28T13:02:33.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.502+0000 7fe253c31b80 1 -- start start 2023-11-28T13:02:33.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe253c31b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b98b0 0x55bc5e38fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe253c31b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3b9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe253c31b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3ba1c0 0x55bc5e3d5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe253c31b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc5e279c60 con 0x55bc5e3b98b0 2023-11-28T13:02:33.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe253c31b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc5e2e83f0 con 0x55bc5e3b71d0 2023-11-28T13:02:33.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe253c31b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc5e2e1920 con 0x55bc5e3ba1c0 2023-11-28T13:02:33.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe245844700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b98b0 0x55bc5e38fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe245844700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b98b0 0x55bc5e38fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37658/0 (socket says 172.21.15.111:37658) 2023-11-28T13:02:33.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe245844700 1 -- 172.21.15.111:0/571135319 learned_addr learned my addr 172.21.15.111:0/571135319 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:33.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe246045700 1 --2- 172.21.15.111:0/571135319 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3ba1c0 0x55bc5e3d5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe245043700 1 --2- 172.21.15.111:0/571135319 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3b9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe245844700 1 -- 172.21.15.111:0/571135319 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3ba1c0 msgr2=0x55bc5e3d5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe245844700 1 --2- 172.21.15.111:0/571135319 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3ba1c0 0x55bc5e3d5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe245844700 1 -- 172.21.15.111:0/571135319 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b71d0 msgr2=0x55bc5e3b9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe245844700 1 --2- 172.21.15.111:0/571135319 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3b9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe245844700 1 -- 172.21.15.111:0/571135319 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc5e163680 con 0x55bc5e3b98b0 2023-11-28T13:02:33.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe245043700 1 --2- 172.21.15.111:0/571135319 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3b9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:02:33.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe245844700 1 --2- 172.21.15.111:0/571135319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b98b0 0x55bc5e38fac0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe23005e4b0 tx=0x7fe23005e4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ffac233d006087 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:33.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe2237fe700 1 -- 172.21.15.111:0/571135319 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe230055b10 con 0x55bc5e3b98b0 2023-11-28T13:02:33.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe2237fe700 1 -- 172.21.15.111:0/571135319 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23005a150 con 0x55bc5e3b98b0 2023-11-28T13:02:33.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.503+0000 7fe2237fe700 1 -- 172.21.15.111:0/571135319 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23005e220 con 0x55bc5e3b98b0 2023-11-28T13:02:33.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.504+0000 7fe253c31b80 1 -- 172.21.15.111:0/571135319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b98b0 msgr2=0x55bc5e38fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.504+0000 7fe253c31b80 1 --2- 172.21.15.111:0/571135319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b98b0 0x55bc5e38fac0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe23005e4b0 tx=0x7fe23005e4e0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.504+0000 7fe253c31b80 1 -- 172.21.15.111:0/571135319 shutdown_connections 2023-11-28T13:02:33.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.504+0000 7fe253c31b80 1 --2- 172.21.15.111:0/571135319 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3ba1c0 0x55bc5e3d5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.504+0000 7fe253c31b80 1 --2- 172.21.15.111:0/571135319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b98b0 0x55bc5e38fac0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.504+0000 7fe253c31b80 1 --2- 172.21.15.111:0/571135319 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3b9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.504+0000 7fe253c31b80 1 -- 172.21.15.111:0/571135319 >> 172.21.15.111:0/571135319 conn(0x55bc5e38ce00 msgr2=0x55bc5e38d790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:33.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.504+0000 7fe253c31b80 1 -- 172.21.15.111:0/571135319 shutdown_connections 2023-11-28T13:02:33.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.504+0000 7fe253c31b80 1 -- 172.21.15.111:0/571135319 wait complete. 2023-11-28T13:02:33.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.505+0000 7fe253c31b80 1 -- start start 2023-11-28T13:02:33.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.505+0000 7fe253c31b80 1 Processor -- start 2023-11-28T13:02:33.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe253c31b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3bd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe253c31b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b98b0 0x55bc5e3b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe253c31b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3ba1c0 0x55bc5e3adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe253c31b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc5e165970 con 0x55bc5e3b71d0 2023-11-28T13:02:33.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe253c31b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc5e3b80a0 con 0x55bc5e3b98b0 2023-11-28T13:02:33.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe253c31b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc5e3acad0 con 0x55bc5e3ba1c0 2023-11-28T13:02:33.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe253c31b80 10 client.0 _ll_register_callbacks cb 0x55bc5e1872e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:02:33.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe245844700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3bd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe245844700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3bd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37664/0 (socket says 172.21.15.111:37664) 2023-11-28T13:02:33.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe245844700 1 -- 172.21.15.111:0/2504339420 learned_addr learned my addr 172.21.15.111:0/2504339420 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:33.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe245043700 1 --2- 172.21.15.111:0/2504339420 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b98b0 0x55bc5e3b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe246045700 1 --2- 172.21.15.111:0/2504339420 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3ba1c0 0x55bc5e3adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe245844700 1 -- 172.21.15.111:0/2504339420 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3ba1c0 msgr2=0x55bc5e3adbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe245844700 1 --2- 172.21.15.111:0/2504339420 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3ba1c0 0x55bc5e3adbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe245844700 1 -- 172.21.15.111:0/2504339420 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b98b0 msgr2=0x55bc5e3b77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe245844700 1 --2- 172.21.15.111:0/2504339420 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b98b0 0x55bc5e3b77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe245844700 1 -- 172.21.15.111:0/2504339420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2400acaf0 con 0x55bc5e3b71d0 2023-11-28T13:02:33.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe245844700 1 --2- 172.21.15.111:0/2504339420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3bd660 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fe23000ffd0 tx=0x7fe230007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55dd3add0713798e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:33.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe220ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:02:33.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe220ff9700 1 -- 172.21.15.111:0/2504339420 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe230055b10 con 0x55bc5e3b71d0 2023-11-28T13:02:33.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe220ff9700 1 -- 172.21.15.111:0/2504339420 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe23005a150 con 0x55bc5e3b71d0 2023-11-28T13:02:33.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.506+0000 7fe253c31b80 1 -- 172.21.15.111:0/2504339420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc5e2af1e0 con 0x55bc5e3b71d0 2023-11-28T13:02:33.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe220ff9700 1 -- 172.21.15.111:0/2504339420 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe230069b60 con 0x55bc5e3b71d0 2023-11-28T13:02:33.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe253c31b80 10 client.6522 fetch_fsmap learned FSMap version 33 2023-11-28T13:02:33.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe253c31b80 1 -- 172.21.15.111:0/2504339420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55bc5e10fe30 con 0x55bc5e3b71d0 2023-11-28T13:02:33.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe220ff9700 1 -- 172.21.15.111:0/2504339420 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bc5e2af1e0 con 0x55bc5e3b71d0 2023-11-28T13:02:33.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe220ff9700 1 -- 172.21.15.111:0/2504339420 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fe230065020 con 0x55bc5e3b71d0 2023-11-28T13:02:33.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe253c31b80 10 client.6522 fetch_fsmap finished waiting for FSMap version 33 2023-11-28T13:02:33.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe253c31b80 10 client.6522 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T13:02:33.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe253c31b80 20 client.6522 populate_metadata read hostname 'smithi111' 2023-11-28T13:02:33.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe253c31b80 1 --2- 172.21.15.111:0/2504339420 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc5e3a9e00 0x55bc5e3aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe253c31b80 4 client.6522 mds_command: new command op to 6252 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-28T13:02:33.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe253c31b80 1 -- 172.21.15.111:0/2504339420 --> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc5e165fb0 con 0x55bc5e3a9e00 2023-11-28T13:02:33.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe245043700 1 --2- 172.21.15.111:0/2504339420 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc5e3a9e00 0x55bc5e3aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe220ff9700 10 client.6522 ms_handle_connect on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:33.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.507+0000 7fe245043700 1 --2- 172.21.15.111:0/2504339420 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc5e3a9e00 0x55bc5e3aa1d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=904487f3c563880f server_cookie=ecec52c76aafc954 in_seq=0 out_seq=0 2023-11-28T13:02:33.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe220ff9700 1 -- 172.21.15.111:0/2504339420 <== mds.0 v2:172.21.15.111:6835/1897250522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55bc5e165fb0 con 0x55bc5e3a9e00 2023-11-28T13:02:33.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe220ff9700 10 client.6522 handle_command_reply: tid=0 2023-11-28T13:02:33.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe253c31b80 1 client.6522 shutdown 2023-11-28T13:02:33.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe253c31b80 1 -- 172.21.15.111:0/2504339420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 msgr2=0x55bc5e3bd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe253c31b80 1 --2- 172.21.15.111:0/2504339420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3bd660 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fe23000ffd0 tx=0x7fe230007f90 comp rx=0 tx=0).stop 2023-11-28T13:02:33.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe253c31b80 1 -- 172.21.15.111:0/2504339420 shutdown_connections 2023-11-28T13:02:33.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe253c31b80 1 --2- 172.21.15.111:0/2504339420 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc5e3a9e00 0x55bc5e3aa1d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe253c31b80 1 --2- 172.21.15.111:0/2504339420 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3ba1c0 0x55bc5e3adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe253c31b80 1 --2- 172.21.15.111:0/2504339420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3bd660 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe220ff9700 0 client.6522 ms_handle_reset on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:33.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe253c31b80 1 --2- 172.21.15.111:0/2504339420 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b98b0 0x55bc5e3b77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe253c31b80 1 -- 172.21.15.111:0/2504339420 >> 172.21.15.111:0/2504339420 conn(0x55bc5e38ce00 msgr2=0x55bc5e38fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:33.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe253c31b80 1 -- 172.21.15.111:0/2504339420 shutdown_connections 2023-11-28T13:02:33.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe253c31b80 1 -- 172.21.15.111:0/2504339420 wait complete. 2023-11-28T13:02:33.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.508+0000 7fe253c31b80 20 client.6522 trim_cache size 0 max 16384 2023-11-28T13:02:33.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.521+0000 7fe253c31b80 1 Processor -- start 2023-11-28T13:02:33.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.521+0000 7fe253c31b80 1 -- start start 2023-11-28T13:02:33.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe253c31b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3bcf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe253c31b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3b98b0 0x55bc5e38f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe253c31b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3ba1c0 0x55bc5e3aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe253c31b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc5e38b4b0 con 0x55bc5e3ba1c0 2023-11-28T13:02:33.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe253c31b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc5e38b260 con 0x55bc5e3b71d0 2023-11-28T13:02:33.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe253c31b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc5e390b10 con 0x55bc5e3b98b0 2023-11-28T13:02:33.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe246045700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3ba1c0 0x55bc5e3aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe246045700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3ba1c0 0x55bc5e3aa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37676/0 (socket says 172.21.15.111:37676) 2023-11-28T13:02:33.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe246045700 1 -- 172.21.15.111:0/1720543949 learned_addr learned my addr 172.21.15.111:0/1720543949 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:33.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe245844700 1 --2- 172.21.15.111:0/1720543949 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3bcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe245043700 1 --2- 172.21.15.111:0/1720543949 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3b98b0 0x55bc5e38f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe246045700 1 -- 172.21.15.111:0/1720543949 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3b98b0 msgr2=0x55bc5e38f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe246045700 1 --2- 172.21.15.111:0/1720543949 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3b98b0 0x55bc5e38f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe246045700 1 -- 172.21.15.111:0/1720543949 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b71d0 msgr2=0x55bc5e3bcf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe246045700 1 --2- 172.21.15.111:0/1720543949 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3bcf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.522+0000 7fe246045700 1 -- 172.21.15.111:0/1720543949 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc5e2a81e0 con 0x55bc5e3ba1c0 2023-11-28T13:02:33.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.523+0000 7fe246045700 1 --2- 172.21.15.111:0/1720543949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3ba1c0 0x55bc5e3aa1d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fe2400b2730 tx=0x7fe238010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf20415c20ff1d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:33.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.523+0000 7fe223fff700 1 -- 172.21.15.111:0/1720543949 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23800bd30 con 0x55bc5e3ba1c0 2023-11-28T13:02:33.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.523+0000 7fe223fff700 1 -- 172.21.15.111:0/1720543949 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23800be90 con 0x55bc5e3ba1c0 2023-11-28T13:02:33.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.523+0000 7fe223fff700 1 -- 172.21.15.111:0/1720543949 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe238003c40 con 0x55bc5e3ba1c0 2023-11-28T13:02:33.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.523+0000 7fe253c31b80 1 -- 172.21.15.111:0/1720543949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3ba1c0 msgr2=0x55bc5e3aa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.523+0000 7fe253c31b80 1 --2- 172.21.15.111:0/1720543949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3ba1c0 0x55bc5e3aa1d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fe2400b2730 tx=0x7fe238010000 comp rx=0 tx=0).stop 2023-11-28T13:02:33.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.523+0000 7fe253c31b80 1 -- 172.21.15.111:0/1720543949 shutdown_connections 2023-11-28T13:02:33.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.523+0000 7fe253c31b80 1 --2- 172.21.15.111:0/1720543949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3ba1c0 0x55bc5e3aa1d0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.523+0000 7fe253c31b80 1 --2- 172.21.15.111:0/1720543949 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3bcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.523+0000 7fe253c31b80 1 --2- 172.21.15.111:0/1720543949 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3b98b0 0x55bc5e38f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.523+0000 7fe253c31b80 1 -- 172.21.15.111:0/1720543949 >> 172.21.15.111:0/1720543949 conn(0x55bc5e38ce00 msgr2=0x55bc5e38d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.523+0000 7fe253c31b80 1 -- 172.21.15.111:0/1720543949 shutdown_connections 2023-11-28T13:02:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.523+0000 7fe253c31b80 1 -- 172.21.15.111:0/1720543949 wait complete. 2023-11-28T13:02:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.524+0000 7fe253c31b80 1 -- start start 2023-11-28T13:02:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.524+0000 7fe253c31b80 1 Processor -- start 2023-11-28T13:02:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.525+0000 7fe253c31b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3b0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.525+0000 7fe253c31b80 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3b98b0 0x55bc5e439ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.525+0000 7fe253c31b80 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3ba1c0 0x55bc5e3d6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.525+0000 7fe253c31b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc5e3b0fc0 con 0x55bc5e3b71d0 2023-11-28T13:02:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.525+0000 7fe253c31b80 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc5e439690 con 0x55bc5e3ba1c0 2023-11-28T13:02:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.525+0000 7fe253c31b80 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc5e3d73a0 con 0x55bc5e3b98b0 2023-11-28T13:02:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe253c31b80 10 client.0 _ll_register_callbacks cb 0x55bc5e1872e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T13:02:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe245844700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3b0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe245844700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3b0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37680/0 (socket says 172.21.15.111:37680) 2023-11-28T13:02:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe245844700 1 -- 172.21.15.111:0/623661954 learned_addr learned my addr 172.21.15.111:0/623661954 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe245844700 1 -- 172.21.15.111:0/623661954 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3b98b0 msgr2=0x55bc5e439ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe245043700 1 --2- 172.21.15.111:0/623661954 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3b98b0 0x55bc5e439ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe245844700 1 --2- 172.21.15.111:0/623661954 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3b98b0 0x55bc5e439ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe245844700 1 -- 172.21.15.111:0/623661954 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3ba1c0 msgr2=0x55bc5e3d6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe246045700 1 --2- 172.21.15.111:0/623661954 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3ba1c0 0x55bc5e3d6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe245844700 1 --2- 172.21.15.111:0/623661954 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3ba1c0 0x55bc5e3d6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe245844700 1 -- 172.21.15.111:0/623661954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2400ac810 con 0x55bc5e3b71d0 2023-11-28T13:02:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe245844700 1 --2- 172.21.15.111:0/623661954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3b0100 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fe23005df80 tx=0x7fe230020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb61e9c7e4f4dd7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe220ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-11-28T13:02:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe220ff9700 1 -- 172.21.15.111:0/623661954 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe230073070 con 0x55bc5e3b71d0 2023-11-28T13:02:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe220ff9700 1 -- 172.21.15.111:0/623661954 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7fe230055910 con 0x55bc5e3b71d0 2023-11-28T13:02:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe253c31b80 1 -- 172.21.15.111:0/623661954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc5e29f200 con 0x55bc5e3b71d0 2023-11-28T13:02:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe220ff9700 1 -- 172.21.15.111:0/623661954 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe230069e80 con 0x55bc5e3b71d0 2023-11-28T13:02:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe253c31b80 10 client.6528 fetch_fsmap learned FSMap version 33 2023-11-28T13:02:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe253c31b80 1 -- 172.21.15.111:0/623661954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x55bc5e1362a0 con 0x55bc5e3b71d0 2023-11-28T13:02:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe220ff9700 1 -- 172.21.15.111:0/623661954 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bc5e29f200 con 0x55bc5e3b71d0 2023-11-28T13:02:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe220ff9700 1 -- 172.21.15.111:0/623661954 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 3106+0+0 (secure 0 0 0) 0x7fe230076020 con 0x55bc5e3b71d0 2023-11-28T13:02:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe253c31b80 10 client.6528 fetch_fsmap finished waiting for FSMap version 33 2023-11-28T13:02:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.526+0000 7fe253c31b80 10 client.6528 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T13:02:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 20 client.6528 populate_metadata read hostname 'smithi111' 2023-11-28T13:02:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 1 --2- 172.21.15.111:0/623661954 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc5e389130 0x55bc5e3cd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 4 client.6528 mds_command: new command op to 6252 tid=0[{"prefix": "damage ls", "format": "json-pretty"}] 2023-11-28T13:02:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 1 -- 172.21.15.111:0/623661954 --> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x55bc5e289210 con 0x55bc5e389130 2023-11-28T13:02:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe245043700 1 --2- 172.21.15.111:0/623661954 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc5e389130 0x55bc5e3cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe220ff9700 10 client.6528 ms_handle_connect on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe245043700 1 --2- 172.21.15.111:0/623661954 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc5e389130 0x55bc5e3cd610 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72b6b3e3a6df282c server_cookie=8aa92935a4fea3bd in_seq=0 out_seq=0 2023-11-28T13:02:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe220ff9700 1 -- 172.21.15.111:0/623661954 <== mds.0 v2:172.21.15.111:6835/1897250522 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+141 (crc 0 0 0) 0x55bc5e289210 con 0x55bc5e389130 2023-11-28T13:02:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe220ff9700 10 client.6528 handle_command_reply: tid=0 2023-11-28T13:02:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 1 client.6528 shutdown 2023-11-28T13:02:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 1 -- 172.21.15.111:0/623661954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 msgr2=0x55bc5e3b0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 1 --2- 172.21.15.111:0/623661954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3b0100 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fe23005df80 tx=0x7fe230020000 comp rx=0 tx=0).stop 2023-11-28T13:02:33.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 1 -- 172.21.15.111:0/623661954 shutdown_connections 2023-11-28T13:02:33.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 1 --2- 172.21.15.111:0/623661954 >> [v2:172.21.15.111:6835/1897250522,v1:172.21.15.111:6837/1897250522] conn(0x55bc5e389130 0x55bc5e3cd610 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 1 --2- 172.21.15.111:0/623661954 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x55bc5e3ba1c0 0x55bc5e3d6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe220ff9700 0 client.6528 ms_handle_reset on v2:172.21.15.111:6835/1897250522 2023-11-28T13:02:33.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 1 --2- 172.21.15.111:0/623661954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bc5e3b71d0 0x55bc5e3b0100 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 1 --2- 172.21.15.111:0/623661954 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x55bc5e3b98b0 0x55bc5e439ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 1 -- 172.21.15.111:0/623661954 >> 172.21.15.111:0/623661954 conn(0x55bc5e38ce00 msgr2=0x55bc5e38fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:33.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.527+0000 7fe253c31b80 1 -- 172.21.15.111:0/623661954 shutdown_connections 2023-11-28T13:02:33.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe253c31b80 1 -- 172.21.15.111:0/623661954 wait complete. 2023-11-28T13:02:33.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe253c31b80 20 client.6528 trim_cache size 0 max 16384 2023-11-28T13:02:33.536 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:02:33.536 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-11-28T13:02:33.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-11-28T13:02:33.536 INFO:teuthology.orchestra.run.smithi111.stdout: "damage_type": "backtrace", 2023-11-28T13:02:33.536 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 1209230853, 2023-11-28T13:02:33.536 INFO:teuthology.orchestra.run.smithi111.stdout: "ino": 1099511627777, 2023-11-28T13:02:33.536 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/dir_test/file" 2023-11-28T13:02:33.536 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-11-28T13:02:33.536 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-11-28T13:02:33.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe247aa8700 1 -- 172.21.15.111:0/1071238891 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2240682f0 msgr2=0x7fe22406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe247aa8700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2240682f0 0x7fe22406a7a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe2401f2890 tx=0x7fe23c008040 comp rx=0 tx=0).stop 2023-11-28T13:02:33.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe247aa8700 1 -- 172.21.15.111:0/1071238891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400aecc0 msgr2=0x7fe240159e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe247aa8700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400aecc0 0x7fe240159e70 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe23000d800 tx=0x7fe23000dbc0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe247aa8700 1 -- 172.21.15.111:0/1071238891 shutdown_connections 2023-11-28T13:02:33.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe247aa8700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe2240682f0 0x7fe22406a7a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe247aa8700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe240159960 0x7fe2400b4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe247aa8700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2400aecc0 0x7fe240159e70 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe247aa8700 1 --2- 172.21.15.111:0/1071238891 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe2400cdbc0 0x7fe2400b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe247aa8700 1 -- 172.21.15.111:0/1071238891 >> 172.21.15.111:0/1071238891 conn(0x7fe240098ee0 msgr2=0x7fe2401565b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:33.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe247aa8700 1 -- 172.21.15.111:0/1071238891 shutdown_connections 2023-11-28T13:02:33.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.528+0000 7fe247aa8700 1 -- 172.21.15.111:0/1071238891 wait complete. 2023-11-28T13:02:33.541 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:02:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.660+0000 7ff8ea81e700 1 Processor -- start 2023-11-28T13:02:33.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.660+0000 7ff8ea81e700 1 -- start start 2023-11-28T13:02:33.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.660+0000 7ff8ea81e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff8e40cef60 0x7ff8e40ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.660+0000 7ff8ea81e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8e40b0160 0x7ff8e40adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.660+0000 7ff8ea81e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e40cc390 con 0x7ff8e415a5d0 2023-11-28T13:02:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.660+0000 7ff8ea81e700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e40cc1b0 con 0x7ff8e40cef60 2023-11-28T13:02:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.660+0000 7ff8ea81e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40cc670 con 0x7ff8e40b0160 2023-11-28T13:02:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e3fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff8e415a5d0 0x7ff8e40ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38464/0 (socket says 172.21.15.111:38464) 2023-11-28T13:02:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e3fff700 1 -- 172.21.15.111:0/3966975916 learned_addr learned my addr 172.21.15.111:0/3966975916 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:02:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e37fe700 1 --2- 172.21.15.111:0/3966975916 >> v2:172.21.15.149:3300/0 conn(0x7ff8e40cef60 0x7ff8e40ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e8dbb700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8e40b0160 0x7ff8e40adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809963298 0 0) 0x7ff8e40cc390 con 0x7ff8e415a5d0 2023-11-28T13:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8cc0035b0 con 0x7ff8e415a5d0 2023-11-28T13:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2062774400 0 0) 0x7ff8cc0035b0 con 0x7ff8e415a5d0 2023-11-28T13:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8e40b0160 msgr2=0x7ff8e40adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8e40b0160 0x7ff8e40adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 >> v2:172.21.15.149:3300/0 conn(0x7ff8e40cef60 msgr2=0x7ff8e40ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 --2- 172.21.15.111:0/3966975916 >> v2:172.21.15.149:3300/0 conn(0x7ff8e40cef60 0x7ff8e40ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e40cc950 con 0x7ff8e415a5d0 2023-11-28T13:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7ff8d40017f0 con 0x7ff8e415a5d0 2023-11-28T13:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 >> v1:172.21.15.111:6789/0 conn(0x7ff8e415a5d0 legacy=0x7ff8e40ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc003a00 0x7ff8cc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8cc004430 0x7ff8cc0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8cc004f60 0x7ff8cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40cc1b0 con 0x7ff8cc003a00 2023-11-28T13:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40cc670 con 0x7ff8cc004430 2023-11-28T13:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc00bf50 con 0x7ff8cc004f60 2023-11-28T13:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e8dbb700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8e40b0160 0x7ff8e40adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e3fff700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc003a00 0x7ff8cc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e37fe700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8cc004430 0x7ff8cc0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.661+0000 7ff8e8dbb700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8cc004f60 0x7ff8cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8e3fff700 1 -- 172.21.15.111:0/3966975916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8cc004f60 msgr2=0x7ff8cc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8e3fff700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8cc004f60 0x7ff8cc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8e3fff700 1 -- 172.21.15.111:0/3966975916 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8cc004430 msgr2=0x7ff8cc0048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:33.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8e3fff700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8cc004430 0x7ff8cc0048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8e3fff700 1 -- 172.21.15.111:0/3966975916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc0035b0 con 0x7ff8cc003a00 2023-11-28T13:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8e8dbb700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8cc004f60 0x7ff8cc00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8e3fff700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc003a00 0x7ff8cc003e10 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400c320 tx=0x7ff8cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8d733afedabc1d9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d400ce60 con 0x7ff8cc003a00 2023-11-28T13:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d4007bf0 con 0x7ff8cc003a00 2023-11-28T13:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8e2ffd700 1 -- 172.21.15.111:0/3966975916 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d40072f0 con 0x7ff8cc003a00 2023-11-28T13:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8ea81e700 1 -- 172.21.15.111:0/3966975916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc003a00 msgr2=0x7ff8cc003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8ea81e700 1 --2- 172.21.15.111:0/3966975916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc003a00 0x7ff8cc003e10 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400c320 tx=0x7ff8cc005880 comp rx=0 tx=0).stop 2023-11-28T13:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8e8dbb700 1 -- 172.21.15.111:0/3966975916 reap_dead start 2023-11-28T13:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8ea81e700 1 -- 172.21.15.111:0/3966975916 shutdown_connections 2023-11-28T13:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8ea81e700 1 -- 172.21.15.111:0/3966975916 >> 172.21.15.111:0/3966975916 conn(0x7ff8e409a040 msgr2=0x7ff8e40d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:33.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8ea81e700 1 -- 172.21.15.111:0/3966975916 shutdown_connections 2023-11-28T13:02:33.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.662+0000 7ff8ea81e700 1 -- 172.21.15.111:0/3966975916 wait complete. 2023-11-28T13:02:33.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.664+0000 7ff8ea81e700 1 Processor -- start 2023-11-28T13:02:33.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.664+0000 7ff8ea81e700 1 -- start start 2023-11-28T13:02:33.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.664+0000 7ff8ea81e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc003a00 0x7ff8e415abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.664+0000 7ff8ea81e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8e414e180 0x7ff8e406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.664+0000 7ff8ea81e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8e414e5f0 0x7ff8e4070480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.664+0000 7ff8ea81e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40cab30 con 0x7ff8cc003a00 2023-11-28T13:02:33.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.664+0000 7ff8ea81e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40ca290 con 0x7ff8e414e180 2023-11-28T13:02:33.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.664+0000 7ff8ea81e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40ca570 con 0x7ff8e414e5f0 2023-11-28T13:02:33.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc003a00 0x7ff8e415abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc003a00 0x7ff8e415abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37698/0 (socket says 172.21.15.111:37698) 2023-11-28T13:02:33.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e3fff700 1 -- 172.21.15.111:0/1109949511 learned_addr learned my addr 172.21.15.111:0/1109949511 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:33.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e37fe700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8e414e180 0x7ff8e406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e8dbb700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8e414e5f0 0x7ff8e4070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e3fff700 1 -- 172.21.15.111:0/1109949511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8e414e5f0 msgr2=0x7ff8e4070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e3fff700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8e414e5f0 0x7ff8e4070480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e3fff700 1 -- 172.21.15.111:0/1109949511 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8e414e180 msgr2=0x7ff8e406ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e3fff700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8e414e180 0x7ff8e406ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e3fff700 1 -- 172.21.15.111:0/1109949511 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e40ccf10 con 0x7ff8cc003a00 2023-11-28T13:02:33.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e3fff700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc003a00 0x7ff8e415abf0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4015040 tx=0x7ff8d4001bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:33.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e0ff9700 1 -- 172.21.15.111:0/1109949511 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d40070d0 con 0x7ff8cc003a00 2023-11-28T13:02:33.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e0ff9700 1 -- 172.21.15.111:0/1109949511 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d4007250 con 0x7ff8cc003a00 2023-11-28T13:02:33.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8e0ff9700 1 -- 172.21.15.111:0/1109949511 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4007600 con 0x7ff8cc003a00 2023-11-28T13:02:33.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8ea81e700 1 -- 172.21.15.111:0/1109949511 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e40ccc30 con 0x7ff8cc003a00 2023-11-28T13:02:33.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.665+0000 7ff8ea81e700 1 -- 172.21.15.111:0/1109949511 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e415a9a0 con 0x7ff8cc003a00 2023-11-28T13:02:33.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.666+0000 7ff8ea81e700 1 -- 172.21.15.111:0/1109949511 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e40caa90 con 0x7ff8cc003a00 2023-11-28T13:02:33.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.667+0000 7ff8e0ff9700 1 -- 172.21.15.111:0/1109949511 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff8d402e070 con 0x7ff8cc003a00 2023-11-28T13:02:33.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.667+0000 7ff8e0ff9700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8d00682f0 0x7ff8d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:33.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.667+0000 7ff8e0ff9700 1 -- 172.21.15.111:0/1109949511 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ff8d4087750 con 0x7ff8cc003a00 2023-11-28T13:02:33.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.667+0000 7ff8e37fe700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8d00682f0 0x7ff8d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:33.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.668+0000 7ff8e37fe700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8d00682f0 0x7ff8d006a7a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40cf350 tx=0x7ff8d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:33.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.671+0000 7ff8e0ff9700 1 -- 172.21.15.111:0/1109949511 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff8d4053f10 con 0x7ff8cc003a00 2023-11-28T13:02:33.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.898+0000 7ff8ea81e700 1 -- 172.21.15.111:0/1109949511 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff8e4067170 con 0x7ff8cc003a00 2023-11-28T13:02:33.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.898+0000 7ff8e0ff9700 1 -- 172.21.15.111:0/1109949511 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7ff8d4057530 con 0x7ff8cc003a00 2023-11-28T13:02:33.900 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:02:33.900 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:02:33.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.900+0000 7ff8ea81e700 1 -- 172.21.15.111:0/1109949511 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8d00682f0 msgr2=0x7ff8d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.900+0000 7ff8ea81e700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8d00682f0 0x7ff8d006a7a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40cf350 tx=0x7ff8d8008040 comp rx=0 tx=0).stop 2023-11-28T13:02:33.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.900+0000 7ff8ea81e700 1 -- 172.21.15.111:0/1109949511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc003a00 msgr2=0x7ff8e415abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:33.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.900+0000 7ff8ea81e700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc003a00 0x7ff8e415abf0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4015040 tx=0x7ff8d4001bf0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.900+0000 7ff8ea81e700 1 -- 172.21.15.111:0/1109949511 shutdown_connections 2023-11-28T13:02:33.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.900+0000 7ff8ea81e700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8e414e5f0 0x7ff8e4070480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.900+0000 7ff8ea81e700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8cc003a00 0x7ff8e415abf0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.900+0000 7ff8ea81e700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8d00682f0 0x7ff8d006a7a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.900+0000 7ff8ea81e700 1 --2- 172.21.15.111:0/1109949511 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8e414e180 0x7ff8e406ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:33.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.900+0000 7ff8ea81e700 1 -- 172.21.15.111:0/1109949511 >> 172.21.15.111:0/1109949511 conn(0x7ff8e409a040 msgr2=0x7ff8e40cd330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:33.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.900+0000 7ff8ea81e700 1 -- 172.21.15.111:0/1109949511 shutdown_connections 2023-11-28T13:02:33.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:33.900+0000 7ff8ea81e700 1 -- 172.21.15.111:0/1109949511 wait complete. 2023-11-28T13:02:33.913 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:02:38.915 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:02:39.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.045+0000 7ffb14a73700 1 Processor -- start 2023-11-28T13:02:39.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.045+0000 7ffb14a73700 1 -- start start 2023-11-28T13:02:39.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb14a73700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffb100b20f0 0x7ffb100b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:39.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb14a73700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb1006ba90 0x7ffb100b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:39.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb14a73700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb100b5300 con 0x7ffb100b40c0 2023-11-28T13:02:39.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb14a73700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb100cfbb0 con 0x7ffb100b20f0 2023-11-28T13:02:39.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb14a73700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100b46f0 con 0x7ffb1006ba90 2023-11-28T13:02:39.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb0e59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffb100b20f0 0x7ffb100b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:39.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb0dd9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ffb100b40c0 0x7ffb1006b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38484/0 (socket says 172.21.15.111:38484) 2023-11-28T13:02:39.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb0dd9b700 1 -- 172.21.15.111:0/3267091549 learned_addr learned my addr 172.21.15.111:0/3267091549 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:02:39.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb0ed9d700 1 --2- 172.21.15.111:0/3267091549 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb1006ba90 0x7ffb100b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:39.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb0e59c700 1 -- 172.21.15.111:0/3267091549 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb1006ba90 msgr2=0x7ffb100b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:39.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb0e59c700 1 --2- 172.21.15.111:0/3267091549 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb1006ba90 0x7ffb100b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:39.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb0e59c700 1 -- 172.21.15.111:0/3267091549 >> v1:172.21.15.111:6789/0 conn(0x7ffb100b40c0 legacy=0x7ffb1006b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:39.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb0d59a700 1 -- 172.21.15.111:0/3267091549 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287182252 0 0) 0x7ffb100b5300 con 0x7ffb100b40c0 2023-11-28T13:02:39.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb0e59c700 1 -- 172.21.15.111:0/3267091549 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb100b49d0 con 0x7ffb100b20f0 2023-11-28T13:02:39.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.046+0000 7ffb0ed9d700 1 --2- 172.21.15.111:0/3267091549 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb1006ba90 0x7ffb100b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:02:39.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.047+0000 7ffb0e59c700 1 --2- 172.21.15.111:0/3267091549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b20f0 0x7ffb100b3b80 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009840 tx=0x7ffb04009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36dbc7c052eb515 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:39.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.047+0000 7ffb0d59a700 1 -- 172.21.15.111:0/3267091549 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04011830 con 0x7ffb100b20f0 2023-11-28T13:02:39.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.047+0000 7ffb0d59a700 1 -- 172.21.15.111:0/3267091549 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb04011990 con 0x7ffb100b20f0 2023-11-28T13:02:39.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.047+0000 7ffb0d59a700 1 -- 172.21.15.111:0/3267091549 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04011d20 con 0x7ffb100b20f0 2023-11-28T13:02:39.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.047+0000 7ffb14a73700 1 -- 172.21.15.111:0/3267091549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b20f0 msgr2=0x7ffb100b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:39.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.047+0000 7ffb14a73700 1 --2- 172.21.15.111:0/3267091549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b20f0 0x7ffb100b3b80 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009840 tx=0x7ffb04009b50 comp rx=0 tx=0).stop 2023-11-28T13:02:39.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.047+0000 7ffb14a73700 1 -- 172.21.15.111:0/3267091549 shutdown_connections 2023-11-28T13:02:39.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.047+0000 7ffb14a73700 1 --2- 172.21.15.111:0/3267091549 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb1006ba90 0x7ffb100b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:39.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.047+0000 7ffb14a73700 1 --2- 172.21.15.111:0/3267091549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b20f0 0x7ffb100b3b80 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:39.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.047+0000 7ffb14a73700 1 -- 172.21.15.111:0/3267091549 >> 172.21.15.111:0/3267091549 conn(0x7ffb1009a040 msgr2=0x7ffb100d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:39.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.047+0000 7ffb14a73700 1 -- 172.21.15.111:0/3267091549 shutdown_connections 2023-11-28T13:02:39.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.047+0000 7ffb14a73700 1 -- 172.21.15.111:0/3267091549 wait complete. 2023-11-28T13:02:39.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.049+0000 7ffb14a73700 1 Processor -- start 2023-11-28T13:02:39.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.049+0000 7ffb14a73700 1 -- start start 2023-11-28T13:02:39.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.049+0000 7ffb14a73700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb1006ba90 0x7ffb100ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:39.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.049+0000 7ffb14a73700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb100b20f0 0x7ffb10157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:39.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.049+0000 7ffb14a73700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b40c0 0x7ffb100cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:39.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.049+0000 7ffb14a73700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100cc1e0 con 0x7ffb100b40c0 2023-11-28T13:02:39.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.049+0000 7ffb14a73700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1015d100 con 0x7ffb1006ba90 2023-11-28T13:02:39.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.049+0000 7ffb14a73700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100cd690 con 0x7ffb100b20f0 2023-11-28T13:02:39.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffb0ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b40c0 0x7ffb100cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:39.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffb0ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b40c0 0x7ffb100cb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37716/0 (socket says 172.21.15.111:37716) 2023-11-28T13:02:39.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffb0ed9d700 1 -- 172.21.15.111:0/2076105515 learned_addr learned my addr 172.21.15.111:0/2076105515 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:39.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffb0e59c700 1 --2- 172.21.15.111:0/2076105515 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb1006ba90 0x7ffb100ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:39.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffb0ed9d700 1 -- 172.21.15.111:0/2076105515 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb100b20f0 msgr2=0x7ffb10157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:39.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffb0ed9d700 1 --2- 172.21.15.111:0/2076105515 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb100b20f0 0x7ffb10157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:39.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffb0ed9d700 1 -- 172.21.15.111:0/2076105515 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb1006ba90 msgr2=0x7ffb100ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:39.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffb0ed9d700 1 --2- 172.21.15.111:0/2076105515 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb1006ba90 0x7ffb100ccf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:39.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffb0ed9d700 1 -- 172.21.15.111:0/2076105515 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb040094a0 con 0x7ffb100b40c0 2023-11-28T13:02:39.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffb0ed9d700 1 --2- 172.21.15.111:0/2076105515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b40c0 0x7ffb100cb360 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800cd50 tx=0x7ffb0800a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:39.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffafeffd700 1 -- 172.21.15.111:0/2076105515 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0800f480 con 0x7ffb100b40c0 2023-11-28T13:02:39.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffafeffd700 1 -- 172.21.15.111:0/2076105515 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0800f600 con 0x7ffb100b40c0 2023-11-28T13:02:39.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffb14a73700 1 -- 172.21.15.111:0/2076105515 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb100b4f90 con 0x7ffb100b40c0 2023-11-28T13:02:39.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffafeffd700 1 -- 172.21.15.111:0/2076105515 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0800f9b0 con 0x7ffb100b40c0 2023-11-28T13:02:39.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.050+0000 7ffb14a73700 1 -- 172.21.15.111:0/2076105515 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb100b4cb0 con 0x7ffb100b40c0 2023-11-28T13:02:39.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.051+0000 7ffb14a73700 1 -- 172.21.15.111:0/2076105515 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb100b24c0 con 0x7ffb100b40c0 2023-11-28T13:02:39.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.052+0000 7ffafeffd700 1 -- 172.21.15.111:0/2076105515 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffb08009030 con 0x7ffb100b40c0 2023-11-28T13:02:39.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.052+0000 7ffafeffd700 1 --2- 172.21.15.111:0/2076105515 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffaf00682f0 0x7ffaf006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:39.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.052+0000 7ffafeffd700 1 -- 172.21.15.111:0/2076105515 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6382+0+0 (secure 0 0 0) 0x7ffb080867b0 con 0x7ffb100b40c0 2023-11-28T13:02:39.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.052+0000 7ffb0e59c700 1 --2- 172.21.15.111:0/2076105515 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffaf00682f0 0x7ffaf006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:39.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.052+0000 7ffb0e59c700 1 --2- 172.21.15.111:0/2076105515 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffaf00682f0 0x7ffaf006a7a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ffb04000c00 tx=0x7ffb04008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:39.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.056+0000 7ffafeffd700 1 -- 172.21.15.111:0/2076105515 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffb08052e50 con 0x7ffb100b40c0 2023-11-28T13:02:39.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.281+0000 7ffb14a73700 1 -- 172.21.15.111:0/2076105515 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ffb10067170 con 0x7ffb100b40c0 2023-11-28T13:02:39.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.281+0000 7ffafeffd700 1 -- 172.21.15.111:0/2076105515 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7ffb08084e60 con 0x7ffb100b40c0 2023-11-28T13:02:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:02:39.283 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:02:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.283+0000 7ffb14a73700 1 -- 172.21.15.111:0/2076105515 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffaf00682f0 msgr2=0x7ffaf006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.283+0000 7ffb14a73700 1 --2- 172.21.15.111:0/2076105515 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffaf00682f0 0x7ffaf006a7a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ffb04000c00 tx=0x7ffb04008030 comp rx=0 tx=0).stop 2023-11-28T13:02:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.283+0000 7ffb14a73700 1 -- 172.21.15.111:0/2076105515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b40c0 msgr2=0x7ffb100cb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.283+0000 7ffb14a73700 1 --2- 172.21.15.111:0/2076105515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b40c0 0x7ffb100cb360 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800cd50 tx=0x7ffb0800a560 comp rx=0 tx=0).stop 2023-11-28T13:02:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.283+0000 7ffb14a73700 1 -- 172.21.15.111:0/2076105515 shutdown_connections 2023-11-28T13:02:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.283+0000 7ffb14a73700 1 --2- 172.21.15.111:0/2076105515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b40c0 0x7ffb100cb360 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.283+0000 7ffb14a73700 1 --2- 172.21.15.111:0/2076105515 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ffaf00682f0 0x7ffaf006a7a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.283+0000 7ffb14a73700 1 --2- 172.21.15.111:0/2076105515 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ffb1006ba90 0x7ffb100ccf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.283+0000 7ffb14a73700 1 --2- 172.21.15.111:0/2076105515 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ffb100b20f0 0x7ffb10157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.283+0000 7ffb14a73700 1 -- 172.21.15.111:0/2076105515 >> 172.21.15.111:0/2076105515 conn(0x7ffb1009a040 msgr2=0x7ffb100d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.283+0000 7ffb14a73700 1 -- 172.21.15.111:0/2076105515 shutdown_connections 2023-11-28T13:02:39.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:39.283+0000 7ffb14a73700 1 -- 172.21.15.111:0/2076105515 wait complete. 2023-11-28T13:02:39.296 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:02:44.297 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:02:44.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.426+0000 7f238704c700 1 Processor -- start 2023-11-28T13:02:44.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.426+0000 7f238704c700 1 -- start start 2023-11-28T13:02:44.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.427+0000 7f238704c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f23800b0d70 0x7f23801604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:44.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.427+0000 7f238704c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f23800b1140 0x7f23800ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:44.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.427+0000 7f238704c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238015cd90 con 0x7f23800cc550 2023-11-28T13:02:44.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.427+0000 7f238704c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23801613e0 con 0x7f23800b1140 2023-11-28T13:02:44.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.427+0000 7f238704c700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2380160b40 con 0x7f23800b0d70 2023-11-28T13:02:44.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.427+0000 7f237ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f23800b1140 0x7f23800ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:44.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.427+0000 7f237ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f23800b1140 0x7f23800ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37720/0 (socket says 172.21.15.111:37720) 2023-11-28T13:02:44.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.427+0000 7f237ffff700 1 -- 172.21.15.111:0/166071574 learned_addr learned my addr 172.21.15.111:0/166071574 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:44.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.427+0000 7f2384de8700 1 --2- 172.21.15.111:0/166071574 >> v2:172.21.15.149:3300/0 conn(0x7f23800b0d70 0x7f23801604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:44.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.427+0000 7f237ffff700 1 -- 172.21.15.111:0/166071574 >> v2:172.21.15.149:3300/0 conn(0x7f23800b0d70 msgr2=0x7f23801604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:44.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.427+0000 7f237ffff700 1 --2- 172.21.15.111:0/166071574 >> v2:172.21.15.149:3300/0 conn(0x7f23800b0d70 0x7f23801604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:44.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.427+0000 7f237ffff700 1 -- 172.21.15.111:0/166071574 >> v1:172.21.15.149:6789/0 conn(0x7f23800cc550 legacy=0x7f23800af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:44.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.427+0000 7f237ffff700 1 -- 172.21.15.111:0/166071574 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2380160e20 con 0x7f23800b1140 2023-11-28T13:02:44.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.428+0000 7f237ffff700 1 --2- 172.21.15.111:0/166071574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23800b1140 0x7f23800ce6c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f2380071960 tx=0x7f236c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df20f06cda90bbdf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:44.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.428+0000 7f237f7fe700 1 -- 172.21.15.111:0/166071574 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c00b870 con 0x7f23800b1140 2023-11-28T13:02:44.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.428+0000 7f237f7fe700 1 -- 172.21.15.111:0/166071574 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236c00b9d0 con 0x7f23800b1140 2023-11-28T13:02:44.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.428+0000 7f237f7fe700 1 -- 172.21.15.111:0/166071574 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c00bd60 con 0x7f23800b1140 2023-11-28T13:02:44.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.428+0000 7f238704c700 1 -- 172.21.15.111:0/166071574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23800b1140 msgr2=0x7f23800ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:44.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.428+0000 7f238704c700 1 --2- 172.21.15.111:0/166071574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23800b1140 0x7f23800ce6c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f2380071960 tx=0x7f236c009bd0 comp rx=0 tx=0).stop 2023-11-28T13:02:44.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.428+0000 7f238704c700 1 -- 172.21.15.111:0/166071574 shutdown_connections 2023-11-28T13:02:44.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.428+0000 7f238704c700 1 --2- 172.21.15.111:0/166071574 >> v2:172.21.15.149:3300/0 conn(0x7f23800b0d70 0x7f23801604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:44.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.428+0000 7f238704c700 1 --2- 172.21.15.111:0/166071574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23800b1140 0x7f23800ce6c0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:44.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.428+0000 7f238704c700 1 -- 172.21.15.111:0/166071574 >> 172.21.15.111:0/166071574 conn(0x7f2380098ee0 msgr2=0x7f238006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:44.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.428+0000 7f238704c700 1 -- 172.21.15.111:0/166071574 shutdown_connections 2023-11-28T13:02:44.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.428+0000 7f238704c700 1 -- 172.21.15.111:0/166071574 wait complete. 2023-11-28T13:02:44.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.430+0000 7f238704c700 1 Processor -- start 2023-11-28T13:02:44.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.430+0000 7f238704c700 1 -- start start 2023-11-28T13:02:44.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f238704c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23800b0d70 0x7f2380144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:44.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f238704c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23800b1140 0x7f23800b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:44.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f238704c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23800cc550 0x7f2380155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:44.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f238704c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2380144ab0 con 0x7f23800cc550 2023-11-28T13:02:44.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f238704c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23800b45e0 con 0x7f23800b1140 2023-11-28T13:02:44.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f238704c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23800b2eb0 con 0x7f23800b0d70 2023-11-28T13:02:44.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f23855e9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23800cc550 0x7f2380155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:44.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f23855e9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23800cc550 0x7f2380155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37730/0 (socket says 172.21.15.111:37730) 2023-11-28T13:02:44.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f23855e9700 1 -- 172.21.15.111:0/3534917798 learned_addr learned my addr 172.21.15.111:0/3534917798 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:44.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f2384de8700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23800b0d70 0x7f2380144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:44.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f237ffff700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23800b1140 0x7f23800b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:44.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f23855e9700 1 -- 172.21.15.111:0/3534917798 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23800b0d70 msgr2=0x7f2380144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:44.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f23855e9700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23800b0d70 0x7f2380144570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:44.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f23855e9700 1 -- 172.21.15.111:0/3534917798 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23800b1140 msgr2=0x7f23800b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:44.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f23855e9700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23800b1140 0x7f23800b2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:44.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f23855e9700 1 -- 172.21.15.111:0/3534917798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236c009550 con 0x7f23800cc550 2023-11-28T13:02:44.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f23855e9700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23800cc550 0x7f2380155910 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f237800cb50 tx=0x7f237800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:44.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f237d7fa700 1 -- 172.21.15.111:0/3534917798 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2378003c00 con 0x7f23800cc550 2023-11-28T13:02:44.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f237d7fa700 1 -- 172.21.15.111:0/3534917798 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2378003d60 con 0x7f23800cc550 2023-11-28T13:02:44.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f238704c700 1 -- 172.21.15.111:0/3534917798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23800af340 con 0x7f23800cc550 2023-11-28T13:02:44.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f237d7fa700 1 -- 172.21.15.111:0/3534917798 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237800aa20 con 0x7f23800cc550 2023-11-28T13:02:44.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.431+0000 7f238704c700 1 -- 172.21.15.111:0/3534917798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2380161100 con 0x7f23800cc550 2023-11-28T13:02:44.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.432+0000 7f238704c700 1 -- 172.21.15.111:0/3534917798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23800cec00 con 0x7f23800cc550 2023-11-28T13:02:44.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.433+0000 7f237d7fa700 1 -- 172.21.15.111:0/3534917798 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2378010050 con 0x7f23800cc550 2023-11-28T13:02:44.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.433+0000 7f237d7fa700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f23640682f0 0x7f236406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:44.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.433+0000 7f237d7fa700 1 -- 172.21.15.111:0/3534917798 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f2378085b00 con 0x7f23800cc550 2023-11-28T13:02:44.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.433+0000 7f2384de8700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f23640682f0 0x7f236406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:44.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.434+0000 7f2384de8700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f23640682f0 0x7f236406a7a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2374006fd0 tx=0x7f2374008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:44.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.437+0000 7f237d7fa700 1 -- 172.21.15.111:0/3534917798 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f23780521e0 con 0x7f23800cc550 2023-11-28T13:02:44.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.662+0000 7f238704c700 1 -- 172.21.15.111:0/3534917798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f238015cd90 con 0x7f23800cc550 2023-11-28T13:02:44.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.663+0000 7f237d7fa700 1 -- 172.21.15.111:0/3534917798 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f238015cd90 con 0x7f23800cc550 2023-11-28T13:02:44.664 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:02:44.664 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:02:44.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.665+0000 7f238704c700 1 -- 172.21.15.111:0/3534917798 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f23640682f0 msgr2=0x7f236406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:44.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.665+0000 7f238704c700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f23640682f0 0x7f236406a7a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2374006fd0 tx=0x7f2374008040 comp rx=0 tx=0).stop 2023-11-28T13:02:44.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.665+0000 7f238704c700 1 -- 172.21.15.111:0/3534917798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23800cc550 msgr2=0x7f2380155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.665+0000 7f238704c700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23800cc550 0x7f2380155910 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f237800cb50 tx=0x7f237800ce60 comp rx=0 tx=0).stop 2023-11-28T13:02:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.665+0000 7f238704c700 1 -- 172.21.15.111:0/3534917798 shutdown_connections 2023-11-28T13:02:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.665+0000 7f238704c700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23800cc550 0x7f2380155910 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.665+0000 7f238704c700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f23800b0d70 0x7f2380144570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.665+0000 7f238704c700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f23640682f0 0x7f236406a7a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.665+0000 7f238704c700 1 --2- 172.21.15.111:0/3534917798 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f23800b1140 0x7f23800b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.665+0000 7f238704c700 1 -- 172.21.15.111:0/3534917798 >> 172.21.15.111:0/3534917798 conn(0x7f2380098ee0 msgr2=0x7f238006dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.665+0000 7f238704c700 1 -- 172.21.15.111:0/3534917798 shutdown_connections 2023-11-28T13:02:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:44.665+0000 7f238704c700 1 -- 172.21.15.111:0/3534917798 wait complete. 2023-11-28T13:02:44.678 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:02:49.679 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:02:49.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.810+0000 7fd498ee0700 1 Processor -- start 2023-11-28T13:02:49.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.810+0000 7fd498ee0700 1 -- start start 2023-11-28T13:02:49.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.811+0000 7fd498ee0700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd4940b0d70 0x7fd4941604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:49.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.811+0000 7fd498ee0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4940b1140 0x7fd4940ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:49.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.811+0000 7fd498ee0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49415cd90 con 0x7fd4940b15b0 2023-11-28T13:02:49.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.811+0000 7fd498ee0700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4941613e0 con 0x7fd4940b0d70 2023-11-28T13:02:49.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.811+0000 7fd498ee0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd494160b40 con 0x7fd4940b1140 2023-11-28T13:02:49.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.811+0000 7fd492d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd4940b15b0 0x7fd4940cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38516/0 (socket says 172.21.15.111:38516) 2023-11-28T13:02:49.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.811+0000 7fd492d9d700 1 -- 172.21.15.111:0/1624911041 learned_addr learned my addr 172.21.15.111:0/1624911041 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:02:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.811+0000 7fd49259c700 1 --2- 172.21.15.111:0/1624911041 >> v2:172.21.15.149:3300/0 conn(0x7fd4940b0d70 0x7fd4941604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.811+0000 7fd491d9b700 1 --2- 172.21.15.111:0/1624911041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4940b1140 0x7fd4940ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.811+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084749568 0 0) 0x7fd48c001490 con 0x7fd4940b15b0 2023-11-28T13:02:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.811+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd47c0035b0 con 0x7fd4940b15b0 2023-11-28T13:02:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3295977374 0 0) 0x7fd47c0035b0 con 0x7fd4940b15b0 2023-11-28T13:02:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4940b1140 msgr2=0x7fd4940ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 --2- 172.21.15.111:0/1624911041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4940b1140 0x7fd4940ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 >> v2:172.21.15.149:3300/0 conn(0x7fd4940b0d70 msgr2=0x7fd4941604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 --2- 172.21.15.111:0/1624911041 >> v2:172.21.15.149:3300/0 conn(0x7fd4940b0d70 0x7fd4941604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd494160e20 con 0x7fd4940b15b0 2023-11-28T13:02:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fd48c001b80 con 0x7fd4940b15b0 2023-11-28T13:02:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 >> v1:172.21.15.111:6789/0 conn(0x7fd4940b15b0 legacy=0x7fd4940cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 --2- 172.21.15.111:0/1624911041 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd47c003a00 0x7fd47c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 --2- 172.21.15.111:0/1624911041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4940b1140 0x7fd47c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 --2- 172.21.15.111:0/1624911041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd47c004d90 0x7fd47c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4941613e0 con 0x7fd47c004d90 2023-11-28T13:02:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd494160b40 con 0x7fd47c003a00 2023-11-28T13:02:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c00ba10 con 0x7fd4940b1140 2023-11-28T13:02:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd492d9d700 1 --2- 172.21.15.111:0/1624911041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd47c004d90 0x7fd47c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49259c700 1 --2- 172.21.15.111:0/1624911041 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd47c003a00 0x7fd47c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd491d9b700 1 --2- 172.21.15.111:0/1624911041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4940b1140 0x7fd47c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd492d9d700 1 -- 172.21.15.111:0/1624911041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4940b1140 msgr2=0x7fd47c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd492d9d700 1 --2- 172.21.15.111:0/1624911041 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd4940b1140 0x7fd47c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd492d9d700 1 -- 172.21.15.111:0/1624911041 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd47c003a00 msgr2=0x7fd47c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd492d9d700 1 --2- 172.21.15.111:0/1624911041 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd47c003a00 0x7fd47c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd492d9d700 1 -- 172.21.15.111:0/1624911041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c0035b0 con 0x7fd47c004d90 2023-11-28T13:02:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd492d9d700 1 --2- 172.21.15.111:0/1624911041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd47c004d90 0x7fd47c005200 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd494070330 tx=0x7fd47c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de42551730bb13c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.812+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c00a4c0 con 0x7fd47c004d90 2023-11-28T13:02:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.813+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48c0072d0 con 0x7fd47c004d90 2023-11-28T13:02:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.813+0000 7fd49159a700 1 -- 172.21.15.111:0/1624911041 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c007680 con 0x7fd47c004d90 2023-11-28T13:02:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.813+0000 7fd498ee0700 1 -- 172.21.15.111:0/1624911041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd47c004d90 msgr2=0x7fd47c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.813+0000 7fd498ee0700 1 --2- 172.21.15.111:0/1624911041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd47c004d90 0x7fd47c005200 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd494070330 tx=0x7fd47c005880 comp rx=0 tx=0).stop 2023-11-28T13:02:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.813+0000 7fd492d9d700 1 -- 172.21.15.111:0/1624911041 reap_dead start 2023-11-28T13:02:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.813+0000 7fd498ee0700 1 -- 172.21.15.111:0/1624911041 shutdown_connections 2023-11-28T13:02:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.813+0000 7fd498ee0700 1 -- 172.21.15.111:0/1624911041 >> 172.21.15.111:0/1624911041 conn(0x7fd494098ee0 msgr2=0x7fd49406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.813+0000 7fd498ee0700 1 -- 172.21.15.111:0/1624911041 shutdown_connections 2023-11-28T13:02:49.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.813+0000 7fd498ee0700 1 -- 172.21.15.111:0/1624911041 wait complete. 2023-11-28T13:02:49.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.814+0000 7fd498ee0700 1 Processor -- start 2023-11-28T13:02:49.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.815+0000 7fd498ee0700 1 -- start start 2023-11-28T13:02:49.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.815+0000 7fd498ee0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd47c004d90 0x7fd49415a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:49.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.815+0000 7fd498ee0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd49415f070 0x7fd4940b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.815+0000 7fd498ee0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd49415f750 0x7fd49415fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.815+0000 7fd498ee0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49415aaf0 con 0x7fd49415f070 2023-11-28T13:02:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.815+0000 7fd498ee0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd494160100 con 0x7fd47c004d90 2023-11-28T13:02:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.815+0000 7fd498ee0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4940b3250 con 0x7fd49415f750 2023-11-28T13:02:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.815+0000 7fd491d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd49415f070 0x7fd4940b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.815+0000 7fd491d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd49415f070 0x7fd4940b2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37748/0 (socket says 172.21.15.111:37748) 2023-11-28T13:02:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.815+0000 7fd491d9b700 1 -- 172.21.15.111:0/2495077938 learned_addr learned my addr 172.21.15.111:0/2495077938 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.815+0000 7fd49259c700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd47c004d90 0x7fd49415a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.816+0000 7fd492d9d700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd49415f750 0x7fd49415fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.816+0000 7fd491d9b700 1 -- 172.21.15.111:0/2495077938 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd49415f750 msgr2=0x7fd49415fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.816+0000 7fd491d9b700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd49415f750 0x7fd49415fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:49.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.816+0000 7fd491d9b700 1 -- 172.21.15.111:0/2495077938 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd47c004d90 msgr2=0x7fd49415a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:49.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.816+0000 7fd491d9b700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd47c004d90 0x7fd49415a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:49.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.816+0000 7fd491d9b700 1 -- 172.21.15.111:0/2495077938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4940af340 con 0x7fd49415f070 2023-11-28T13:02:49.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.816+0000 7fd491d9b700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd49415f070 0x7fd4940b2970 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd484011aa0 tx=0x7fd484011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:49.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.816+0000 7fd482ffd700 1 -- 172.21.15.111:0/2495077938 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48400f8c0 con 0x7fd49415f070 2023-11-28T13:02:49.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.816+0000 7fd482ffd700 1 -- 172.21.15.111:0/2495077938 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48400fa20 con 0x7fd49415f070 2023-11-28T13:02:49.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.816+0000 7fd498ee0700 1 -- 172.21.15.111:0/2495077938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd494161100 con 0x7fd49415f070 2023-11-28T13:02:49.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.816+0000 7fd482ffd700 1 -- 172.21.15.111:0/2495077938 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48400fdd0 con 0x7fd49415f070 2023-11-28T13:02:49.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.816+0000 7fd498ee0700 1 -- 172.21.15.111:0/2495077938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4940ccc60 con 0x7fd49415f070 2023-11-28T13:02:49.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.817+0000 7fd498ee0700 1 -- 172.21.15.111:0/2495077938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4940cb0f0 con 0x7fd49415f070 2023-11-28T13:02:49.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.817+0000 7fd482ffd700 1 -- 172.21.15.111:0/2495077938 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd484014440 con 0x7fd49415f070 2023-11-28T13:02:49.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.818+0000 7fd482ffd700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4740682f0 0x7fd47406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:49.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.818+0000 7fd482ffd700 1 -- 172.21.15.111:0/2495077938 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fd484086640 con 0x7fd49415f070 2023-11-28T13:02:49.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.818+0000 7fd49259c700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4740682f0 0x7fd47406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:49.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.818+0000 7fd49259c700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4740682f0 0x7fd47406a7a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd488009cc0 tx=0x7fd4880091c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:49.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:49.822+0000 7fd482ffd700 1 -- 172.21.15.111:0/2495077938 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd484052d80 con 0x7fd49415f070 2023-11-28T13:02:50.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.048+0000 7fd498ee0700 1 -- 172.21.15.111:0/2495077938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd494067170 con 0x7fd49415f070 2023-11-28T13:02:50.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.048+0000 7fd482ffd700 1 -- 172.21.15.111:0/2495077938 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7fd4840563a0 con 0x7fd49415f070 2023-11-28T13:02:50.050 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:02:50.050 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:02:50.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.050+0000 7fd498ee0700 1 -- 172.21.15.111:0/2495077938 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4740682f0 msgr2=0x7fd47406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:50.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.050+0000 7fd498ee0700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4740682f0 0x7fd47406a7a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd488009cc0 tx=0x7fd4880091c0 comp rx=0 tx=0).stop 2023-11-28T13:02:50.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.050+0000 7fd498ee0700 1 -- 172.21.15.111:0/2495077938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd49415f070 msgr2=0x7fd4940b2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:50.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.050+0000 7fd498ee0700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd49415f070 0x7fd4940b2970 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd484011aa0 tx=0x7fd484011e60 comp rx=0 tx=0).stop 2023-11-28T13:02:50.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.050+0000 7fd498ee0700 1 -- 172.21.15.111:0/2495077938 shutdown_connections 2023-11-28T13:02:50.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.050+0000 7fd498ee0700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd49415f750 0x7fd49415fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:50.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.050+0000 7fd498ee0700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd4740682f0 0x7fd47406a7a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.050+0000 7fd498ee0700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd47c004d90 0x7fd49415a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.050+0000 7fd498ee0700 1 --2- 172.21.15.111:0/2495077938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd49415f070 0x7fd4940b2970 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.050+0000 7fd498ee0700 1 -- 172.21.15.111:0/2495077938 >> 172.21.15.111:0/2495077938 conn(0x7fd494098ee0 msgr2=0x7fd4940cb300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.050+0000 7fd498ee0700 1 -- 172.21.15.111:0/2495077938 shutdown_connections 2023-11-28T13:02:50.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:50.050+0000 7fd498ee0700 1 -- 172.21.15.111:0/2495077938 wait complete. 2023-11-28T13:02:50.064 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:02:55.065 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:02:55.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.197+0000 7f7d7caaf700 1 Processor -- start 2023-11-28T13:02:55.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.197+0000 7f7d7caaf700 1 -- start start 2023-11-28T13:02:55.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.198+0000 7f7d7caaf700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f7d78159960 0x7f7d780b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:55.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.198+0000 7f7d7caaf700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d780aecc0 0x7f7d780cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:55.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.198+0000 7f7d7caaf700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7d780cdbc0 0x7f7d780afa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:55.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.198+0000 7f7d7caaf700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d780ce040 con 0x7f7d780cdbc0 2023-11-28T13:02:55.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.198+0000 7f7d7caaf700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d78159d30 con 0x7f7d78159960 2023-11-28T13:02:55.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.198+0000 7f7d7caaf700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780ad420 con 0x7f7d780aecc0 2023-11-28T13:02:55.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d76d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7d780cdbc0 0x7f7d780afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:55.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d76d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7d780cdbc0 0x7f7d780afa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37758/0 (socket says 172.21.15.111:37758) 2023-11-28T13:02:55.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d76d9d700 1 -- 172.21.15.111:0/805472810 learned_addr learned my addr 172.21.15.111:0/805472810 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:55.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d7659c700 1 --2- 172.21.15.111:0/805472810 >> v2:172.21.15.149:3300/0 conn(0x7f7d78159960 0x7f7d780b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:55.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d75d9b700 1 --2- 172.21.15.111:0/805472810 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d780aecc0 0x7f7d780cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:55.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d76d9d700 1 -- 172.21.15.111:0/805472810 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d780aecc0 msgr2=0x7f7d780cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:55.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d76d9d700 1 --2- 172.21.15.111:0/805472810 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d780aecc0 0x7f7d780cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:55.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d76d9d700 1 -- 172.21.15.111:0/805472810 >> v2:172.21.15.149:3300/0 conn(0x7f7d78159960 msgr2=0x7f7d780b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:02:55.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d76d9d700 1 --2- 172.21.15.111:0/805472810 >> v2:172.21.15.149:3300/0 conn(0x7f7d78159960 0x7f7d780b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:55.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d76d9d700 1 -- 172.21.15.111:0/805472810 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d780ac810 con 0x7f7d780cdbc0 2023-11-28T13:02:55.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d7659c700 1 --2- 172.21.15.111:0/805472810 >> v2:172.21.15.149:3300/0 conn(0x7f7d78159960 0x7f7d780b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:02:55.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d76d9d700 1 --2- 172.21.15.111:0/805472810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d780cdbc0 0x7f7d780afa70 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7d7806c6a0 tx=0x7f7d7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00824f8ad492ecc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:55.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d7559a700 1 -- 172.21.15.111:0/805472810 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7000e070 con 0x7f7d780cdbc0 2023-11-28T13:02:55.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d7559a700 1 -- 172.21.15.111:0/805472810 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7000b9c0 con 0x7f7d780cdbc0 2023-11-28T13:02:55.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.199+0000 7f7d7559a700 1 -- 172.21.15.111:0/805472810 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7000bd70 con 0x7f7d780cdbc0 2023-11-28T13:02:55.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.200+0000 7f7d7caaf700 1 -- 172.21.15.111:0/805472810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d780cdbc0 msgr2=0x7f7d780afa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:55.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.200+0000 7f7d7caaf700 1 --2- 172.21.15.111:0/805472810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d780cdbc0 0x7f7d780afa70 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7d7806c6a0 tx=0x7f7d7000b330 comp rx=0 tx=0).stop 2023-11-28T13:02:55.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.200+0000 7f7d7caaf700 1 -- 172.21.15.111:0/805472810 shutdown_connections 2023-11-28T13:02:55.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.200+0000 7f7d7caaf700 1 --2- 172.21.15.111:0/805472810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d780cdbc0 0x7f7d780afa70 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:55.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.200+0000 7f7d7caaf700 1 --2- 172.21.15.111:0/805472810 >> v2:172.21.15.149:3300/0 conn(0x7f7d78159960 0x7f7d780b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:55.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.200+0000 7f7d7caaf700 1 --2- 172.21.15.111:0/805472810 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d780aecc0 0x7f7d780cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:55.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.200+0000 7f7d7caaf700 1 -- 172.21.15.111:0/805472810 >> 172.21.15.111:0/805472810 conn(0x7f7d78098ee0 msgr2=0x7f7d7806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:55.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.200+0000 7f7d7caaf700 1 -- 172.21.15.111:0/805472810 shutdown_connections 2023-11-28T13:02:55.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.200+0000 7f7d7caaf700 1 -- 172.21.15.111:0/805472810 wait complete. 2023-11-28T13:02:55.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.201+0000 7f7d7caaf700 1 Processor -- start 2023-11-28T13:02:55.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.201+0000 7f7d7caaf700 1 -- start start 2023-11-28T13:02:55.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d7caaf700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d780aecc0 0x7f7d780c2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:55.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d7caaf700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d780cdbc0 0x7f7d78145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:55.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d7caaf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d78159960 0x7f7d7815b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:55.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d7caaf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780c21d0 con 0x7f7d78159960 2023-11-28T13:02:55.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d7caaf700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d78148220 con 0x7f7d780aecc0 2023-11-28T13:02:55.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d7caaf700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d78146110 con 0x7f7d780cdbc0 2023-11-28T13:02:55.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d76d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d78159960 0x7f7d7815b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:55.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d76d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d78159960 0x7f7d7815b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37764/0 (socket says 172.21.15.111:37764) 2023-11-28T13:02:55.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d76d9d700 1 -- 172.21.15.111:0/2784254631 learned_addr learned my addr 172.21.15.111:0/2784254631 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:02:55.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d75d9b700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d780cdbc0 0x7f7d78145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:55.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d7659c700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d780aecc0 0x7f7d780c2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:55.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d76d9d700 1 -- 172.21.15.111:0/2784254631 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d780cdbc0 msgr2=0x7f7d78145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:55.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d76d9d700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d780cdbc0 0x7f7d78145bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:55.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d76d9d700 1 -- 172.21.15.111:0/2784254631 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d780aecc0 msgr2=0x7f7d780c2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:55.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d76d9d700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d780aecc0 0x7f7d780c2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:55.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.202+0000 7f7d76d9d700 1 -- 172.21.15.111:0/2784254631 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d70009b30 con 0x7f7d78159960 2023-11-28T13:02:55.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.203+0000 7f7d76d9d700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d78159960 0x7f7d7815b190 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7d780b2730 tx=0x7f7d70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:55.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.203+0000 7f7d66ffd700 1 -- 172.21.15.111:0/2784254631 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d700099b0 con 0x7f7d78159960 2023-11-28T13:02:55.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.203+0000 7f7d66ffd700 1 -- 172.21.15.111:0/2784254631 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d70004250 con 0x7f7d78159960 2023-11-28T13:02:55.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.203+0000 7f7d7caaf700 1 -- 172.21.15.111:0/2784254631 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d780acdd0 con 0x7f7d78159960 2023-11-28T13:02:55.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.203+0000 7f7d66ffd700 1 -- 172.21.15.111:0/2784254631 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d70004600 con 0x7f7d78159960 2023-11-28T13:02:55.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.203+0000 7f7d7caaf700 1 -- 172.21.15.111:0/2784254631 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d780acaf0 con 0x7f7d78159960 2023-11-28T13:02:55.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.204+0000 7f7d7caaf700 1 -- 172.21.15.111:0/2784254631 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d780ad0b0 con 0x7f7d78159960 2023-11-28T13:02:55.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.204+0000 7f7d66ffd700 1 -- 172.21.15.111:0/2784254631 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7d70004760 con 0x7f7d78159960 2023-11-28T13:02:55.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.205+0000 7f7d66ffd700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7d580682f0 0x7f7d5806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:02:55.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.205+0000 7f7d66ffd700 1 -- 172.21.15.111:0/2784254631 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f7d700873d0 con 0x7f7d78159960 2023-11-28T13:02:55.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.205+0000 7f7d7659c700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7d580682f0 0x7f7d5806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:02:55.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.205+0000 7f7d7659c700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7d580682f0 0x7f7d5806a7a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00c960 tx=0x7f7d6c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:02:55.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.209+0000 7f7d66ffd700 1 -- 172.21.15.111:0/2784254631 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7d70053b60 con 0x7f7d78159960 2023-11-28T13:02:55.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.435+0000 7f7d7caaf700 1 -- 172.21.15.111:0/2784254631 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7d7806c4f0 con 0x7f7d78159960 2023-11-28T13:02:55.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.436+0000 7f7d66ffd700 1 -- 172.21.15.111:0/2784254631 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f7d70057180 con 0x7f7d78159960 2023-11-28T13:02:55.437 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:02:55.437 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:02:55.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.438+0000 7f7d7caaf700 1 -- 172.21.15.111:0/2784254631 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7d580682f0 msgr2=0x7f7d5806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:55.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.438+0000 7f7d7caaf700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7d580682f0 0x7f7d5806a7a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00c960 tx=0x7f7d6c008040 comp rx=0 tx=0).stop 2023-11-28T13:02:55.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.438+0000 7f7d7caaf700 1 -- 172.21.15.111:0/2784254631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d78159960 msgr2=0x7f7d7815b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:02:55.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.438+0000 7f7d7caaf700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d78159960 0x7f7d7815b190 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f7d780b2730 tx=0x7f7d70004ad0 comp rx=0 tx=0).stop 2023-11-28T13:02:55.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.438+0000 7f7d7caaf700 1 -- 172.21.15.111:0/2784254631 shutdown_connections 2023-11-28T13:02:55.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.438+0000 7f7d7caaf700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d78159960 0x7f7d7815b190 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:55.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.438+0000 7f7d7caaf700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f7d580682f0 0x7f7d5806a7a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:55.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.438+0000 7f7d7caaf700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f7d780aecc0 0x7f7d780c2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:55.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.438+0000 7f7d7caaf700 1 --2- 172.21.15.111:0/2784254631 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f7d780cdbc0 0x7f7d78145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:02:55.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.438+0000 7f7d7caaf700 1 -- 172.21.15.111:0/2784254631 >> 172.21.15.111:0/2784254631 conn(0x7f7d78098ee0 msgr2=0x7f7d7806dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:02:55.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.438+0000 7f7d7caaf700 1 -- 172.21.15.111:0/2784254631 shutdown_connections 2023-11-28T13:02:55.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:02:55.438+0000 7f7d7caaf700 1 -- 172.21.15.111:0/2784254631 wait complete. 2023-11-28T13:02:55.454 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:02:59.136 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:02:59.139 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:03:00.455 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:03:00.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.585+0000 7f52ed8ab700 1 Processor -- start 2023-11-28T13:03:00.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.585+0000 7f52ed8ab700 1 -- start start 2023-11-28T13:03:00.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52ed8ab700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52e80b8510 0x7f52e806b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:00.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52ed8ab700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52e80d0d60 0x7f52e80d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:00.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52ed8ab700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e80b88e0 con 0x7f52e80cd460 2023-11-28T13:03:00.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52ed8ab700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e80cdbd0 con 0x7f52e80d0d60 2023-11-28T13:03:00.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52ed8ab700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e80d02d0 con 0x7f52e80b8510 2023-11-28T13:03:00.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52e77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52e80d0d60 0x7f52e80d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:00.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52e77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52e80d0d60 0x7f52e80d11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37772/0 (socket says 172.21.15.111:37772) 2023-11-28T13:03:00.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52e77fe700 1 -- 172.21.15.111:0/2402033825 learned_addr learned my addr 172.21.15.111:0/2402033825 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:00.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52e77fe700 1 -- 172.21.15.111:0/2402033825 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52e80b8510 msgr2=0x7f52e806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:00.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52e77fe700 1 --2- 172.21.15.111:0/2402033825 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52e80b8510 0x7f52e806b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:00.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52e77fe700 1 -- 172.21.15.111:0/2402033825 >> v1:172.21.15.149:6789/0 conn(0x7f52e80cd460 legacy=0x7f52e806beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:00.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52e77fe700 1 -- 172.21.15.111:0/2402033825 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e80cd8f0 con 0x7f52e80d0d60 2023-11-28T13:03:00.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52e77fe700 1 --2- 172.21.15.111:0/2402033825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e80d0d60 0x7f52e80d11a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009e80 tx=0x7f52e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac31a6372e7de8c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:00.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52e5ffb700 1 -- 172.21.15.111:0/2402033825 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e000e070 con 0x7f52e80d0d60 2023-11-28T13:03:00.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52e5ffb700 1 -- 172.21.15.111:0/2402033825 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e0003a20 con 0x7f52e80d0d60 2023-11-28T13:03:00.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.586+0000 7f52e5ffb700 1 -- 172.21.15.111:0/2402033825 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e0003dd0 con 0x7f52e80d0d60 2023-11-28T13:03:00.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.587+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2402033825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e80d0d60 msgr2=0x7f52e80d11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:00.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.587+0000 7f52ed8ab700 1 --2- 172.21.15.111:0/2402033825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e80d0d60 0x7f52e80d11a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009e80 tx=0x7f52e000b330 comp rx=0 tx=0).stop 2023-11-28T13:03:00.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.587+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2402033825 shutdown_connections 2023-11-28T13:03:00.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.587+0000 7f52ed8ab700 1 --2- 172.21.15.111:0/2402033825 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52e80b8510 0x7f52e806b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:00.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.587+0000 7f52ed8ab700 1 --2- 172.21.15.111:0/2402033825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e80d0d60 0x7f52e80d11a0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:00.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.587+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2402033825 >> 172.21.15.111:0/2402033825 conn(0x7f52e80719a0 msgr2=0x7f52e8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:00.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.587+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2402033825 shutdown_connections 2023-11-28T13:03:00.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.587+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2402033825 wait complete. 2023-11-28T13:03:00.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.588+0000 7f52ed8ab700 1 Processor -- start 2023-11-28T13:03:00.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.588+0000 7f52ed8ab700 1 -- start start 2023-11-28T13:03:00.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.589+0000 7f52ed8ab700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52e80b8510 0x7f52e80b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:00.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.589+0000 7f52ed8ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e80cd460 0x7f52e80b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:00.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.589+0000 7f52ed8ab700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52e80d0d60 0x7f52e80b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:00.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.589+0000 7f52ed8ab700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e81f5120 con 0x7f52e80cd460 2023-11-28T13:03:00.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.589+0000 7f52ed8ab700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e81f28d0 con 0x7f52e80d0d60 2023-11-28T13:03:00.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.589+0000 7f52ed8ab700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e81f5370 con 0x7f52e80b8510 2023-11-28T13:03:00.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.589+0000 7f52e67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e80cd460 0x7f52e80b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:00.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.589+0000 7f52e67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e80cd460 0x7f52e80b3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37776/0 (socket says 172.21.15.111:37776) 2023-11-28T13:03:00.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.589+0000 7f52e67fc700 1 -- 172.21.15.111:0/2041901585 learned_addr learned my addr 172.21.15.111:0/2041901585 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:00.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.589+0000 7f52e77fe700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52e80d0d60 0x7f52e80b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:00.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.589+0000 7f52e6ffd700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52e80b8510 0x7f52e80b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:00.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.590+0000 7f52e67fc700 1 -- 172.21.15.111:0/2041901585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52e80b8510 msgr2=0x7f52e80b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:00.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.590+0000 7f52e67fc700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52e80b8510 0x7f52e80b1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:00.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.590+0000 7f52e67fc700 1 -- 172.21.15.111:0/2041901585 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52e80d0d60 msgr2=0x7f52e80b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:00.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.590+0000 7f52e67fc700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52e80d0d60 0x7f52e80b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:00.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.590+0000 7f52e67fc700 1 -- 172.21.15.111:0/2041901585 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e0009b30 con 0x7f52e80cd460 2023-11-28T13:03:00.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.590+0000 7f52e67fc700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e80cd460 0x7f52e80b3070 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00c4d0 tx=0x7f52dc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:00.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.590+0000 7f52cffff700 1 -- 172.21.15.111:0/2041901585 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc011790 con 0x7f52e80cd460 2023-11-28T13:03:00.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.590+0000 7f52cffff700 1 -- 172.21.15.111:0/2041901585 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52dc0118f0 con 0x7f52e80cd460 2023-11-28T13:03:00.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.590+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2041901585 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e80cfff0 con 0x7f52e80cd460 2023-11-28T13:03:00.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.590+0000 7f52cffff700 1 -- 172.21.15.111:0/2041901585 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc011ca0 con 0x7f52e80cd460 2023-11-28T13:03:00.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.590+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2041901585 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e80cfd10 con 0x7f52e80cd460 2023-11-28T13:03:00.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.591+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2041901585 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e8067880 con 0x7f52e80cd460 2023-11-28T13:03:00.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.591+0000 7f52cffff700 1 -- 172.21.15.111:0/2041901585 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f52dc010050 con 0x7f52e80cd460 2023-11-28T13:03:00.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.592+0000 7f52cffff700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52d40682f0 0x7f52d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:00.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.592+0000 7f52cffff700 1 -- 172.21.15.111:0/2041901585 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f52dc085b00 con 0x7f52e80cd460 2023-11-28T13:03:00.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.592+0000 7f52e6ffd700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52d40682f0 0x7f52d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:00.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.592+0000 7f52e6ffd700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52d40682f0 0x7f52d406a7a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f52d8006fd0 tx=0x7f52d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:00.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.596+0000 7f52cffff700 1 -- 172.21.15.111:0/2041901585 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f52dc0521e0 con 0x7f52e80cd460 2023-11-28T13:03:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.821+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2041901585 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f52e80d2940 con 0x7f52e80cd460 2023-11-28T13:03:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.822+0000 7f52cffff700 1 -- 172.21.15.111:0/2041901585 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f52dc085030 con 0x7f52e80cd460 2023-11-28T13:03:00.823 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:03:00.823 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:03:00.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.823+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2041901585 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52d40682f0 msgr2=0x7f52d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:00.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.823+0000 7f52ed8ab700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52d40682f0 0x7f52d406a7a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f52d8006fd0 tx=0x7f52d8008040 comp rx=0 tx=0).stop 2023-11-28T13:03:00.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.823+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2041901585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e80cd460 msgr2=0x7f52e80b3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.823+0000 7f52ed8ab700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e80cd460 0x7f52e80b3070 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00c4d0 tx=0x7f52dc00c890 comp rx=0 tx=0).stop 2023-11-28T13:03:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.824+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2041901585 shutdown_connections 2023-11-28T13:03:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.824+0000 7f52ed8ab700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f52d40682f0 0x7f52d406a7a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.824+0000 7f52ed8ab700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f52e80d0d60 0x7f52e80b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.824+0000 7f52ed8ab700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f52e80b8510 0x7f52e80b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.824+0000 7f52ed8ab700 1 --2- 172.21.15.111:0/2041901585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52e80cd460 0x7f52e80b3070 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.824+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2041901585 >> 172.21.15.111:0/2041901585 conn(0x7f52e80719a0 msgr2=0x7f52e80d1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.824+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2041901585 shutdown_connections 2023-11-28T13:03:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:00.824+0000 7f52ed8ab700 1 -- 172.21.15.111:0/2041901585 wait complete. 2023-11-28T13:03:00.837 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:03:05.838 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:03:05.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.969+0000 7f2f8b6fa700 1 Processor -- start 2023-11-28T13:03:05.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.969+0000 7f2f8b6fa700 1 -- start start 2023-11-28T13:03:05.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.970+0000 7f2f8b6fa700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f840d1d70 0x7f2f84167270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:05.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.970+0000 7f2f8b6fa700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8406c790 con 0x7f2f840b8510 2023-11-28T13:03:05.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.970+0000 7f2f8b6fa700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f840b88e0 con 0x7f2f840cd460 2023-11-28T13:03:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.970+0000 7f2f8b6fa700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f840cdbd0 con 0x7f2f840d1d70 2023-11-28T13:03:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.970+0000 7f2f89496700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2f840b8510 0x7f2f8406b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38560/0 (socket says 172.21.15.111:38560) 2023-11-28T13:03:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.970+0000 7f2f89496700 1 -- 172.21.15.111:0/3724292579 learned_addr learned my addr 172.21.15.111:0/3724292579 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:03:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f89c97700 1 --2- 172.21.15.111:0/3724292579 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f840d1d70 0x7f2f84167270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220927291 0 0) 0x7f2f8406c790 con 0x7f2f840b8510 2023-11-28T13:03:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f700035b0 con 0x7f2f840b8510 2023-11-28T13:03:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593428 0 0) 0x7f2f840b88e0 con 0x7f2f840cd460 2023-11-28T13:03:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f8406c790 con 0x7f2f840cd460 2023-11-28T13:03:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2512489720 0 0) 0x7f2f700035b0 con 0x7f2f840b8510 2023-11-28T13:03:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f840d1d70 msgr2=0x7f2f84167270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 --2- 172.21.15.111:0/3724292579 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f840d1d70 0x7f2f84167270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:05.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 >> v1:172.21.15.149:6789/0 conn(0x7f2f840cd460 legacy=0x7f2f8406c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f840cd8f0 con 0x7f2f840b8510 2023-11-28T13:03:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f2f74001670 con 0x7f2f840b8510 2023-11-28T13:03:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 >> v1:172.21.15.111:6789/0 conn(0x7f2f840b8510 legacy=0x7f2f8406b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 --2- 172.21.15.111:0/3724292579 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f70003bc0 0x7f2f70003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 --2- 172.21.15.111:0/3724292579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f70004850 0x7f2f70004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 --2- 172.21.15.111:0/3724292579 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f840d1d70 0x7f2f70005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f840cdbd0 con 0x7f2f70004850 2023-11-28T13:03:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7000bac0 con 0x7f2f70003bc0 2023-11-28T13:03:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7000bc80 con 0x7f2f840d1d70 2023-11-28T13:03:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f88c95700 1 --2- 172.21.15.111:0/3724292579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f70004850 0x7f2f70004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f89496700 1 --2- 172.21.15.111:0/3724292579 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f70003bc0 0x7f2f70003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:05.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f89c97700 1 --2- 172.21.15.111:0/3724292579 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f840d1d70 0x7f2f70005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f88c95700 1 -- 172.21.15.111:0/3724292579 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f840d1d70 msgr2=0x7f2f70005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.971+0000 7f2f88c95700 1 --2- 172.21.15.111:0/3724292579 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f840d1d70 0x7f2f70005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f88c95700 1 -- 172.21.15.111:0/3724292579 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f70003bc0 msgr2=0x7f2f70003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f88c95700 1 --2- 172.21.15.111:0/3724292579 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f70003bc0 0x7f2f70003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f88c95700 1 -- 172.21.15.111:0/3724292579 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f700035b0 con 0x7f2f70004850 2023-11-28T13:03:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f88c95700 1 --2- 172.21.15.111:0/3724292579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f70004850 0x7f2f70004cc0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2f8408af10 tx=0x7f2f70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6341aee94f4209 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f80001430 con 0x7f2f70004850 2023-11-28T13:03:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f800052d0 con 0x7f2f70004850 2023-11-28T13:03:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f7bfff700 1 -- 172.21.15.111:0/3724292579 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f80001d40 con 0x7f2f70004850 2023-11-28T13:03:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3724292579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f70004850 msgr2=0x7f2f70004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f8b6fa700 1 --2- 172.21.15.111:0/3724292579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f70004850 0x7f2f70004cc0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2f8408af10 tx=0x7f2f70005880 comp rx=0 tx=0).stop 2023-11-28T13:03:05.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f89c97700 1 -- 172.21.15.111:0/3724292579 reap_dead start 2023-11-28T13:03:05.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3724292579 shutdown_connections 2023-11-28T13:03:05.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3724292579 >> 172.21.15.111:0/3724292579 conn(0x7f2f840719a0 msgr2=0x7f2f84071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:05.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3724292579 shutdown_connections 2023-11-28T13:03:05.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.972+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3724292579 wait complete. 2023-11-28T13:03:05.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.974+0000 7f2f8b6fa700 1 Processor -- start 2023-11-28T13:03:05.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.974+0000 7f2f8b6fa700 1 -- start start 2023-11-28T13:03:05.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.974+0000 7f2f8b6fa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f70004850 0x7f2f841f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:05.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.974+0000 7f2f8b6fa700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f840c6940 0x7f2f840c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:05.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.974+0000 7f2f8b6fa700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f841572b0 0x7f2f84157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:05.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.974+0000 7f2f8b6fa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f84157c60 con 0x7f2f70004850 2023-11-28T13:03:05.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.974+0000 7f2f8b6fa700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f84158660 con 0x7f2f840c6940 2023-11-28T13:03:05.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.974+0000 7f2f8b6fa700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f84158c20 con 0x7f2f841572b0 2023-11-28T13:03:05.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f89496700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f70004850 0x7f2f841f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:05.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f89496700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f70004850 0x7f2f841f1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37792/0 (socket says 172.21.15.111:37792) 2023-11-28T13:03:05.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f89496700 1 -- 172.21.15.111:0/3275753794 learned_addr learned my addr 172.21.15.111:0/3275753794 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:05.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f88c95700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f840c6940 0x7f2f840c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:05.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f89c97700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f841572b0 0x7f2f84157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:05.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f89496700 1 -- 172.21.15.111:0/3275753794 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f841572b0 msgr2=0x7f2f84157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:05.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f89496700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f841572b0 0x7f2f84157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:05.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f89496700 1 -- 172.21.15.111:0/3275753794 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f840c6940 msgr2=0x7f2f840c6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:05.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f89496700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f840c6940 0x7f2f840c6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:05.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f89496700 1 -- 172.21.15.111:0/3275753794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f840cfff0 con 0x7f2f70004850 2023-11-28T13:03:05.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f89496700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f70004850 0x7f2f841f1bf0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2f74007340 tx=0x7f2f74007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:05.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f79ffb700 1 -- 172.21.15.111:0/3275753794 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7400f870 con 0x7f2f70004850 2023-11-28T13:03:05.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f79ffb700 1 -- 172.21.15.111:0/3275753794 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7400f9f0 con 0x7f2f70004850 2023-11-28T13:03:05.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3275753794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f840cfd10 con 0x7f2f70004850 2023-11-28T13:03:05.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f79ffb700 1 -- 172.21.15.111:0/3275753794 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7400fda0 con 0x7f2f70004850 2023-11-28T13:03:05.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.975+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3275753794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f840d29c0 con 0x7f2f70004850 2023-11-28T13:03:05.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.976+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3275753794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f84067880 con 0x7f2f70004850 2023-11-28T13:03:05.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.977+0000 7f2f79ffb700 1 -- 172.21.15.111:0/3275753794 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f7401d460 con 0x7f2f70004850 2023-11-28T13:03:05.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.977+0000 7f2f79ffb700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f680682f0 0x7f2f6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:05.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.977+0000 7f2f79ffb700 1 -- 172.21.15.111:0/3275753794 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f2f74086040 con 0x7f2f70004850 2023-11-28T13:03:05.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.977+0000 7f2f88c95700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f680682f0 0x7f2f6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:05.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.977+0000 7f2f88c95700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f680682f0 0x7f2f6806a7a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2f840b2bc0 tx=0x7f2f8000a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:05.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:05.981+0000 7f2f79ffb700 1 -- 172.21.15.111:0/3275753794 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f740526c0 con 0x7f2f70004850 2023-11-28T13:03:06.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.204+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3275753794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2f8408a970 con 0x7f2f70004850 2023-11-28T13:03:06.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.204+0000 7f2f79ffb700 1 -- 172.21.15.111:0/3275753794 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f2f74055ce0 con 0x7f2f70004850 2023-11-28T13:03:06.208 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:03:06.208 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:03:06.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.206+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3275753794 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f680682f0 msgr2=0x7f2f6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:06.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.206+0000 7f2f8b6fa700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f680682f0 0x7f2f6806a7a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2f840b2bc0 tx=0x7f2f8000a000 comp rx=0 tx=0).stop 2023-11-28T13:03:06.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.206+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3275753794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f70004850 msgr2=0x7f2f841f1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:06.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.206+0000 7f2f8b6fa700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f70004850 0x7f2f841f1bf0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2f74007340 tx=0x7f2f74007650 comp rx=0 tx=0).stop 2023-11-28T13:03:06.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.206+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3275753794 shutdown_connections 2023-11-28T13:03:06.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.206+0000 7f2f8b6fa700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f841572b0 0x7f2f84157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:06.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.206+0000 7f2f8b6fa700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f70004850 0x7f2f841f1bf0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:06.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.206+0000 7f2f8b6fa700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f680682f0 0x7f2f6806a7a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:06.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.206+0000 7f2f8b6fa700 1 --2- 172.21.15.111:0/3275753794 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f840c6940 0x7f2f840c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:06.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.206+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3275753794 >> 172.21.15.111:0/3275753794 conn(0x7f2f840719a0 msgr2=0x7f2f840b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:06.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.206+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3275753794 shutdown_connections 2023-11-28T13:03:06.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:06.206+0000 7f2f8b6fa700 1 -- 172.21.15.111:0/3275753794 wait complete. 2023-11-28T13:03:06.219 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:03:11.220 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:03:11.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.351+0000 7f5752e70700 1 Processor -- start 2023-11-28T13:03:11.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.352+0000 7f5752e70700 1 -- start start 2023-11-28T13:03:11.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.352+0000 7f5752e70700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f574c0b8510 0x7f574c06b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:11.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.352+0000 7f5752e70700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f574c0d0d60 0x7f574c0d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:11.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.352+0000 7f5752e70700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574c0b88e0 con 0x7f574c0cd460 2023-11-28T13:03:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.352+0000 7f5752e70700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574c0cdbd0 con 0x7f574c0d0d60 2023-11-28T13:03:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.352+0000 7f5752e70700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0d02d0 con 0x7f574c0b8510 2023-11-28T13:03:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.352+0000 7f574bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f574c0cd460 0x7f574c06beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38580/0 (socket says 172.21.15.111:38580) 2023-11-28T13:03:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.352+0000 7f574bfff700 1 -- 172.21.15.111:0/3302738428 learned_addr learned my addr 172.21.15.111:0/3302738428 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:03:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f575140d700 1 --2- 172.21.15.111:0/3302738428 >> v2:172.21.15.149:3300/0 conn(0x7f574c0d0d60 0x7f574c0d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f5750c0c700 1 --2- 172.21.15.111:0/3302738428 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f574c0b8510 0x7f574c06b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686634636 0 0) 0x7f574c0b88e0 con 0x7f574c0cd460 2023-11-28T13:03:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573c0035b0 con 0x7f574c0cd460 2023-11-28T13:03:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2702044114 0 0) 0x7f573c0035b0 con 0x7f574c0cd460 2023-11-28T13:03:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f574c0b8510 msgr2=0x7f574c06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 --2- 172.21.15.111:0/3302738428 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f574c0b8510 0x7f574c06b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:11.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 >> v2:172.21.15.149:3300/0 conn(0x7f574c0d0d60 msgr2=0x7f574c0d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 --2- 172.21.15.111:0/3302738428 >> v2:172.21.15.149:3300/0 conn(0x7f574c0d0d60 0x7f574c0d11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c0cd8f0 con 0x7f574c0cd460 2023-11-28T13:03:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f5738001670 con 0x7f574c0cd460 2023-11-28T13:03:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 >> v1:172.21.15.111:6789/0 conn(0x7f574c0cd460 legacy=0x7f574c06beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 --2- 172.21.15.111:0/3302738428 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f574c0b8510 0x7f573c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 --2- 172.21.15.111:0/3302738428 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f573c003fb0 0x7f573c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 --2- 172.21.15.111:0/3302738428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f573c004d90 0x7f573c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0cdbd0 con 0x7f573c004d90 2023-11-28T13:03:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0d02d0 con 0x7f573c003fb0 2023-11-28T13:03:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c00ba10 con 0x7f574c0b8510 2023-11-28T13:03:11.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f575140d700 1 --2- 172.21.15.111:0/3302738428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f573c004d90 0x7f573c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:11.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f574bfff700 1 --2- 172.21.15.111:0/3302738428 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f573c003fb0 0x7f573c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:11.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f5750c0c700 1 --2- 172.21.15.111:0/3302738428 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f574c0b8510 0x7f573c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:11.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f575140d700 1 -- 172.21.15.111:0/3302738428 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f574c0b8510 msgr2=0x7f573c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:11.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.353+0000 7f575140d700 1 --2- 172.21.15.111:0/3302738428 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f574c0b8510 0x7f573c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:11.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f575140d700 1 -- 172.21.15.111:0/3302738428 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f573c003fb0 msgr2=0x7f573c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:11.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f575140d700 1 --2- 172.21.15.111:0/3302738428 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f573c003fb0 0x7f573c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:11.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f575140d700 1 -- 172.21.15.111:0/3302738428 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c0035b0 con 0x7f573c004d90 2023-11-28T13:03:11.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f575140d700 1 --2- 172.21.15.111:0/3302738428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f573c004d90 0x7f573c005200 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5744009fc0 tx=0x7f573c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a20aa9524bde74c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574400ec30 con 0x7f573c004d90 2023-11-28T13:03:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574400f040 con 0x7f573c004d90 2023-11-28T13:03:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f574b7fe700 1 -- 172.21.15.111:0/3302738428 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744014480 con 0x7f573c004d90 2023-11-28T13:03:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f5752e70700 1 -- 172.21.15.111:0/3302738428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f573c004d90 msgr2=0x7f573c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f5752e70700 1 --2- 172.21.15.111:0/3302738428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f573c004d90 0x7f573c005200 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5744009fc0 tx=0x7f573c005880 comp rx=0 tx=0).stop 2023-11-28T13:03:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f575140d700 1 -- 172.21.15.111:0/3302738428 reap_dead start 2023-11-28T13:03:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f5752e70700 1 -- 172.21.15.111:0/3302738428 shutdown_connections 2023-11-28T13:03:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f5752e70700 1 -- 172.21.15.111:0/3302738428 >> 172.21.15.111:0/3302738428 conn(0x7f574c0719a0 msgr2=0x7f574c071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f5752e70700 1 -- 172.21.15.111:0/3302738428 shutdown_connections 2023-11-28T13:03:11.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.354+0000 7f5752e70700 1 -- 172.21.15.111:0/3302738428 wait complete. 2023-11-28T13:03:11.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.356+0000 7f5752e70700 1 Processor -- start 2023-11-28T13:03:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.356+0000 7f5752e70700 1 -- start start 2023-11-28T13:03:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.356+0000 7f5752e70700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f574c0b3660 0x7f574c0ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.356+0000 7f5752e70700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f574c0b3a30 0x7f574c1587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.356+0000 7f5752e70700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f574c0c8870 0x7f574c0c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.356+0000 7f5752e70700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0c9250 con 0x7f574c0b3a30 2023-11-28T13:03:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.356+0000 7f5752e70700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c159ac0 con 0x7f574c0b3660 2023-11-28T13:03:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.356+0000 7f5752e70700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574c158d10 con 0x7f574c0c8870 2023-11-28T13:03:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f574bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f574c0b3a30 0x7f574c1587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f574bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f574c0b3a30 0x7f574c1587d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37814/0 (socket says 172.21.15.111:37814) 2023-11-28T13:03:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f574bfff700 1 -- 172.21.15.111:0/3180907801 learned_addr learned my addr 172.21.15.111:0/3180907801 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:11.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f5750c0c700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f574c0b3660 0x7f574c0ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f575140d700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f574c0c8870 0x7f574c0c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f574bfff700 1 -- 172.21.15.111:0/3180907801 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f574c0c8870 msgr2=0x7f574c0c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f574bfff700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f574c0c8870 0x7f574c0c8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f574bfff700 1 -- 172.21.15.111:0/3180907801 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f574c0b3660 msgr2=0x7f574c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f574bfff700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f574c0b3660 0x7f574c0ca0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f574bfff700 1 -- 172.21.15.111:0/3180907801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c0cfff0 con 0x7f574c0b3a30 2023-11-28T13:03:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f574bfff700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f574c0b3a30 0x7f574c1587d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f57380072c0 tx=0x7f5738007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f57497fa700 1 -- 172.21.15.111:0/3180907801 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738001450 con 0x7f574c0b3a30 2023-11-28T13:03:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f57497fa700 1 -- 172.21.15.111:0/3180907801 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573800f850 con 0x7f574c0b3a30 2023-11-28T13:03:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f5752e70700 1 -- 172.21.15.111:0/3180907801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574c0cfd10 con 0x7f574c0b3a30 2023-11-28T13:03:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f57497fa700 1 -- 172.21.15.111:0/3180907801 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573800fc00 con 0x7f574c0b3a30 2023-11-28T13:03:11.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.357+0000 7f5752e70700 1 -- 172.21.15.111:0/3180907801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574c0d29c0 con 0x7f574c0b3a30 2023-11-28T13:03:11.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.358+0000 7f5752e70700 1 -- 172.21.15.111:0/3180907801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574c067880 con 0x7f574c0b3a30 2023-11-28T13:03:11.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.359+0000 7f57497fa700 1 -- 172.21.15.111:0/3180907801 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f573801c030 con 0x7f574c0b3a30 2023-11-28T13:03:11.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.359+0000 7f57497fa700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f57300682f0 0x7f573006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:11.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.359+0000 7f5750c0c700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f57300682f0 0x7f573006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:11.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.359+0000 7f57497fa700 1 -- 172.21.15.111:0/3180907801 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f5738085860 con 0x7f574c0b3a30 2023-11-28T13:03:11.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.360+0000 7f5750c0c700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f57300682f0 0x7f573006a7a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5740009aa0 tx=0x7f5740008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:11.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.363+0000 7f57497fa700 1 -- 172.21.15.111:0/3180907801 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5738051f40 con 0x7f574c0b3a30 2023-11-28T13:03:11.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.586+0000 7f5752e70700 1 -- 172.21.15.111:0/3180907801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f574c15d450 con 0x7f574c0b3a30 2023-11-28T13:03:11.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.586+0000 7f57497fa700 1 -- 172.21.15.111:0/3180907801 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f5738055560 con 0x7f574c0b3a30 2023-11-28T13:03:11.587 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:03:11.587 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:03:11.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.588+0000 7f5752e70700 1 -- 172.21.15.111:0/3180907801 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f57300682f0 msgr2=0x7f573006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:11.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.588+0000 7f5752e70700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f57300682f0 0x7f573006a7a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5740009aa0 tx=0x7f5740008040 comp rx=0 tx=0).stop 2023-11-28T13:03:11.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.588+0000 7f5752e70700 1 -- 172.21.15.111:0/3180907801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f574c0b3a30 msgr2=0x7f574c1587d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:11.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.588+0000 7f5752e70700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f574c0b3a30 0x7f574c1587d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f57380072c0 tx=0x7f5738007680 comp rx=0 tx=0).stop 2023-11-28T13:03:11.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.588+0000 7f5752e70700 1 -- 172.21.15.111:0/3180907801 shutdown_connections 2023-11-28T13:03:11.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.588+0000 7f5752e70700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f574c0c8870 0x7f574c0c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:11.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.588+0000 7f5752e70700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f57300682f0 0x7f573006a7a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:11.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.588+0000 7f5752e70700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f574c0b3660 0x7f574c0ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:11.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.588+0000 7f5752e70700 1 --2- 172.21.15.111:0/3180907801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f574c0b3a30 0x7f574c1587d0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:11.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.588+0000 7f5752e70700 1 -- 172.21.15.111:0/3180907801 >> 172.21.15.111:0/3180907801 conn(0x7f574c0719a0 msgr2=0x7f574c0b77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:11.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.588+0000 7f5752e70700 1 -- 172.21.15.111:0/3180907801 shutdown_connections 2023-11-28T13:03:11.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:11.588+0000 7f5752e70700 1 -- 172.21.15.111:0/3180907801 wait complete. 2023-11-28T13:03:11.601 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:03:16.602 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:03:16.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.734+0000 7f97a4bd0700 1 Processor -- start 2023-11-28T13:03:16.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.735+0000 7f97a4bd0700 1 -- start start 2023-11-28T13:03:16.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.735+0000 7f97a4bd0700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f97a015c7b0 0x7f97a00cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:16.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.735+0000 7f97a4bd0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97a006b380 0x7f97a006b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:16.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.735+0000 7f97a4bd0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a00cb810 con 0x7f97a00d0490 2023-11-28T13:03:16.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.735+0000 7f97a4bd0700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a006bd40 con 0x7f97a015c7b0 2023-11-28T13:03:16.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.735+0000 7f97a4bd0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a00cc650 con 0x7f97a006b380 2023-11-28T13:03:16.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979dd9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f97a00d0490 0x7f97a00ad8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38598/0 (socket says 172.21.15.111:38598) 2023-11-28T13:03:16.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979dd9b700 1 -- 172.21.15.111:0/557934813 learned_addr learned my addr 172.21.15.111:0/557934813 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:03:16.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979ed9d700 1 --2- 172.21.15.111:0/557934813 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97a006b380 0x7f97a006b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:16.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979e59c700 1 --2- 172.21.15.111:0/557934813 >> v2:172.21.15.149:3300/0 conn(0x7f97a015c7b0 0x7f97a00cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:16.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045838841 0 0) 0x7f97a00cb810 con 0x7f97a00d0490 2023-11-28T13:03:16.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f978c0035b0 con 0x7f97a00d0490 2023-11-28T13:03:16.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 36060751 0 0) 0x7f978c0035b0 con 0x7f97a00d0490 2023-11-28T13:03:16.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97a006b380 msgr2=0x7f97a006b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:16.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 --2- 172.21.15.111:0/557934813 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97a006b380 0x7f97a006b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:16.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 >> v2:172.21.15.149:3300/0 conn(0x7f97a015c7b0 msgr2=0x7f97a00cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:16.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 --2- 172.21.15.111:0/557934813 >> v2:172.21.15.149:3300/0 conn(0x7f97a015c7b0 0x7f97a00cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:16.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a00cba40 con 0x7f97a00d0490 2023-11-28T13:03:16.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f9790001670 con 0x7f97a00d0490 2023-11-28T13:03:16.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 >> v1:172.21.15.111:6789/0 conn(0x7f97a00d0490 legacy=0x7f97a00ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:16.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 --2- 172.21.15.111:0/557934813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f978c003a00 0x7f978c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:16.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 --2- 172.21.15.111:0/557934813 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f978c004430 0x7f978c0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:16.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 --2- 172.21.15.111:0/557934813 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f978c004fb0 0x7f978c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:16.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a006bd40 con 0x7f978c003a00 2023-11-28T13:03:16.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a00cc650 con 0x7f978c004fb0 2023-11-28T13:03:16.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c00bf50 con 0x7f978c004430 2023-11-28T13:03:16.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979ed9d700 1 --2- 172.21.15.111:0/557934813 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97a006b380 0x7f97a006b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:03:16.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979e59c700 1 --2- 172.21.15.111:0/557934813 >> v2:172.21.15.149:3300/0 conn(0x7f97a015c7b0 0x7f97a00cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:03:16.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979dd9b700 1 --2- 172.21.15.111:0/557934813 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f978c004430 0x7f978c0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:16.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.736+0000 7f979e59c700 1 --2- 172.21.15.111:0/557934813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f978c003a00 0x7f978c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:16.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.737+0000 7f979ed9d700 1 --2- 172.21.15.111:0/557934813 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f978c004fb0 0x7f978c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:16.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.737+0000 7f979e59c700 1 -- 172.21.15.111:0/557934813 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f978c004430 msgr2=0x7f978c0048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:16.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.737+0000 7f979e59c700 1 --2- 172.21.15.111:0/557934813 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f978c004430 0x7f978c0048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:16.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.737+0000 7f979e59c700 1 -- 172.21.15.111:0/557934813 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f978c004fb0 msgr2=0x7f978c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:16.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.737+0000 7f979e59c700 1 --2- 172.21.15.111:0/557934813 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f978c004fb0 0x7f978c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:16.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.737+0000 7f979e59c700 1 -- 172.21.15.111:0/557934813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c0035b0 con 0x7f978c003a00 2023-11-28T13:03:16.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.737+0000 7f979e59c700 1 --2- 172.21.15.111:0/557934813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f978c003a00 0x7f978c003e10 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9788009cf0 tx=0x7f978c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a5088f0915b6622 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:16.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.737+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9788004030 con 0x7f978c003a00 2023-11-28T13:03:16.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.737+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9788004190 con 0x7f978c003a00 2023-11-28T13:03:16.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.737+0000 7f979d59a700 1 -- 172.21.15.111:0/557934813 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9788004500 con 0x7f978c003a00 2023-11-28T13:03:16.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.738+0000 7f97a4bd0700 1 -- 172.21.15.111:0/557934813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f978c003a00 msgr2=0x7f978c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:16.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.738+0000 7f97a4bd0700 1 --2- 172.21.15.111:0/557934813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f978c003a00 0x7f978c003e10 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9788009cf0 tx=0x7f978c005880 comp rx=0 tx=0).stop 2023-11-28T13:03:16.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.738+0000 7f979ed9d700 1 -- 172.21.15.111:0/557934813 reap_dead start 2023-11-28T13:03:16.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.738+0000 7f97a4bd0700 1 -- 172.21.15.111:0/557934813 shutdown_connections 2023-11-28T13:03:16.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.738+0000 7f97a4bd0700 1 -- 172.21.15.111:0/557934813 >> 172.21.15.111:0/557934813 conn(0x7f97a009a040 msgr2=0x7f97a00d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:16.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.738+0000 7f97a4bd0700 1 -- 172.21.15.111:0/557934813 shutdown_connections 2023-11-28T13:03:16.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.738+0000 7f97a4bd0700 1 -- 172.21.15.111:0/557934813 wait complete. 2023-11-28T13:03:16.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.739+0000 7f97a4bd0700 1 Processor -- start 2023-11-28T13:03:16.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.739+0000 7f97a4bd0700 1 -- start start 2023-11-28T13:03:16.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f97a4bd0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f978c003a00 0x7f97a00b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:16.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f97a4bd0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97a00b3fb0 0x7f97a01575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:16.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f97a4bd0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97a00ca440 0x7f97a00ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:16.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f97a4bd0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a00b51d0 con 0x7f97a00b3fb0 2023-11-28T13:03:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f97a4bd0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a00cadf0 con 0x7f97a00ca440 2023-11-28T13:03:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f97a4bd0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a00b3560 con 0x7f978c003a00 2023-11-28T13:03:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f979dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97a00b3fb0 0x7f97a01575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f979dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97a00b3fb0 0x7f97a01575a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37834/0 (socket says 172.21.15.111:37834) 2023-11-28T13:03:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f979dd9b700 1 -- 172.21.15.111:0/3103796781 learned_addr learned my addr 172.21.15.111:0/3103796781 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f979e59c700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f978c003a00 0x7f97a00b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f979ed9d700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97a00ca440 0x7f97a00ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f979dd9b700 1 -- 172.21.15.111:0/3103796781 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f978c003a00 msgr2=0x7f97a00b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f979dd9b700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f978c003a00 0x7f97a00b3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f979dd9b700 1 -- 172.21.15.111:0/3103796781 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97a00ca440 msgr2=0x7f97a00ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f979dd9b700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97a00ca440 0x7f97a00ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f979dd9b700 1 -- 172.21.15.111:0/3103796781 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a00cc000 con 0x7f97a00b3fb0 2023-11-28T13:03:16.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f979dd9b700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97a00b3fb0 0x7f97a01575a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f97900073a0 tx=0x7f97900076b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:16.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.740+0000 7f9796ffd700 1 -- 172.21.15.111:0/3103796781 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979000f6c0 con 0x7f97a00b3fb0 2023-11-28T13:03:16.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.741+0000 7f9796ffd700 1 -- 172.21.15.111:0/3103796781 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979000f820 con 0x7f97a00b3fb0 2023-11-28T13:03:16.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.741+0000 7f97a4bd0700 1 -- 172.21.15.111:0/3103796781 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a00cbd20 con 0x7f97a00b3fb0 2023-11-28T13:03:16.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.741+0000 7f9796ffd700 1 -- 172.21.15.111:0/3103796781 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979000fbd0 con 0x7f97a00b3fb0 2023-11-28T13:03:16.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.741+0000 7f97a4bd0700 1 -- 172.21.15.111:0/3103796781 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a00cc2e0 con 0x7f97a00b3fb0 2023-11-28T13:03:16.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.741+0000 7f97a4bd0700 1 -- 172.21.15.111:0/3103796781 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a0067170 con 0x7f97a00b3fb0 2023-11-28T13:03:16.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.742+0000 7f9796ffd700 1 -- 172.21.15.111:0/3103796781 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f979001c030 con 0x7f97a00b3fb0 2023-11-28T13:03:16.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.742+0000 7f9796ffd700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f97800682f0 0x7f978006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:16.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.742+0000 7f9796ffd700 1 -- 172.21.15.111:0/3103796781 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f9790085690 con 0x7f97a00b3fb0 2023-11-28T13:03:16.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.742+0000 7f979e59c700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f97800682f0 0x7f978006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:16.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.743+0000 7f979e59c700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f97800682f0 0x7f978006a7a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9788009c10 tx=0x7f9788008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:16.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.747+0000 7f9796ffd700 1 -- 172.21.15.111:0/3103796781 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9790051d70 con 0x7f97a00b3fb0 2023-11-28T13:03:16.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.975+0000 7f97a4bd0700 1 -- 172.21.15.111:0/3103796781 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f97a00caf70 con 0x7f97a00b3fb0 2023-11-28T13:03:16.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.976+0000 7f9796ffd700 1 -- 172.21.15.111:0/3103796781 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f9790055390 con 0x7f97a00b3fb0 2023-11-28T13:03:16.977 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:03:16.977 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:03:16.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.978+0000 7f97a4bd0700 1 -- 172.21.15.111:0/3103796781 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f97800682f0 msgr2=0x7f978006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:16.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.978+0000 7f97a4bd0700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f97800682f0 0x7f978006a7a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9788009c10 tx=0x7f9788008030 comp rx=0 tx=0).stop 2023-11-28T13:03:16.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.978+0000 7f97a4bd0700 1 -- 172.21.15.111:0/3103796781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97a00b3fb0 msgr2=0x7f97a01575a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:16.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.978+0000 7f97a4bd0700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97a00b3fb0 0x7f97a01575a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f97900073a0 tx=0x7f97900076b0 comp rx=0 tx=0).stop 2023-11-28T13:03:16.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.978+0000 7f97a4bd0700 1 -- 172.21.15.111:0/3103796781 shutdown_connections 2023-11-28T13:03:16.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.978+0000 7f97a4bd0700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f97800682f0 0x7f978006a7a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:16.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.978+0000 7f97a4bd0700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97a00ca440 0x7f97a00ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:16.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.978+0000 7f97a4bd0700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f978c003a00 0x7f97a00b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:16.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.978+0000 7f97a4bd0700 1 --2- 172.21.15.111:0/3103796781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97a00b3fb0 0x7f97a01575a0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:16.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.978+0000 7f97a4bd0700 1 -- 172.21.15.111:0/3103796781 >> 172.21.15.111:0/3103796781 conn(0x7f97a009a040 msgr2=0x7f97a0070090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:16.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.978+0000 7f97a4bd0700 1 -- 172.21.15.111:0/3103796781 shutdown_connections 2023-11-28T13:03:16.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:16.978+0000 7f97a4bd0700 1 -- 172.21.15.111:0/3103796781 wait complete. 2023-11-28T13:03:16.990 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:03:21.991 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:03:22.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.122+0000 7f402c8b4700 1 Processor -- start 2023-11-28T13:03:22.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.122+0000 7f402c8b4700 1 -- start start 2023-11-28T13:03:22.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.123+0000 7f402c8b4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f40280d2df0 0x7f40280cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:22.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.123+0000 7f402c8b4700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f40280b4fc0 0x7f40280d31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:22.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.123+0000 7f402c8b4700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40280d1950 con 0x7f40280b1c60 2023-11-28T13:03:22.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.123+0000 7f402c8b4700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40280d3d30 con 0x7f40280d2df0 2023-11-28T13:03:22.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.123+0000 7f402c8b4700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40280b2500 con 0x7f40280b4fc0 2023-11-28T13:03:22.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.123+0000 7f402659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f40280d2df0 0x7f40280cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:22.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.123+0000 7f402659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f40280d2df0 0x7f40280cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37840/0 (socket says 172.21.15.111:37840) 2023-11-28T13:03:22.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.123+0000 7f402659c700 1 -- 172.21.15.111:0/2768673820 learned_addr learned my addr 172.21.15.111:0/2768673820 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:22.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.123+0000 7f4025d9b700 1 --2- 172.21.15.111:0/2768673820 >> v2:172.21.15.149:3300/0 conn(0x7f40280b4fc0 0x7f40280d31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:22.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.123+0000 7f402659c700 1 -- 172.21.15.111:0/2768673820 >> v2:172.21.15.149:3300/0 conn(0x7f40280b4fc0 msgr2=0x7f40280d31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:22.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.123+0000 7f402659c700 1 --2- 172.21.15.111:0/2768673820 >> v2:172.21.15.149:3300/0 conn(0x7f40280b4fc0 0x7f40280d31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:22.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.123+0000 7f402659c700 1 -- 172.21.15.111:0/2768673820 >> v1:172.21.15.149:6789/0 conn(0x7f40280b1c60 legacy=0x7f40280cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:22.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.123+0000 7f402659c700 1 -- 172.21.15.111:0/2768673820 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40280d3a50 con 0x7f40280d2df0 2023-11-28T13:03:22.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.124+0000 7f402659c700 1 --2- 172.21.15.111:0/2768673820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40280d2df0 0x7f40280cd620 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f40281601b0 tx=0x7f401c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de695cb234e797ba server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:22.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.124+0000 7f402559a700 1 -- 172.21.15.111:0/2768673820 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c004230 con 0x7f40280d2df0 2023-11-28T13:03:22.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.124+0000 7f402559a700 1 -- 172.21.15.111:0/2768673820 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401c004390 con 0x7f40280d2df0 2023-11-28T13:03:22.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.124+0000 7f402559a700 1 -- 172.21.15.111:0/2768673820 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c0036a0 con 0x7f40280d2df0 2023-11-28T13:03:22.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.124+0000 7f402c8b4700 1 -- 172.21.15.111:0/2768673820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40280d2df0 msgr2=0x7f40280cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:22.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.124+0000 7f402c8b4700 1 --2- 172.21.15.111:0/2768673820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40280d2df0 0x7f40280cd620 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f40281601b0 tx=0x7f401c009bd0 comp rx=0 tx=0).stop 2023-11-28T13:03:22.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.124+0000 7f402c8b4700 1 -- 172.21.15.111:0/2768673820 shutdown_connections 2023-11-28T13:03:22.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.124+0000 7f402c8b4700 1 --2- 172.21.15.111:0/2768673820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40280d2df0 0x7f40280cd620 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:22.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.124+0000 7f402c8b4700 1 --2- 172.21.15.111:0/2768673820 >> v2:172.21.15.149:3300/0 conn(0x7f40280b4fc0 0x7f40280d31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:22.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.124+0000 7f402c8b4700 1 -- 172.21.15.111:0/2768673820 >> 172.21.15.111:0/2768673820 conn(0x7f40280719a0 msgr2=0x7f4028071da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:22.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.124+0000 7f402c8b4700 1 -- 172.21.15.111:0/2768673820 shutdown_connections 2023-11-28T13:03:22.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.124+0000 7f402c8b4700 1 -- 172.21.15.111:0/2768673820 wait complete. 2023-11-28T13:03:22.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.126+0000 7f402c8b4700 1 Processor -- start 2023-11-28T13:03:22.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.126+0000 7f402c8b4700 1 -- start start 2023-11-28T13:03:22.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f402c8b4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f40280b1c60 0x7f40280ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:22.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f402c8b4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40280b4fc0 0x7f4028154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:22.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f402c8b4700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40280d2df0 0x7f402815ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:22.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f402c8b4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4028155b00 con 0x7f40280b4fc0 2023-11-28T13:03:22.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f402c8b4700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40280cbac0 con 0x7f40280d2df0 2023-11-28T13:03:22.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f402c8b4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40280cb220 con 0x7f40280b1c60 2023-11-28T13:03:22.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4025d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40280b4fc0 0x7f4028154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:22.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4025d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40280b4fc0 0x7f4028154410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37850/0 (socket says 172.21.15.111:37850) 2023-11-28T13:03:22.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4025d9b700 1 -- 172.21.15.111:0/2749684242 learned_addr learned my addr 172.21.15.111:0/2749684242 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:22.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4026d9d700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40280d2df0 0x7f402815ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:22.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f402659c700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f40280b1c60 0x7f40280ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:22.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4025d9b700 1 -- 172.21.15.111:0/2749684242 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f40280b1c60 msgr2=0x7f40280ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:22.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4025d9b700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f40280b1c60 0x7f40280ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:22.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4025d9b700 1 -- 172.21.15.111:0/2749684242 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40280d2df0 msgr2=0x7f402815ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:22.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4025d9b700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40280d2df0 0x7f402815ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:22.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4025d9b700 1 -- 172.21.15.111:0/2749684242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c009550 con 0x7f40280b4fc0 2023-11-28T13:03:22.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4026d9d700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40280d2df0 0x7f402815ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:03:22.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f402659c700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f40280b1c60 0x7f40280ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:03:22.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4025d9b700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40280b4fc0 0x7f4028154410 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f401800d800 tx=0x7f401800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:22.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4016ffd700 1 -- 172.21.15.111:0/2749684242 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40180125f0 con 0x7f40280b4fc0 2023-11-28T13:03:22.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4016ffd700 1 -- 172.21.15.111:0/2749684242 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4018012750 con 0x7f40280b4fc0 2023-11-28T13:03:22.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f402c8b4700 1 -- 172.21.15.111:0/2749684242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40280b88f0 con 0x7f40280b4fc0 2023-11-28T13:03:22.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f4016ffd700 1 -- 172.21.15.111:0/2749684242 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4018012b00 con 0x7f40280b4fc0 2023-11-28T13:03:22.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.127+0000 7f402c8b4700 1 -- 172.21.15.111:0/2749684242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40280d15e0 con 0x7f40280b4fc0 2023-11-28T13:03:22.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.128+0000 7f402c8b4700 1 -- 172.21.15.111:0/2749684242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40280b21b0 con 0x7f40280b4fc0 2023-11-28T13:03:22.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.129+0000 7f4016ffd700 1 -- 172.21.15.111:0/2749684242 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4018012c60 con 0x7f40280b4fc0 2023-11-28T13:03:22.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.129+0000 7f4016ffd700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f40080682f0 0x7f400806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:22.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.129+0000 7f4016ffd700 1 -- 172.21.15.111:0/2749684242 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f4018086390 con 0x7f40280b4fc0 2023-11-28T13:03:22.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.129+0000 7f402659c700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f40080682f0 0x7f400806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:22.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.130+0000 7f402659c700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f40080682f0 0x7f400806a7a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f401c009c10 tx=0x7f401c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:22.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.134+0000 7f4016ffd700 1 -- 172.21.15.111:0/2749684242 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4018052a70 con 0x7f40280b4fc0 2023-11-28T13:03:22.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.361+0000 7f402c8b4700 1 -- 172.21.15.111:0/2749684242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4028067170 con 0x7f40280b4fc0 2023-11-28T13:03:22.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.362+0000 7f4016ffd700 1 -- 172.21.15.111:0/2749684242 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f4018056090 con 0x7f40280b4fc0 2023-11-28T13:03:22.362 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:03:22.362 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:03:22.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.364+0000 7f402c8b4700 1 -- 172.21.15.111:0/2749684242 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f40080682f0 msgr2=0x7f400806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:22.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.364+0000 7f402c8b4700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f40080682f0 0x7f400806a7a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f401c009c10 tx=0x7f401c008030 comp rx=0 tx=0).stop 2023-11-28T13:03:22.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.364+0000 7f402c8b4700 1 -- 172.21.15.111:0/2749684242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40280b4fc0 msgr2=0x7f4028154410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:22.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.364+0000 7f402c8b4700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40280b4fc0 0x7f4028154410 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f401800d800 tx=0x7f401800dbc0 comp rx=0 tx=0).stop 2023-11-28T13:03:22.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.364+0000 7f402c8b4700 1 -- 172.21.15.111:0/2749684242 shutdown_connections 2023-11-28T13:03:22.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.364+0000 7f402c8b4700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f40080682f0 0x7f400806a7a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:22.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.364+0000 7f402c8b4700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f40280d2df0 0x7f402815ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:22.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.364+0000 7f402c8b4700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f40280b1c60 0x7f40280ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:22.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.364+0000 7f402c8b4700 1 --2- 172.21.15.111:0/2749684242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40280b4fc0 0x7f4028154410 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:22.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.364+0000 7f402c8b4700 1 -- 172.21.15.111:0/2749684242 >> 172.21.15.111:0/2749684242 conn(0x7f40280719a0 msgr2=0x7f40280b7540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:22.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.364+0000 7f402c8b4700 1 -- 172.21.15.111:0/2749684242 shutdown_connections 2023-11-28T13:03:22.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:22.364+0000 7f402c8b4700 1 -- 172.21.15.111:0/2749684242 wait complete. 2023-11-28T13:03:22.376 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:03:27.377 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:03:27.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.506+0000 7efdca921700 1 Processor -- start 2023-11-28T13:03:27.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.507+0000 7efdca921700 1 -- start start 2023-11-28T13:03:27.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.507+0000 7efdca921700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7efdc415c800 0x7efdc4160e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:27.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.507+0000 7efdca921700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdc406bfd0 0x7efdc406a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:27.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.507+0000 7efdca921700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc415cbd0 con 0x7efdc40cd790 2023-11-28T13:03:27.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.507+0000 7efdca921700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc406af30 con 0x7efdc415c800 2023-11-28T13:03:27.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.507+0000 7efdca921700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc406b210 con 0x7efdc406bfd0 2023-11-28T13:03:27.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc8ebe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efdc40cd790 0x7efdc40b29f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38636/0 (socket says 172.21.15.111:38636) 2023-11-28T13:03:27.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc8ebe700 1 -- 172.21.15.111:0/3477804650 learned_addr learned my addr 172.21.15.111:0/3477804650 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:03:27.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc3fff700 1 --2- 172.21.15.111:0/3477804650 >> v2:172.21.15.149:3300/0 conn(0x7efdc415c800 0x7efdc4160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:27.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc37fe700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdc406bfd0 0x7efdc406a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:27.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413382600 0 0) 0x7efdc415cbd0 con 0x7efdc40cd790 2023-11-28T13:03:27.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdac0035b0 con 0x7efdc40cd790 2023-11-28T13:03:27.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1002325955 0 0) 0x7efdac0035b0 con 0x7efdc40cd790 2023-11-28T13:03:27.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdc406bfd0 msgr2=0x7efdc406a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:27.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdc406bfd0 0x7efdc406a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:27.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 >> v2:172.21.15.149:3300/0 conn(0x7efdc415c800 msgr2=0x7efdc4160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:27.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 --2- 172.21.15.111:0/3477804650 >> v2:172.21.15.149:3300/0 conn(0x7efdc415c800 0x7efdc4160e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:27.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc406b4f0 con 0x7efdc40cd790 2023-11-28T13:03:27.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7efdbc0019f0 con 0x7efdc40cd790 2023-11-28T13:03:27.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 >> v1:172.21.15.111:6789/0 conn(0x7efdc40cd790 legacy=0x7efdc40b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:27.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efdac003a00 0x7efdac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:27.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdac004430 0x7efdac0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:27.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdac004f10 0x7efdac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:27.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc406af30 con 0x7efdac004f10 2023-11-28T13:03:27.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc406b210 con 0x7efdac003a00 2023-11-28T13:03:27.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdac00bf50 con 0x7efdac004430 2023-11-28T13:03:27.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc37fe700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdc406bfd0 0x7efdc406a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:03:27.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc3fff700 1 --2- 172.21.15.111:0/3477804650 >> v2:172.21.15.149:3300/0 conn(0x7efdc415c800 0x7efdc4160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:03:27.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc8ebe700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdac004f10 0x7efdac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:27.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc3fff700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efdac003a00 0x7efdac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:27.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.508+0000 7efdc37fe700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdac004430 0x7efdac0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:27.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdc8ebe700 1 -- 172.21.15.111:0/3477804650 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdac004430 msgr2=0x7efdac0048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:27.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdc8ebe700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdac004430 0x7efdac0048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:27.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdc8ebe700 1 -- 172.21.15.111:0/3477804650 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efdac003a00 msgr2=0x7efdac003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:27.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdc8ebe700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efdac003a00 0x7efdac003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdc8ebe700 1 -- 172.21.15.111:0/3477804650 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdac0035b0 con 0x7efdac004f10 2023-11-28T13:03:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdc37fe700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdac004430 0x7efdac0048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:03:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdc8ebe700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdac004f10 0x7efdac00ba10 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7efdbc00a860 tx=0x7efdac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e22ecc0753f414c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc012600 con 0x7efdac004f10 2023-11-28T13:03:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdbc012780 con 0x7efdac004f10 2023-11-28T13:03:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdc2ffd700 1 -- 172.21.15.111:0/3477804650 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc012b30 con 0x7efdac004f10 2023-11-28T13:03:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdca921700 1 -- 172.21.15.111:0/3477804650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdac004f10 msgr2=0x7efdac00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdca921700 1 --2- 172.21.15.111:0/3477804650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdac004f10 0x7efdac00ba10 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7efdbc00a860 tx=0x7efdac005880 comp rx=0 tx=0).stop 2023-11-28T13:03:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdc8ebe700 1 -- 172.21.15.111:0/3477804650 reap_dead start 2023-11-28T13:03:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdca921700 1 -- 172.21.15.111:0/3477804650 shutdown_connections 2023-11-28T13:03:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdca921700 1 -- 172.21.15.111:0/3477804650 >> 172.21.15.111:0/3477804650 conn(0x7efdc406db10 msgr2=0x7efdc406df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.509+0000 7efdca921700 1 -- 172.21.15.111:0/3477804650 shutdown_connections 2023-11-28T13:03:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.510+0000 7efdca921700 1 -- 172.21.15.111:0/3477804650 wait complete. 2023-11-28T13:03:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.511+0000 7efdca921700 1 Processor -- start 2023-11-28T13:03:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.511+0000 7efdca921700 1 -- start start 2023-11-28T13:03:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdca921700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdc40ad000 0x7efdc415f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdca921700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc40ad3d0 0x7efdc40cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdca921700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efdc40cc2f0 0x7efdc40c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdca921700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc415ef30 con 0x7efdc40ad3d0 2023-11-28T13:03:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdca921700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc40cc790 con 0x7efdc40cc2f0 2023-11-28T13:03:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdca921700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc415e5d0 con 0x7efdc40ad000 2023-11-28T13:03:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc40ad3d0 0x7efdc40cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc40ad3d0 0x7efdc40cbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37868/0 (socket says 172.21.15.111:37868) 2023-11-28T13:03:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc37fe700 1 -- 172.21.15.111:0/1729577987 learned_addr learned my addr 172.21.15.111:0/1729577987 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc3fff700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdc40ad000 0x7efdc415f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc8ebe700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efdc40cc2f0 0x7efdc40c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc37fe700 1 -- 172.21.15.111:0/1729577987 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdc40ad000 msgr2=0x7efdc415f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc37fe700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdc40ad000 0x7efdc415f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc37fe700 1 -- 172.21.15.111:0/1729577987 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efdc40cc2f0 msgr2=0x7efdc40c37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc37fe700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efdc40cc2f0 0x7efdc40c37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc37fe700 1 -- 172.21.15.111:0/1729577987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc406bab0 con 0x7efdc40ad3d0 2023-11-28T13:03:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc3fff700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdc40ad000 0x7efdc415f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:03:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc8ebe700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efdc40cc2f0 0x7efdc40c37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:03:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc37fe700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc40ad3d0 0x7efdc40cbdb0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7efdb80118e0 tx=0x7efdb8011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc0ff9700 1 -- 172.21.15.111:0/1729577987 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb800fae0 con 0x7efdc40ad3d0 2023-11-28T13:03:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc0ff9700 1 -- 172.21.15.111:0/1729577987 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb800fc40 con 0x7efdc40ad3d0 2023-11-28T13:03:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdca921700 1 -- 172.21.15.111:0/1729577987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc406b7d0 con 0x7efdc40ad3d0 2023-11-28T13:03:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdc0ff9700 1 -- 172.21.15.111:0/1729577987 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb8011e80 con 0x7efdc40ad3d0 2023-11-28T13:03:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.512+0000 7efdca921700 1 -- 172.21.15.111:0/1729577987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc40cdc10 con 0x7efdc40ad3d0 2023-11-28T13:03:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.513+0000 7efdca921700 1 -- 172.21.15.111:0/1729577987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc415f070 con 0x7efdc40ad3d0 2023-11-28T13:03:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.514+0000 7efdc0ff9700 1 -- 172.21.15.111:0/1729577987 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efdb8012040 con 0x7efdc40ad3d0 2023-11-28T13:03:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.514+0000 7efdc0ff9700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efdb00682f0 0x7efdb006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.514+0000 7efdc0ff9700 1 -- 172.21.15.111:0/1729577987 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7efdb8086590 con 0x7efdc40ad3d0 2023-11-28T13:03:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.514+0000 7efdc3fff700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efdb00682f0 0x7efdb006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.515+0000 7efdc3fff700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efdb00682f0 0x7efdb006a7a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009df0 tx=0x7efdb4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:27.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.518+0000 7efdc0ff9700 1 -- 172.21.15.111:0/1729577987 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efdc415f070 con 0x7efdc40ad3d0 2023-11-28T13:03:27.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.744+0000 7efdca921700 1 -- 172.21.15.111:0/1729577987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efdc415f070 con 0x7efdc40ad3d0 2023-11-28T13:03:27.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.744+0000 7efdc0ff9700 1 -- 172.21.15.111:0/1729577987 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7efdc415f070 con 0x7efdc40ad3d0 2023-11-28T13:03:27.746 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:03:27.746 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:03:27.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.746+0000 7efdca921700 1 -- 172.21.15.111:0/1729577987 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efdb00682f0 msgr2=0x7efdb006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:27.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.746+0000 7efdca921700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efdb00682f0 0x7efdb006a7a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009df0 tx=0x7efdb4008040 comp rx=0 tx=0).stop 2023-11-28T13:03:27.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.746+0000 7efdca921700 1 -- 172.21.15.111:0/1729577987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc40ad3d0 msgr2=0x7efdc40cbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:27.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.746+0000 7efdca921700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc40ad3d0 0x7efdc40cbdb0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7efdb80118e0 tx=0x7efdb8011ca0 comp rx=0 tx=0).stop 2023-11-28T13:03:27.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.746+0000 7efdca921700 1 -- 172.21.15.111:0/1729577987 shutdown_connections 2023-11-28T13:03:27.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.746+0000 7efdca921700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efdb00682f0 0x7efdb006a7a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:27.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.746+0000 7efdca921700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efdc40cc2f0 0x7efdc40c37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:27.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.746+0000 7efdca921700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efdc40ad000 0x7efdc415f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:27.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.746+0000 7efdca921700 1 --2- 172.21.15.111:0/1729577987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdc40ad3d0 0x7efdc40cbdb0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:27.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.746+0000 7efdca921700 1 -- 172.21.15.111:0/1729577987 >> 172.21.15.111:0/1729577987 conn(0x7efdc406db10 msgr2=0x7efdc40b5180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:27.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.746+0000 7efdca921700 1 -- 172.21.15.111:0/1729577987 shutdown_connections 2023-11-28T13:03:27.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:27.746+0000 7efdca921700 1 -- 172.21.15.111:0/1729577987 wait complete. 2023-11-28T13:03:27.759 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:03:29.236 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:03:29.239 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:03:32.760 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:03:32.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.890+0000 7f1398e75700 1 Processor -- start 2023-11-28T13:03:32.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.890+0000 7f1398e75700 1 -- start start 2023-11-28T13:03:32.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.891+0000 7f1398e75700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f139415a5d0 0x7f13940ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:32.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.891+0000 7f1398e75700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f13940cb2d0 0x7f13940ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:32.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.891+0000 7f1398e75700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13940ae8b0 con 0x7f13940b0160 2023-11-28T13:03:32.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.891+0000 7f1398e75700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13940cc1b0 con 0x7f13940cb2d0 2023-11-28T13:03:32.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.891+0000 7f1398e75700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13940cc390 con 0x7f139415a5d0 2023-11-28T13:03:32.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.891+0000 7f1391d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f13940b0160 0x7f139415e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38652/0 (socket says 172.21.15.111:38652) 2023-11-28T13:03:32.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.891+0000 7f1391d9b700 1 -- 172.21.15.111:0/849062135 learned_addr learned my addr 172.21.15.111:0/849062135 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:03:32.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f1392d9d700 1 --2- 172.21.15.111:0/849062135 >> v2:172.21.15.149:3300/0 conn(0x7f13940cb2d0 0x7f13940ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:32.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139259c700 1 --2- 172.21.15.111:0/849062135 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f139415a5d0 0x7f13940ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:32.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22809318 0 0) 0x7f13940ae8b0 con 0x7f13940b0160 2023-11-28T13:03:32.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f137c0035b0 con 0x7f13940b0160 2023-11-28T13:03:32.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4128892074 0 0) 0x7f137c0035b0 con 0x7f13940b0160 2023-11-28T13:03:32.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f139415a5d0 msgr2=0x7f13940ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:32.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 --2- 172.21.15.111:0/849062135 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f139415a5d0 0x7f13940ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:32.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 >> v2:172.21.15.149:3300/0 conn(0x7f13940cb2d0 msgr2=0x7f13940ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:32.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 --2- 172.21.15.111:0/849062135 >> v2:172.21.15.149:3300/0 conn(0x7f13940cb2d0 0x7f13940ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:32.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13940cc670 con 0x7f13940b0160 2023-11-28T13:03:32.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f1384001670 con 0x7f13940b0160 2023-11-28T13:03:32.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 >> v1:172.21.15.111:6789/0 conn(0x7f13940b0160 legacy=0x7f139415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:32.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 --2- 172.21.15.111:0/849062135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f137c003a00 0x7f137c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:32.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 --2- 172.21.15.111:0/849062135 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f139415a5d0 0x7f137c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:32.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 --2- 172.21.15.111:0/849062135 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f137c004d90 0x7f137c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:32.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13940cc1b0 con 0x7f137c003a00 2023-11-28T13:03:32.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13940cc390 con 0x7f137c004d90 2023-11-28T13:03:32.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137c00ba10 con 0x7f139415a5d0 2023-11-28T13:03:32.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139259c700 1 --2- 172.21.15.111:0/849062135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f137c003a00 0x7f137c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:32.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f1391d9b700 1 --2- 172.21.15.111:0/849062135 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f139415a5d0 0x7f137c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:32.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f1392d9d700 1 --2- 172.21.15.111:0/849062135 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f137c004d90 0x7f137c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:32.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139259c700 1 -- 172.21.15.111:0/849062135 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f139415a5d0 msgr2=0x7f137c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:32.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139259c700 1 --2- 172.21.15.111:0/849062135 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f139415a5d0 0x7f137c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:32.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139259c700 1 -- 172.21.15.111:0/849062135 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f137c004d90 msgr2=0x7f137c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:32.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139259c700 1 --2- 172.21.15.111:0/849062135 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f137c004d90 0x7f137c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:32.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f139259c700 1 -- 172.21.15.111:0/849062135 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137c0035b0 con 0x7f137c003a00 2023-11-28T13:03:32.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.892+0000 7f1392d9d700 1 --2- 172.21.15.111:0/849062135 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f137c004d90 0x7f137c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:03:32.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.893+0000 7f139259c700 1 --2- 172.21.15.111:0/849062135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f137c003a00 0x7f137c003e10 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f1388009b40 tx=0x7f137c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac66bd4868bb5b3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:32.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.893+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388005a20 con 0x7f137c003a00 2023-11-28T13:03:32.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.893+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1388005b80 con 0x7f137c003a00 2023-11-28T13:03:32.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.893+0000 7f139159a700 1 -- 172.21.15.111:0/849062135 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138800e660 con 0x7f137c003a00 2023-11-28T13:03:32.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.893+0000 7f1398e75700 1 -- 172.21.15.111:0/849062135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f137c003a00 msgr2=0x7f137c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:32.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.893+0000 7f1398e75700 1 --2- 172.21.15.111:0/849062135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f137c003a00 0x7f137c003e10 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f1388009b40 tx=0x7f137c005880 comp rx=0 tx=0).stop 2023-11-28T13:03:32.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.893+0000 7f1392d9d700 1 -- 172.21.15.111:0/849062135 reap_dead start 2023-11-28T13:03:32.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.893+0000 7f1398e75700 1 -- 172.21.15.111:0/849062135 shutdown_connections 2023-11-28T13:03:32.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.893+0000 7f1398e75700 1 -- 172.21.15.111:0/849062135 >> 172.21.15.111:0/849062135 conn(0x7f139409a040 msgr2=0x7f13940d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:32.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.893+0000 7f1398e75700 1 -- 172.21.15.111:0/849062135 shutdown_connections 2023-11-28T13:03:32.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.893+0000 7f1398e75700 1 -- 172.21.15.111:0/849062135 wait complete. 2023-11-28T13:03:32.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.895+0000 7f1398e75700 1 Processor -- start 2023-11-28T13:03:32.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.895+0000 7f1398e75700 1 -- start start 2023-11-28T13:03:32.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.895+0000 7f1398e75700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f137c003a00 0x7f1394156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:32.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.895+0000 7f1398e75700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1394154b70 0x7f13940b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:32.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.895+0000 7f1398e75700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13940b2d80 0x7f13940c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:32.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.895+0000 7f1398e75700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13940b3220 con 0x7f137c003a00 2023-11-28T13:03:32.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.895+0000 7f1398e75700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1394156d60 con 0x7f1394154b70 2023-11-28T13:03:32.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.895+0000 7f1398e75700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13940b50f0 con 0x7f13940b2d80 2023-11-28T13:03:32.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f139259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f137c003a00 0x7f1394156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:32.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f139259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f137c003a00 0x7f1394156820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37886/0 (socket says 172.21.15.111:37886) 2023-11-28T13:03:32.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f139259c700 1 -- 172.21.15.111:0/552780794 learned_addr learned my addr 172.21.15.111:0/552780794 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:32.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f1392d9d700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13940b2d80 0x7f13940c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:32.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f1391d9b700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1394154b70 0x7f13940b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:32.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f139259c700 1 -- 172.21.15.111:0/552780794 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13940b2d80 msgr2=0x7f13940c6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:32.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f139259c700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13940b2d80 0x7f13940c6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:32.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f139259c700 1 -- 172.21.15.111:0/552780794 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1394154b70 msgr2=0x7f13940b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:32.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f139259c700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1394154b70 0x7f13940b2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:32.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f139259c700 1 -- 172.21.15.111:0/552780794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13940ccc30 con 0x7f137c003a00 2023-11-28T13:03:32.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f1391d9b700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1394154b70 0x7f13940b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:03:32.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f139259c700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f137c003a00 0x7f1394156820 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f138800e4b0 tx=0x7f13880047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:32.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f1382ffd700 1 -- 172.21.15.111:0/552780794 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388020070 con 0x7f137c003a00 2023-11-28T13:03:32.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f1382ffd700 1 -- 172.21.15.111:0/552780794 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1388004050 con 0x7f137c003a00 2023-11-28T13:03:32.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f1398e75700 1 -- 172.21.15.111:0/552780794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13940cc950 con 0x7f137c003a00 2023-11-28T13:03:32.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f1382ffd700 1 -- 172.21.15.111:0/552780794 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388004400 con 0x7f137c003a00 2023-11-28T13:03:32.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.896+0000 7f1398e75700 1 -- 172.21.15.111:0/552780794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13940ccf10 con 0x7f137c003a00 2023-11-28T13:03:32.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.897+0000 7f1398e75700 1 -- 172.21.15.111:0/552780794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1394067170 con 0x7f137c003a00 2023-11-28T13:03:32.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.898+0000 7f1382ffd700 1 -- 172.21.15.111:0/552780794 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f138802b030 con 0x7f137c003a00 2023-11-28T13:03:32.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.898+0000 7f1382ffd700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13740682f0 0x7f137406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:32.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.898+0000 7f1382ffd700 1 -- 172.21.15.111:0/552780794 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f1388086e40 con 0x7f137c003a00 2023-11-28T13:03:32.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.898+0000 7f1391d9b700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13740682f0 0x7f137406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:32.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.899+0000 7f1391d9b700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13740682f0 0x7f137406a7a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1384000c00 tx=0x7f1384009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:32.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:32.903+0000 7f1382ffd700 1 -- 172.21.15.111:0/552780794 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f13880534a0 con 0x7f137c003a00 2023-11-28T13:03:33.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.127+0000 7f1398e75700 1 -- 172.21.15.111:0/552780794 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f13940ce150 con 0x7f137c003a00 2023-11-28T13:03:33.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.127+0000 7f1382ffd700 1 -- 172.21.15.111:0/552780794 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f1388029080 con 0x7f137c003a00 2023-11-28T13:03:33.129 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:03:33.129 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:03:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.129+0000 7f1398e75700 1 -- 172.21.15.111:0/552780794 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13740682f0 msgr2=0x7f137406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.129+0000 7f1398e75700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13740682f0 0x7f137406a7a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1384000c00 tx=0x7f1384009040 comp rx=0 tx=0).stop 2023-11-28T13:03:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.129+0000 7f1398e75700 1 -- 172.21.15.111:0/552780794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f137c003a00 msgr2=0x7f1394156820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.129+0000 7f1398e75700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f137c003a00 0x7f1394156820 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f138800e4b0 tx=0x7f13880047c0 comp rx=0 tx=0).stop 2023-11-28T13:03:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.129+0000 7f1398e75700 1 -- 172.21.15.111:0/552780794 shutdown_connections 2023-11-28T13:03:33.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.129+0000 7f1398e75700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13940b2d80 0x7f13940c6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:33.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.129+0000 7f1398e75700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f137c003a00 0x7f1394156820 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:33.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.129+0000 7f1398e75700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f13740682f0 0x7f137406a7a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:33.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.129+0000 7f1398e75700 1 --2- 172.21.15.111:0/552780794 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1394154b70 0x7f13940b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:33.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.129+0000 7f1398e75700 1 -- 172.21.15.111:0/552780794 >> 172.21.15.111:0/552780794 conn(0x7f139409a040 msgr2=0x7f13940cd330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:33.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.129+0000 7f1398e75700 1 -- 172.21.15.111:0/552780794 shutdown_connections 2023-11-28T13:03:33.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:33.129+0000 7f1398e75700 1 -- 172.21.15.111:0/552780794 wait complete. 2023-11-28T13:03:33.143 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:03:38.145 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:03:38.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.276+0000 7fcf892f6700 1 Processor -- start 2023-11-28T13:03:38.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.276+0000 7fcf892f6700 1 -- start start 2023-11-28T13:03:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.277+0000 7fcf892f6700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fcf840af160 0x7fcf840ade20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.277+0000 7fcf892f6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf8415aa70 con 0x7fcf8415a6a0 2023-11-28T13:03:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.277+0000 7fcf892f6700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf840ad280 con 0x7fcf8406a980 2023-11-28T13:03:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.277+0000 7fcf892f6700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf840ac950 con 0x7fcf840af160 2023-11-28T13:03:38.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.277+0000 7fcf82ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcf8415a6a0 0x7fcf8406a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38672/0 (socket says 172.21.15.111:38672) 2023-11-28T13:03:38.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.277+0000 7fcf82ffd700 1 -- 172.21.15.111:0/3450653475 learned_addr learned my addr 172.21.15.111:0/3450653475 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:03:38.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.277+0000 7fcf837fe700 1 --2- 172.21.15.111:0/3450653475 >> v2:172.21.15.149:3300/0 conn(0x7fcf840af160 0x7fcf840ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:38.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.277+0000 7fcf81ffb700 1 -- 172.21.15.111:0/3450653475 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936570494 0 0) 0x7fcf74001110 con 0x7fcf8415a6a0 2023-11-28T13:03:38.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.277+0000 7fcf81ffb700 1 -- 172.21.15.111:0/3450653475 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf6c0035b0 con 0x7fcf8415a6a0 2023-11-28T13:03:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.277+0000 7fcf837fe700 1 -- 172.21.15.111:0/3450653475 >> v1:172.21.15.149:6789/0 conn(0x7fcf8406a980 legacy=0x7fcf840ad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.278+0000 7fcf837fe700 1 -- 172.21.15.111:0/3450653475 >> v1:172.21.15.111:6789/0 conn(0x7fcf8415a6a0 legacy=0x7fcf8406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.278+0000 7fcf837fe700 1 -- 172.21.15.111:0/3450653475 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf840acc30 con 0x7fcf840af160 2023-11-28T13:03:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.278+0000 7fcf837fe700 1 --2- 172.21.15.111:0/3450653475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf840af160 0x7fcf840ade20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009d90 tx=0x7fcf7c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c0c542c0aa38933 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:38.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.278+0000 7fcf81ffb700 1 -- 172.21.15.111:0/3450653475 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c00e070 con 0x7fcf840af160 2023-11-28T13:03:38.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.278+0000 7fcf81ffb700 1 -- 172.21.15.111:0/3450653475 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c00bb20 con 0x7fcf840af160 2023-11-28T13:03:38.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.278+0000 7fcf81ffb700 1 -- 172.21.15.111:0/3450653475 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c012430 con 0x7fcf840af160 2023-11-28T13:03:38.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.279+0000 7fcf892f6700 1 -- 172.21.15.111:0/3450653475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf840af160 msgr2=0x7fcf840ade20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:38.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.279+0000 7fcf892f6700 1 --2- 172.21.15.111:0/3450653475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf840af160 0x7fcf840ade20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009d90 tx=0x7fcf7c00b330 comp rx=0 tx=0).stop 2023-11-28T13:03:38.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.279+0000 7fcf892f6700 1 -- 172.21.15.111:0/3450653475 shutdown_connections 2023-11-28T13:03:38.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.279+0000 7fcf892f6700 1 --2- 172.21.15.111:0/3450653475 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf840af160 0x7fcf840ade20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:38.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.279+0000 7fcf892f6700 1 -- 172.21.15.111:0/3450653475 >> 172.21.15.111:0/3450653475 conn(0x7fcf84098ee0 msgr2=0x7fcf8406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:38.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.279+0000 7fcf892f6700 1 -- 172.21.15.111:0/3450653475 shutdown_connections 2023-11-28T13:03:38.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.279+0000 7fcf892f6700 1 -- 172.21.15.111:0/3450653475 wait complete. 2023-11-28T13:03:38.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.280+0000 7fcf892f6700 1 Processor -- start 2023-11-28T13:03:38.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.280+0000 7fcf892f6700 1 -- start start 2023-11-28T13:03:38.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf892f6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8406a980 0x7fcf840c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:38.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf892f6700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf840af160 0x7fcf840b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:38.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf892f6700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf8415a6a0 0x7fcf8415d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:38.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf892f6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8415c1b0 con 0x7fcf8406a980 2023-11-28T13:03:38.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf892f6700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8415c490 con 0x7fcf840af160 2023-11-28T13:03:38.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf892f6700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8415c770 con 0x7fcf8415a6a0 2023-11-28T13:03:38.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf82ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8406a980 0x7fcf840c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:38.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf82ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8406a980 0x7fcf840c5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37900/0 (socket says 172.21.15.111:37900) 2023-11-28T13:03:38.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf82ffd700 1 -- 172.21.15.111:0/1441178175 learned_addr learned my addr 172.21.15.111:0/1441178175 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:38.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf827fc700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf840af160 0x7fcf840b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:38.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf837fe700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf8415a6a0 0x7fcf8415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:38.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf82ffd700 1 -- 172.21.15.111:0/1441178175 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf8415a6a0 msgr2=0x7fcf8415d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:38.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf82ffd700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf8415a6a0 0x7fcf8415d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf82ffd700 1 -- 172.21.15.111:0/1441178175 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf840af160 msgr2=0x7fcf840b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf82ffd700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf840af160 0x7fcf840b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf82ffd700 1 -- 172.21.15.111:0/1441178175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c009a70 con 0x7fcf8406a980 2023-11-28T13:03:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.281+0000 7fcf827fc700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf840af160 0x7fcf840b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:03:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.282+0000 7fcf82ffd700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8406a980 0x7fcf840c5aa0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fcf74007fd0 tx=0x7fcf7400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.282+0000 7fcf6b7fe700 1 -- 172.21.15.111:0/1441178175 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf74005d70 con 0x7fcf8406a980 2023-11-28T13:03:38.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.282+0000 7fcf6b7fe700 1 -- 172.21.15.111:0/1441178175 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf740070d0 con 0x7fcf8406a980 2023-11-28T13:03:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.282+0000 7fcf892f6700 1 -- 172.21.15.111:0/1441178175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf840992e0 con 0x7fcf8406a980 2023-11-28T13:03:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.282+0000 7fcf6b7fe700 1 -- 172.21.15.111:0/1441178175 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf74007460 con 0x7fcf8406a980 2023-11-28T13:03:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.282+0000 7fcf892f6700 1 -- 172.21.15.111:0/1441178175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf840acf10 con 0x7fcf8406a980 2023-11-28T13:03:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.283+0000 7fcf892f6700 1 -- 172.21.15.111:0/1441178175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf840aec10 con 0x7fcf8406a980 2023-11-28T13:03:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.283+0000 7fcf6b7fe700 1 -- 172.21.15.111:0/1441178175 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcf74019030 con 0x7fcf8406a980 2023-11-28T13:03:38.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.284+0000 7fcf6b7fe700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcf700682f0 0x7fcf7006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:38.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.284+0000 7fcf6b7fe700 1 -- 172.21.15.111:0/1441178175 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fcf74085020 con 0x7fcf8406a980 2023-11-28T13:03:38.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.284+0000 7fcf827fc700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcf700682f0 0x7fcf7006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:38.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.284+0000 7fcf827fc700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcf700682f0 0x7fcf7006a7a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcf841479a0 tx=0x7fcf78009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:38.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.289+0000 7fcf6b7fe700 1 -- 172.21.15.111:0/1441178175 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcf74051700 con 0x7fcf8406a980 2023-11-28T13:03:38.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.515+0000 7fcf892f6700 1 -- 172.21.15.111:0/1441178175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcf84067170 con 0x7fcf8406a980 2023-11-28T13:03:38.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.516+0000 7fcf6b7fe700 1 -- 172.21.15.111:0/1441178175 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7fcf740170c0 con 0x7fcf8406a980 2023-11-28T13:03:38.517 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:03:38.518 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:03:38.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.517+0000 7fcf892f6700 1 -- 172.21.15.111:0/1441178175 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcf700682f0 msgr2=0x7fcf7006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:38.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.517+0000 7fcf892f6700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcf700682f0 0x7fcf7006a7a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcf841479a0 tx=0x7fcf78009040 comp rx=0 tx=0).stop 2023-11-28T13:03:38.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.518+0000 7fcf892f6700 1 -- 172.21.15.111:0/1441178175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8406a980 msgr2=0x7fcf840c5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:38.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.518+0000 7fcf892f6700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8406a980 0x7fcf840c5aa0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fcf74007fd0 tx=0x7fcf7400c860 comp rx=0 tx=0).stop 2023-11-28T13:03:38.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.518+0000 7fcf892f6700 1 -- 172.21.15.111:0/1441178175 shutdown_connections 2023-11-28T13:03:38.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.518+0000 7fcf892f6700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcf8415a6a0 0x7fcf8415d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:38.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.518+0000 7fcf892f6700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf8406a980 0x7fcf840c5aa0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:38.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.518+0000 7fcf892f6700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcf700682f0 0x7fcf7006a7a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:38.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.518+0000 7fcf892f6700 1 --2- 172.21.15.111:0/1441178175 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcf840af160 0x7fcf840b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:38.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.518+0000 7fcf892f6700 1 -- 172.21.15.111:0/1441178175 >> 172.21.15.111:0/1441178175 conn(0x7fcf84098ee0 msgr2=0x7fcf84159510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:38.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.518+0000 7fcf892f6700 1 -- 172.21.15.111:0/1441178175 shutdown_connections 2023-11-28T13:03:38.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:38.518+0000 7fcf892f6700 1 -- 172.21.15.111:0/1441178175 wait complete. 2023-11-28T13:03:38.531 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:03:43.532 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:03:43.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.663+0000 7f3842b89700 1 Processor -- start 2023-11-28T13:03:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.663+0000 7f3842b89700 1 -- start start 2023-11-28T13:03:43.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.664+0000 7f3842b89700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c15c6c0 0x7f383c0afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:43.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.664+0000 7f3842b89700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f383c0cb300 0x7f383c0b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:43.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.664+0000 7f3842b89700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c0cc750 con 0x7f383c0b0440 2023-11-28T13:03:43.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.664+0000 7f3842b89700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383c0b0df0 con 0x7f383c0cb300 2023-11-28T13:03:43.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.664+0000 7f3842b89700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0cca30 con 0x7f383c15c6c0 2023-11-28T13:03:43.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.664+0000 7f3840925700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c15c6c0 0x7f383c0afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:43.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.664+0000 7f3841126700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f383c0cb300 0x7f383c0b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:43.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.664+0000 7f3840925700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c15c6c0 0x7f383c0afb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:59656/0 (socket says 172.21.15.111:59656) 2023-11-28T13:03:43.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.664+0000 7f3840925700 1 -- 172.21.15.111:0/2742907803 learned_addr learned my addr 172.21.15.111:0/2742907803 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:43.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.664+0000 7f383bfff700 1 --1- 172.21.15.111:0/2742907803 >> v1:172.21.15.149:6789/0 conn(0x7f383c0b0440 0x7f383c1602a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:47228/0 (socket says 172.21.15.111:47228) 2023-11-28T13:03:43.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.664+0000 7f3841126700 1 -- 172.21.15.111:0/2742907803 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c15c6c0 msgr2=0x7f383c0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:43.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.664+0000 7f3841126700 1 --2- 172.21.15.111:0/2742907803 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c15c6c0 0x7f383c0afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:43.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.664+0000 7f3841126700 1 -- 172.21.15.111:0/2742907803 >> v1:172.21.15.149:6789/0 conn(0x7f383c0b0440 legacy=0x7f383c1602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:43.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.665+0000 7f3841126700 1 -- 172.21.15.111:0/2742907803 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c0ccd10 con 0x7f383c0cb300 2023-11-28T13:03:43.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.665+0000 7f383b7fe700 1 -- 172.21.15.111:0/2742907803 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816168624 0 0) 0x7f383c0cc750 con 0x7f383c0b0440 2023-11-28T13:03:43.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.665+0000 7f3840925700 1 --2- 172.21.15.111:0/2742907803 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c15c6c0 0x7f383c0afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:03:43.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.665+0000 7f3841126700 1 --2- 172.21.15.111:0/2742907803 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c0cb300 0x7f383c0b08b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3834009d90 tx=0x7f383400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=851489817a4f57fc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:43.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.665+0000 7f383b7fe700 1 -- 172.21.15.111:0/2742907803 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383400e070 con 0x7f383c0cb300 2023-11-28T13:03:43.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.665+0000 7f383b7fe700 1 -- 172.21.15.111:0/2742907803 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383400bb20 con 0x7f383c0cb300 2023-11-28T13:03:43.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.665+0000 7f383b7fe700 1 -- 172.21.15.111:0/2742907803 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834012430 con 0x7f383c0cb300 2023-11-28T13:03:43.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.665+0000 7f3842b89700 1 -- 172.21.15.111:0/2742907803 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c0cb300 msgr2=0x7f383c0b08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:43.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.665+0000 7f3842b89700 1 --2- 172.21.15.111:0/2742907803 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c0cb300 0x7f383c0b08b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3834009d90 tx=0x7f383400b330 comp rx=0 tx=0).stop 2023-11-28T13:03:43.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.666+0000 7f3842b89700 1 -- 172.21.15.111:0/2742907803 shutdown_connections 2023-11-28T13:03:43.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.666+0000 7f3842b89700 1 --2- 172.21.15.111:0/2742907803 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c0cb300 0x7f383c0b08b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:43.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.666+0000 7f3842b89700 1 --2- 172.21.15.111:0/2742907803 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c15c6c0 0x7f383c0afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:43.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.666+0000 7f3842b89700 1 -- 172.21.15.111:0/2742907803 >> 172.21.15.111:0/2742907803 conn(0x7f383c098ee0 msgr2=0x7f383c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:43.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.666+0000 7f3842b89700 1 -- 172.21.15.111:0/2742907803 shutdown_connections 2023-11-28T13:03:43.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.666+0000 7f3842b89700 1 -- 172.21.15.111:0/2742907803 wait complete. 2023-11-28T13:03:43.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.667+0000 7f3842b89700 1 Processor -- start 2023-11-28T13:03:43.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.667+0000 7f3842b89700 1 -- start start 2023-11-28T13:03:43.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f3842b89700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0b0440 0x7f383c1e7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:43.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f3842b89700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0cb300 0x7f383c0b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:43.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f3842b89700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c15c6c0 0x7f383c0b5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:43.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f3842b89700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c159a40 con 0x7f383c0cb300 2023-11-28T13:03:43.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f3842b89700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c158e30 con 0x7f383c15c6c0 2023-11-28T13:03:43.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f3842b89700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c159110 con 0x7f383c0b0440 2023-11-28T13:03:43.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f383bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0cb300 0x7f383c0b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:43.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f383bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0cb300 0x7f383c0b4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37916/0 (socket says 172.21.15.111:37916) 2023-11-28T13:03:43.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f383bfff700 1 -- 172.21.15.111:0/176718085 learned_addr learned my addr 172.21.15.111:0/176718085 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:43.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f3840925700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0b0440 0x7f383c1e7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:43.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f3841126700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c15c6c0 0x7f383c0b5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:43.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f383bfff700 1 -- 172.21.15.111:0/176718085 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0b0440 msgr2=0x7f383c1e7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:43.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f383bfff700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0b0440 0x7f383c1e7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:43.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f383bfff700 1 -- 172.21.15.111:0/176718085 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c15c6c0 msgr2=0x7f383c0b5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:43.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f383bfff700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c15c6c0 0x7f383c0b5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:43.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f383bfff700 1 -- 172.21.15.111:0/176718085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3830009550 con 0x7f383c0cb300 2023-11-28T13:03:43.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f3841126700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c15c6c0 0x7f383c0b5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:03:43.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f3840925700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0b0440 0x7f383c1e7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:03:43.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f383bfff700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0cb300 0x7f383c0b4c10 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f382800c550 tx=0x7f382800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:43.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.668+0000 7f38397fa700 1 -- 172.21.15.111:0/176718085 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828004c80 con 0x7f383c0cb300 2023-11-28T13:03:43.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.669+0000 7f38397fa700 1 -- 172.21.15.111:0/176718085 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3828004e00 con 0x7f383c0cb300 2023-11-28T13:03:43.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.669+0000 7f3842b89700 1 -- 172.21.15.111:0/176718085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3834009a70 con 0x7f383c0cb300 2023-11-28T13:03:43.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.669+0000 7f38397fa700 1 -- 172.21.15.111:0/176718085 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828007310 con 0x7f383c0cb300 2023-11-28T13:03:43.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.669+0000 7f3842b89700 1 -- 172.21.15.111:0/176718085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383c0cd2d0 con 0x7f383c0cb300 2023-11-28T13:03:43.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.669+0000 7f3842b89700 1 -- 172.21.15.111:0/176718085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383c0ccff0 con 0x7f383c0cb300 2023-11-28T13:03:43.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.670+0000 7f38397fa700 1 -- 172.21.15.111:0/176718085 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f382801a020 con 0x7f383c0cb300 2023-11-28T13:03:43.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.670+0000 7f38397fa700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38200682f0 0x7f382006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:43.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.670+0000 7f38397fa700 1 -- 172.21.15.111:0/176718085 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f3828085a60 con 0x7f383c0cb300 2023-11-28T13:03:43.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.670+0000 7f3840925700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38200682f0 0x7f382006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:43.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.671+0000 7f3840925700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38200682f0 0x7f382006a7a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3830009c40 tx=0x7f3830008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:43.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.675+0000 7f38397fa700 1 -- 172.21.15.111:0/176718085 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3828052160 con 0x7f383c0cb300 2023-11-28T13:03:43.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.897+0000 7f3842b89700 1 -- 172.21.15.111:0/176718085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f383c0cdf80 con 0x7f383c0cb300 2023-11-28T13:03:43.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.897+0000 7f38397fa700 1 -- 172.21.15.111:0/176718085 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f3828055780 con 0x7f383c0cb300 2023-11-28T13:03:43.899 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:03:43.899 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:03:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.899+0000 7f3842b89700 1 -- 172.21.15.111:0/176718085 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38200682f0 msgr2=0x7f382006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.899+0000 7f3842b89700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38200682f0 0x7f382006a7a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3830009c40 tx=0x7f3830008040 comp rx=0 tx=0).stop 2023-11-28T13:03:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.899+0000 7f3842b89700 1 -- 172.21.15.111:0/176718085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0cb300 msgr2=0x7f383c0b4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.899+0000 7f3842b89700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0cb300 0x7f383c0b4c10 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f382800c550 tx=0x7f382800c910 comp rx=0 tx=0).stop 2023-11-28T13:03:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.899+0000 7f3842b89700 1 -- 172.21.15.111:0/176718085 shutdown_connections 2023-11-28T13:03:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.899+0000 7f3842b89700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f38200682f0 0x7f382006a7a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.899+0000 7f3842b89700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f383c15c6c0 0x7f383c0b5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.899+0000 7f3842b89700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f383c0b0440 0x7f383c1e7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.899+0000 7f3842b89700 1 --2- 172.21.15.111:0/176718085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f383c0cb300 0x7f383c0b4c10 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.899+0000 7f3842b89700 1 -- 172.21.15.111:0/176718085 >> 172.21.15.111:0/176718085 conn(0x7f383c098ee0 msgr2=0x7f383c06ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.899+0000 7f3842b89700 1 -- 172.21.15.111:0/176718085 shutdown_connections 2023-11-28T13:03:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:43.900+0000 7f3842b89700 1 -- 172.21.15.111:0/176718085 wait complete. 2023-11-28T13:03:43.913 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:03:48.914 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:03:49.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.044+0000 7fd93789c700 1 Processor -- start 2023-11-28T13:03:49.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.044+0000 7fd93789c700 1 -- start start 2023-11-28T13:03:49.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.045+0000 7fd93789c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd93015c7b0 0x7fd9300cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:49.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.045+0000 7fd93789c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd9300d0490 0x7fd9300ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:49.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.045+0000 7fd93789c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9300cb810 con 0x7fd93006b380 2023-11-28T13:03:49.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.045+0000 7fd93789c700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9300cc650 con 0x7fd9300d0490 2023-11-28T13:03:49.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.045+0000 7fd93789c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9300cba40 con 0x7fd93015c7b0 2023-11-28T13:03:49.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd935638700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd93015c7b0 0x7fd9300cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:49.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd934e37700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fd9300d0490 0x7fd9300ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:49.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd935e39700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7fd93006b380 0x7fd9300ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:47246/0 (socket says 172.21.15.111:47246) 2023-11-28T13:03:49.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd935e39700 1 -- 172.21.15.111:0/4027202630 learned_addr learned my addr 172.21.15.111:0/4027202630 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:03:49.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd927fff700 1 -- 172.21.15.111:0/4027202630 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687425042 0 0) 0x7fd9300cb810 con 0x7fd93006b380 2023-11-28T13:03:49.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd927fff700 1 -- 172.21.15.111:0/4027202630 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd91c0035b0 con 0x7fd93006b380 2023-11-28T13:03:49.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd935638700 1 -- 172.21.15.111:0/4027202630 >> v2:172.21.15.149:3300/0 conn(0x7fd9300d0490 msgr2=0x7fd9300ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:49.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd935638700 1 --2- 172.21.15.111:0/4027202630 >> v2:172.21.15.149:3300/0 conn(0x7fd9300d0490 0x7fd9300ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:49.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd935638700 1 -- 172.21.15.111:0/4027202630 >> v1:172.21.15.149:6789/0 conn(0x7fd93006b380 legacy=0x7fd9300ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd935638700 1 -- 172.21.15.111:0/4027202630 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9300cbd20 con 0x7fd93015c7b0 2023-11-28T13:03:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd934e37700 1 --2- 172.21.15.111:0/4027202630 >> v2:172.21.15.149:3300/0 conn(0x7fd9300d0490 0x7fd9300ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:03:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd935638700 1 --2- 172.21.15.111:0/4027202630 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd93015c7b0 0x7fd9300cb2d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd920009820 tx=0x7fd920009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=121bd6f0fb09c8e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd927fff700 1 -- 172.21.15.111:0/4027202630 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd920005e30 con 0x7fd93015c7b0 2023-11-28T13:03:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.046+0000 7fd927fff700 1 -- 172.21.15.111:0/4027202630 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd920006030 con 0x7fd93015c7b0 2023-11-28T13:03:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.047+0000 7fd927fff700 1 -- 172.21.15.111:0/4027202630 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9200063c0 con 0x7fd93015c7b0 2023-11-28T13:03:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.047+0000 7fd93789c700 1 -- 172.21.15.111:0/4027202630 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd93015c7b0 msgr2=0x7fd9300cb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.047+0000 7fd93789c700 1 --2- 172.21.15.111:0/4027202630 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd93015c7b0 0x7fd9300cb2d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd920009820 tx=0x7fd920009b30 comp rx=0 tx=0).stop 2023-11-28T13:03:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.047+0000 7fd93789c700 1 -- 172.21.15.111:0/4027202630 shutdown_connections 2023-11-28T13:03:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.047+0000 7fd93789c700 1 --2- 172.21.15.111:0/4027202630 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd93015c7b0 0x7fd9300cb2d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.047+0000 7fd93789c700 1 --2- 172.21.15.111:0/4027202630 >> v2:172.21.15.149:3300/0 conn(0x7fd9300d0490 0x7fd9300ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:49.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.047+0000 7fd93789c700 1 -- 172.21.15.111:0/4027202630 >> 172.21.15.111:0/4027202630 conn(0x7fd93009a040 msgr2=0x7fd9300d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:49.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.047+0000 7fd93789c700 1 -- 172.21.15.111:0/4027202630 shutdown_connections 2023-11-28T13:03:49.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.047+0000 7fd93789c700 1 -- 172.21.15.111:0/4027202630 wait complete. 2023-11-28T13:03:49.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.048+0000 7fd93789c700 1 Processor -- start 2023-11-28T13:03:49.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.048+0000 7fd93789c700 1 -- start start 2023-11-28T13:03:49.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.049+0000 7fd93789c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd93006b380 0x7fd9301554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:49.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.049+0000 7fd93789c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9300d0490 0x7fd930159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.049+0000 7fd93789c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd93015c7b0 0x7fd9300af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.049+0000 7fd93789c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9300af700 con 0x7fd9300d0490 2023-11-28T13:03:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.049+0000 7fd93789c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9300afef0 con 0x7fd93015c7b0 2023-11-28T13:03:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.049+0000 7fd93789c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93015a770 con 0x7fd93006b380 2023-11-28T13:03:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.049+0000 7fd934e37700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9300d0490 0x7fd930159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.049+0000 7fd934e37700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9300d0490 0x7fd930159fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37930/0 (socket says 172.21.15.111:37930) 2023-11-28T13:03:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.049+0000 7fd934e37700 1 -- 172.21.15.111:0/3916328717 learned_addr learned my addr 172.21.15.111:0/3916328717 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.049+0000 7fd935638700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd93006b380 0x7fd9301554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.050+0000 7fd935e39700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd93015c7b0 0x7fd9300af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.050+0000 7fd934e37700 1 -- 172.21.15.111:0/3916328717 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd93006b380 msgr2=0x7fd9301554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.050+0000 7fd934e37700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd93006b380 0x7fd9301554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.050+0000 7fd934e37700 1 -- 172.21.15.111:0/3916328717 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd93015c7b0 msgr2=0x7fd9300af1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.050+0000 7fd934e37700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd93015c7b0 0x7fd9300af1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.050+0000 7fd934e37700 1 -- 172.21.15.111:0/3916328717 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c0094a0 con 0x7fd9300d0490 2023-11-28T13:03:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.050+0000 7fd934e37700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9300d0490 0x7fd930159fe0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00ebe0 tx=0x7fd92c009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.050+0000 7fd925ffb700 1 -- 172.21.15.111:0/3916328717 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c00c910 con 0x7fd9300d0490 2023-11-28T13:03:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.050+0000 7fd925ffb700 1 -- 172.21.15.111:0/3916328717 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92c00ca70 con 0x7fd9300d0490 2023-11-28T13:03:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.050+0000 7fd93789c700 1 -- 172.21.15.111:0/3916328717 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd920009430 con 0x7fd9300d0490 2023-11-28T13:03:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.050+0000 7fd925ffb700 1 -- 172.21.15.111:0/3916328717 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c00cde0 con 0x7fd9300d0490 2023-11-28T13:03:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.050+0000 7fd93789c700 1 -- 172.21.15.111:0/3916328717 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9300cc2e0 con 0x7fd9300d0490 2023-11-28T13:03:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.051+0000 7fd93789c700 1 -- 172.21.15.111:0/3916328717 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9300cc000 con 0x7fd9300d0490 2023-11-28T13:03:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.052+0000 7fd925ffb700 1 -- 172.21.15.111:0/3916328717 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd92c019030 con 0x7fd9300d0490 2023-11-28T13:03:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.052+0000 7fd925ffb700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9140682f0 0x7fd91406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.052+0000 7fd925ffb700 1 -- 172.21.15.111:0/3916328717 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fd92c085ed0 con 0x7fd9300d0490 2023-11-28T13:03:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.052+0000 7fd935638700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9140682f0 0x7fd91406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:49.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.052+0000 7fd935638700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9140682f0 0x7fd91406a7a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd920000c00 tx=0x7fd920008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:49.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.056+0000 7fd925ffb700 1 -- 172.21.15.111:0/3916328717 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd92c0525b0 con 0x7fd9300d0490 2023-11-28T13:03:49.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.281+0000 7fd93789c700 1 -- 172.21.15.111:0/3916328717 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd93015ad80 con 0x7fd9300d0490 2023-11-28T13:03:49.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.281+0000 7fd925ffb700 1 -- 172.21.15.111:0/3916328717 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7fd92c0170c0 con 0x7fd9300d0490 2023-11-28T13:03:49.283 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:03:49.283 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:03:49.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.283+0000 7fd93789c700 1 -- 172.21.15.111:0/3916328717 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9140682f0 msgr2=0x7fd91406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:49.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.283+0000 7fd93789c700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9140682f0 0x7fd91406a7a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd920000c00 tx=0x7fd920008070 comp rx=0 tx=0).stop 2023-11-28T13:03:49.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.283+0000 7fd93789c700 1 -- 172.21.15.111:0/3916328717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9300d0490 msgr2=0x7fd930159fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:49.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.283+0000 7fd93789c700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9300d0490 0x7fd930159fe0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00ebe0 tx=0x7fd92c009ec0 comp rx=0 tx=0).stop 2023-11-28T13:03:49.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.283+0000 7fd93789c700 1 -- 172.21.15.111:0/3916328717 shutdown_connections 2023-11-28T13:03:49.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.283+0000 7fd93789c700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd9140682f0 0x7fd91406a7a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:49.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.283+0000 7fd93789c700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd93015c7b0 0x7fd9300af1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:49.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.283+0000 7fd93789c700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd93006b380 0x7fd9301554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:49.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.283+0000 7fd93789c700 1 --2- 172.21.15.111:0/3916328717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9300d0490 0x7fd930159fe0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:49.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.283+0000 7fd93789c700 1 -- 172.21.15.111:0/3916328717 >> 172.21.15.111:0/3916328717 conn(0x7fd93009a040 msgr2=0x7fd9300cf6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:49.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.283+0000 7fd93789c700 1 -- 172.21.15.111:0/3916328717 shutdown_connections 2023-11-28T13:03:49.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:49.284+0000 7fd93789c700 1 -- 172.21.15.111:0/3916328717 wait complete. 2023-11-28T13:03:49.298 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:03:54.299 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:03:54.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.428+0000 7f0691359700 1 Processor -- start 2023-11-28T13:03:54.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.428+0000 7f0691359700 1 -- start start 2023-11-28T13:03:54.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.428+0000 7f0691359700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f068c0ca1f0 0x7f068c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:54.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.428+0000 7f0691359700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f068c06a270 0x7f068c070910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:54.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.428+0000 7f0691359700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068c06d870 con 0x7f068c0c9300 2023-11-28T13:03:54.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.428+0000 7f0691359700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f068c0ca640 con 0x7f068c06a270 2023-11-28T13:03:54.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.428+0000 7f0691359700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068c06cc60 con 0x7f068c0ca1f0 2023-11-28T13:03:54.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.429+0000 7f068a7fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f068c0ca1f0 0x7f068c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:54.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.429+0000 7f068a7fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f068c0ca1f0 0x7f068c06dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:59686/0 (socket says 172.21.15.111:59686) 2023-11-28T13:03:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.429+0000 7f068a7fc700 1 -- 172.21.15.111:0/4043879162 learned_addr learned my addr 172.21.15.111:0/4043879162 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.429+0000 7f068b7fe700 1 --2- 172.21.15.111:0/4043879162 >> v2:172.21.15.149:3300/0 conn(0x7f068c06a270 0x7f068c070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.429+0000 7f068b7fe700 1 -- 172.21.15.111:0/4043879162 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f068c0ca1f0 msgr2=0x7f068c06dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.429+0000 7f068b7fe700 1 --2- 172.21.15.111:0/4043879162 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f068c0ca1f0 0x7f068c06dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.429+0000 7f068b7fe700 1 -- 172.21.15.111:0/4043879162 >> v1:172.21.15.149:6789/0 conn(0x7f068c0c9300 legacy=0x7f068c158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.429+0000 7f068b7fe700 1 -- 172.21.15.111:0/4043879162 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068c06cf40 con 0x7f068c06a270 2023-11-28T13:03:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.429+0000 7f068a7fc700 1 --2- 172.21.15.111:0/4043879162 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f068c0ca1f0 0x7f068c06dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:03:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.429+0000 7f068b7fe700 1 --2- 172.21.15.111:0/4043879162 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f068c06a270 0x7f068c070910 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0684009d90 tx=0x7f068400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b5bd9d38905e801 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.430+0000 7f0689ffb700 1 -- 172.21.15.111:0/4043879162 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068400e070 con 0x7f068c06a270 2023-11-28T13:03:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.430+0000 7f0689ffb700 1 -- 172.21.15.111:0/4043879162 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068400bb20 con 0x7f068c06a270 2023-11-28T13:03:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.430+0000 7f0689ffb700 1 -- 172.21.15.111:0/4043879162 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684012430 con 0x7f068c06a270 2023-11-28T13:03:54.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.430+0000 7f0691359700 1 -- 172.21.15.111:0/4043879162 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f068c06a270 msgr2=0x7f068c070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:54.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.430+0000 7f0691359700 1 --2- 172.21.15.111:0/4043879162 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f068c06a270 0x7f068c070910 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0684009d90 tx=0x7f068400b330 comp rx=0 tx=0).stop 2023-11-28T13:03:54.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.430+0000 7f0691359700 1 -- 172.21.15.111:0/4043879162 shutdown_connections 2023-11-28T13:03:54.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.430+0000 7f0691359700 1 --2- 172.21.15.111:0/4043879162 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f068c06a270 0x7f068c070910 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:54.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.430+0000 7f0691359700 1 --2- 172.21.15.111:0/4043879162 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f068c0ca1f0 0x7f068c06dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:54.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.430+0000 7f0691359700 1 -- 172.21.15.111:0/4043879162 >> 172.21.15.111:0/4043879162 conn(0x7f068c0cb490 msgr2=0x7f068c0cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:54.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.430+0000 7f0691359700 1 -- 172.21.15.111:0/4043879162 shutdown_connections 2023-11-28T13:03:54.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.430+0000 7f0691359700 1 -- 172.21.15.111:0/4043879162 wait complete. 2023-11-28T13:03:54.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.432+0000 7f0691359700 1 Processor -- start 2023-11-28T13:03:54.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.432+0000 7f0691359700 1 -- start start 2023-11-28T13:03:54.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.432+0000 7f0691359700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f068c06a270 0x7f068c1e4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:54.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.432+0000 7f0691359700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f068c0c9300 0x7f068c0c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:54.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.432+0000 7f0691359700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f068c0ca1f0 0x7f068c1e2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:54.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.432+0000 7f0691359700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c0c8680 con 0x7f068c0c9300 2023-11-28T13:03:54.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.432+0000 7f0691359700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c0c8860 con 0x7f068c06a270 2023-11-28T13:03:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.432+0000 7f0691359700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068c0ac380 con 0x7f068c0ca1f0 2023-11-28T13:03:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f068a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f068c0c9300 0x7f068c0c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f068a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f068c0c9300 0x7f068c0c8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37944/0 (socket says 172.21.15.111:37944) 2023-11-28T13:03:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f068a7fc700 1 -- 172.21.15.111:0/1785748013 learned_addr learned my addr 172.21.15.111:0/1785748013 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f068b7fe700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f068c0ca1f0 0x7f068c1e2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f068affd700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f068c06a270 0x7f068c1e4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f068a7fc700 1 -- 172.21.15.111:0/1785748013 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f068c0ca1f0 msgr2=0x7f068c1e2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f068a7fc700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f068c0ca1f0 0x7f068c1e2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f068a7fc700 1 -- 172.21.15.111:0/1785748013 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f068c06a270 msgr2=0x7f068c1e4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f068a7fc700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f068c06a270 0x7f068c1e4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:54.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f068a7fc700 1 -- 172.21.15.111:0/1785748013 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068000b7f0 con 0x7f068c0c9300 2023-11-28T13:03:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f068b7fe700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f068c0ca1f0 0x7f068c1e2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:03:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f068affd700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f068c06a270 0x7f068c1e4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:03:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f068a7fc700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f068c0c9300 0x7f068c0c8140 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f068000ed70 tx=0x7f068000d630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f06737fe700 1 -- 172.21.15.111:0/1785748013 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0680010640 con 0x7f068c0c9300 2023-11-28T13:03:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f06737fe700 1 -- 172.21.15.111:0/1785748013 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06800107a0 con 0x7f068c0c9300 2023-11-28T13:03:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f0691359700 1 -- 172.21.15.111:0/1785748013 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0684009a70 con 0x7f068c0c9300 2023-11-28T13:03:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f06737fe700 1 -- 172.21.15.111:0/1785748013 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0680010b10 con 0x7f068c0c9300 2023-11-28T13:03:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.433+0000 7f0691359700 1 -- 172.21.15.111:0/1785748013 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f068c06d500 con 0x7f068c0c9300 2023-11-28T13:03:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.434+0000 7f0691359700 1 -- 172.21.15.111:0/1785748013 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068c06d220 con 0x7f068c0c9300 2023-11-28T13:03:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.435+0000 7f06737fe700 1 -- 172.21.15.111:0/1785748013 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0680010c70 con 0x7f068c0c9300 2023-11-28T13:03:54.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.435+0000 7f06737fe700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06780682f0 0x7f067806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:54.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.435+0000 7f06737fe700 1 -- 172.21.15.111:0/1785748013 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f0680086470 con 0x7f068c0c9300 2023-11-28T13:03:54.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.435+0000 7f068affd700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06780682f0 0x7f067806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:54.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.435+0000 7f068affd700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06780682f0 0x7f067806a7a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f067c000f60 tx=0x7f067c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:54.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.439+0000 7f06737fe700 1 -- 172.21.15.111:0/1785748013 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0680052b50 con 0x7f068c0c9300 2023-11-28T13:03:54.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.664+0000 7f0691359700 1 -- 172.21.15.111:0/1785748013 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f068c071b20 con 0x7f068c0c9300 2023-11-28T13:03:54.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.664+0000 7f06737fe700 1 -- 172.21.15.111:0/1785748013 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f0680056170 con 0x7f068c0c9300 2023-11-28T13:03:54.666 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:03:54.666 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:03:54.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.666+0000 7f0691359700 1 -- 172.21.15.111:0/1785748013 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06780682f0 msgr2=0x7f067806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:54.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.666+0000 7f0691359700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06780682f0 0x7f067806a7a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f067c000f60 tx=0x7f067c009040 comp rx=0 tx=0).stop 2023-11-28T13:03:54.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.666+0000 7f0691359700 1 -- 172.21.15.111:0/1785748013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f068c0c9300 msgr2=0x7f068c0c8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:54.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.666+0000 7f0691359700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f068c0c9300 0x7f068c0c8140 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f068000ed70 tx=0x7f068000d630 comp rx=0 tx=0).stop 2023-11-28T13:03:54.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.666+0000 7f0691359700 1 -- 172.21.15.111:0/1785748013 shutdown_connections 2023-11-28T13:03:54.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.666+0000 7f0691359700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f068c0ca1f0 0x7f068c1e2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:54.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.666+0000 7f0691359700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06780682f0 0x7f067806a7a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:54.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.666+0000 7f0691359700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f068c06a270 0x7f068c1e4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:54.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.666+0000 7f0691359700 1 --2- 172.21.15.111:0/1785748013 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f068c0c9300 0x7f068c0c8140 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:54.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.666+0000 7f0691359700 1 -- 172.21.15.111:0/1785748013 >> 172.21.15.111:0/1785748013 conn(0x7f068c0cb490 msgr2=0x7f068c0b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:54.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.666+0000 7f0691359700 1 -- 172.21.15.111:0/1785748013 shutdown_connections 2023-11-28T13:03:54.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:54.666+0000 7f0691359700 1 -- 172.21.15.111:0/1785748013 wait complete. 2023-11-28T13:03:54.679 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:03:59.298 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:03:59.301 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:03:59.680 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:03:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.808+0000 7ff506956700 1 Processor -- start 2023-11-28T13:03:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.808+0000 7ff506956700 1 -- start start 2023-11-28T13:03:59.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.809+0000 7ff506956700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff5001604b0 0x7ff5000ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:59.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.809+0000 7ff506956700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50015cd90 con 0x7ff5000cc550 2023-11-28T13:03:59.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.809+0000 7ff506956700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5001613e0 con 0x7ff5000b0d70 2023-11-28T13:03:59.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.809+0000 7ff506956700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff500160950 con 0x7ff5001604b0 2023-11-28T13:03:59.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.809+0000 7ff504ef3700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff5000cc550 0x7ff5000af860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38732/0 (socket says 172.21.15.111:38732) 2023-11-28T13:03:59.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.809+0000 7ff504ef3700 1 -- 172.21.15.111:0/3366380116 learned_addr learned my addr 172.21.15.111:0/3366380116 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:03:59.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4ff7fe700 1 --2- 172.21.15.111:0/3366380116 >> v2:172.21.15.149:3300/0 conn(0x7ff5001604b0 0x7ff5000ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:59.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303537598 0 0) 0x7ff4f8001490 con 0x7ff5000cc550 2023-11-28T13:03:59.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4ec0035b0 con 0x7ff5000cc550 2023-11-28T13:03:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3661064010 0 0) 0x7ff4ec0035b0 con 0x7ff5000cc550 2023-11-28T13:03:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 >> v2:172.21.15.149:3300/0 conn(0x7ff5001604b0 msgr2=0x7ff5000ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 --2- 172.21.15.111:0/3366380116 >> v2:172.21.15.149:3300/0 conn(0x7ff5001604b0 0x7ff5000ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 >> v1:172.21.15.149:6789/0 conn(0x7ff5000b0d70 legacy=0x7ff5000b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff500160b40 con 0x7ff5000cc550 2023-11-28T13:03:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7ff4f8001b80 con 0x7ff5000cc550 2023-11-28T13:03:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 >> v1:172.21.15.111:6789/0 conn(0x7ff5000cc550 legacy=0x7ff5000af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 --2- 172.21.15.111:0/3366380116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4ec003a00 0x7ff4ec003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 --2- 172.21.15.111:0/3366380116 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff4ec004850 0x7ff4ec004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 --2- 172.21.15.111:0/3366380116 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4ec005200 0x7ff4ec00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:59.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec00c050 con 0x7ff4ec003a00 2023-11-28T13:03:59.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec00c1c0 con 0x7ff4ec005200 2023-11-28T13:03:59.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec00c380 con 0x7ff4ec004850 2023-11-28T13:03:59.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4fffff700 1 --2- 172.21.15.111:0/3366380116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4ec003a00 0x7ff4ec003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:59.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff504ef3700 1 --2- 172.21.15.111:0/3366380116 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4ec005200 0x7ff4ec00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:59.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4ff7fe700 1 --2- 172.21.15.111:0/3366380116 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff4ec004850 0x7ff4ec004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:59.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4fffff700 1 -- 172.21.15.111:0/3366380116 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff4ec004850 msgr2=0x7ff4ec004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:59.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4fffff700 1 --2- 172.21.15.111:0/3366380116 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff4ec004850 0x7ff4ec004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:59.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4fffff700 1 -- 172.21.15.111:0/3366380116 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4ec005200 msgr2=0x7ff4ec00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:03:59.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4fffff700 1 --2- 172.21.15.111:0/3366380116 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4ec005200 0x7ff4ec00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:59.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff504ef3700 1 -- 172.21.15.111:0/3366380116 reap_dead start 2023-11-28T13:03:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4fffff700 1 -- 172.21.15.111:0/3366380116 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ec0035b0 con 0x7ff4ec003a00 2023-11-28T13:03:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.810+0000 7ff4ff7fe700 1 --2- 172.21.15.111:0/3366380116 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff4ec004850 0x7ff4ec004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:03:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.811+0000 7ff4fffff700 1 --2- 172.21.15.111:0/3366380116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4ec003a00 0x7ff4ec003e10 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0001820 tx=0x7ff4ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129a95f7fb6d966c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.811+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f00012e0 con 0x7ff4ec003a00 2023-11-28T13:03:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.811+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f00052d0 con 0x7ff4ec003a00 2023-11-28T13:03:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.811+0000 7ff4feffd700 1 -- 172.21.15.111:0/3366380116 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f00036d0 con 0x7ff4ec003a00 2023-11-28T13:03:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.811+0000 7ff506956700 1 -- 172.21.15.111:0/3366380116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4ec003a00 msgr2=0x7ff4ec003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.811+0000 7ff506956700 1 --2- 172.21.15.111:0/3366380116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4ec003a00 0x7ff4ec003e10 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0001820 tx=0x7ff4ec005880 comp rx=0 tx=0).stop 2023-11-28T13:03:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.811+0000 7ff506956700 1 -- 172.21.15.111:0/3366380116 shutdown_connections 2023-11-28T13:03:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.811+0000 7ff506956700 1 --2- 172.21.15.111:0/3366380116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4ec003a00 0x7ff4ec003e10 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:59.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.811+0000 7ff506956700 1 -- 172.21.15.111:0/3366380116 >> 172.21.15.111:0/3366380116 conn(0x7ff500098ee0 msgr2=0x7ff50006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:03:59.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.811+0000 7ff506956700 1 -- 172.21.15.111:0/3366380116 shutdown_connections 2023-11-28T13:03:59.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.811+0000 7ff506956700 1 -- 172.21.15.111:0/3366380116 wait complete. 2023-11-28T13:03:59.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.812+0000 7ff506956700 1 Processor -- start 2023-11-28T13:03:59.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.813+0000 7ff506956700 1 -- start start 2023-11-28T13:03:59.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.813+0000 7ff506956700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4ec003a00 0x7ff50015a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:59.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.813+0000 7ff506956700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff5000c35c0 0x7ff5000c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:59.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.813+0000 7ff506956700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5000b2d90 0x7ff5001f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:59.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.813+0000 7ff506956700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5000c4700 con 0x7ff5000b2d90 2023-11-28T13:03:59.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.813+0000 7ff506956700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5001f6710 con 0x7ff4ec003a00 2023-11-28T13:03:59.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.813+0000 7ff506956700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50015a8d0 con 0x7ff5000c35c0 2023-11-28T13:03:59.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.813+0000 7ff504ef3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5000b2d90 0x7ff5001f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:59.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.813+0000 7ff504ef3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5000b2d90 0x7ff5001f58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37966/0 (socket says 172.21.15.111:37966) 2023-11-28T13:03:59.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.813+0000 7ff504ef3700 1 -- 172.21.15.111:0/3537982323 learned_addr learned my addr 172.21.15.111:0/3537982323 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:03:59.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.813+0000 7ff4fffff700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4ec003a00 0x7ff50015a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:59.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.813+0000 7ff4ff7fe700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff5000c35c0 0x7ff5000c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:59.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.814+0000 7ff504ef3700 1 -- 172.21.15.111:0/3537982323 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff5000c35c0 msgr2=0x7ff5000c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:59.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.814+0000 7ff504ef3700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff5000c35c0 0x7ff5000c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:59.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.814+0000 7ff504ef3700 1 -- 172.21.15.111:0/3537982323 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4ec003a00 msgr2=0x7ff50015a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:03:59.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.814+0000 7ff504ef3700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4ec003a00 0x7ff50015a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:03:59.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.814+0000 7ff504ef3700 1 -- 172.21.15.111:0/3537982323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff500161100 con 0x7ff5000b2d90 2023-11-28T13:03:59.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.814+0000 7ff504ef3700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5000b2d90 0x7ff5001f58b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ff5000b1ee0 tx=0x7ff4f8007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:59.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.814+0000 7ff4fcff9700 1 -- 172.21.15.111:0/3537982323 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8007a00 con 0x7ff5000b2d90 2023-11-28T13:03:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.814+0000 7ff4fcff9700 1 -- 172.21.15.111:0/3537982323 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f8007b60 con 0x7ff5000b2d90 2023-11-28T13:03:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.814+0000 7ff4fcff9700 1 -- 172.21.15.111:0/3537982323 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f80135f0 con 0x7ff5000b2d90 2023-11-28T13:03:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.814+0000 7ff506956700 1 -- 172.21.15.111:0/3537982323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff500160e20 con 0x7ff5000b2d90 2023-11-28T13:03:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.814+0000 7ff506956700 1 -- 172.21.15.111:0/3537982323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5000af340 con 0x7ff5000b2d90 2023-11-28T13:03:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.815+0000 7ff506956700 1 -- 172.21.15.111:0/3537982323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff500067170 con 0x7ff5000b2d90 2023-11-28T13:03:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.815+0000 7ff4fcff9700 1 -- 172.21.15.111:0/3537982323 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff4f8013750 con 0x7ff5000b2d90 2023-11-28T13:03:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.816+0000 7ff4fcff9700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff4e00682f0 0x7ff4e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:03:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.816+0000 7ff4fcff9700 1 -- 172.21.15.111:0/3537982323 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7ff4f80865f0 con 0x7ff5000b2d90 2023-11-28T13:03:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.816+0000 7ff4fffff700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff4e00682f0 0x7ff4e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:03:59.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.816+0000 7ff4fffff700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff4e00682f0 0x7ff4e006a7a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0000c00 tx=0x7ff4f0011000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:03:59.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:03:59.820+0000 7ff4fcff9700 1 -- 172.21.15.111:0/3537982323 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff4f8052c70 con 0x7ff5000b2d90 2023-11-28T13:04:00.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.043+0000 7ff506956700 1 -- 172.21.15.111:0/3537982323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff50015cd90 con 0x7ff5000b2d90 2023-11-28T13:04:00.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.044+0000 7ff4fcff9700 1 -- 172.21.15.111:0/3537982323 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7ff50015cd90 con 0x7ff5000b2d90 2023-11-28T13:04:00.046 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:04:00.046 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:04:00.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.046+0000 7ff506956700 1 -- 172.21.15.111:0/3537982323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff4e00682f0 msgr2=0x7ff4e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:00.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.046+0000 7ff506956700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff4e00682f0 0x7ff4e006a7a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0000c00 tx=0x7ff4f0011000 comp rx=0 tx=0).stop 2023-11-28T13:04:00.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.046+0000 7ff506956700 1 -- 172.21.15.111:0/3537982323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5000b2d90 msgr2=0x7ff5001f58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:00.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.046+0000 7ff506956700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5000b2d90 0x7ff5001f58b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ff5000b1ee0 tx=0x7ff4f8007340 comp rx=0 tx=0).stop 2023-11-28T13:04:00.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.046+0000 7ff506956700 1 -- 172.21.15.111:0/3537982323 shutdown_connections 2023-11-28T13:04:00.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.046+0000 7ff506956700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5000b2d90 0x7ff5001f58b0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:00.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.046+0000 7ff506956700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff4e00682f0 0x7ff4e006a7a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:00.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.046+0000 7ff506956700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff4ec003a00 0x7ff50015a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:00.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.046+0000 7ff506956700 1 --2- 172.21.15.111:0/3537982323 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff5000c35c0 0x7ff5000c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:00.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.046+0000 7ff506956700 1 -- 172.21.15.111:0/3537982323 >> 172.21.15.111:0/3537982323 conn(0x7ff500098ee0 msgr2=0x7ff50015b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:00.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.046+0000 7ff506956700 1 -- 172.21.15.111:0/3537982323 shutdown_connections 2023-11-28T13:04:00.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:00.046+0000 7ff506956700 1 -- 172.21.15.111:0/3537982323 wait complete. 2023-11-28T13:04:00.059 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:04:05.060 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:04:05.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.191+0000 7f1b977b1700 1 Processor -- start 2023-11-28T13:04:05.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.191+0000 7f1b977b1700 1 -- start start 2023-11-28T13:04:05.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b977b1700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f1b900cfc90 0x7f1b900d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:05.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b977b1700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b900d05a0 con 0x7f1b900cc5a0 2023-11-28T13:04:05.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b977b1700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b900ca260 con 0x7f1b900cdef0 2023-11-28T13:04:05.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b977b1700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b900ccaa0 con 0x7f1b900cfc90 2023-11-28T13:04:05.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b94d4c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1b900cc5a0 0x7f1b900cd7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38748/0 (socket says 172.21.15.111:38748) 2023-11-28T13:04:05.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b94d4c700 1 -- 172.21.15.111:0/2006959385 learned_addr learned my addr 172.21.15.111:0/2006959385 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:04:05.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b9554d700 1 --2- 172.21.15.111:0/2006959385 >> v2:172.21.15.149:3300/0 conn(0x7f1b900cfc90 0x7f1b900d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:05.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098700154 0 0) 0x7f1b80001110 con 0x7f1b900cc5a0 2023-11-28T13:04:05.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b7c0035b0 con 0x7f1b900cc5a0 2023-11-28T13:04:05.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2583824644 0 0) 0x7f1b7c0035b0 con 0x7f1b900cc5a0 2023-11-28T13:04:05.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 >> v2:172.21.15.149:3300/0 conn(0x7f1b900cfc90 msgr2=0x7f1b900d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b87fff700 1 --2- 172.21.15.111:0/2006959385 >> v2:172.21.15.149:3300/0 conn(0x7f1b900cfc90 0x7f1b900d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 >> v1:172.21.15.149:6789/0 conn(0x7f1b900cdef0 legacy=0x7f1b900ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.192+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b900ccd80 con 0x7f1b900cc5a0 2023-11-28T13:04:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f1b80001800 con 0x7f1b900cc5a0 2023-11-28T13:04:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 >> v1:172.21.15.111:6789/0 conn(0x7f1b900cc5a0 legacy=0x7f1b900cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b87fff700 1 --2- 172.21.15.111:0/2006959385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b7c003a00 0x7f1b7c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b87fff700 1 --2- 172.21.15.111:0/2006959385 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b7c004850 0x7f1b7c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b87fff700 1 --2- 172.21.15.111:0/2006959385 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b7c005200 0x7f1b7c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b900ccaa0 con 0x7f1b7c003a00 2023-11-28T13:04:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c00c080 con 0x7f1b7c004850 2023-11-28T13:04:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c00c240 con 0x7f1b7c005200 2023-11-28T13:04:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b9554d700 1 --2- 172.21.15.111:0/2006959385 >> v2:172.21.15.149:3300/0 conn(0x7f1b900cfc90 0x7f1b900d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:04:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b9554d700 1 --2- 172.21.15.111:0/2006959385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b7c003a00 0x7f1b7c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b94d4c700 1 --2- 172.21.15.111:0/2006959385 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b7c004850 0x7f1b7c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b95d4e700 1 --2- 172.21.15.111:0/2006959385 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b7c005200 0x7f1b7c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b9554d700 1 -- 172.21.15.111:0/2006959385 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b7c005200 msgr2=0x7f1b7c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b9554d700 1 --2- 172.21.15.111:0/2006959385 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b7c005200 0x7f1b7c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b9554d700 1 -- 172.21.15.111:0/2006959385 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b7c004850 msgr2=0x7f1b7c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b9554d700 1 --2- 172.21.15.111:0/2006959385 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b7c004850 0x7f1b7c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b9554d700 1 -- 172.21.15.111:0/2006959385 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c0035b0 con 0x7f1b7c003a00 2023-11-28T13:04:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b95d4e700 1 -- 172.21.15.111:0/2006959385 reap_dead start 2023-11-28T13:04:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b9554d700 1 --2- 172.21.15.111:0/2006959385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b7c003a00 0x7f1b7c003e10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009d10 tx=0x7f1b7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e19c023a0c6850 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c009e60 con 0x7f1b7c003a00 2023-11-28T13:04:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8c00e9a0 con 0x7f1b7c003a00 2023-11-28T13:04:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.193+0000 7f1b87fff700 1 -- 172.21.15.111:0/2006959385 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c00ed10 con 0x7f1b7c003a00 2023-11-28T13:04:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.194+0000 7f1b977b1700 1 -- 172.21.15.111:0/2006959385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b7c003a00 msgr2=0x7f1b7c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.194+0000 7f1b977b1700 1 --2- 172.21.15.111:0/2006959385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b7c003a00 0x7f1b7c003e10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009d10 tx=0x7f1b7c005880 comp rx=0 tx=0).stop 2023-11-28T13:04:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.194+0000 7f1b977b1700 1 -- 172.21.15.111:0/2006959385 shutdown_connections 2023-11-28T13:04:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.194+0000 7f1b977b1700 1 --2- 172.21.15.111:0/2006959385 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b7c003a00 0x7f1b7c003e10 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.194+0000 7f1b977b1700 1 -- 172.21.15.111:0/2006959385 >> 172.21.15.111:0/2006959385 conn(0x7f1b90098ee0 msgr2=0x7f1b9006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.194+0000 7f1b977b1700 1 -- 172.21.15.111:0/2006959385 shutdown_connections 2023-11-28T13:04:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.194+0000 7f1b977b1700 1 -- 172.21.15.111:0/2006959385 wait complete. 2023-11-28T13:04:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.195+0000 7f1b977b1700 1 Processor -- start 2023-11-28T13:04:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.195+0000 7f1b977b1700 1 -- start start 2023-11-28T13:04:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b977b1700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b7c003a00 0x7f1b900c9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b977b1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b900af340 0x7f1b900af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b977b1700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b900c0870 0x7f1b900c0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b977b1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b900afcf0 con 0x7f1b900af340 2023-11-28T13:04:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b977b1700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b900c7d30 con 0x7f1b7c003a00 2023-11-28T13:04:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b977b1700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b900c7f80 con 0x7f1b900c0870 2023-11-28T13:04:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b94d4c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b900af340 0x7f1b900af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b94d4c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b900af340 0x7f1b900af7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37984/0 (socket says 172.21.15.111:37984) 2023-11-28T13:04:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b94d4c700 1 -- 172.21.15.111:0/3009034565 learned_addr learned my addr 172.21.15.111:0/3009034565 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b9554d700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b7c003a00 0x7f1b900c9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b95d4e700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b900c0870 0x7f1b900c0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b94d4c700 1 -- 172.21.15.111:0/3009034565 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b900c0870 msgr2=0x7f1b900c0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b94d4c700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b900c0870 0x7f1b900c0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b94d4c700 1 -- 172.21.15.111:0/3009034565 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b7c003a00 msgr2=0x7f1b900c9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b94d4c700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b7c003a00 0x7f1b900c9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.196+0000 7f1b94d4c700 1 -- 172.21.15.111:0/3009034565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b900cd340 con 0x7f1b900af340 2023-11-28T13:04:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.197+0000 7f1b94d4c700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b900af340 0x7f1b900af7b0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f1b800070e0 tx=0x7f1b800018f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.197+0000 7f1b85ffb700 1 -- 172.21.15.111:0/3009034565 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b80011580 con 0x7f1b900af340 2023-11-28T13:04:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.197+0000 7f1b85ffb700 1 -- 172.21.15.111:0/3009034565 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b80011720 con 0x7f1b900af340 2023-11-28T13:04:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.197+0000 7f1b85ffb700 1 -- 172.21.15.111:0/3009034565 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b80011ad0 con 0x7f1b900af340 2023-11-28T13:04:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.197+0000 7f1b977b1700 1 -- 172.21.15.111:0/3009034565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b900cd060 con 0x7f1b900af340 2023-11-28T13:04:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.197+0000 7f1b977b1700 1 -- 172.21.15.111:0/3009034565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b90143280 con 0x7f1b900af340 2023-11-28T13:04:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.198+0000 7f1b977b1700 1 -- 172.21.15.111:0/3009034565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b90067170 con 0x7f1b900af340 2023-11-28T13:04:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.198+0000 7f1b85ffb700 1 -- 172.21.15.111:0/3009034565 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1b80011c30 con 0x7f1b900af340 2023-11-28T13:04:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.198+0000 7f1b85ffb700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b740682f0 0x7f1b7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:05.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.198+0000 7f1b85ffb700 1 -- 172.21.15.111:0/3009034565 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f1b80085d70 con 0x7f1b900af340 2023-11-28T13:04:05.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.199+0000 7f1b9554d700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b740682f0 0x7f1b7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:05.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.199+0000 7f1b9554d700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b740682f0 0x7f1b7406a7a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0090b0 tx=0x7f1b8c01e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:05.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.203+0000 7f1b85ffb700 1 -- 172.21.15.111:0/3009034565 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1b80052500 con 0x7f1b900af340 2023-11-28T13:04:05.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.423+0000 7f1b977b1700 1 -- 172.21.15.111:0/3009034565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1b900d05a0 con 0x7f1b900af340 2023-11-28T13:04:05.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.423+0000 7f1b85ffb700 1 -- 172.21.15.111:0/3009034565 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f1b900d05a0 con 0x7f1b900af340 2023-11-28T13:04:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:04:05.425 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:04:05.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.425+0000 7f1b977b1700 1 -- 172.21.15.111:0/3009034565 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b740682f0 msgr2=0x7f1b7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:05.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.425+0000 7f1b977b1700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b740682f0 0x7f1b7406a7a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0090b0 tx=0x7f1b8c01e040 comp rx=0 tx=0).stop 2023-11-28T13:04:05.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.425+0000 7f1b977b1700 1 -- 172.21.15.111:0/3009034565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b900af340 msgr2=0x7f1b900af7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:05.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.425+0000 7f1b977b1700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b900af340 0x7f1b900af7b0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f1b800070e0 tx=0x7f1b800018f0 comp rx=0 tx=0).stop 2023-11-28T13:04:05.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.425+0000 7f1b977b1700 1 -- 172.21.15.111:0/3009034565 shutdown_connections 2023-11-28T13:04:05.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.425+0000 7f1b977b1700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1b900c0870 0x7f1b900c0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:05.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.425+0000 7f1b977b1700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1b740682f0 0x7f1b7406a7a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:05.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.425+0000 7f1b977b1700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1b7c003a00 0x7f1b900c9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:05.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.425+0000 7f1b977b1700 1 --2- 172.21.15.111:0/3009034565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b900af340 0x7f1b900af7b0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:05.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.425+0000 7f1b977b1700 1 -- 172.21.15.111:0/3009034565 >> 172.21.15.111:0/3009034565 conn(0x7f1b90098ee0 msgr2=0x7f1b900cd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:05.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.425+0000 7f1b977b1700 1 -- 172.21.15.111:0/3009034565 shutdown_connections 2023-11-28T13:04:05.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:05.425+0000 7f1b977b1700 1 -- 172.21.15.111:0/3009034565 wait complete. 2023-11-28T13:04:05.438 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:04:10.439 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:04:10.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.570+0000 7f06b151d700 1 Processor -- start 2023-11-28T13:04:10.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.570+0000 7f06b151d700 1 -- start start 2023-11-28T13:04:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.571+0000 7f06b151d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06ac0cc550 0x7f06ac0cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.571+0000 7f06b151d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ac15cd90 con 0x7f06ac1604b0 2023-11-28T13:04:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.571+0000 7f06b151d700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ac1613e0 con 0x7f06ac0b0d70 2023-11-28T13:04:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.571+0000 7f06b151d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac160950 con 0x7f06ac0cc550 2023-11-28T13:04:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.571+0000 7f06aa7fc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f06ac1604b0 0x7f06ac0ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38770/0 (socket says 172.21.15.111:38770) 2023-11-28T13:04:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.571+0000 7f06aa7fc700 1 -- 172.21.15.111:0/2625298751 learned_addr learned my addr 172.21.15.111:0/2625298751 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:04:10.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06ab7fe700 1 --2- 172.21.15.111:0/2625298751 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06ac0cc550 0x7f06ac0cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540586172 0 0) 0x7f069c001110 con 0x7f06ac1604b0 2023-11-28T13:04:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06940035b0 con 0x7f06ac1604b0 2023-11-28T13:04:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3597733385 0 0) 0x7f06940035b0 con 0x7f06ac1604b0 2023-11-28T13:04:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06ac0cc550 msgr2=0x7f06ac0cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 --2- 172.21.15.111:0/2625298751 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06ac0cc550 0x7f06ac0cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 >> v1:172.21.15.149:6789/0 conn(0x7f06ac0b0d70 legacy=0x7f06ac0b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac160b40 con 0x7f06ac1604b0 2023-11-28T13:04:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f069c001800 con 0x7f06ac1604b0 2023-11-28T13:04:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 >> v1:172.21.15.111:6789/0 conn(0x7f06ac1604b0 legacy=0x7f06ac0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 --2- 172.21.15.111:0/2625298751 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06ac0cc550 0x7f0694003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 --2- 172.21.15.111:0/2625298751 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0694003f70 0x7f0694004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:10.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 --2- 172.21.15.111:0/2625298751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0694004d90 0x7f0694005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:10.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069400ba10 con 0x7f0694004d90 2023-11-28T13:04:10.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069400bb80 con 0x7f0694003f70 2023-11-28T13:04:10.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069400bd40 con 0x7f06ac0cc550 2023-11-28T13:04:10.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06ab7fe700 1 --2- 172.21.15.111:0/2625298751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0694004d90 0x7f0694005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:10.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06aa7fc700 1 --2- 172.21.15.111:0/2625298751 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0694003f70 0x7f0694004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:10.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06aaffd700 1 --2- 172.21.15.111:0/2625298751 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06ac0cc550 0x7f0694003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:10.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06ab7fe700 1 -- 172.21.15.111:0/2625298751 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06ac0cc550 msgr2=0x7f0694003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:10.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06ab7fe700 1 --2- 172.21.15.111:0/2625298751 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06ac0cc550 0x7f0694003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:10.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06ab7fe700 1 -- 172.21.15.111:0/2625298751 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0694003f70 msgr2=0x7f0694004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:10.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06ab7fe700 1 --2- 172.21.15.111:0/2625298751 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0694003f70 0x7f0694004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:10.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06ab7fe700 1 -- 172.21.15.111:0/2625298751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06940035b0 con 0x7f0694004d90 2023-11-28T13:04:10.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.572+0000 7f06ab7fe700 1 --2- 172.21.15.111:0/2625298751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0694004d90 0x7f0694005200 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f06a4009fc0 tx=0x7f0694005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78dcdef675f25af9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:10.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.573+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a400ec30 con 0x7f0694004d90 2023-11-28T13:04:10.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.573+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a400f040 con 0x7f0694004d90 2023-11-28T13:04:10.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.573+0000 7f06a9ffb700 1 -- 172.21.15.111:0/2625298751 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a4014480 con 0x7f0694004d90 2023-11-28T13:04:10.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.573+0000 7f06b151d700 1 -- 172.21.15.111:0/2625298751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0694004d90 msgr2=0x7f0694005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:10.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.573+0000 7f06b151d700 1 --2- 172.21.15.111:0/2625298751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0694004d90 0x7f0694005200 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f06a4009fc0 tx=0x7f0694005880 comp rx=0 tx=0).stop 2023-11-28T13:04:10.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.573+0000 7f06ab7fe700 1 -- 172.21.15.111:0/2625298751 reap_dead start 2023-11-28T13:04:10.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.573+0000 7f06b151d700 1 -- 172.21.15.111:0/2625298751 shutdown_connections 2023-11-28T13:04:10.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.573+0000 7f06b151d700 1 -- 172.21.15.111:0/2625298751 >> 172.21.15.111:0/2625298751 conn(0x7f06ac098ee0 msgr2=0x7f06ac06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:10.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.573+0000 7f06b151d700 1 -- 172.21.15.111:0/2625298751 shutdown_connections 2023-11-28T13:04:10.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.573+0000 7f06b151d700 1 -- 172.21.15.111:0/2625298751 wait complete. 2023-11-28T13:04:10.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.575+0000 7f06b151d700 1 Processor -- start 2023-11-28T13:04:10.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.575+0000 7f06b151d700 1 -- start start 2023-11-28T13:04:10.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.575+0000 7f06b151d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ac14b1a0 0x7f06ac14e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:10.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.575+0000 7f06b151d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06ac15bb00 0x7f06ac0c5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:10.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.575+0000 7f06b151d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06ac0c5d70 0x7f06ac0c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:10.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.575+0000 7f06b151d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac0c5410 con 0x7f06ac14b1a0 2023-11-28T13:04:10.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.575+0000 7f06b151d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac14e8e0 con 0x7f06ac0c5d70 2023-11-28T13:04:10.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.575+0000 7f06b151d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac0c4790 con 0x7f06ac15bb00 2023-11-28T13:04:10.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.575+0000 7f06aaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ac14b1a0 0x7f06ac14e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:10.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.575+0000 7f06aaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ac14b1a0 0x7f06ac14e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38002/0 (socket says 172.21.15.111:38002) 2023-11-28T13:04:10.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.575+0000 7f06aaffd700 1 -- 172.21.15.111:0/2831105120 learned_addr learned my addr 172.21.15.111:0/2831105120 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:10.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.576+0000 7f06ab7fe700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06ac0c5d70 0x7f06ac0c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:10.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.576+0000 7f06aa7fc700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06ac15bb00 0x7f06ac0c5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:10.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.576+0000 7f06aaffd700 1 -- 172.21.15.111:0/2831105120 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06ac15bb00 msgr2=0x7f06ac0c5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:10.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.576+0000 7f06aaffd700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06ac15bb00 0x7f06ac0c5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:10.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.576+0000 7f06aaffd700 1 -- 172.21.15.111:0/2831105120 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06ac0c5d70 msgr2=0x7f06ac0c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:10.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.576+0000 7f06aaffd700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06ac0c5d70 0x7f06ac0c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:10.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.576+0000 7f06aaffd700 1 -- 172.21.15.111:0/2831105120 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac161100 con 0x7f06ac14b1a0 2023-11-28T13:04:10.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.576+0000 7f06aaffd700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ac14b1a0 0x7f06ac14e3a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f06a000f820 tx=0x7f06a000fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:10.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.576+0000 7f06937fe700 1 -- 172.21.15.111:0/2831105120 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a00052e0 con 0x7f06ac14b1a0 2023-11-28T13:04:10.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.576+0000 7f06937fe700 1 -- 172.21.15.111:0/2831105120 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a0005440 con 0x7f06ac14b1a0 2023-11-28T13:04:10.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.576+0000 7f06b151d700 1 -- 172.21.15.111:0/2831105120 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ac160e20 con 0x7f06ac14b1a0 2023-11-28T13:04:10.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.576+0000 7f06937fe700 1 -- 172.21.15.111:0/2831105120 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a00055b0 con 0x7f06ac14b1a0 2023-11-28T13:04:10.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.576+0000 7f06b151d700 1 -- 172.21.15.111:0/2831105120 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ac0af340 con 0x7f06ac14b1a0 2023-11-28T13:04:10.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.577+0000 7f06b151d700 1 -- 172.21.15.111:0/2831105120 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ac0c5550 con 0x7f06ac14b1a0 2023-11-28T13:04:10.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.577+0000 7f06937fe700 1 -- 172.21.15.111:0/2831105120 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f06a0003910 con 0x7f06ac14b1a0 2023-11-28T13:04:10.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.578+0000 7f06937fe700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06980682f0 0x7f069806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:10.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.578+0000 7f06937fe700 1 -- 172.21.15.111:0/2831105120 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f06a0085e50 con 0x7f06ac14b1a0 2023-11-28T13:04:10.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.578+0000 7f06aa7fc700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06980682f0 0x7f069806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:10.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.578+0000 7f06aa7fc700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06980682f0 0x7f069806a7a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f069c001050 tx=0x7f069c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:10.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.582+0000 7f06937fe700 1 -- 172.21.15.111:0/2831105120 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f06ac0c5550 con 0x7f06ac14b1a0 2023-11-28T13:04:10.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.808+0000 7f06b151d700 1 -- 172.21.15.111:0/2831105120 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f06ac0c5550 con 0x7f06ac14b1a0 2023-11-28T13:04:10.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.808+0000 7f06937fe700 1 -- 172.21.15.111:0/2831105120 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f06ac0c5550 con 0x7f06ac14b1a0 2023-11-28T13:04:10.809 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:04:10.809 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:04:10.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.811+0000 7f06b151d700 1 -- 172.21.15.111:0/2831105120 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06980682f0 msgr2=0x7f069806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:10.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.811+0000 7f06b151d700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06980682f0 0x7f069806a7a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f069c001050 tx=0x7f069c009040 comp rx=0 tx=0).stop 2023-11-28T13:04:10.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.811+0000 7f06b151d700 1 -- 172.21.15.111:0/2831105120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ac14b1a0 msgr2=0x7f06ac14e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:10.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.811+0000 7f06b151d700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ac14b1a0 0x7f06ac14e3a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f06a000f820 tx=0x7f06a000fb30 comp rx=0 tx=0).stop 2023-11-28T13:04:10.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.811+0000 7f06b151d700 1 -- 172.21.15.111:0/2831105120 shutdown_connections 2023-11-28T13:04:10.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.811+0000 7f06b151d700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f06980682f0 0x7f069806a7a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:10.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.811+0000 7f06b151d700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f06ac0c5d70 0x7f06ac0c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:10.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.811+0000 7f06b151d700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06ac14b1a0 0x7f06ac14e3a0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:10.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.811+0000 7f06b151d700 1 --2- 172.21.15.111:0/2831105120 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f06ac15bb00 0x7f06ac0c5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:10.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.811+0000 7f06b151d700 1 -- 172.21.15.111:0/2831105120 >> 172.21.15.111:0/2831105120 conn(0x7f06ac098ee0 msgr2=0x7f06ac0ae680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:10.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.811+0000 7f06b151d700 1 -- 172.21.15.111:0/2831105120 shutdown_connections 2023-11-28T13:04:10.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:10.811+0000 7f06b151d700 1 -- 172.21.15.111:0/2831105120 wait complete. 2023-11-28T13:04:10.823 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:04:15.824 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:04:15.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.955+0000 7f2f16b12700 1 Processor -- start 2023-11-28T13:04:15.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.955+0000 7f2f16b12700 1 -- start start 2023-11-28T13:04:15.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.956+0000 7f2f16b12700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2f100cc5a0 0x7f2f100d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:15.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.956+0000 7f2f16b12700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f100d05a0 con 0x7f2f100cdef0 2023-11-28T13:04:15.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.956+0000 7f2f16b12700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f100ca260 con 0x7f2f100cfc90 2023-11-28T13:04:15.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.956+0000 7f2f16b12700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f100ccaa0 con 0x7f2f100cc5a0 2023-11-28T13:04:15.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.956+0000 7f2f150af700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2f100cc5a0 0x7f2f100d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:15.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.956+0000 7f2f0ffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2f100cdef0 0x7f2f100ca6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38790/0 (socket says 172.21.15.111:38790) 2023-11-28T13:04:15.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.956+0000 7f2f0ffff700 1 -- 172.21.15.111:0/1942605658 learned_addr learned my addr 172.21.15.111:0/1942605658 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:04:15.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.956+0000 7f2f150af700 1 -- 172.21.15.111:0/1942605658 >> v1:172.21.15.149:6789/0 conn(0x7f2f100cfc90 legacy=0x7f2f100cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:15.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.956+0000 7f2f150af700 1 -- 172.21.15.111:0/1942605658 >> v1:172.21.15.111:6789/0 conn(0x7f2f100cdef0 legacy=0x7f2f100ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:15.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.956+0000 7f2f150af700 1 -- 172.21.15.111:0/1942605658 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f100ccd80 con 0x7f2f100cc5a0 2023-11-28T13:04:15.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.956+0000 7f2f150af700 1 --2- 172.21.15.111:0/1942605658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f100cc5a0 0x7f2f100d0060 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009d90 tx=0x7f2f0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acfa01e0ef3d6ca9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:15.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.957+0000 7f2f0f7fe700 1 -- 172.21.15.111:0/1942605658 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0800bdd0 con 0x7f2f100cc5a0 2023-11-28T13:04:15.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.957+0000 7f2f0f7fe700 1 -- 172.21.15.111:0/1942605658 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f08011460 con 0x7f2f100cc5a0 2023-11-28T13:04:15.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.957+0000 7f2f0f7fe700 1 -- 172.21.15.111:0/1942605658 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08011810 con 0x7f2f100cc5a0 2023-11-28T13:04:15.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.957+0000 7f2f16b12700 1 -- 172.21.15.111:0/1942605658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f100cc5a0 msgr2=0x7f2f100d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:15.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.957+0000 7f2f16b12700 1 --2- 172.21.15.111:0/1942605658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f100cc5a0 0x7f2f100d0060 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009d90 tx=0x7f2f0800b330 comp rx=0 tx=0).stop 2023-11-28T13:04:15.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.957+0000 7f2f16b12700 1 -- 172.21.15.111:0/1942605658 shutdown_connections 2023-11-28T13:04:15.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.957+0000 7f2f16b12700 1 --2- 172.21.15.111:0/1942605658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f100cc5a0 0x7f2f100d0060 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:15.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.957+0000 7f2f16b12700 1 -- 172.21.15.111:0/1942605658 >> 172.21.15.111:0/1942605658 conn(0x7f2f10098ee0 msgr2=0x7f2f1006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:15.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.957+0000 7f2f16b12700 1 -- 172.21.15.111:0/1942605658 shutdown_connections 2023-11-28T13:04:15.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.957+0000 7f2f16b12700 1 -- 172.21.15.111:0/1942605658 wait complete. 2023-11-28T13:04:15.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.958+0000 7f2f16b12700 1 Processor -- start 2023-11-28T13:04:15.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.958+0000 7f2f16b12700 1 -- start start 2023-11-28T13:04:15.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.959+0000 7f2f16b12700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f100cc5a0 0x7f2f100b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:15.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.959+0000 7f2f16b12700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f100cdef0 0x7f2f100c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:15.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.959+0000 7f2f16b12700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f100cfc90 0x7f2f100c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:15.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.959+0000 7f2f16b12700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f100c7db0 con 0x7f2f100cdef0 2023-11-28T13:04:15.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.959+0000 7f2f16b12700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f100b2620 con 0x7f2f100cc5a0 2023-11-28T13:04:15.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.959+0000 7f2f16b12700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f100c7230 con 0x7f2f100cfc90 2023-11-28T13:04:15.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.959+0000 7f2f0ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f100cdef0 0x7f2f100c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:15.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.959+0000 7f2f0ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f100cdef0 0x7f2f100c82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38018/0 (socket says 172.21.15.111:38018) 2023-11-28T13:04:15.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.959+0000 7f2f0ffff700 1 -- 172.21.15.111:0/3291040201 learned_addr learned my addr 172.21.15.111:0/3291040201 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:15.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.959+0000 7f2f148ae700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f100cc5a0 0x7f2f100b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:15.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.960+0000 7f2f150af700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f100cfc90 0x7f2f100c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:15.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.960+0000 7f2f0ffff700 1 -- 172.21.15.111:0/3291040201 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f100cfc90 msgr2=0x7f2f100c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.960+0000 7f2f0ffff700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f100cfc90 0x7f2f100c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.960+0000 7f2f0ffff700 1 -- 172.21.15.111:0/3291040201 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f100cc5a0 msgr2=0x7f2f100b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.960+0000 7f2f0ffff700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f100cc5a0 0x7f2f100b20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.960+0000 7f2f0ffff700 1 -- 172.21.15.111:0/3291040201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f08009a70 con 0x7f2f100cdef0 2023-11-28T13:04:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.960+0000 7f2f150af700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f100cfc90 0x7f2f100c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:04:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.960+0000 7f2f0ffff700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f100cdef0 0x7f2f100c82c0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00c450 tx=0x7f2efc00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.960+0000 7f2f0d7fa700 1 -- 172.21.15.111:0/3291040201 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc005d70 con 0x7f2f100cdef0 2023-11-28T13:04:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.960+0000 7f2f0d7fa700 1 -- 172.21.15.111:0/3291040201 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2efc0070d0 con 0x7f2f100cdef0 2023-11-28T13:04:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.960+0000 7f2f16b12700 1 -- 172.21.15.111:0/3291040201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f100cd340 con 0x7f2f100cdef0 2023-11-28T13:04:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.960+0000 7f2f0d7fa700 1 -- 172.21.15.111:0/3291040201 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc007440 con 0x7f2f100cdef0 2023-11-28T13:04:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.960+0000 7f2f16b12700 1 -- 172.21.15.111:0/3291040201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f100cd060 con 0x7f2f100cdef0 2023-11-28T13:04:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.961+0000 7f2f16b12700 1 -- 172.21.15.111:0/3291040201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f10143280 con 0x7f2f100cdef0 2023-11-28T13:04:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.961+0000 7f2f0d7fa700 1 -- 172.21.15.111:0/3291040201 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2efc019030 con 0x7f2f100cdef0 2023-11-28T13:04:15.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.962+0000 7f2f0d7fa700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2ef40682f0 0x7f2ef406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:15.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.962+0000 7f2f148ae700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2ef40682f0 0x7f2ef406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:15.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.962+0000 7f2f0d7fa700 1 -- 172.21.15.111:0/3291040201 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f2efc084e90 con 0x7f2f100cdef0 2023-11-28T13:04:15.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.962+0000 7f2f148ae700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2ef40682f0 0x7f2ef406a7a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2f04000f60 tx=0x7f2f04009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:15.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:15.966+0000 7f2f0d7fa700 1 -- 172.21.15.111:0/3291040201 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2efc04b5e0 con 0x7f2f100cdef0 2023-11-28T13:04:16.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.188+0000 7f2f16b12700 1 -- 172.21.15.111:0/3291040201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2f100d05a0 con 0x7f2f100cdef0 2023-11-28T13:04:16.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.188+0000 7f2f0d7fa700 1 -- 172.21.15.111:0/3291040201 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f2efc0170c0 con 0x7f2f100cdef0 2023-11-28T13:04:16.189 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:04:16.189 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:04:16.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.190+0000 7f2f16b12700 1 -- 172.21.15.111:0/3291040201 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2ef40682f0 msgr2=0x7f2ef406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:16.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.190+0000 7f2f16b12700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2ef40682f0 0x7f2ef406a7a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2f04000f60 tx=0x7f2f04009040 comp rx=0 tx=0).stop 2023-11-28T13:04:16.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.190+0000 7f2f16b12700 1 -- 172.21.15.111:0/3291040201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f100cdef0 msgr2=0x7f2f100c82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:16.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.190+0000 7f2f16b12700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f100cdef0 0x7f2f100c82c0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00c450 tx=0x7f2efc00c810 comp rx=0 tx=0).stop 2023-11-28T13:04:16.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.190+0000 7f2f16b12700 1 -- 172.21.15.111:0/3291040201 shutdown_connections 2023-11-28T13:04:16.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.190+0000 7f2f16b12700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f100cfc90 0x7f2f100c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:16.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.190+0000 7f2f16b12700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2ef40682f0 0x7f2ef406a7a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:16.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.190+0000 7f2f16b12700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f100cc5a0 0x7f2f100b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:16.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.190+0000 7f2f16b12700 1 --2- 172.21.15.111:0/3291040201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f100cdef0 0x7f2f100c82c0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:16.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.190+0000 7f2f16b12700 1 -- 172.21.15.111:0/3291040201 >> 172.21.15.111:0/3291040201 conn(0x7f2f10098ee0 msgr2=0x7f2f100acec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:16.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.190+0000 7f2f16b12700 1 -- 172.21.15.111:0/3291040201 shutdown_connections 2023-11-28T13:04:16.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:16.190+0000 7f2f16b12700 1 -- 172.21.15.111:0/3291040201 wait complete. 2023-11-28T13:04:16.202 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:04:21.203 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:04:21.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.334+0000 7f13c07fc700 1 Processor -- start 2023-11-28T13:04:21.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.334+0000 7f13c07fc700 1 -- start start 2023-11-28T13:04:21.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.335+0000 7f13c07fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f13b815a5d0 0x7f13b80ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:21.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.335+0000 7f13c07fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b80b0160 0x7f13b80ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:21.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.335+0000 7f13c07fc700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b80ae8b0 con 0x7f13b80cb2d0 2023-11-28T13:04:21.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.335+0000 7f13c07fc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b80cc1b0 con 0x7f13b815a5d0 2023-11-28T13:04:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.335+0000 7f13c07fc700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b80cc390 con 0x7f13b80b0160 2023-11-28T13:04:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.336+0000 7f13be598700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f13b815a5d0 0x7f13b80ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.336+0000 7f13be598700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f13b815a5d0 0x7f13b80ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38024/0 (socket says 172.21.15.111:38024) 2023-11-28T13:04:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.336+0000 7f13be598700 1 -- 172.21.15.111:0/1110498527 learned_addr learned my addr 172.21.15.111:0/1110498527 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.336+0000 7f13bdd97700 1 --2- 172.21.15.111:0/1110498527 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b80b0160 0x7f13b80ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.336+0000 7f13be598700 1 -- 172.21.15.111:0/1110498527 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b80b0160 msgr2=0x7f13b80ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.336+0000 7f13be598700 1 --2- 172.21.15.111:0/1110498527 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b80b0160 0x7f13b80ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.336+0000 7f13be598700 1 -- 172.21.15.111:0/1110498527 >> v1:172.21.15.149:6789/0 conn(0x7f13b80cb2d0 legacy=0x7f13b815e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.336+0000 7f13be598700 1 -- 172.21.15.111:0/1110498527 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b80cc670 con 0x7f13b815a5d0 2023-11-28T13:04:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.336+0000 7f13be598700 1 --2- 172.21.15.111:0/1110498527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b815a5d0 0x7f13b80ce850 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f13a8006fd0 tx=0x7f13a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5496e939ad2cbfe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.336+0000 7f13bd596700 1 -- 172.21.15.111:0/1110498527 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8005e30 con 0x7f13b815a5d0 2023-11-28T13:04:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.336+0000 7f13bd596700 1 -- 172.21.15.111:0/1110498527 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a8004030 con 0x7f13b815a5d0 2023-11-28T13:04:21.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.336+0000 7f13bd596700 1 -- 172.21.15.111:0/1110498527 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a80043e0 con 0x7f13b815a5d0 2023-11-28T13:04:21.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.337+0000 7f13c07fc700 1 -- 172.21.15.111:0/1110498527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b815a5d0 msgr2=0x7f13b80ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:21.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.337+0000 7f13c07fc700 1 --2- 172.21.15.111:0/1110498527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b815a5d0 0x7f13b80ce850 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f13a8006fd0 tx=0x7f13a8009bd0 comp rx=0 tx=0).stop 2023-11-28T13:04:21.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.337+0000 7f13c07fc700 1 -- 172.21.15.111:0/1110498527 shutdown_connections 2023-11-28T13:04:21.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.337+0000 7f13c07fc700 1 --2- 172.21.15.111:0/1110498527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b815a5d0 0x7f13b80ce850 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:21.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.337+0000 7f13c07fc700 1 --2- 172.21.15.111:0/1110498527 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b80b0160 0x7f13b80ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:21.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.337+0000 7f13c07fc700 1 -- 172.21.15.111:0/1110498527 >> 172.21.15.111:0/1110498527 conn(0x7f13b809a040 msgr2=0x7f13b80d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:21.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.337+0000 7f13c07fc700 1 -- 172.21.15.111:0/1110498527 shutdown_connections 2023-11-28T13:04:21.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.337+0000 7f13c07fc700 1 -- 172.21.15.111:0/1110498527 wait complete. 2023-11-28T13:04:21.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.338+0000 7f13c07fc700 1 Processor -- start 2023-11-28T13:04:21.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.338+0000 7f13c07fc700 1 -- start start 2023-11-28T13:04:21.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13c07fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b80b0160 0x7f13b8154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:21.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13c07fc700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f13b80cb2d0 0x7f13b80c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:21.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13c07fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b815a5d0 0x7f13b80ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:21.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13c07fc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b81550b0 con 0x7f13b815a5d0 2023-11-28T13:04:21.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13c07fc700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b80c7130 con 0x7f13b80cb2d0 2023-11-28T13:04:21.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13c07fc700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b80c75c0 con 0x7f13b80b0160 2023-11-28T13:04:21.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13bed99700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b815a5d0 0x7f13b80ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:21.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13bed99700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b815a5d0 0x7f13b80ca460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38034/0 (socket says 172.21.15.111:38034) 2023-11-28T13:04:21.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13bdd97700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f13b80cb2d0 0x7f13b80c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:21.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13bed99700 1 -- 172.21.15.111:0/3152668391 learned_addr learned my addr 172.21.15.111:0/3152668391 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:21.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13be598700 1 --2- 172.21.15.111:0/3152668391 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b80b0160 0x7f13b8154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:21.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13bed99700 1 -- 172.21.15.111:0/3152668391 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b80b0160 msgr2=0x7f13b8154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:21.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13bed99700 1 --2- 172.21.15.111:0/3152668391 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b80b0160 0x7f13b8154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:21.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13bed99700 1 -- 172.21.15.111:0/3152668391 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f13b80cb2d0 msgr2=0x7f13b80c9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:21.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13bed99700 1 --2- 172.21.15.111:0/3152668391 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f13b80cb2d0 0x7f13b80c9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:21.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13bed99700 1 -- 172.21.15.111:0/3152668391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a8009500 con 0x7f13b815a5d0 2023-11-28T13:04:21.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.339+0000 7f13bed99700 1 --2- 172.21.15.111:0/3152668391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b815a5d0 0x7f13b80ca460 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f13b000cbf0 tx=0x7f13b000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:21.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.340+0000 7f13aeffd700 1 -- 172.21.15.111:0/3152668391 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b0003c00 con 0x7f13b815a5d0 2023-11-28T13:04:21.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.340+0000 7f13aeffd700 1 -- 172.21.15.111:0/3152668391 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b0003d60 con 0x7f13b815a5d0 2023-11-28T13:04:21.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.340+0000 7f13c07fc700 1 -- 172.21.15.111:0/3152668391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b80ccc30 con 0x7f13b815a5d0 2023-11-28T13:04:21.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.340+0000 7f13aeffd700 1 -- 172.21.15.111:0/3152668391 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b000aaa0 con 0x7f13b815a5d0 2023-11-28T13:04:21.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.340+0000 7f13c07fc700 1 -- 172.21.15.111:0/3152668391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b80cc950 con 0x7f13b815a5d0 2023-11-28T13:04:21.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.341+0000 7f13c07fc700 1 -- 172.21.15.111:0/3152668391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b80ccf10 con 0x7f13b815a5d0 2023-11-28T13:04:21.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.341+0000 7f13aeffd700 1 -- 172.21.15.111:0/3152668391 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f13b0010050 con 0x7f13b815a5d0 2023-11-28T13:04:21.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.341+0000 7f13aeffd700 1 --2- 172.21.15.111:0/3152668391 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f139c0682f0 0x7f139c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:21.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.341+0000 7f13aeffd700 1 -- 172.21.15.111:0/3152668391 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f13b0085cb0 con 0x7f13b815a5d0 2023-11-28T13:04:21.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.341+0000 7f13be598700 1 --2- 172.21.15.111:0/3152668391 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f139c0682f0 0x7f139c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:21.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.342+0000 7f13be598700 1 --2- 172.21.15.111:0/3152668391 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f139c0682f0 0x7f139c06a7a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009130 tx=0x7f13a8008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:21.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.346+0000 7f13aeffd700 1 -- 172.21.15.111:0/3152668391 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f13b0052310 con 0x7f13b815a5d0 2023-11-28T13:04:21.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.572+0000 7f13c07fc700 1 -- 172.21.15.111:0/3152668391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f13b8067170 con 0x7f13b815a5d0 2023-11-28T13:04:21.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.573+0000 7f13aeffd700 1 -- 172.21.15.111:0/3152668391 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f13b0055930 con 0x7f13b815a5d0 2023-11-28T13:04:21.574 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:04:21.574 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:04:21.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.574+0000 7f13c07fc700 1 -- 172.21.15.111:0/3152668391 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f139c0682f0 msgr2=0x7f139c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:21.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.574+0000 7f13c07fc700 1 --2- 172.21.15.111:0/3152668391 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f139c0682f0 0x7f139c06a7a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009130 tx=0x7f13a8008070 comp rx=0 tx=0).stop 2023-11-28T13:04:21.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.574+0000 7f13c07fc700 1 -- 172.21.15.111:0/3152668391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b815a5d0 msgr2=0x7f13b80ca460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:21.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.574+0000 7f13c07fc700 1 --2- 172.21.15.111:0/3152668391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b815a5d0 0x7f13b80ca460 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f13b000cbf0 tx=0x7f13b000cf00 comp rx=0 tx=0).stop 2023-11-28T13:04:21.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.574+0000 7f13c07fc700 1 -- 172.21.15.111:0/3152668391 shutdown_connections 2023-11-28T13:04:21.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.574+0000 7f13c07fc700 1 --2- 172.21.15.111:0/3152668391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13b815a5d0 0x7f13b80ca460 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:21.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.574+0000 7f13c07fc700 1 --2- 172.21.15.111:0/3152668391 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f13b80b0160 0x7f13b8154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:21.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.574+0000 7f13c07fc700 1 --2- 172.21.15.111:0/3152668391 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f139c0682f0 0x7f139c06a7a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:21.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.574+0000 7f13c07fc700 1 --2- 172.21.15.111:0/3152668391 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f13b80cb2d0 0x7f13b80c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:21.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.575+0000 7f13c07fc700 1 -- 172.21.15.111:0/3152668391 >> 172.21.15.111:0/3152668391 conn(0x7f13b809a040 msgr2=0x7f13b806f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:21.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.575+0000 7f13c07fc700 1 -- 172.21.15.111:0/3152668391 shutdown_connections 2023-11-28T13:04:21.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:21.575+0000 7f13c07fc700 1 -- 172.21.15.111:0/3152668391 wait complete. 2023-11-28T13:04:21.588 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:04:26.588 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:04:26.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.720+0000 7fc9faf47700 1 Processor -- start 2023-11-28T13:04:26.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.720+0000 7fc9faf47700 1 -- start start 2023-11-28T13:04:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.721+0000 7fc9faf47700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc9f40b08e0 0x7fc9f40caa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.721+0000 7fc9faf47700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f40b0d40 con 0x7fc9f40b0440 2023-11-28T13:04:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.721+0000 7fc9faf47700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f40cc750 con 0x7fc9f415c6c0 2023-11-28T13:04:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.721+0000 7fc9faf47700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9f40b0260 con 0x7fc9f40b08e0 2023-11-28T13:04:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.721+0000 7fc9f3fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc9f40b0440 0x7fc9f40ca2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38818/0 (socket says 172.21.15.111:38818) 2023-11-28T13:04:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.721+0000 7fc9f3fff700 1 -- 172.21.15.111:0/192425194 learned_addr learned my addr 172.21.15.111:0/192425194 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:04:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.721+0000 7fc9f94e4700 1 --2- 172.21.15.111:0/192425194 >> v2:172.21.15.149:3300/0 conn(0x7fc9f40b08e0 0x7fc9f40caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.721+0000 7fc9f37fe700 1 -- 172.21.15.111:0/192425194 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462540408 0 0) 0x7fc9f40b0d40 con 0x7fc9f40b0440 2023-11-28T13:04:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.721+0000 7fc9f37fe700 1 -- 172.21.15.111:0/192425194 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e40035b0 con 0x7fc9f40b0440 2023-11-28T13:04:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.721+0000 7fc9f94e4700 1 -- 172.21.15.111:0/192425194 >> v1:172.21.15.149:6789/0 conn(0x7fc9f415c6c0 legacy=0x7fc9f40afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.721+0000 7fc9f94e4700 1 -- 172.21.15.111:0/192425194 >> v1:172.21.15.111:6789/0 conn(0x7fc9f40b0440 legacy=0x7fc9f40ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.721+0000 7fc9f94e4700 1 -- 172.21.15.111:0/192425194 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f40cca30 con 0x7fc9f40b08e0 2023-11-28T13:04:26.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.721+0000 7fc9f94e4700 1 --2- 172.21.15.111:0/192425194 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc9f40b08e0 0x7fc9f40caa00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec009d90 tx=0x7fc9ec00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2b620365b23f7c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:26.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.722+0000 7fc9f37fe700 1 -- 172.21.15.111:0/192425194 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec00e070 con 0x7fc9f40b08e0 2023-11-28T13:04:26.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.722+0000 7fc9f37fe700 1 -- 172.21.15.111:0/192425194 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ec00bb20 con 0x7fc9f40b08e0 2023-11-28T13:04:26.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.722+0000 7fc9f37fe700 1 -- 172.21.15.111:0/192425194 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec012430 con 0x7fc9f40b08e0 2023-11-28T13:04:26.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.722+0000 7fc9faf47700 1 -- 172.21.15.111:0/192425194 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc9f40b08e0 msgr2=0x7fc9f40caa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:26.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.722+0000 7fc9faf47700 1 --2- 172.21.15.111:0/192425194 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc9f40b08e0 0x7fc9f40caa00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec009d90 tx=0x7fc9ec00b330 comp rx=0 tx=0).stop 2023-11-28T13:04:26.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.722+0000 7fc9faf47700 1 -- 172.21.15.111:0/192425194 shutdown_connections 2023-11-28T13:04:26.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.722+0000 7fc9faf47700 1 --2- 172.21.15.111:0/192425194 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc9f40b08e0 0x7fc9f40caa00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:26.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.722+0000 7fc9faf47700 1 -- 172.21.15.111:0/192425194 >> 172.21.15.111:0/192425194 conn(0x7fc9f4098ee0 msgr2=0x7fc9f406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:26.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.722+0000 7fc9faf47700 1 -- 172.21.15.111:0/192425194 shutdown_connections 2023-11-28T13:04:26.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.722+0000 7fc9faf47700 1 -- 172.21.15.111:0/192425194 wait complete. 2023-11-28T13:04:26.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.723+0000 7fc9faf47700 1 Processor -- start 2023-11-28T13:04:26.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.723+0000 7fc9faf47700 1 -- start start 2023-11-28T13:04:26.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.724+0000 7fc9faf47700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc9f40b0440 0x7fc9f4157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:26.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.724+0000 7fc9faf47700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f40b08e0 0x7fc9f4155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:26.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.724+0000 7fc9faf47700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc9f415c6c0 0x7fc9f40b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:26.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.724+0000 7fc9faf47700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f4156fa0 con 0x7fc9f40b08e0 2023-11-28T13:04:26.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.724+0000 7fc9faf47700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f4156390 con 0x7fc9f415c6c0 2023-11-28T13:04:26.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.724+0000 7fc9faf47700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f4156670 con 0x7fc9f40b0440 2023-11-28T13:04:26.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.724+0000 7fc9f3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f40b08e0 0x7fc9f4155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:26.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.724+0000 7fc9f3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f40b08e0 0x7fc9f4155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38046/0 (socket says 172.21.15.111:38046) 2023-11-28T13:04:26.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.724+0000 7fc9f3fff700 1 -- 172.21.15.111:0/2404022787 learned_addr learned my addr 172.21.15.111:0/2404022787 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:26.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.724+0000 7fc9f94e4700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc9f415c6c0 0x7fc9f40b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:26.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.724+0000 7fc9f8ce3700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc9f40b0440 0x7fc9f4157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:26.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.725+0000 7fc9f3fff700 1 -- 172.21.15.111:0/2404022787 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc9f40b0440 msgr2=0x7fc9f4157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:26.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.725+0000 7fc9f3fff700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc9f40b0440 0x7fc9f4157430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:26.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.725+0000 7fc9f3fff700 1 -- 172.21.15.111:0/2404022787 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc9f415c6c0 msgr2=0x7fc9f40b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:26.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.725+0000 7fc9f3fff700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc9f415c6c0 0x7fc9f40b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:26.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.725+0000 7fc9f3fff700 1 -- 172.21.15.111:0/2404022787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ec009a70 con 0x7fc9f40b08e0 2023-11-28T13:04:26.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.725+0000 7fc9f3fff700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f40b08e0 0x7fc9f4155c90 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0001eb0 tx=0x7fc9e000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:26.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.725+0000 7fc9f17fa700 1 -- 172.21.15.111:0/2404022787 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0005d70 con 0x7fc9f40b08e0 2023-11-28T13:04:26.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.725+0000 7fc9f17fa700 1 -- 172.21.15.111:0/2404022787 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e00070d0 con 0x7fc9f40b08e0 2023-11-28T13:04:26.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.725+0000 7fc9f17fa700 1 -- 172.21.15.111:0/2404022787 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0007460 con 0x7fc9f40b08e0 2023-11-28T13:04:26.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.725+0000 7fc9faf47700 1 -- 172.21.15.111:0/2404022787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f40ccff0 con 0x7fc9f40b08e0 2023-11-28T13:04:26.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.725+0000 7fc9faf47700 1 -- 172.21.15.111:0/2404022787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f40ccd10 con 0x7fc9f40b08e0 2023-11-28T13:04:26.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.726+0000 7fc9faf47700 1 -- 172.21.15.111:0/2404022787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f40cd2d0 con 0x7fc9f40b08e0 2023-11-28T13:04:26.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.726+0000 7fc9f17fa700 1 -- 172.21.15.111:0/2404022787 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc9e0019030 con 0x7fc9f40b08e0 2023-11-28T13:04:26.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.727+0000 7fc9f17fa700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc9d80682f0 0x7fc9d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:26.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.727+0000 7fc9f17fa700 1 -- 172.21.15.111:0/2404022787 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fc9e00850c0 con 0x7fc9f40b08e0 2023-11-28T13:04:26.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.727+0000 7fc9f8ce3700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc9d80682f0 0x7fc9d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:26.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.727+0000 7fc9f8ce3700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc9d80682f0 0x7fc9d806a7a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8007fd0 tx=0x7fc9e8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:26.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.731+0000 7fc9f17fa700 1 -- 172.21.15.111:0/2404022787 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc9e00517a0 con 0x7fc9f40b08e0 2023-11-28T13:04:26.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.956+0000 7fc9faf47700 1 -- 172.21.15.111:0/2404022787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc9f4067170 con 0x7fc9f40b08e0 2023-11-28T13:04:26.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.956+0000 7fc9f17fa700 1 -- 172.21.15.111:0/2404022787 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7fc9e00170c0 con 0x7fc9f40b08e0 2023-11-28T13:04:26.958 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:04:26.958 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:04:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.958+0000 7fc9faf47700 1 -- 172.21.15.111:0/2404022787 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc9d80682f0 msgr2=0x7fc9d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.958+0000 7fc9faf47700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc9d80682f0 0x7fc9d806a7a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8007fd0 tx=0x7fc9e8009040 comp rx=0 tx=0).stop 2023-11-28T13:04:26.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.958+0000 7fc9faf47700 1 -- 172.21.15.111:0/2404022787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f40b08e0 msgr2=0x7fc9f4155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.958+0000 7fc9faf47700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f40b08e0 0x7fc9f4155c90 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0001eb0 tx=0x7fc9e000c7b0 comp rx=0 tx=0).stop 2023-11-28T13:04:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.959+0000 7fc9faf47700 1 -- 172.21.15.111:0/2404022787 shutdown_connections 2023-11-28T13:04:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.959+0000 7fc9faf47700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc9d80682f0 0x7fc9d806a7a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.959+0000 7fc9faf47700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc9f415c6c0 0x7fc9f40b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.959+0000 7fc9faf47700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc9f40b0440 0x7fc9f4157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.959+0000 7fc9faf47700 1 --2- 172.21.15.111:0/2404022787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f40b08e0 0x7fc9f4155c90 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.959+0000 7fc9faf47700 1 -- 172.21.15.111:0/2404022787 >> 172.21.15.111:0/2404022787 conn(0x7fc9f4098ee0 msgr2=0x7fc9f41586a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.959+0000 7fc9faf47700 1 -- 172.21.15.111:0/2404022787 shutdown_connections 2023-11-28T13:04:26.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:26.959+0000 7fc9faf47700 1 -- 172.21.15.111:0/2404022787 wait complete. 2023-11-28T13:04:26.972 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:04:29.365 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:04:29.368 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:04:31.973 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:04:32.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.103+0000 7f2c9369e700 1 Processor -- start 2023-11-28T13:04:32.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.103+0000 7f2c9369e700 1 -- start start 2023-11-28T13:04:32.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c9369e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c06a980 0x7f2c8c06f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:32.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c9369e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c8c06b490 con 0x7f2c8c0cb790 2023-11-28T13:04:32.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c9369e700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c8c06f680 con 0x7f2c8c15a3c0 2023-11-28T13:04:32.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c9369e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c15a790 con 0x7f2c8c06a980 2023-11-28T13:04:32.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c90c39700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2c8c0cb790 0x7f2c8c0712e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38832/0 (socket says 172.21.15.111:38832) 2023-11-28T13:04:32.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c90c39700 1 -- 172.21.15.111:0/806338277 learned_addr learned my addr 172.21.15.111:0/806338277 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:04:32.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c91c3b700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c06a980 0x7f2c8c06f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:32.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153242092 0 0) 0x7f2c8c06b490 con 0x7f2c8c0cb790 2023-11-28T13:04:32.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c780035b0 con 0x7f2c8c0cb790 2023-11-28T13:04:32.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4261458003 0 0) 0x7f2c780035b0 con 0x7f2c8c0cb790 2023-11-28T13:04:32.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c06a980 msgr2=0x7f2c8c06f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:32.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c83fff700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c06a980 0x7f2c8c06f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:32.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 >> v1:172.21.15.149:6789/0 conn(0x7f2c8c15a3c0 legacy=0x7f2c8c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c06b6e0 con 0x7f2c8c0cb790 2023-11-28T13:04:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f2c7c001670 con 0x7f2c8c0cb790 2023-11-28T13:04:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.104+0000 7f2c91c3b700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c06a980 0x7f2c8c06f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:04:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 >> v1:172.21.15.111:6789/0 conn(0x7f2c8c0cb790 legacy=0x7f2c8c0712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c83fff700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c78003a30 0x7f2c78003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c83fff700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c78004850 0x7f2c78004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c83fff700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c06a980 0x7f2c78005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c15a790 con 0x7f2c78003a30 2023-11-28T13:04:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7800bba0 con 0x7f2c78004850 2023-11-28T13:04:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7800bd60 con 0x7f2c8c06a980 2023-11-28T13:04:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c9143a700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c78003a30 0x7f2c78003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:32.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c90c39700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c78004850 0x7f2c78004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:32.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c91c3b700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c06a980 0x7f2c78005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:32.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c9143a700 1 -- 172.21.15.111:0/806338277 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c06a980 msgr2=0x7f2c78005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:32.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c9143a700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c06a980 0x7f2c78005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:32.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c9143a700 1 -- 172.21.15.111:0/806338277 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c78004850 msgr2=0x7f2c78004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:32.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c9143a700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c78004850 0x7f2c78004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:32.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c9143a700 1 -- 172.21.15.111:0/806338277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c780035b0 con 0x7f2c78003a30 2023-11-28T13:04:32.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c91c3b700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c06a980 0x7f2c78005290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:04:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c9143a700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c78003a30 0x7f2c78003e40 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f2c880017c0 tx=0x7f2c78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d71e4a4eaed389c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c880052d0 con 0x7f2c78003a30 2023-11-28T13:04:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c88005430 con 0x7f2c78003a30 2023-11-28T13:04:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.105+0000 7f2c83fff700 1 -- 172.21.15.111:0/806338277 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c880038c0 con 0x7f2c78003a30 2023-11-28T13:04:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.106+0000 7f2c9369e700 1 -- 172.21.15.111:0/806338277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c78003a30 msgr2=0x7f2c78003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.106+0000 7f2c9369e700 1 --2- 172.21.15.111:0/806338277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c78003a30 0x7f2c78003e40 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f2c880017c0 tx=0x7f2c78005880 comp rx=0 tx=0).stop 2023-11-28T13:04:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.106+0000 7f2c91c3b700 1 -- 172.21.15.111:0/806338277 reap_dead start 2023-11-28T13:04:32.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.106+0000 7f2c9369e700 1 -- 172.21.15.111:0/806338277 shutdown_connections 2023-11-28T13:04:32.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.106+0000 7f2c9369e700 1 -- 172.21.15.111:0/806338277 >> 172.21.15.111:0/806338277 conn(0x7f2c8c0b0db0 msgr2=0x7f2c8c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:32.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.106+0000 7f2c9369e700 1 -- 172.21.15.111:0/806338277 shutdown_connections 2023-11-28T13:04:32.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.106+0000 7f2c9369e700 1 -- 172.21.15.111:0/806338277 wait complete. 2023-11-28T13:04:32.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.107+0000 7f2c9369e700 1 Processor -- start 2023-11-28T13:04:32.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.107+0000 7f2c9369e700 1 -- start start 2023-11-28T13:04:32.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9369e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c78003a30 0x7f2c8c0c50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:32.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9369e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c8c148350 0x7f2c8c0c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:32.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9369e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c0c7350 0x7f2c8c158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:32.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9369e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c15c4e0 con 0x7f2c78003a30 2023-11-28T13:04:32.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9369e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c0c5620 con 0x7f2c8c148350 2023-11-28T13:04:32.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9369e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c0c77c0 con 0x7f2c8c0c7350 2023-11-28T13:04:32.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9143a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c78003a30 0x7f2c8c0c50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:32.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9143a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c78003a30 0x7f2c8c0c50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38064/0 (socket says 172.21.15.111:38064) 2023-11-28T13:04:32.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9143a700 1 -- 172.21.15.111:0/570092293 learned_addr learned my addr 172.21.15.111:0/570092293 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:32.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c91c3b700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c0c7350 0x7f2c8c158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:32.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c90c39700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c8c148350 0x7f2c8c0c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:32.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9143a700 1 -- 172.21.15.111:0/570092293 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c0c7350 msgr2=0x7f2c8c158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:32.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9143a700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c0c7350 0x7f2c8c158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:32.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9143a700 1 -- 172.21.15.111:0/570092293 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c8c148350 msgr2=0x7f2c8c0c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:32.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9143a700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c8c148350 0x7f2c8c0c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:32.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9143a700 1 -- 172.21.15.111:0/570092293 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c0c9ab0 con 0x7f2c78003a30 2023-11-28T13:04:32.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.108+0000 7f2c9143a700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c78003a30 0x7f2c8c0c50e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f2c88006b40 tx=0x7f2c88001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:32.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.109+0000 7f2c81ffb700 1 -- 172.21.15.111:0/570092293 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c88007240 con 0x7f2c78003a30 2023-11-28T13:04:32.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.109+0000 7f2c81ffb700 1 -- 172.21.15.111:0/570092293 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c880073a0 con 0x7f2c78003a30 2023-11-28T13:04:32.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.109+0000 7f2c9369e700 1 -- 172.21.15.111:0/570092293 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8c06b9c0 con 0x7f2c78003a30 2023-11-28T13:04:32.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.109+0000 7f2c81ffb700 1 -- 172.21.15.111:0/570092293 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c88001290 con 0x7f2c78003a30 2023-11-28T13:04:32.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.109+0000 7f2c9369e700 1 -- 172.21.15.111:0/570092293 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8c0c9d90 con 0x7f2c78003a30 2023-11-28T13:04:32.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.110+0000 7f2c9369e700 1 -- 172.21.15.111:0/570092293 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c8c0ca850 con 0x7f2c78003a30 2023-11-28T13:04:32.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.110+0000 7f2c81ffb700 1 -- 172.21.15.111:0/570092293 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2c8802e070 con 0x7f2c78003a30 2023-11-28T13:04:32.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.111+0000 7f2c81ffb700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c700682f0 0x7f2c7006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:32.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.111+0000 7f2c81ffb700 1 -- 172.21.15.111:0/570092293 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f2c88086ba0 con 0x7f2c78003a30 2023-11-28T13:04:32.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.111+0000 7f2c90c39700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c700682f0 0x7f2c7006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:32.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.111+0000 7f2c90c39700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c700682f0 0x7f2c7006a7a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c001c10 tx=0x7f2c7c001b20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:32.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.115+0000 7f2c81ffb700 1 -- 172.21.15.111:0/570092293 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2c88053330 con 0x7f2c78003a30 2023-11-28T13:04:32.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.343+0000 7f2c9369e700 1 -- 172.21.15.111:0/570092293 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2c8c0b1ce0 con 0x7f2c78003a30 2023-11-28T13:04:32.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.343+0000 7f2c81ffb700 1 -- 172.21.15.111:0/570092293 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f2c88056950 con 0x7f2c78003a30 2023-11-28T13:04:32.345 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:04:32.345 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:04:32.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.345+0000 7f2c9369e700 1 -- 172.21.15.111:0/570092293 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c700682f0 msgr2=0x7f2c7006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:32.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.345+0000 7f2c9369e700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c700682f0 0x7f2c7006a7a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c001c10 tx=0x7f2c7c001b20 comp rx=0 tx=0).stop 2023-11-28T13:04:32.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.345+0000 7f2c9369e700 1 -- 172.21.15.111:0/570092293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c78003a30 msgr2=0x7f2c8c0c50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:32.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.345+0000 7f2c9369e700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c78003a30 0x7f2c8c0c50e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f2c88006b40 tx=0x7f2c88001120 comp rx=0 tx=0).stop 2023-11-28T13:04:32.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.345+0000 7f2c9369e700 1 -- 172.21.15.111:0/570092293 shutdown_connections 2023-11-28T13:04:32.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.346+0000 7f2c9369e700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c8c0c7350 0x7f2c8c158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:32.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.346+0000 7f2c9369e700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c78003a30 0x7f2c8c0c50e0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:32.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.346+0000 7f2c9369e700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c700682f0 0x7f2c7006a7a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:32.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.346+0000 7f2c9369e700 1 --2- 172.21.15.111:0/570092293 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c8c148350 0x7f2c8c0c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:32.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.346+0000 7f2c9369e700 1 -- 172.21.15.111:0/570092293 >> 172.21.15.111:0/570092293 conn(0x7f2c8c0b0db0 msgr2=0x7f2c8c0b02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:32.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.346+0000 7f2c9369e700 1 -- 172.21.15.111:0/570092293 shutdown_connections 2023-11-28T13:04:32.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:32.346+0000 7f2c9369e700 1 -- 172.21.15.111:0/570092293 wait complete. 2023-11-28T13:04:32.358 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:04:37.360 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:04:37.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.490+0000 7f77847e9700 1 Processor -- start 2023-11-28T13:04:37.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.490+0000 7f77847e9700 1 -- start start 2023-11-28T13:04:37.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f77847e9700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f777c15c6c0 0x7f777c0afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:37.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f77847e9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f777c0b0440 0x7f777c0b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:37.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f77847e9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f777c1602a0 0x7f777c160720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f77847e9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f777c0b0e20 con 0x7f777c0b0440 2023-11-28T13:04:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f77847e9700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f777c0cc750 con 0x7f777c15c6c0 2023-11-28T13:04:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f77847e9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777c0cca30 con 0x7f777c1602a0 2023-11-28T13:04:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f7781d84700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f777c0b0440 0x7f777c0b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f7781d84700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f777c0b0440 0x7f777c0b08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38072/0 (socket says 172.21.15.111:38072) 2023-11-28T13:04:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f7781d84700 1 -- 172.21.15.111:0/4092856465 learned_addr learned my addr 172.21.15.111:0/4092856465 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f7782585700 1 --2- 172.21.15.111:0/4092856465 >> v2:172.21.15.149:3300/0 conn(0x7f777c15c6c0 0x7f777c0afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f7782d86700 1 --2- 172.21.15.111:0/4092856465 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f777c1602a0 0x7f777c160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f7781d84700 1 -- 172.21.15.111:0/4092856465 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f777c1602a0 msgr2=0x7f777c160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f7781d84700 1 --2- 172.21.15.111:0/4092856465 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f777c1602a0 0x7f777c160720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.491+0000 7f7781d84700 1 -- 172.21.15.111:0/4092856465 >> v2:172.21.15.149:3300/0 conn(0x7f777c15c6c0 msgr2=0x7f777c0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f7781d84700 1 --2- 172.21.15.111:0/4092856465 >> v2:172.21.15.149:3300/0 conn(0x7f777c15c6c0 0x7f777c0afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f7781d84700 1 -- 172.21.15.111:0/4092856465 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c0ccd10 con 0x7f777c0b0440 2023-11-28T13:04:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f7782d86700 1 --2- 172.21.15.111:0/4092856465 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f777c1602a0 0x7f777c160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:04:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f7781d84700 1 --2- 172.21.15.111:0/4092856465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f777c0b0440 0x7f777c0b08e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f777c071950 tx=0x7f7778009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728eb85aa5995769 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f7781583700 1 -- 172.21.15.111:0/4092856465 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777800b870 con 0x7f777c0b0440 2023-11-28T13:04:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f7781583700 1 -- 172.21.15.111:0/4092856465 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777800b9d0 con 0x7f777c0b0440 2023-11-28T13:04:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f7781583700 1 -- 172.21.15.111:0/4092856465 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777800bd60 con 0x7f777c0b0440 2023-11-28T13:04:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f77847e9700 1 -- 172.21.15.111:0/4092856465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f777c0b0440 msgr2=0x7f777c0b08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f77847e9700 1 --2- 172.21.15.111:0/4092856465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f777c0b0440 0x7f777c0b08e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f777c071950 tx=0x7f7778009bd0 comp rx=0 tx=0).stop 2023-11-28T13:04:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f77847e9700 1 -- 172.21.15.111:0/4092856465 shutdown_connections 2023-11-28T13:04:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f77847e9700 1 --2- 172.21.15.111:0/4092856465 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f777c1602a0 0x7f777c160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f77847e9700 1 --2- 172.21.15.111:0/4092856465 >> v2:172.21.15.149:3300/0 conn(0x7f777c15c6c0 0x7f777c0afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:37.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f77847e9700 1 --2- 172.21.15.111:0/4092856465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f777c0b0440 0x7f777c0b08e0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:37.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.492+0000 7f77847e9700 1 -- 172.21.15.111:0/4092856465 >> 172.21.15.111:0/4092856465 conn(0x7f777c098ee0 msgr2=0x7f777c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:37.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.493+0000 7f77847e9700 1 -- 172.21.15.111:0/4092856465 shutdown_connections 2023-11-28T13:04:37.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.493+0000 7f77847e9700 1 -- 172.21.15.111:0/4092856465 wait complete. 2023-11-28T13:04:37.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.494+0000 7f77847e9700 1 Processor -- start 2023-11-28T13:04:37.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.494+0000 7f77847e9700 1 -- start start 2023-11-28T13:04:37.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.494+0000 7f77847e9700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f777c0b0440 0x7f777c0c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:37.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.494+0000 7f77847e9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f777c15c6c0 0x7f777c0c3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:37.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f77847e9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f777c1602a0 0x7f777c151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:37.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f77847e9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c15f0d0 con 0x7f777c15c6c0 2023-11-28T13:04:37.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f77847e9700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c15e260 con 0x7f777c0b0440 2023-11-28T13:04:37.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f77847e9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777c15e450 con 0x7f777c1602a0 2023-11-28T13:04:37.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7781d84700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f777c15c6c0 0x7f777c0c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:37.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7781d84700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f777c15c6c0 0x7f777c0c3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38078/0 (socket says 172.21.15.111:38078) 2023-11-28T13:04:37.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7781d84700 1 -- 172.21.15.111:0/825245665 learned_addr learned my addr 172.21.15.111:0/825245665 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:37.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7782585700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f777c0b0440 0x7f777c0c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:37.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7782d86700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f777c1602a0 0x7f777c151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:37.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7781d84700 1 -- 172.21.15.111:0/825245665 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f777c1602a0 msgr2=0x7f777c151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:37.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7781d84700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f777c1602a0 0x7f777c151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:37.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7781d84700 1 -- 172.21.15.111:0/825245665 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f777c0b0440 msgr2=0x7f777c0c65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:37.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7781d84700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f777c0b0440 0x7f777c0c65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:37.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7781d84700 1 -- 172.21.15.111:0/825245665 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7778009550 con 0x7f777c15c6c0 2023-11-28T13:04:37.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7782d86700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f777c1602a0 0x7f777c151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:04:37.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7781d84700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f777c15c6c0 0x7f777c0c3700 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f7778000c00 tx=0x7f7778004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:37.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7772ffd700 1 -- 172.21.15.111:0/825245665 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77780129c0 con 0x7f777c15c6c0 2023-11-28T13:04:37.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7772ffd700 1 -- 172.21.15.111:0/825245665 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777801e070 con 0x7f777c15c6c0 2023-11-28T13:04:37.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f77847e9700 1 -- 172.21.15.111:0/825245665 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777c0cd2d0 con 0x7f777c15c6c0 2023-11-28T13:04:37.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f7772ffd700 1 -- 172.21.15.111:0/825245665 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778012d40 con 0x7f777c15c6c0 2023-11-28T13:04:37.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.495+0000 7f77847e9700 1 -- 172.21.15.111:0/825245665 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777c0ccff0 con 0x7f777c15c6c0 2023-11-28T13:04:37.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.496+0000 7f77847e9700 1 -- 172.21.15.111:0/825245665 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777c15ca90 con 0x7f777c15c6c0 2023-11-28T13:04:37.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.497+0000 7f7772ffd700 1 -- 172.21.15.111:0/825245665 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f777800f050 con 0x7f777c15c6c0 2023-11-28T13:04:37.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.497+0000 7f7772ffd700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f77600682f0 0x7f776006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:37.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.497+0000 7f7772ffd700 1 -- 172.21.15.111:0/825245665 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f7778086ed0 con 0x7f777c15c6c0 2023-11-28T13:04:37.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.497+0000 7f7782585700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f77600682f0 0x7f776006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:37.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.497+0000 7f7782585700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f77600682f0 0x7f776006a7a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f776c00b9b0 tx=0x7f776c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:37.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.501+0000 7f7772ffd700 1 -- 172.21.15.111:0/825245665 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f77780535b0 con 0x7f777c15c6c0 2023-11-28T13:04:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.724+0000 7f77847e9700 1 -- 172.21.15.111:0/825245665 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f777c0cdf80 con 0x7f777c15c6c0 2023-11-28T13:04:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.725+0000 7f7772ffd700 1 -- 172.21.15.111:0/825245665 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f77780150c0 con 0x7f777c15c6c0 2023-11-28T13:04:37.726 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:04:37.726 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:04:37.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.726+0000 7f77847e9700 1 -- 172.21.15.111:0/825245665 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f77600682f0 msgr2=0x7f776006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:37.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.726+0000 7f77847e9700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f77600682f0 0x7f776006a7a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f776c00b9b0 tx=0x7f776c008040 comp rx=0 tx=0).stop 2023-11-28T13:04:37.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.726+0000 7f77847e9700 1 -- 172.21.15.111:0/825245665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f777c15c6c0 msgr2=0x7f777c0c3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:37.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.726+0000 7f77847e9700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f777c15c6c0 0x7f777c0c3700 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f7778000c00 tx=0x7f7778004750 comp rx=0 tx=0).stop 2023-11-28T13:04:37.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.727+0000 7f77847e9700 1 -- 172.21.15.111:0/825245665 shutdown_connections 2023-11-28T13:04:37.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.727+0000 7f77847e9700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f777c1602a0 0x7f777c151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:37.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.727+0000 7f77847e9700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f77600682f0 0x7f776006a7a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:37.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.727+0000 7f77847e9700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f777c0b0440 0x7f777c0c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:37.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.727+0000 7f77847e9700 1 --2- 172.21.15.111:0/825245665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f777c15c6c0 0x7f777c0c3700 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:37.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.727+0000 7f77847e9700 1 -- 172.21.15.111:0/825245665 >> 172.21.15.111:0/825245665 conn(0x7f777c098ee0 msgr2=0x7f777c0afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:37.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.727+0000 7f77847e9700 1 -- 172.21.15.111:0/825245665 shutdown_connections 2023-11-28T13:04:37.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:37.727+0000 7f77847e9700 1 -- 172.21.15.111:0/825245665 wait complete. 2023-11-28T13:04:37.740 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:04:42.741 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:04:42.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.870+0000 7f706ffc7700 1 Processor -- start 2023-11-28T13:04:42.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.870+0000 7f706ffc7700 1 -- start start 2023-11-28T13:04:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706ffc7700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f706806a980 0x7f70680ad710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706ffc7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f70680af160 0x7f70680adc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706ffc7700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706815aa70 con 0x7f706815a6a0 2023-11-28T13:04:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706ffc7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70680ad280 con 0x7f70680af160 2023-11-28T13:04:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706ffc7700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70680ac950 con 0x7f706806a980 2023-11-28T13:04:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706e564700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f70680af160 0x7f70680adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706e564700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f70680af160 0x7f70680adc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38090/0 (socket says 172.21.15.111:38090) 2023-11-28T13:04:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706e564700 1 -- 172.21.15.111:0/908168819 learned_addr learned my addr 172.21.15.111:0/908168819 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706d562700 1 --2- 172.21.15.111:0/908168819 >> v2:172.21.15.149:3300/0 conn(0x7f706806a980 0x7f70680ad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706e564700 1 -- 172.21.15.111:0/908168819 >> v2:172.21.15.149:3300/0 conn(0x7f706806a980 msgr2=0x7f70680ad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706e564700 1 --2- 172.21.15.111:0/908168819 >> v2:172.21.15.149:3300/0 conn(0x7f706806a980 0x7f70680ad710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:42.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706e564700 1 -- 172.21.15.111:0/908168819 >> v1:172.21.15.149:6789/0 conn(0x7f706815a6a0 legacy=0x7f706806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:42.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706e564700 1 -- 172.21.15.111:0/908168819 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70680acc30 con 0x7f70680af160 2023-11-28T13:04:42.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706e564700 1 --2- 172.21.15.111:0/908168819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70680af160 0x7f70680adc50 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f70680c3710 tx=0x7f706000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6259ec63f1f6ec63 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:42.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706cd61700 1 -- 172.21.15.111:0/908168819 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706000e070 con 0x7f70680af160 2023-11-28T13:04:42.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706cd61700 1 -- 172.21.15.111:0/908168819 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706000b910 con 0x7f70680af160 2023-11-28T13:04:42.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.871+0000 7f706cd61700 1 -- 172.21.15.111:0/908168819 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706000bcc0 con 0x7f70680af160 2023-11-28T13:04:42.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.872+0000 7f706ffc7700 1 -- 172.21.15.111:0/908168819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70680af160 msgr2=0x7f70680adc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:42.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.872+0000 7f706ffc7700 1 --2- 172.21.15.111:0/908168819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70680af160 0x7f70680adc50 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f70680c3710 tx=0x7f706000b330 comp rx=0 tx=0).stop 2023-11-28T13:04:42.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.872+0000 7f706ffc7700 1 -- 172.21.15.111:0/908168819 shutdown_connections 2023-11-28T13:04:42.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.872+0000 7f706ffc7700 1 --2- 172.21.15.111:0/908168819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70680af160 0x7f70680adc50 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:42.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.872+0000 7f706ffc7700 1 --2- 172.21.15.111:0/908168819 >> v2:172.21.15.149:3300/0 conn(0x7f706806a980 0x7f70680ad710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:42.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.872+0000 7f706ffc7700 1 -- 172.21.15.111:0/908168819 >> 172.21.15.111:0/908168819 conn(0x7f7068098ee0 msgr2=0x7f706806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:42.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.872+0000 7f706ffc7700 1 -- 172.21.15.111:0/908168819 shutdown_connections 2023-11-28T13:04:42.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.872+0000 7f706ffc7700 1 -- 172.21.15.111:0/908168819 wait complete. 2023-11-28T13:04:42.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.873+0000 7f706ffc7700 1 Processor -- start 2023-11-28T13:04:42.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.873+0000 7f706ffc7700 1 -- start start 2023-11-28T13:04:42.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706ffc7700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f706806a980 0x7f70680ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:42.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706ffc7700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70680af160 0x7f706815c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:42.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706ffc7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f706815a6a0 0x7f706806ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:42.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706ffc7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70680c4fb0 con 0x7f706815a6a0 2023-11-28T13:04:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706ffc7700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70680c5290 con 0x7f70680af160 2023-11-28T13:04:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706ffc7700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70680c5570 con 0x7f706806a980 2023-11-28T13:04:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706e564700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f706815a6a0 0x7f706806ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706e564700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f706815a6a0 0x7f706806ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38096/0 (socket says 172.21.15.111:38096) 2023-11-28T13:04:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706e564700 1 -- 172.21.15.111:0/1618983588 learned_addr learned my addr 172.21.15.111:0/1618983588 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706dd63700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f706806a980 0x7f70680ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706d562700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70680af160 0x7f706815c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706e564700 1 -- 172.21.15.111:0/1618983588 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f706806a980 msgr2=0x7f70680ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706e564700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f706806a980 0x7f70680ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706e564700 1 -- 172.21.15.111:0/1618983588 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70680af160 msgr2=0x7f706815c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706e564700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70680af160 0x7f706815c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.874+0000 7f706e564700 1 -- 172.21.15.111:0/1618983588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7060009b30 con 0x7f706815a6a0 2023-11-28T13:04:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.875+0000 7f706e564700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f706815a6a0 0x7f706806ec40 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f7068067c00 tx=0x7f7060004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.875+0000 7f705e7fc700 1 -- 172.21.15.111:0/1618983588 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70600099b0 con 0x7f706815a6a0 2023-11-28T13:04:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.875+0000 7f705e7fc700 1 -- 172.21.15.111:0/1618983588 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70600092e0 con 0x7f706815a6a0 2023-11-28T13:04:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.875+0000 7f706ffc7700 1 -- 172.21.15.111:0/1618983588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70680992e0 con 0x7f706815a6a0 2023-11-28T13:04:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.875+0000 7f705e7fc700 1 -- 172.21.15.111:0/1618983588 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706001c640 con 0x7f706815a6a0 2023-11-28T13:04:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.875+0000 7f706ffc7700 1 -- 172.21.15.111:0/1618983588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70680acf10 con 0x7f706815a6a0 2023-11-28T13:04:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.876+0000 7f706ffc7700 1 -- 172.21.15.111:0/1618983588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70680aec10 con 0x7f706815a6a0 2023-11-28T13:04:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.876+0000 7f705e7fc700 1 -- 172.21.15.111:0/1618983588 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f706002c070 con 0x7f706815a6a0 2023-11-28T13:04:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.876+0000 7f705e7fc700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f704c0682f0 0x7f704c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.876+0000 7f706dd63700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f704c0682f0 0x7f704c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.876+0000 7f705e7fc700 1 -- 172.21.15.111:0/1618983588 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f70600879f0 con 0x7f706815a6a0 2023-11-28T13:04:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.877+0000 7f706dd63700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f704c0682f0 0x7f704c06a7a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7064000f60 tx=0x7f7064009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:42.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:42.881+0000 7f705e7fc700 1 -- 172.21.15.111:0/1618983588 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7060089e60 con 0x7f706815a6a0 2023-11-28T13:04:43.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.106+0000 7f706ffc7700 1 -- 172.21.15.111:0/1618983588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7068071070 con 0x7f706815a6a0 2023-11-28T13:04:43.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.106+0000 7f705e7fc700 1 -- 172.21.15.111:0/1618983588 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f70600540d0 con 0x7f706815a6a0 2023-11-28T13:04:43.108 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:04:43.108 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:04:43.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.108+0000 7f706ffc7700 1 -- 172.21.15.111:0/1618983588 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f704c0682f0 msgr2=0x7f704c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:43.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.108+0000 7f706ffc7700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f704c0682f0 0x7f704c06a7a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7064000f60 tx=0x7f7064009040 comp rx=0 tx=0).stop 2023-11-28T13:04:43.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.108+0000 7f706ffc7700 1 -- 172.21.15.111:0/1618983588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f706815a6a0 msgr2=0x7f706806ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:43.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.108+0000 7f706ffc7700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f706815a6a0 0x7f706806ec40 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f7068067c00 tx=0x7f7060004ad0 comp rx=0 tx=0).stop 2023-11-28T13:04:43.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.108+0000 7f706ffc7700 1 -- 172.21.15.111:0/1618983588 shutdown_connections 2023-11-28T13:04:43.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.108+0000 7f706ffc7700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f706815a6a0 0x7f706806ec40 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:43.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.108+0000 7f706ffc7700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f706806a980 0x7f70680ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:43.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.108+0000 7f706ffc7700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f704c0682f0 0x7f704c06a7a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:43.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.108+0000 7f706ffc7700 1 --2- 172.21.15.111:0/1618983588 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70680af160 0x7f706815c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:43.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.108+0000 7f706ffc7700 1 -- 172.21.15.111:0/1618983588 >> 172.21.15.111:0/1618983588 conn(0x7f7068098ee0 msgr2=0x7f70680b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:43.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.108+0000 7f706ffc7700 1 -- 172.21.15.111:0/1618983588 shutdown_connections 2023-11-28T13:04:43.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:43.108+0000 7f706ffc7700 1 -- 172.21.15.111:0/1618983588 wait complete. 2023-11-28T13:04:43.121 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:04:48.123 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:04:48.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.253+0000 7f2f43943700 1 Processor -- start 2023-11-28T13:04:48.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.253+0000 7f2f43943700 1 -- start start 2023-11-28T13:04:48.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f43943700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f2f3c0af9f0 0x7f2f3c0ca220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:48.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f43943700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3c06ac80 con 0x7f2f3c15fed0 2023-11-28T13:04:48.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f43943700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3c0cee00 con 0x7f2f3c0af5a0 2023-11-28T13:04:48.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f43943700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f3c0cf9a0 con 0x7f2f3c0af9f0 2023-11-28T13:04:48.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f416df700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2f3c15fed0 0x7f2f3c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38878/0 (socket says 172.21.15.111:38878) 2023-11-28T13:04:48.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f416df700 1 -- 172.21.15.111:0/1539738710 learned_addr learned my addr 172.21.15.111:0/1539738710 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:04:48.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f41ee0700 1 --2- 172.21.15.111:0/1539738710 >> v2:172.21.15.149:3300/0 conn(0x7f2f3c0af9f0 0x7f2f3c0ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:48.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207104179 0 0) 0x7f2f3c06ac80 con 0x7f2f3c15fed0 2023-11-28T13:04:48.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f280035b0 con 0x7f2f3c15fed0 2023-11-28T13:04:48.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1256583145 0 0) 0x7f2f280035b0 con 0x7f2f3c15fed0 2023-11-28T13:04:48.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 >> v2:172.21.15.149:3300/0 conn(0x7f2f3c0af9f0 msgr2=0x7f2f3c0ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:48.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f33fff700 1 --2- 172.21.15.111:0/1539738710 >> v2:172.21.15.149:3300/0 conn(0x7f2f3c0af9f0 0x7f2f3c0ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:48.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 >> v1:172.21.15.149:6789/0 conn(0x7f2f3c0af5a0 legacy=0x7f2f3c0b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:48.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c06aed0 con 0x7f2f3c15fed0 2023-11-28T13:04:48.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f2f38001670 con 0x7f2f3c15fed0 2023-11-28T13:04:48.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 >> v1:172.21.15.111:6789/0 conn(0x7f2f3c15fed0 legacy=0x7f2f3c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:48.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.254+0000 7f2f33fff700 1 --2- 172.21.15.111:0/1539738710 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f28003a00 0x7f2f28003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:48.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f33fff700 1 --2- 172.21.15.111:0/1539738710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f28004850 0x7f2f28004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:48.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f33fff700 1 --2- 172.21.15.111:0/1539738710 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f28005200 0x7f2f2800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:48.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c0cf9a0 con 0x7f2f28004850 2023-11-28T13:04:48.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2800c080 con 0x7f2f28003a00 2023-11-28T13:04:48.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2800c240 con 0x7f2f28005200 2023-11-28T13:04:48.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f40ede700 1 --2- 172.21.15.111:0/1539738710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f28004850 0x7f2f28004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f416df700 1 --2- 172.21.15.111:0/1539738710 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f28003a00 0x7f2f28003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f41ee0700 1 --2- 172.21.15.111:0/1539738710 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f28005200 0x7f2f2800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f40ede700 1 -- 172.21.15.111:0/1539738710 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f28005200 msgr2=0x7f2f2800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f40ede700 1 --2- 172.21.15.111:0/1539738710 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f28005200 0x7f2f2800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f40ede700 1 -- 172.21.15.111:0/1539738710 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f28003a00 msgr2=0x7f2f28003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f40ede700 1 --2- 172.21.15.111:0/1539738710 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f28003a00 0x7f2f28003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f40ede700 1 -- 172.21.15.111:0/1539738710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f280035b0 con 0x7f2f28004850 2023-11-28T13:04:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f41ee0700 1 --2- 172.21.15.111:0/1539738710 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f28005200 0x7f2f2800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:04:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f41ee0700 1 -- 172.21.15.111:0/1539738710 reap_dead start 2023-11-28T13:04:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f40ede700 1 --2- 172.21.15.111:0/1539738710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f28004850 0x7f2f28004cc0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0af970 tx=0x7f2f28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90bda127604a2bee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c0052d0 con 0x7f2f28004850 2023-11-28T13:04:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2c005430 con 0x7f2f28004850 2023-11-28T13:04:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.255+0000 7f2f33fff700 1 -- 172.21.15.111:0/1539738710 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c0038c0 con 0x7f2f28004850 2023-11-28T13:04:48.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.256+0000 7f2f43943700 1 -- 172.21.15.111:0/1539738710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f28004850 msgr2=0x7f2f28004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:48.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.256+0000 7f2f43943700 1 --2- 172.21.15.111:0/1539738710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f28004850 0x7f2f28004cc0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0af970 tx=0x7f2f28005880 comp rx=0 tx=0).stop 2023-11-28T13:04:48.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.256+0000 7f2f43943700 1 -- 172.21.15.111:0/1539738710 shutdown_connections 2023-11-28T13:04:48.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.256+0000 7f2f43943700 1 --2- 172.21.15.111:0/1539738710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f28004850 0x7f2f28004cc0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:48.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.256+0000 7f2f43943700 1 -- 172.21.15.111:0/1539738710 >> 172.21.15.111:0/1539738710 conn(0x7f2f3c06bca0 msgr2=0x7f2f3c0cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:48.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.256+0000 7f2f43943700 1 -- 172.21.15.111:0/1539738710 shutdown_connections 2023-11-28T13:04:48.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.256+0000 7f2f43943700 1 -- 172.21.15.111:0/1539738710 wait complete. 2023-11-28T13:04:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.257+0000 7f2f43943700 1 Processor -- start 2023-11-28T13:04:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.257+0000 7f2f43943700 1 -- start start 2023-11-28T13:04:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f43943700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f28004850 0x7f2f3c159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f43943700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f3c0c5e80 0x7f2f3c0c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f43943700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f3c0c62f0 0x7f2f3c0c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f43943700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c0c8a80 con 0x7f2f3c0c5e80 2023-11-28T13:04:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f43943700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c0b3dd0 con 0x7f2f3c0c62f0 2023-11-28T13:04:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f43943700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c0c8ea0 con 0x7f2f28004850 2023-11-28T13:04:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f40ede700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f3c0c5e80 0x7f2f3c0c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f40ede700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f3c0c5e80 0x7f2f3c0c9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38114/0 (socket says 172.21.15.111:38114) 2023-11-28T13:04:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f40ede700 1 -- 172.21.15.111:0/617805750 learned_addr learned my addr 172.21.15.111:0/617805750 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f41ee0700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f3c0c62f0 0x7f2f3c0c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f416df700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f28004850 0x7f2f3c159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f40ede700 1 -- 172.21.15.111:0/617805750 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f28004850 msgr2=0x7f2f3c159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f40ede700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f28004850 0x7f2f3c159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f40ede700 1 -- 172.21.15.111:0/617805750 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f3c0c62f0 msgr2=0x7f2f3c0c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f40ede700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f3c0c62f0 0x7f2f3c0c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f40ede700 1 -- 172.21.15.111:0/617805750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c1602a0 con 0x7f2f3c0c5e80 2023-11-28T13:04:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f40ede700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f3c0c5e80 0x7f2f3c0c9260 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c000c00 tx=0x7f2f2c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.258+0000 7f2f31ffb700 1 -- 172.21.15.111:0/617805750 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c003e40 con 0x7f2f3c0c5e80 2023-11-28T13:04:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.259+0000 7f2f31ffb700 1 -- 172.21.15.111:0/617805750 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2c001290 con 0x7f2f3c0c5e80 2023-11-28T13:04:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.259+0000 7f2f43943700 1 -- 172.21.15.111:0/617805750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f3c0cea90 con 0x7f2f3c0c5e80 2023-11-28T13:04:48.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.259+0000 7f2f31ffb700 1 -- 172.21.15.111:0/617805750 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c01d6c0 con 0x7f2f3c0c5e80 2023-11-28T13:04:48.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.259+0000 7f2f43943700 1 -- 172.21.15.111:0/617805750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3c0ce6b0 con 0x7f2f3c0c5e80 2023-11-28T13:04:48.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.260+0000 7f2f43943700 1 -- 172.21.15.111:0/617805750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3c0c8cf0 con 0x7f2f3c0c5e80 2023-11-28T13:04:48.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.260+0000 7f2f31ffb700 1 -- 172.21.15.111:0/617805750 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f2c02f070 con 0x7f2f3c0c5e80 2023-11-28T13:04:48.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.260+0000 7f2f31ffb700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f200682f0 0x7f2f2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:48.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.261+0000 7f2f31ffb700 1 -- 172.21.15.111:0/617805750 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f2f2c0868e0 con 0x7f2f3c0c5e80 2023-11-28T13:04:48.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.261+0000 7f2f416df700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f200682f0 0x7f2f2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:48.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.261+0000 7f2f416df700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f200682f0 0x7f2f2006a7a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2f380014a0 tx=0x7f2f38001430 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:48.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.265+0000 7f2f31ffb700 1 -- 172.21.15.111:0/617805750 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f2c052fc0 con 0x7f2f3c0c5e80 2023-11-28T13:04:48.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.490+0000 7f2f43943700 1 -- 172.21.15.111:0/617805750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2f3c067170 con 0x7f2f3c0c5e80 2023-11-28T13:04:48.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.490+0000 7f2f31ffb700 1 -- 172.21.15.111:0/617805750 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f2f2c0565e0 con 0x7f2f3c0c5e80 2023-11-28T13:04:48.492 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:04:48.492 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:04:48.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.492+0000 7f2f43943700 1 -- 172.21.15.111:0/617805750 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f200682f0 msgr2=0x7f2f2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:48.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.492+0000 7f2f43943700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f200682f0 0x7f2f2006a7a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2f380014a0 tx=0x7f2f38001430 comp rx=0 tx=0).stop 2023-11-28T13:04:48.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.492+0000 7f2f43943700 1 -- 172.21.15.111:0/617805750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f3c0c5e80 msgr2=0x7f2f3c0c9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:48.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.492+0000 7f2f43943700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f3c0c5e80 0x7f2f3c0c9260 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c000c00 tx=0x7f2f2c001120 comp rx=0 tx=0).stop 2023-11-28T13:04:48.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.492+0000 7f2f43943700 1 -- 172.21.15.111:0/617805750 shutdown_connections 2023-11-28T13:04:48.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.492+0000 7f2f43943700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f200682f0 0x7f2f2006a7a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:48.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.492+0000 7f2f43943700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f3c0c62f0 0x7f2f3c0c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:48.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.492+0000 7f2f43943700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2f28004850 0x7f2f3c159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:48.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.492+0000 7f2f43943700 1 --2- 172.21.15.111:0/617805750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f3c0c5e80 0x7f2f3c0c9260 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:48.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.492+0000 7f2f43943700 1 -- 172.21.15.111:0/617805750 >> 172.21.15.111:0/617805750 conn(0x7f2f3c06bca0 msgr2=0x7f2f3c15eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:48.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.492+0000 7f2f43943700 1 -- 172.21.15.111:0/617805750 shutdown_connections 2023-11-28T13:04:48.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:48.493+0000 7f2f43943700 1 -- 172.21.15.111:0/617805750 wait complete. 2023-11-28T13:04:48.505 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:04:53.506 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:04:53.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.636+0000 7f099910f700 1 Processor -- start 2023-11-28T13:04:53.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.636+0000 7f099910f700 1 -- start start 2023-11-28T13:04:53.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099910f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09941604b0 0x7f09940ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:53.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099910f700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f09940cc550 0x7f09940cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:53.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099910f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099415cd90 con 0x7f09940b0d70 2023-11-28T13:04:53.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099910f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0994160950 con 0x7f09941604b0 2023-11-28T13:04:53.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099910f700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09940cced0 con 0x7f09940cc550 2023-11-28T13:04:53.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09941604b0 0x7f09940ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:53.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f0992d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f09940b0d70 0x7f09940b1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38898/0 (socket says 172.21.15.111:38898) 2023-11-28T13:04:53.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09941604b0 0x7f09940ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38122/0 (socket says 172.21.15.111:38122) 2023-11-28T13:04:53.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f0992d9d700 1 -- 172.21.15.111:0/196058112 learned_addr learned my addr 172.21.15.111:0/196058112 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:04:53.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099359e700 1 --2- 172.21.15.111:0/196058112 >> v2:172.21.15.149:3300/0 conn(0x7f09940cc550 0x7f09940cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:53.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099259c700 1 -- 172.21.15.111:0/196058112 >> v2:172.21.15.149:3300/0 conn(0x7f09940cc550 msgr2=0x7f09940cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:53.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099259c700 1 --2- 172.21.15.111:0/196058112 >> v2:172.21.15.149:3300/0 conn(0x7f09940cc550 0x7f09940cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:53.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099259c700 1 -- 172.21.15.111:0/196058112 >> v1:172.21.15.111:6789/0 conn(0x7f09940b0d70 legacy=0x7f09940b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:53.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099259c700 1 -- 172.21.15.111:0/196058112 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09941613e0 con 0x7f09941604b0 2023-11-28T13:04:53.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f0991d9b700 1 -- 172.21.15.111:0/196058112 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506673871 0 0) 0x7f0988001290 con 0x7f09940b0d70 2023-11-28T13:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099359e700 1 --2- 172.21.15.111:0/196058112 >> v2:172.21.15.149:3300/0 conn(0x7f09940cc550 0x7f09940cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.637+0000 7f099259c700 1 --2- 172.21.15.111:0/196058112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09941604b0 0x7f09940ce6c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f0994070310 tx=0x7f0984009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9636ae2169861b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.638+0000 7f0991d9b700 1 -- 172.21.15.111:0/196058112 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984011830 con 0x7f09941604b0 2023-11-28T13:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.638+0000 7f0991d9b700 1 -- 172.21.15.111:0/196058112 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0984011990 con 0x7f09941604b0 2023-11-28T13:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.638+0000 7f0991d9b700 1 -- 172.21.15.111:0/196058112 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984011d20 con 0x7f09941604b0 2023-11-28T13:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.638+0000 7f099910f700 1 -- 172.21.15.111:0/196058112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09941604b0 msgr2=0x7f09940ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.638+0000 7f099910f700 1 --2- 172.21.15.111:0/196058112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09941604b0 0x7f09940ce6c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f0994070310 tx=0x7f0984009b20 comp rx=0 tx=0).stop 2023-11-28T13:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.638+0000 7f099910f700 1 -- 172.21.15.111:0/196058112 shutdown_connections 2023-11-28T13:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.638+0000 7f099910f700 1 --2- 172.21.15.111:0/196058112 >> v2:172.21.15.149:3300/0 conn(0x7f09940cc550 0x7f09940cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.638+0000 7f099910f700 1 --2- 172.21.15.111:0/196058112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09941604b0 0x7f09940ce6c0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:53.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.638+0000 7f099910f700 1 -- 172.21.15.111:0/196058112 >> 172.21.15.111:0/196058112 conn(0x7f0994098ee0 msgr2=0x7f099406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:53.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.638+0000 7f099910f700 1 -- 172.21.15.111:0/196058112 shutdown_connections 2023-11-28T13:04:53.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.638+0000 7f099910f700 1 -- 172.21.15.111:0/196058112 wait complete. 2023-11-28T13:04:53.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.640+0000 7f099910f700 1 Processor -- start 2023-11-28T13:04:53.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.640+0000 7f099910f700 1 -- start start 2023-11-28T13:04:53.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.641+0000 7f099910f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09940b0d70 0x7f09940c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:53.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.641+0000 7f099910f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09940cc550 0x7f09940b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:53.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.641+0000 7f099910f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09941604b0 0x7f09940c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:53.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.641+0000 7f099910f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099415a290 con 0x7f09940cc550 2023-11-28T13:04:53.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.641+0000 7f099910f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09940c7800 con 0x7f09940b0d70 2023-11-28T13:04:53.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.641+0000 7f099910f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09940c7ae0 con 0x7f09941604b0 2023-11-28T13:04:53.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.641+0000 7f099259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09940cc550 0x7f09940b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:53.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.641+0000 7f099259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09940cc550 0x7f09940b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38128/0 (socket says 172.21.15.111:38128) 2023-11-28T13:04:53.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.641+0000 7f099259c700 1 -- 172.21.15.111:0/2809414135 learned_addr learned my addr 172.21.15.111:0/2809414135 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:53.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.641+0000 7f0992d9d700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09940b0d70 0x7f09940c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:53.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.642+0000 7f099359e700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09941604b0 0x7f09940c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:53.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.642+0000 7f099259c700 1 -- 172.21.15.111:0/2809414135 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09941604b0 msgr2=0x7f09940c35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:53.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.642+0000 7f099259c700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09941604b0 0x7f09940c35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:53.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.642+0000 7f099259c700 1 -- 172.21.15.111:0/2809414135 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09940b0d70 msgr2=0x7f09940c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:53.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.642+0000 7f099259c700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09940b0d70 0x7f09940c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:53.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.642+0000 7f099259c700 1 -- 172.21.15.111:0/2809414135 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09840094a0 con 0x7f09940cc550 2023-11-28T13:04:53.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.642+0000 7f099359e700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09941604b0 0x7f09940c35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:04:53.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.642+0000 7f099259c700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09940cc550 0x7f09940b51a0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f09840090b0 tx=0x7f0984004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:53.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.642+0000 7f097b7fe700 1 -- 172.21.15.111:0/2809414135 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09840036a0 con 0x7f09940cc550 2023-11-28T13:04:53.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.642+0000 7f097b7fe700 1 -- 172.21.15.111:0/2809414135 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0984003800 con 0x7f09940cc550 2023-11-28T13:04:53.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.642+0000 7f099910f700 1 -- 172.21.15.111:0/2809414135 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0994160e20 con 0x7f09940cc550 2023-11-28T13:04:53.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.642+0000 7f097b7fe700 1 -- 172.21.15.111:0/2809414135 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984003b70 con 0x7f09940cc550 2023-11-28T13:04:53.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.642+0000 7f099910f700 1 -- 172.21.15.111:0/2809414135 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0994160b40 con 0x7f09940cc550 2023-11-28T13:04:53.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.643+0000 7f099910f700 1 -- 172.21.15.111:0/2809414135 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0994161100 con 0x7f09940cc550 2023-11-28T13:04:53.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.644+0000 7f097b7fe700 1 -- 172.21.15.111:0/2809414135 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0984023070 con 0x7f09940cc550 2023-11-28T13:04:53.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.644+0000 7f097b7fe700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f09800682f0 0x7f098006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:53.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.644+0000 7f097b7fe700 1 -- 172.21.15.111:0/2809414135 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f09840874e0 con 0x7f09940cc550 2023-11-28T13:04:53.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.644+0000 7f0992d9d700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f09800682f0 0x7f098006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:53.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.644+0000 7f0992d9d700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f09800682f0 0x7f098006a7a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0988007fd0 tx=0x7f0988009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:53.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.648+0000 7f097b7fe700 1 -- 172.21.15.111:0/2809414135 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0984053bc0 con 0x7f09940cc550 2023-11-28T13:04:53.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.874+0000 7f099910f700 1 -- 172.21.15.111:0/2809414135 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0994067170 con 0x7f09940cc550 2023-11-28T13:04:53.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.874+0000 7f097b7fe700 1 -- 172.21.15.111:0/2809414135 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f09840571e0 con 0x7f09940cc550 2023-11-28T13:04:53.876 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:04:53.876 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:04:53.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.876+0000 7f099910f700 1 -- 172.21.15.111:0/2809414135 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f09800682f0 msgr2=0x7f098006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:53.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.876+0000 7f099910f700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f09800682f0 0x7f098006a7a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0988007fd0 tx=0x7f0988009040 comp rx=0 tx=0).stop 2023-11-28T13:04:53.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.876+0000 7f099910f700 1 -- 172.21.15.111:0/2809414135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09940cc550 msgr2=0x7f09940b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:53.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.876+0000 7f099910f700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09940cc550 0x7f09940b51a0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f09840090b0 tx=0x7f0984004750 comp rx=0 tx=0).stop 2023-11-28T13:04:53.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.876+0000 7f099910f700 1 -- 172.21.15.111:0/2809414135 shutdown_connections 2023-11-28T13:04:53.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.876+0000 7f099910f700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f09941604b0 0x7f09940c35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:53.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.876+0000 7f099910f700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f09800682f0 0x7f098006a7a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:53.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.876+0000 7f099910f700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f09940b0d70 0x7f09940c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:53.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.876+0000 7f099910f700 1 --2- 172.21.15.111:0/2809414135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09940cc550 0x7f09940b51a0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:53.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.876+0000 7f099910f700 1 -- 172.21.15.111:0/2809414135 >> 172.21.15.111:0/2809414135 conn(0x7f0994098ee0 msgr2=0x7f099406e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:53.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.876+0000 7f099910f700 1 -- 172.21.15.111:0/2809414135 shutdown_connections 2023-11-28T13:04:53.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:53.876+0000 7f099910f700 1 -- 172.21.15.111:0/2809414135 wait complete. 2023-11-28T13:04:53.889 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:04:58.891 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:04:59.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.020+0000 7f63b15b9700 1 Processor -- start 2023-11-28T13:04:59.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.020+0000 7f63b15b9700 1 -- start start 2023-11-28T13:04:59.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.021+0000 7f63b15b9700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f63ac15fed0 0x7f63ac0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:59.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.021+0000 7f63b15b9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63ac0af5a0 0x7f63ac0ca220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:59.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.021+0000 7f63b15b9700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac06ac80 con 0x7f63ac0cf7d0 2023-11-28T13:04:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.021+0000 7f63b15b9700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ac0cee00 con 0x7f63ac15fed0 2023-11-28T13:04:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.021+0000 7f63b15b9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac06aed0 con 0x7f63ac0af5a0 2023-11-28T13:04:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.021+0000 7f63aaffd700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f63ac15fed0 0x7f63ac0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.022+0000 7f63ab7fe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63ac0af5a0 0x7f63ac0ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.022+0000 7f63aa7fc700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f63ac0cf7d0 0x7f63ac0b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:47456/0 (socket says 172.21.15.111:47456) 2023-11-28T13:04:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.022+0000 7f63aa7fc700 1 -- 172.21.15.111:0/222071103 learned_addr learned my addr 172.21.15.111:0/222071103 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:04:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.022+0000 7f63ab7fe700 1 -- 172.21.15.111:0/222071103 >> v2:172.21.15.149:3300/0 conn(0x7f63ac15fed0 msgr2=0x7f63ac0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:04:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.022+0000 7f63ab7fe700 1 --2- 172.21.15.111:0/222071103 >> v2:172.21.15.149:3300/0 conn(0x7f63ac15fed0 0x7f63ac0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.022+0000 7f63ab7fe700 1 -- 172.21.15.111:0/222071103 >> v1:172.21.15.149:6789/0 conn(0x7f63ac0cf7d0 legacy=0x7f63ac0b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.022+0000 7f63ab7fe700 1 -- 172.21.15.111:0/222071103 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac0cea90 con 0x7f63ac0af5a0 2023-11-28T13:04:59.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.022+0000 7f63aaffd700 1 --2- 172.21.15.111:0/222071103 >> v2:172.21.15.149:3300/0 conn(0x7f63ac15fed0 0x7f63ac0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:04:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.022+0000 7f63ab7fe700 1 --2- 172.21.15.111:0/222071103 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63ac0af5a0 0x7f63ac0ca220 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f63a4009d90 tx=0x7f63a400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2de467d8252a3ecf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.022+0000 7f63a9ffb700 1 -- 172.21.15.111:0/222071103 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a400e070 con 0x7f63ac0af5a0 2023-11-28T13:04:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.022+0000 7f63a9ffb700 1 -- 172.21.15.111:0/222071103 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a400bb20 con 0x7f63ac0af5a0 2023-11-28T13:04:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.022+0000 7f63a9ffb700 1 -- 172.21.15.111:0/222071103 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a4012430 con 0x7f63ac0af5a0 2023-11-28T13:04:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.023+0000 7f63b15b9700 1 -- 172.21.15.111:0/222071103 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63ac0af5a0 msgr2=0x7f63ac0ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.023+0000 7f63b15b9700 1 --2- 172.21.15.111:0/222071103 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63ac0af5a0 0x7f63ac0ca220 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f63a4009d90 tx=0x7f63a400b330 comp rx=0 tx=0).stop 2023-11-28T13:04:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.023+0000 7f63b15b9700 1 -- 172.21.15.111:0/222071103 shutdown_connections 2023-11-28T13:04:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.023+0000 7f63b15b9700 1 --2- 172.21.15.111:0/222071103 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63ac0af5a0 0x7f63ac0ca220 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.023+0000 7f63b15b9700 1 --2- 172.21.15.111:0/222071103 >> v2:172.21.15.149:3300/0 conn(0x7f63ac15fed0 0x7f63ac0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.023+0000 7f63b15b9700 1 -- 172.21.15.111:0/222071103 >> 172.21.15.111:0/222071103 conn(0x7f63ac06bca0 msgr2=0x7f63ac0cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:59.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.023+0000 7f63b15b9700 1 -- 172.21.15.111:0/222071103 shutdown_connections 2023-11-28T13:04:59.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.023+0000 7f63b15b9700 1 -- 172.21.15.111:0/222071103 wait complete. 2023-11-28T13:04:59.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.024+0000 7f63b15b9700 1 Processor -- start 2023-11-28T13:04:59.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.024+0000 7f63b15b9700 1 -- start start 2023-11-28T13:04:59.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63b15b9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac0af5a0 0x7f63ac0c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:59.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63b15b9700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63ac0cf7d0 0x7f63ac0c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:59.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63b15b9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63ac15fed0 0x7f63ac0c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:59.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63b15b9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac0c9a90 con 0x7f63ac0af5a0 2023-11-28T13:04:59.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63b15b9700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac0c7d40 con 0x7f63ac0cf7d0 2023-11-28T13:04:59.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63b15b9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac0c8020 con 0x7f63ac15fed0 2023-11-28T13:04:59.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63aaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac0af5a0 0x7f63ac0c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:59.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63aaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac0af5a0 0x7f63ac0c76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38140/0 (socket says 172.21.15.111:38140) 2023-11-28T13:04:59.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63aaffd700 1 -- 172.21.15.111:0/3792280047 learned_addr learned my addr 172.21.15.111:0/3792280047 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:04:59.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63aa7fc700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63ac0cf7d0 0x7f63ac0c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:59.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63ab7fe700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63ac15fed0 0x7f63ac0c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:59.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63aaffd700 1 -- 172.21.15.111:0/3792280047 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63ac15fed0 msgr2=0x7f63ac0c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:59.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63aaffd700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63ac15fed0 0x7f63ac0c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:59.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63aaffd700 1 -- 172.21.15.111:0/3792280047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63ac0cf7d0 msgr2=0x7f63ac0c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:59.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.025+0000 7f63aaffd700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63ac0cf7d0 0x7f63ac0c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:59.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.026+0000 7f63aaffd700 1 -- 172.21.15.111:0/3792280047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a00094a0 con 0x7f63ac0af5a0 2023-11-28T13:04:59.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.026+0000 7f63aaffd700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac0af5a0 0x7f63ac0c76b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f63a000ebe0 tx=0x7f63a0009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:59.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.026+0000 7f63937fe700 1 -- 172.21.15.111:0/3792280047 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a000c8b0 con 0x7f63ac0af5a0 2023-11-28T13:04:59.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.026+0000 7f63937fe700 1 -- 172.21.15.111:0/3792280047 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a000ca10 con 0x7f63ac0af5a0 2023-11-28T13:04:59.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.026+0000 7f63b15b9700 1 -- 172.21.15.111:0/3792280047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a4009a70 con 0x7f63ac0af5a0 2023-11-28T13:04:59.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.026+0000 7f63937fe700 1 -- 172.21.15.111:0/3792280047 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a000cda0 con 0x7f63ac0af5a0 2023-11-28T13:04:59.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.026+0000 7f63b15b9700 1 -- 172.21.15.111:0/3792280047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ac0ce6b0 con 0x7f63ac0af5a0 2023-11-28T13:04:59.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.027+0000 7f63b15b9700 1 -- 172.21.15.111:0/3792280047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63ac1602a0 con 0x7f63ac0af5a0 2023-11-28T13:04:59.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.027+0000 7f63937fe700 1 -- 172.21.15.111:0/3792280047 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f63a0019030 con 0x7f63ac0af5a0 2023-11-28T13:04:59.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.027+0000 7f63937fe700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63980682f0 0x7f639806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:04:59.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.028+0000 7f63937fe700 1 -- 172.21.15.111:0/3792280047 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f63a0085e10 con 0x7f63ac0af5a0 2023-11-28T13:04:59.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.028+0000 7f63aa7fc700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63980682f0 0x7f639806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:04:59.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.028+0000 7f63aa7fc700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63980682f0 0x7f639806a7a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f63ac0c9280 tx=0x7f639c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:04:59.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.032+0000 7f63937fe700 1 -- 172.21.15.111:0/3792280047 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f63a00524f0 con 0x7f63ac0af5a0 2023-11-28T13:04:59.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.253+0000 7f63b15b9700 1 -- 172.21.15.111:0/3792280047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f63ac0c4620 con 0x7f63ac0af5a0 2023-11-28T13:04:59.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.253+0000 7f63937fe700 1 -- 172.21.15.111:0/3792280047 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f63a00210c0 con 0x7f63ac0af5a0 2023-11-28T13:04:59.255 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:04:59.255 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:04:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.255+0000 7f63b15b9700 1 -- 172.21.15.111:0/3792280047 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63980682f0 msgr2=0x7f639806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.255+0000 7f63b15b9700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63980682f0 0x7f639806a7a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f63ac0c9280 tx=0x7f639c009040 comp rx=0 tx=0).stop 2023-11-28T13:04:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.255+0000 7f63b15b9700 1 -- 172.21.15.111:0/3792280047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac0af5a0 msgr2=0x7f63ac0c76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:04:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.255+0000 7f63b15b9700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac0af5a0 0x7f63ac0c76b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f63a000ebe0 tx=0x7f63a0009e20 comp rx=0 tx=0).stop 2023-11-28T13:04:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.255+0000 7f63b15b9700 1 -- 172.21.15.111:0/3792280047 shutdown_connections 2023-11-28T13:04:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.255+0000 7f63b15b9700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f63ac15fed0 0x7f63ac0c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.255+0000 7f63b15b9700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63ac0af5a0 0x7f63ac0c76b0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.255+0000 7f63b15b9700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f63980682f0 0x7f639806a7a0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.255+0000 7f63b15b9700 1 --2- 172.21.15.111:0/3792280047 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f63ac0cf7d0 0x7f63ac0c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:04:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.255+0000 7f63b15b9700 1 -- 172.21.15.111:0/3792280047 >> 172.21.15.111:0/3792280047 conn(0x7f63ac06bca0 msgr2=0x7f63ac0ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:04:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.255+0000 7f63b15b9700 1 -- 172.21.15.111:0/3792280047 shutdown_connections 2023-11-28T13:04:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:04:59.255+0000 7f63b15b9700 1 -- 172.21.15.111:0/3792280047 wait complete. 2023-11-28T13:04:59.268 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:04:59.429 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:04:59.432 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:05:04.269 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:05:04.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.400+0000 7f3d3f8d4700 1 Processor -- start 2023-11-28T13:05:04.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.400+0000 7f3d3f8d4700 1 -- start start 2023-11-28T13:05:04.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3f8d4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3d380af5a0 0x7f3d380b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:04.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3f8d4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d380af9f0 0x7f3d380b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:04.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3f8d4700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3806ac80 con 0x7f3d3815fed0 2023-11-28T13:05:04.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3f8d4700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d380cee00 con 0x7f3d380af5a0 2023-11-28T13:05:04.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3f8d4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380cf9a0 con 0x7f3d380af9f0 2023-11-28T13:05:04.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3ce6f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3d380af5a0 0x7f3d380b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:04.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3ce6f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3d380af5a0 0x7f3d380b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38150/0 (socket says 172.21.15.111:38150) 2023-11-28T13:05:04.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3ce6f700 1 -- 172.21.15.111:0/3278622881 learned_addr learned my addr 172.21.15.111:0/3278622881 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:04.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3ce6f700 1 -- 172.21.15.111:0/3278622881 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d380af9f0 msgr2=0x7f3d380b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:04.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3de71700 1 --2- 172.21.15.111:0/3278622881 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d380af9f0 0x7f3d380b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:04.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3ce6f700 1 --2- 172.21.15.111:0/3278622881 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d380af9f0 0x7f3d380b09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:04.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3ce6f700 1 -- 172.21.15.111:0/3278622881 >> v1:172.21.15.149:6789/0 conn(0x7f3d3815fed0 legacy=0x7f3d380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:04.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3ce6f700 1 -- 172.21.15.111:0/3278622881 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3806aed0 con 0x7f3d380af5a0 2023-11-28T13:05:04.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3de71700 1 --2- 172.21.15.111:0/3278622881 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d380af9f0 0x7f3d380b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:05:04.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.401+0000 7f3d3ce6f700 1 --2- 172.21.15.111:0/3278622881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d380af5a0 0x7f3d380b04b0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f3d280098f0 tx=0x7f3d28009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d220658d516c6db server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:04.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.402+0000 7f3d2ffff700 1 -- 172.21.15.111:0/3278622881 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d28004230 con 0x7f3d380af5a0 2023-11-28T13:05:04.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.402+0000 7f3d2ffff700 1 -- 172.21.15.111:0/3278622881 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d28004390 con 0x7f3d380af5a0 2023-11-28T13:05:04.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.402+0000 7f3d2ffff700 1 -- 172.21.15.111:0/3278622881 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d280036a0 con 0x7f3d380af5a0 2023-11-28T13:05:04.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.402+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/3278622881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d380af5a0 msgr2=0x7f3d380b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:04.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.402+0000 7f3d3f8d4700 1 --2- 172.21.15.111:0/3278622881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d380af5a0 0x7f3d380b04b0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f3d280098f0 tx=0x7f3d28009c00 comp rx=0 tx=0).stop 2023-11-28T13:05:04.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.402+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/3278622881 shutdown_connections 2023-11-28T13:05:04.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.402+0000 7f3d3f8d4700 1 --2- 172.21.15.111:0/3278622881 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d380af9f0 0x7f3d380b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:04.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.402+0000 7f3d3f8d4700 1 --2- 172.21.15.111:0/3278622881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d380af5a0 0x7f3d380b04b0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:04.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.402+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/3278622881 >> 172.21.15.111:0/3278622881 conn(0x7f3d3806bca0 msgr2=0x7f3d380cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:04.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.402+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/3278622881 shutdown_connections 2023-11-28T13:05:04.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.402+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/3278622881 wait complete. 2023-11-28T13:05:04.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.403+0000 7f3d3f8d4700 1 Processor -- start 2023-11-28T13:05:04.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.404+0000 7f3d3f8d4700 1 -- start start 2023-11-28T13:05:04.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.404+0000 7f3d3f8d4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d380af5a0 0x7f3d380b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:04.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.404+0000 7f3d3f8d4700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d380af9f0 0x7f3d380c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:04.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.404+0000 7f3d3f8d4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d3815fed0 0x7f3d381512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:04.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.404+0000 7f3d3f8d4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380b3a30 con 0x7f3d380af5a0 2023-11-28T13:05:04.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.404+0000 7f3d3f8d4700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380b3d10 con 0x7f3d380af9f0 2023-11-28T13:05:04.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.404+0000 7f3d3f8d4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380b3ff0 con 0x7f3d3815fed0 2023-11-28T13:05:04.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.404+0000 7f3d3d670700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d380af5a0 0x7f3d380b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:04.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.404+0000 7f3d3d670700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d380af5a0 0x7f3d380b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38154/0 (socket says 172.21.15.111:38154) 2023-11-28T13:05:04.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.404+0000 7f3d3d670700 1 -- 172.21.15.111:0/2190289991 learned_addr learned my addr 172.21.15.111:0/2190289991 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:04.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d3ce6f700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d380af9f0 0x7f3d380c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:04.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d3de71700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d3815fed0 0x7f3d381512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:04.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d3d670700 1 -- 172.21.15.111:0/2190289991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d3815fed0 msgr2=0x7f3d381512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:04.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d3d670700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d3815fed0 0x7f3d381512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:04.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d3d670700 1 -- 172.21.15.111:0/2190289991 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d380af9f0 msgr2=0x7f3d380c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:04.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d3d670700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d380af9f0 0x7f3d380c6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:04.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d3d670700 1 -- 172.21.15.111:0/2190289991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d28009550 con 0x7f3d380af5a0 2023-11-28T13:05:04.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d3de71700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d3815fed0 0x7f3d381512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:05:04.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d3d670700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d380af5a0 0x7f3d380b4ad0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400b4a0 tx=0x7f3d3400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:04.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d2dffb700 1 -- 172.21.15.111:0/2190289991 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34011830 con 0x7f3d380af5a0 2023-11-28T13:05:04.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d2dffb700 1 -- 172.21.15.111:0/2190289991 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d34011990 con 0x7f3d380af5a0 2023-11-28T13:05:04.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/2190289991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d381602a0 con 0x7f3d380af5a0 2023-11-28T13:05:04.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d2dffb700 1 -- 172.21.15.111:0/2190289991 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34011d40 con 0x7f3d380af5a0 2023-11-28T13:05:04.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.405+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/2190289991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d380cea90 con 0x7f3d380af5a0 2023-11-28T13:05:04.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.406+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/2190289991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d380ce6b0 con 0x7f3d380af5a0 2023-11-28T13:05:04.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.406+0000 7f3d2dffb700 1 -- 172.21.15.111:0/2190289991 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d3400f050 con 0x7f3d380af5a0 2023-11-28T13:05:04.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.407+0000 7f3d2dffb700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d1c0682f0 0x7f3d1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:04.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.407+0000 7f3d3ce6f700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d1c0682f0 0x7f3d1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:04.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.407+0000 7f3d2dffb700 1 -- 172.21.15.111:0/2190289991 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f3d340859f0 con 0x7f3d380af5a0 2023-11-28T13:05:04.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.407+0000 7f3d3ce6f700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d1c0682f0 0x7f3d1c06a7a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3d280098f0 tx=0x7f3d28008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:04.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.411+0000 7f3d2dffb700 1 -- 172.21.15.111:0/2190289991 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d34087e60 con 0x7f3d380af5a0 2023-11-28T13:05:04.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.640+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/2190289991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3d380c3a50 con 0x7f3d380af5a0 2023-11-28T13:05:04.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.640+0000 7f3d2dffb700 1 -- 172.21.15.111:0/2190289991 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f3d380c3a50 con 0x7f3d380af5a0 2023-11-28T13:05:04.642 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:05:04.643 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:05:04.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.642+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/2190289991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d1c0682f0 msgr2=0x7f3d1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:04.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.642+0000 7f3d3f8d4700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d1c0682f0 0x7f3d1c06a7a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3d280098f0 tx=0x7f3d28008030 comp rx=0 tx=0).stop 2023-11-28T13:05:04.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.642+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/2190289991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d380af5a0 msgr2=0x7f3d380b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:04.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.642+0000 7f3d3f8d4700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d380af5a0 0x7f3d380b4ad0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400b4a0 tx=0x7f3d3400b860 comp rx=0 tx=0).stop 2023-11-28T13:05:04.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.643+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/2190289991 shutdown_connections 2023-11-28T13:05:04.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.643+0000 7f3d3f8d4700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d3815fed0 0x7f3d381512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:04.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.643+0000 7f3d3f8d4700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d380af5a0 0x7f3d380b4ad0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:04.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.643+0000 7f3d3f8d4700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d1c0682f0 0x7f3d1c06a7a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:04.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.643+0000 7f3d3f8d4700 1 --2- 172.21.15.111:0/2190289991 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d380af9f0 0x7f3d380c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:04.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.643+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/2190289991 >> 172.21.15.111:0/2190289991 conn(0x7f3d3806bca0 msgr2=0x7f3d3806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:04.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.643+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/2190289991 shutdown_connections 2023-11-28T13:05:04.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:04.643+0000 7f3d3f8d4700 1 -- 172.21.15.111:0/2190289991 wait complete. 2023-11-28T13:05:04.655 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:05:09.656 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:05:09.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.788+0000 7fe290c34700 1 Processor -- start 2023-11-28T13:05:09.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.788+0000 7fe290c34700 1 -- start start 2023-11-28T13:05:09.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe290c34700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe28c0b20f0 0x7fe28c0b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:09.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe290c34700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c0b5300 con 0x7fe28c06ba90 2023-11-28T13:05:09.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe290c34700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28c0cfbb0 con 0x7fe28c0b40c0 2023-11-28T13:05:09.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe290c34700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe28c0b46f0 con 0x7fe28c0b20f0 2023-11-28T13:05:09.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe28a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe28c0b20f0 0x7fe28c0b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:09.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe28ad9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe28c06ba90 0x7fe28c0b5790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38944/0 (socket says 172.21.15.111:38944) 2023-11-28T13:05:09.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe28ad9d700 1 -- 172.21.15.111:0/2523821823 learned_addr learned my addr 172.21.15.111:0/2523821823 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:05:09.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe28a59c700 1 -- 172.21.15.111:0/2523821823 >> v1:172.21.15.149:6789/0 conn(0x7fe28c0b40c0 legacy=0x7fe28c06b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:09.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe28a59c700 1 -- 172.21.15.111:0/2523821823 >> v1:172.21.15.111:6789/0 conn(0x7fe28c06ba90 legacy=0x7fe28c0b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:09.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe28959a700 1 -- 172.21.15.111:0/2523821823 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912230973 0 0) 0x7fe28c0b5300 con 0x7fe28c06ba90 2023-11-28T13:05:09.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe28a59c700 1 -- 172.21.15.111:0/2523821823 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c0b49d0 con 0x7fe28c0b20f0 2023-11-28T13:05:09.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe28a59c700 1 --2- 172.21.15.111:0/2523821823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe28c0b20f0 0x7fe28c0b3b80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009840 tx=0x7fe27c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9180170a9b0bd40 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:09.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe28959a700 1 -- 172.21.15.111:0/2523821823 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c011830 con 0x7fe28c0b20f0 2023-11-28T13:05:09.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe28959a700 1 -- 172.21.15.111:0/2523821823 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27c011990 con 0x7fe28c0b20f0 2023-11-28T13:05:09.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.789+0000 7fe28959a700 1 -- 172.21.15.111:0/2523821823 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c011d20 con 0x7fe28c0b20f0 2023-11-28T13:05:09.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.790+0000 7fe290c34700 1 -- 172.21.15.111:0/2523821823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe28c0b20f0 msgr2=0x7fe28c0b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:09.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.790+0000 7fe290c34700 1 --2- 172.21.15.111:0/2523821823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe28c0b20f0 0x7fe28c0b3b80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009840 tx=0x7fe27c009b50 comp rx=0 tx=0).stop 2023-11-28T13:05:09.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.790+0000 7fe290c34700 1 -- 172.21.15.111:0/2523821823 shutdown_connections 2023-11-28T13:05:09.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.790+0000 7fe290c34700 1 --2- 172.21.15.111:0/2523821823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe28c0b20f0 0x7fe28c0b3b80 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:09.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.790+0000 7fe290c34700 1 -- 172.21.15.111:0/2523821823 >> 172.21.15.111:0/2523821823 conn(0x7fe28c09a040 msgr2=0x7fe28c0d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:09.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.790+0000 7fe290c34700 1 -- 172.21.15.111:0/2523821823 shutdown_connections 2023-11-28T13:05:09.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.790+0000 7fe290c34700 1 -- 172.21.15.111:0/2523821823 wait complete. 2023-11-28T13:05:09.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.791+0000 7fe290c34700 1 Processor -- start 2023-11-28T13:05:09.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.792+0000 7fe290c34700 1 -- start start 2023-11-28T13:05:09.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.792+0000 7fe290c34700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe28c06ba90 0x7fe28c0ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:09.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.792+0000 7fe290c34700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe28c0b20f0 0x7fe28c158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:09.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.792+0000 7fe290c34700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe28c0b40c0 0x7fe28c15a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:09.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.792+0000 7fe290c34700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c1e4e50 con 0x7fe28c06ba90 2023-11-28T13:05:09.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.792+0000 7fe290c34700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c0c98b0 con 0x7fe28c0b40c0 2023-11-28T13:05:09.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.792+0000 7fe290c34700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c0c9b90 con 0x7fe28c0b20f0 2023-11-28T13:05:09.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.792+0000 7fe28a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe28c06ba90 0x7fe28c0ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:09.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.792+0000 7fe28a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe28c06ba90 0x7fe28c0ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38168/0 (socket says 172.21.15.111:38168) 2023-11-28T13:05:09.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.792+0000 7fe28a59c700 1 -- 172.21.15.111:0/2474538143 learned_addr learned my addr 172.21.15.111:0/2474538143 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:09.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.792+0000 7fe28ad9d700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe28c0b40c0 0x7fe28c15a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:09.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.792+0000 7fe289d9b700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe28c0b20f0 0x7fe28c158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:09.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.793+0000 7fe28a59c700 1 -- 172.21.15.111:0/2474538143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe28c0b20f0 msgr2=0x7fe28c158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:09.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.793+0000 7fe28a59c700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe28c0b20f0 0x7fe28c158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:09.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.793+0000 7fe28a59c700 1 -- 172.21.15.111:0/2474538143 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe28c0b40c0 msgr2=0x7fe28c15a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:09.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.793+0000 7fe28a59c700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe28c0b40c0 0x7fe28c15a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:09.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.793+0000 7fe28a59c700 1 -- 172.21.15.111:0/2474538143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c0094a0 con 0x7fe28c06ba90 2023-11-28T13:05:09.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.793+0000 7fe28ad9d700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe28c0b40c0 0x7fe28c15a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:05:09.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.793+0000 7fe28a59c700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe28c06ba90 0x7fe28c0ca950 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe27c014040 tx=0x7fe27c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:09.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.793+0000 7fe27affd700 1 -- 172.21.15.111:0/2474538143 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c01a9c0 con 0x7fe28c06ba90 2023-11-28T13:05:09.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.793+0000 7fe27affd700 1 -- 172.21.15.111:0/2474538143 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27c01ab20 con 0x7fe28c06ba90 2023-11-28T13:05:09.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.793+0000 7fe27affd700 1 -- 172.21.15.111:0/2474538143 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c024420 con 0x7fe28c06ba90 2023-11-28T13:05:09.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.793+0000 7fe290c34700 1 -- 172.21.15.111:0/2474538143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c0b4f90 con 0x7fe28c06ba90 2023-11-28T13:05:09.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.793+0000 7fe290c34700 1 -- 172.21.15.111:0/2474538143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c0b4cb0 con 0x7fe28c06ba90 2023-11-28T13:05:09.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.794+0000 7fe290c34700 1 -- 172.21.15.111:0/2474538143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28c0b24c0 con 0x7fe28c06ba90 2023-11-28T13:05:09.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.795+0000 7fe27affd700 1 -- 172.21.15.111:0/2474538143 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe27c00f050 con 0x7fe28c06ba90 2023-11-28T13:05:09.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.795+0000 7fe27affd700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe26c0682f0 0x7fe26c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:09.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.796+0000 7fe289d9b700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe26c0682f0 0x7fe26c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:09.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.796+0000 7fe27affd700 1 -- 172.21.15.111:0/2474538143 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fe27c087330 con 0x7fe28c06ba90 2023-11-28T13:05:09.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.796+0000 7fe289d9b700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe26c0682f0 0x7fe26c06a7a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe28c1f5320 tx=0x7fe280009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:09.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:09.799+0000 7fe27affd700 1 -- 172.21.15.111:0/2474538143 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe27c053ab0 con 0x7fe28c06ba90 2023-11-28T13:05:10.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.025+0000 7fe290c34700 1 -- 172.21.15.111:0/2474538143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe28c067170 con 0x7fe28c06ba90 2023-11-28T13:05:10.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.026+0000 7fe27affd700 1 -- 172.21.15.111:0/2474538143 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7fe27c0570d0 con 0x7fe28c06ba90 2023-11-28T13:05:10.026 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:05:10.026 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:05:10.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.027+0000 7fe290c34700 1 -- 172.21.15.111:0/2474538143 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe26c0682f0 msgr2=0x7fe26c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:10.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.027+0000 7fe290c34700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe26c0682f0 0x7fe26c06a7a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe28c1f5320 tx=0x7fe280009040 comp rx=0 tx=0).stop 2023-11-28T13:05:10.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.027+0000 7fe290c34700 1 -- 172.21.15.111:0/2474538143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe28c06ba90 msgr2=0x7fe28c0ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:10.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.027+0000 7fe290c34700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe28c06ba90 0x7fe28c0ca950 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe27c014040 tx=0x7fe27c004750 comp rx=0 tx=0).stop 2023-11-28T13:05:10.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.027+0000 7fe290c34700 1 -- 172.21.15.111:0/2474538143 shutdown_connections 2023-11-28T13:05:10.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.028+0000 7fe290c34700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe26c0682f0 0x7fe26c06a7a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:10.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.028+0000 7fe290c34700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe28c0b40c0 0x7fe28c15a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:10.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.028+0000 7fe290c34700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe28c06ba90 0x7fe28c0ca950 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:10.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.028+0000 7fe290c34700 1 --2- 172.21.15.111:0/2474538143 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe28c0b20f0 0x7fe28c158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:10.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.028+0000 7fe290c34700 1 -- 172.21.15.111:0/2474538143 >> 172.21.15.111:0/2474538143 conn(0x7fe28c09a040 msgr2=0x7fe28c06ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:10.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.028+0000 7fe290c34700 1 -- 172.21.15.111:0/2474538143 shutdown_connections 2023-11-28T13:05:10.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:10.028+0000 7fe290c34700 1 -- 172.21.15.111:0/2474538143 wait complete. 2023-11-28T13:05:10.040 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:05:15.041 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:05:15.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.171+0000 7f458afe8700 1 Processor -- start 2023-11-28T13:05:15.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.171+0000 7f458afe8700 1 -- start start 2023-11-28T13:05:15.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f458afe8700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f458415fed0 0x7f45840cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:15.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f458afe8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f45840af5a0 0x7f45840ca220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:15.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f458afe8700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f458406ac80 con 0x7f45840cf7d0 2023-11-28T13:05:15.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f458afe8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45840cee00 con 0x7f45840af5a0 2023-11-28T13:05:15.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f458afe8700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f458406aed0 con 0x7f458415fed0 2023-11-28T13:05:15.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f4589585700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f45840af5a0 0x7f45840ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:15.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f4589585700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f45840af5a0 0x7f45840ca220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38180/0 (socket says 172.21.15.111:38180) 2023-11-28T13:05:15.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f4589585700 1 -- 172.21.15.111:0/4185101776 learned_addr learned my addr 172.21.15.111:0/4185101776 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:15.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f4589585700 1 -- 172.21.15.111:0/4185101776 >> v2:172.21.15.149:3300/0 conn(0x7f458415fed0 msgr2=0x7f45840cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:15.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f4589585700 1 --2- 172.21.15.111:0/4185101776 >> v2:172.21.15.149:3300/0 conn(0x7f458415fed0 0x7f45840cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:15.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f4589585700 1 -- 172.21.15.111:0/4185101776 >> v1:172.21.15.149:6789/0 conn(0x7f45840cf7d0 legacy=0x7f45840b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:15.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f4589585700 1 -- 172.21.15.111:0/4185101776 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45840cea90 con 0x7f45840af5a0 2023-11-28T13:05:15.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f4589585700 1 --2- 172.21.15.111:0/4185101776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45840af5a0 0x7f45840ca220 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f458406ced0 tx=0x7f457c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9583e9ebb9148dd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:15.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f45837fe700 1 -- 172.21.15.111:0/4185101776 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c00e070 con 0x7f45840af5a0 2023-11-28T13:05:15.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f45837fe700 1 -- 172.21.15.111:0/4185101776 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c003a20 con 0x7f45840af5a0 2023-11-28T13:05:15.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.172+0000 7f45837fe700 1 -- 172.21.15.111:0/4185101776 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c003dd0 con 0x7f45840af5a0 2023-11-28T13:05:15.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.173+0000 7f458afe8700 1 -- 172.21.15.111:0/4185101776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45840af5a0 msgr2=0x7f45840ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:15.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.173+0000 7f458afe8700 1 --2- 172.21.15.111:0/4185101776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45840af5a0 0x7f45840ca220 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f458406ced0 tx=0x7f457c00b330 comp rx=0 tx=0).stop 2023-11-28T13:05:15.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.173+0000 7f458afe8700 1 -- 172.21.15.111:0/4185101776 shutdown_connections 2023-11-28T13:05:15.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.173+0000 7f458afe8700 1 --2- 172.21.15.111:0/4185101776 >> v2:172.21.15.149:3300/0 conn(0x7f458415fed0 0x7f45840cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:15.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.173+0000 7f458afe8700 1 --2- 172.21.15.111:0/4185101776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45840af5a0 0x7f45840ca220 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:15.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.173+0000 7f458afe8700 1 -- 172.21.15.111:0/4185101776 >> 172.21.15.111:0/4185101776 conn(0x7f458406bca0 msgr2=0x7f45840cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:15.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.173+0000 7f458afe8700 1 -- 172.21.15.111:0/4185101776 shutdown_connections 2023-11-28T13:05:15.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.173+0000 7f458afe8700 1 -- 172.21.15.111:0/4185101776 wait complete. 2023-11-28T13:05:15.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.174+0000 7f458afe8700 1 Processor -- start 2023-11-28T13:05:15.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.174+0000 7f458afe8700 1 -- start start 2023-11-28T13:05:15.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f458afe8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45840af5a0 0x7f45841552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:15.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f458afe8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f45840cf7d0 0x7f45840c4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:15.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f458afe8700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f458415fed0 0x7f458415d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:15.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f458afe8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45841f2da0 con 0x7f45840af5a0 2023-11-28T13:05:15.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f458afe8700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4584155820 con 0x7f458415fed0 2023-11-28T13:05:15.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f458afe8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45841f2420 con 0x7f45840cf7d0 2023-11-28T13:05:15.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f4588d84700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45840af5a0 0x7f45841552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:15.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f4588d84700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45840af5a0 0x7f45841552e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38182/0 (socket says 172.21.15.111:38182) 2023-11-28T13:05:15.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f4588d84700 1 -- 172.21.15.111:0/2070189821 learned_addr learned my addr 172.21.15.111:0/2070189821 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:15.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f4589585700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f458415fed0 0x7f458415d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:15.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f4583fff700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f45840cf7d0 0x7f45840c4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:15.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f4588d84700 1 -- 172.21.15.111:0/2070189821 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f45840cf7d0 msgr2=0x7f45840c4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:15.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f4588d84700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f45840cf7d0 0x7f45840c4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:15.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f4588d84700 1 -- 172.21.15.111:0/2070189821 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f458415fed0 msgr2=0x7f458415d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:15.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f4588d84700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f458415fed0 0x7f458415d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:15.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.175+0000 7f4588d84700 1 -- 172.21.15.111:0/2070189821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c009b30 con 0x7f45840af5a0 2023-11-28T13:05:15.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.176+0000 7f4588d84700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45840af5a0 0x7f45841552e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f457800b4a0 tx=0x7f457800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:15.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.176+0000 7f45817fa700 1 -- 172.21.15.111:0/2070189821 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578011830 con 0x7f45840af5a0 2023-11-28T13:05:15.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.176+0000 7f45817fa700 1 -- 172.21.15.111:0/2070189821 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4578011990 con 0x7f45840af5a0 2023-11-28T13:05:15.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.176+0000 7f458afe8700 1 -- 172.21.15.111:0/2070189821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45840ce6b0 con 0x7f45840af5a0 2023-11-28T13:05:15.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.176+0000 7f45817fa700 1 -- 172.21.15.111:0/2070189821 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578011d40 con 0x7f45840af5a0 2023-11-28T13:05:15.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.176+0000 7f458afe8700 1 -- 172.21.15.111:0/2070189821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45841602a0 con 0x7f45840af5a0 2023-11-28T13:05:15.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.177+0000 7f458afe8700 1 -- 172.21.15.111:0/2070189821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45840ce840 con 0x7f45840af5a0 2023-11-28T13:05:15.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.177+0000 7f45817fa700 1 -- 172.21.15.111:0/2070189821 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4578021070 con 0x7f45840af5a0 2023-11-28T13:05:15.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.177+0000 7f45817fa700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f45680682f0 0x7f456806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:15.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.178+0000 7f4583fff700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f45680682f0 0x7f456806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:15.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.178+0000 7f45817fa700 1 -- 172.21.15.111:0/2070189821 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f4578085d70 con 0x7f45840af5a0 2023-11-28T13:05:15.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.178+0000 7f4583fff700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f45680682f0 0x7f456806a7a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f45840c98a0 tx=0x7f4570009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:15.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.182+0000 7f45817fa700 1 -- 172.21.15.111:0/2070189821 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f45780523d0 con 0x7f45840af5a0 2023-11-28T13:05:15.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.408+0000 7f458afe8700 1 -- 172.21.15.111:0/2070189821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4584067170 con 0x7f45840af5a0 2023-11-28T13:05:15.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.408+0000 7f45817fa700 1 -- 172.21.15.111:0/2070189821 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f4578015080 con 0x7f45840af5a0 2023-11-28T13:05:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:05:15.409 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:05:15.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.410+0000 7f458afe8700 1 -- 172.21.15.111:0/2070189821 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f45680682f0 msgr2=0x7f456806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:15.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.410+0000 7f458afe8700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f45680682f0 0x7f456806a7a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f45840c98a0 tx=0x7f4570009040 comp rx=0 tx=0).stop 2023-11-28T13:05:15.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.410+0000 7f458afe8700 1 -- 172.21.15.111:0/2070189821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45840af5a0 msgr2=0x7f45841552e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:15.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.410+0000 7f458afe8700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45840af5a0 0x7f45841552e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f457800b4a0 tx=0x7f457800b860 comp rx=0 tx=0).stop 2023-11-28T13:05:15.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.410+0000 7f458afe8700 1 -- 172.21.15.111:0/2070189821 shutdown_connections 2023-11-28T13:05:15.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.410+0000 7f458afe8700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f45680682f0 0x7f456806a7a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:15.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.410+0000 7f458afe8700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f458415fed0 0x7f458415d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:15.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.410+0000 7f458afe8700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45840af5a0 0x7f45841552e0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:15.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.410+0000 7f458afe8700 1 --2- 172.21.15.111:0/2070189821 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f45840cf7d0 0x7f45840c4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:15.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.410+0000 7f458afe8700 1 -- 172.21.15.111:0/2070189821 >> 172.21.15.111:0/2070189821 conn(0x7f458406bca0 msgr2=0x7f458406e470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:15.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.410+0000 7f458afe8700 1 -- 172.21.15.111:0/2070189821 shutdown_connections 2023-11-28T13:05:15.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:15.410+0000 7f458afe8700 1 -- 172.21.15.111:0/2070189821 wait complete. 2023-11-28T13:05:15.423 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:05:20.424 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:05:20.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.554+0000 7f4ce2e13700 1 Processor -- start 2023-11-28T13:05:20.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.554+0000 7f4ce2e13700 1 -- start start 2023-11-28T13:05:20.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4ce2e13700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cdc1604b0 0x7f4cdc0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4ce2e13700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4cdc0cc550 0x7f4cdc0cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4ce2e13700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cdc15cd90 con 0x7f4cdc0b0d70 2023-11-28T13:05:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4ce2e13700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cdc0ccf10 con 0x7f4cdc0cc550 2023-11-28T13:05:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4ce2e13700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc160920 con 0x7f4cdc1604b0 2023-11-28T13:05:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4ce0baf700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4cdc0b0d70 0x7f4cdc0b1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38968/0 (socket says 172.21.15.111:38968) 2023-11-28T13:05:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4ce0baf700 1 -- 172.21.15.111:0/3621675127 learned_addr learned my addr 172.21.15.111:0/3621675127 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:05:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4ce13b0700 1 --2- 172.21.15.111:0/3621675127 >> v2:172.21.15.149:3300/0 conn(0x7f4cdc0cc550 0x7f4cdc0cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdbfff700 1 --2- 172.21.15.111:0/3621675127 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cdc1604b0 0x7f4cdc0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1143169475 0 0) 0x7f4cd0001290 con 0x7f4cdc0b0d70 2023-11-28T13:05:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ccc0035b0 con 0x7f4cdc0b0d70 2023-11-28T13:05:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 209746997 0 0) 0x7f4ccc0035b0 con 0x7f4cdc0b0d70 2023-11-28T13:05:20.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cdc1604b0 msgr2=0x7f4cdc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:20.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 --2- 172.21.15.111:0/3621675127 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cdc1604b0 0x7f4cdc0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:20.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 >> v2:172.21.15.149:3300/0 conn(0x7f4cdc0cc550 msgr2=0x7f4cdc0cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:20.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 --2- 172.21.15.111:0/3621675127 >> v2:172.21.15.149:3300/0 conn(0x7f4cdc0cc550 0x7f4cdc0cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:20.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cdc1613e0 con 0x7f4cdc0b0d70 2023-11-28T13:05:20.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f4cd0001980 con 0x7f4cdc0b0d70 2023-11-28T13:05:20.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 >> v1:172.21.15.111:6789/0 conn(0x7f4cdc0b0d70 legacy=0x7f4cdc0b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:20.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 --2- 172.21.15.111:0/3621675127 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cdc1604b0 0x7f4ccc003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:20.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 --2- 172.21.15.111:0/3621675127 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ccc003f80 0x7f4ccc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:20.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 --2- 172.21.15.111:0/3621675127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc004d90 0x7f4ccc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:20.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc0ccf10 con 0x7f4ccc004d90 2023-11-28T13:05:20.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc160920 con 0x7f4ccc003f80 2023-11-28T13:05:20.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.555+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc00ba10 con 0x7f4cdc1604b0 2023-11-28T13:05:20.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.556+0000 7f4ce13b0700 1 --2- 172.21.15.111:0/3621675127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc004d90 0x7f4ccc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:20.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.556+0000 7f4cdbfff700 1 --2- 172.21.15.111:0/3621675127 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ccc003f80 0x7f4ccc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:20.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.556+0000 7f4ce0baf700 1 --2- 172.21.15.111:0/3621675127 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cdc1604b0 0x7f4ccc003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:20.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.556+0000 7f4ce13b0700 1 -- 172.21.15.111:0/3621675127 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cdc1604b0 msgr2=0x7f4ccc003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:20.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.556+0000 7f4ce13b0700 1 --2- 172.21.15.111:0/3621675127 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cdc1604b0 0x7f4ccc003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:20.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.556+0000 7f4ce13b0700 1 -- 172.21.15.111:0/3621675127 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ccc003f80 msgr2=0x7f4ccc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:20.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.556+0000 7f4ce13b0700 1 --2- 172.21.15.111:0/3621675127 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ccc003f80 0x7f4ccc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:20.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.556+0000 7f4ce13b0700 1 -- 172.21.15.111:0/3621675127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc0035b0 con 0x7f4ccc004d90 2023-11-28T13:05:20.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.556+0000 7f4ce13b0700 1 --2- 172.21.15.111:0/3621675127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc004d90 0x7f4ccc005200 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009fc0 tx=0x7f4ccc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9086c7d6c20c03 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:20.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.556+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd400ec30 con 0x7f4ccc004d90 2023-11-28T13:05:20.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.556+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd400f040 con 0x7f4ccc004d90 2023-11-28T13:05:20.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.556+0000 7f4cdb7fe700 1 -- 172.21.15.111:0/3621675127 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4014480 con 0x7f4ccc004d90 2023-11-28T13:05:20.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.557+0000 7f4ce2e13700 1 -- 172.21.15.111:0/3621675127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc004d90 msgr2=0x7f4ccc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:20.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.557+0000 7f4ce2e13700 1 --2- 172.21.15.111:0/3621675127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc004d90 0x7f4ccc005200 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009fc0 tx=0x7f4ccc005880 comp rx=0 tx=0).stop 2023-11-28T13:05:20.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.557+0000 7f4ce13b0700 1 -- 172.21.15.111:0/3621675127 reap_dead start 2023-11-28T13:05:20.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.557+0000 7f4ce2e13700 1 -- 172.21.15.111:0/3621675127 shutdown_connections 2023-11-28T13:05:20.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.557+0000 7f4ce2e13700 1 -- 172.21.15.111:0/3621675127 >> 172.21.15.111:0/3621675127 conn(0x7f4cdc098ee0 msgr2=0x7f4cdc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:20.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.557+0000 7f4ce2e13700 1 -- 172.21.15.111:0/3621675127 shutdown_connections 2023-11-28T13:05:20.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.557+0000 7f4ce2e13700 1 -- 172.21.15.111:0/3621675127 wait complete. 2023-11-28T13:05:20.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.558+0000 7f4ce2e13700 1 Processor -- start 2023-11-28T13:05:20.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.558+0000 7f4ce2e13700 1 -- start start 2023-11-28T13:05:20.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce2e13700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc004d90 0x7f4cdc0c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:20.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce2e13700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cdc14b420 0x7f4cdc147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:20.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce2e13700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cdc14a810 0x7f4cdc14ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:20.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce2e13700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc0c9780 con 0x7f4ccc004d90 2023-11-28T13:05:20.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce2e13700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc14b1d0 con 0x7f4cdc14b420 2023-11-28T13:05:20.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce2e13700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc0c9ba0 con 0x7f4cdc14a810 2023-11-28T13:05:20.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce0baf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc004d90 0x7f4cdc0c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:20.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce0baf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc004d90 0x7f4cdc0c6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38202/0 (socket says 172.21.15.111:38202) 2023-11-28T13:05:20.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce0baf700 1 -- 172.21.15.111:0/482429098 learned_addr learned my addr 172.21.15.111:0/482429098 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:20.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4cdbfff700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cdc14b420 0x7f4cdc147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:20.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce13b0700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cdc14a810 0x7f4cdc14ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:20.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce0baf700 1 -- 172.21.15.111:0/482429098 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cdc14a810 msgr2=0x7f4cdc14ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:20.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce0baf700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cdc14a810 0x7f4cdc14ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:20.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce0baf700 1 -- 172.21.15.111:0/482429098 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cdc14b420 msgr2=0x7f4cdc147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:20.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce0baf700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cdc14b420 0x7f4cdc147e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:20.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce0baf700 1 -- 172.21.15.111:0/482429098 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cdc160e20 con 0x7f4ccc004d90 2023-11-28T13:05:20.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4cdbfff700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cdc14b420 0x7f4cdc147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:05:20.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.559+0000 7f4ce0baf700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc004d90 0x7f4cdc0c6db0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0006fd0 tx=0x7f4cd0007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:20.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.560+0000 7f4cd97fa700 1 -- 172.21.15.111:0/482429098 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd000f8a0 con 0x7f4ccc004d90 2023-11-28T13:05:20.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.560+0000 7f4cd97fa700 1 -- 172.21.15.111:0/482429098 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd000fa20 con 0x7f4ccc004d90 2023-11-28T13:05:20.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.560+0000 7f4ce2e13700 1 -- 172.21.15.111:0/482429098 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cdc160b40 con 0x7f4ccc004d90 2023-11-28T13:05:20.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.560+0000 7f4cd97fa700 1 -- 172.21.15.111:0/482429098 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd000fdd0 con 0x7f4ccc004d90 2023-11-28T13:05:20.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.560+0000 7f4ce2e13700 1 -- 172.21.15.111:0/482429098 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cdc161100 con 0x7f4ccc004d90 2023-11-28T13:05:20.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.561+0000 7f4ce2e13700 1 -- 172.21.15.111:0/482429098 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cdc0c98c0 con 0x7f4ccc004d90 2023-11-28T13:05:20.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.561+0000 7f4cd97fa700 1 -- 172.21.15.111:0/482429098 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4cd00133f0 con 0x7f4ccc004d90 2023-11-28T13:05:20.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.561+0000 7f4cd97fa700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4cc00682f0 0x7f4cc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:20.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.562+0000 7f4cd97fa700 1 -- 172.21.15.111:0/482429098 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f4cd0085eb0 con 0x7f4ccc004d90 2023-11-28T13:05:20.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.562+0000 7f4cdbfff700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4cc00682f0 0x7f4cc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:20.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.562+0000 7f4cdbfff700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4cc00682f0 0x7f4cc006a7a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc14b7f0 tx=0x7f4cc8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:20.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.566+0000 7f4cd97fa700 1 -- 172.21.15.111:0/482429098 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4cdc0c98c0 con 0x7f4ccc004d90 2023-11-28T13:05:20.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.788+0000 7f4ce2e13700 1 -- 172.21.15.111:0/482429098 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4cdc15cd90 con 0x7f4ccc004d90 2023-11-28T13:05:20.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.789+0000 7f4cd97fa700 1 -- 172.21.15.111:0/482429098 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f4cdc15cd90 con 0x7f4ccc004d90 2023-11-28T13:05:20.790 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:05:20.791 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:05:20.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.791+0000 7f4ce2e13700 1 -- 172.21.15.111:0/482429098 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4cc00682f0 msgr2=0x7f4cc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:20.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.791+0000 7f4ce2e13700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4cc00682f0 0x7f4cc006a7a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc14b7f0 tx=0x7f4cc8008040 comp rx=0 tx=0).stop 2023-11-28T13:05:20.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.791+0000 7f4ce2e13700 1 -- 172.21.15.111:0/482429098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc004d90 msgr2=0x7f4cdc0c6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:20.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.791+0000 7f4ce2e13700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc004d90 0x7f4cdc0c6db0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0006fd0 tx=0x7f4cd0007680 comp rx=0 tx=0).stop 2023-11-28T13:05:20.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.791+0000 7f4ce2e13700 1 -- 172.21.15.111:0/482429098 shutdown_connections 2023-11-28T13:05:20.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.791+0000 7f4ce2e13700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4cdc14a810 0x7f4cdc14ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:20.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.791+0000 7f4ce2e13700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc004d90 0x7f4cdc0c6db0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:20.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.791+0000 7f4ce2e13700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4cc00682f0 0x7f4cc006a7a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:20.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.791+0000 7f4ce2e13700 1 --2- 172.21.15.111:0/482429098 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4cdc14b420 0x7f4cdc147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:20.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.791+0000 7f4ce2e13700 1 -- 172.21.15.111:0/482429098 >> 172.21.15.111:0/482429098 conn(0x7f4cdc098ee0 msgr2=0x7f4cdc0ae680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:20.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.791+0000 7f4ce2e13700 1 -- 172.21.15.111:0/482429098 shutdown_connections 2023-11-28T13:05:20.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:20.791+0000 7f4ce2e13700 1 -- 172.21.15.111:0/482429098 wait complete. 2023-11-28T13:05:20.804 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:05:25.805 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:05:25.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.935+0000 7f883d133700 1 Processor -- start 2023-11-28T13:05:25.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.935+0000 7f883d133700 1 -- start start 2023-11-28T13:05:25.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f883d133700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f88380cdef0 0x7f88380d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:25.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f883d133700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88380cc5a0 0x7f88380ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:25.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f883d133700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88380d05a0 con 0x7f88380cfc90 2023-11-28T13:05:25.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f883d133700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88380ca260 con 0x7f88380cdef0 2023-11-28T13:05:25.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f883d133700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88380ccaa0 con 0x7f88380cc5a0 2023-11-28T13:05:25.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f8836d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f88380cfc90 0x7f88380cd7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:38988/0 (socket says 172.21.15.111:38988) 2023-11-28T13:05:25.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f8836d9d700 1 -- 172.21.15.111:0/3553650547 learned_addr learned my addr 172.21.15.111:0/3553650547 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:05:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f883659c700 1 --2- 172.21.15.111:0/3553650547 >> v2:172.21.15.149:3300/0 conn(0x7f88380cdef0 0x7f88380d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f883759e700 1 --2- 172.21.15.111:0/3553650547 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88380cc5a0 0x7f88380ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f8835d9b700 1 -- 172.21.15.111:0/3553650547 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783903466 0 0) 0x7f882c001110 con 0x7f88380cfc90 2023-11-28T13:05:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f8835d9b700 1 -- 172.21.15.111:0/3553650547 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88200035b0 con 0x7f88380cfc90 2023-11-28T13:05:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f883759e700 1 -- 172.21.15.111:0/3553650547 >> v2:172.21.15.149:3300/0 conn(0x7f88380cdef0 msgr2=0x7f88380d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f883759e700 1 --2- 172.21.15.111:0/3553650547 >> v2:172.21.15.149:3300/0 conn(0x7f88380cdef0 0x7f88380d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f883759e700 1 -- 172.21.15.111:0/3553650547 >> v1:172.21.15.111:6789/0 conn(0x7f88380cfc90 legacy=0x7f88380cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f883759e700 1 -- 172.21.15.111:0/3553650547 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88380ccd80 con 0x7f88380cc5a0 2023-11-28T13:05:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.936+0000 7f883659c700 1 --2- 172.21.15.111:0/3553650547 >> v2:172.21.15.149:3300/0 conn(0x7f88380cdef0 0x7f88380d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:05:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.937+0000 7f883759e700 1 --2- 172.21.15.111:0/3553650547 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88380cc5a0 0x7f88380ca6f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8830009d90 tx=0x7f883000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=598ab0c15a8b7ee1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.937+0000 7f8835d9b700 1 -- 172.21.15.111:0/3553650547 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883000e070 con 0x7f88380cc5a0 2023-11-28T13:05:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.937+0000 7f8835d9b700 1 -- 172.21.15.111:0/3553650547 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883000bb20 con 0x7f88380cc5a0 2023-11-28T13:05:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.937+0000 7f8835d9b700 1 -- 172.21.15.111:0/3553650547 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883000be90 con 0x7f88380cc5a0 2023-11-28T13:05:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.937+0000 7f883d133700 1 -- 172.21.15.111:0/3553650547 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88380cc5a0 msgr2=0x7f88380ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.937+0000 7f883d133700 1 --2- 172.21.15.111:0/3553650547 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88380cc5a0 0x7f88380ca6f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8830009d90 tx=0x7f883000b330 comp rx=0 tx=0).stop 2023-11-28T13:05:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.937+0000 7f883d133700 1 -- 172.21.15.111:0/3553650547 shutdown_connections 2023-11-28T13:05:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.937+0000 7f883d133700 1 --2- 172.21.15.111:0/3553650547 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88380cc5a0 0x7f88380ca6f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.937+0000 7f883d133700 1 --2- 172.21.15.111:0/3553650547 >> v2:172.21.15.149:3300/0 conn(0x7f88380cdef0 0x7f88380d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.937+0000 7f883d133700 1 -- 172.21.15.111:0/3553650547 >> 172.21.15.111:0/3553650547 conn(0x7f8838098ee0 msgr2=0x7f883806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.937+0000 7f883d133700 1 -- 172.21.15.111:0/3553650547 shutdown_connections 2023-11-28T13:05:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.937+0000 7f883d133700 1 -- 172.21.15.111:0/3553650547 wait complete. 2023-11-28T13:05:25.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.939+0000 7f883d133700 1 Processor -- start 2023-11-28T13:05:25.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.939+0000 7f883d133700 1 -- start start 2023-11-28T13:05:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883d133700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f88380cc5a0 0x7f88380b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883d133700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88380cdef0 0x7f88380af310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883d133700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88380cfc90 0x7f88381f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883d133700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88380af9a0 con 0x7f88380cdef0 2023-11-28T13:05:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883d133700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88381425d0 con 0x7f88380cc5a0 2023-11-28T13:05:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883d133700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88380c7aa0 con 0x7f88380cfc90 2023-11-28T13:05:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88380cdef0 0x7f88380af310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88380cdef0 0x7f88380af310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38218/0 (socket says 172.21.15.111:38218) 2023-11-28T13:05:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883659c700 1 -- 172.21.15.111:0/1262504659 learned_addr learned my addr 172.21.15.111:0/1262504659 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f8836d9d700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f88380cc5a0 0x7f88380b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883759e700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88380cfc90 0x7f88381f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883659c700 1 -- 172.21.15.111:0/1262504659 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88380cfc90 msgr2=0x7f88381f5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883659c700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88380cfc90 0x7f88381f5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883659c700 1 -- 172.21.15.111:0/1262504659 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f88380cc5a0 msgr2=0x7f88380b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883659c700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f88380cc5a0 0x7f88380b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883659c700 1 -- 172.21.15.111:0/1262504659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88280094a0 con 0x7f88380cdef0 2023-11-28T13:05:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883759e700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88380cfc90 0x7f88381f5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:05:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f8836d9d700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f88380cc5a0 0x7f88380b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:05:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883659c700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88380cdef0 0x7f88380af310 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f882800ebe0 tx=0x7f8828009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f881f7fe700 1 -- 172.21.15.111:0/1262504659 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882800c890 con 0x7f88380cdef0 2023-11-28T13:05:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f881f7fe700 1 -- 172.21.15.111:0/1262504659 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882800c9f0 con 0x7f88380cdef0 2023-11-28T13:05:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883d133700 1 -- 172.21.15.111:0/1262504659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8830009a70 con 0x7f88380cdef0 2023-11-28T13:05:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f881f7fe700 1 -- 172.21.15.111:0/1262504659 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882800cda0 con 0x7f88380cdef0 2023-11-28T13:05:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.940+0000 7f883d133700 1 -- 172.21.15.111:0/1262504659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88380cd340 con 0x7f88380cdef0 2023-11-28T13:05:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.942+0000 7f883d133700 1 -- 172.21.15.111:0/1262504659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88380cd060 con 0x7f88380cdef0 2023-11-28T13:05:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.942+0000 7f881f7fe700 1 -- 172.21.15.111:0/1262504659 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8828019030 con 0x7f88380cdef0 2023-11-28T13:05:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.942+0000 7f881f7fe700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f88240682f0 0x7f882406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.942+0000 7f881f7fe700 1 -- 172.21.15.111:0/1262504659 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f8828085b70 con 0x7f88380cdef0 2023-11-28T13:05:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.942+0000 7f8836d9d700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f88240682f0 0x7f882406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.943+0000 7f8836d9d700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f88240682f0 0x7f882406a7a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f882c005910 tx=0x7f882c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:25.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:25.947+0000 7f881f7fe700 1 -- 172.21.15.111:0/1262504659 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f88280521d0 con 0x7f88380cdef0 2023-11-28T13:05:26.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.170+0000 7f883d133700 1 -- 172.21.15.111:0/1262504659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8838143280 con 0x7f88380cdef0 2023-11-28T13:05:26.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.171+0000 7f881f7fe700 1 -- 172.21.15.111:0/1262504659 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f88280557f0 con 0x7f88380cdef0 2023-11-28T13:05:26.172 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:05:26.172 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:05:26.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.173+0000 7f883d133700 1 -- 172.21.15.111:0/1262504659 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f88240682f0 msgr2=0x7f882406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:26.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.173+0000 7f883d133700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f88240682f0 0x7f882406a7a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f882c005910 tx=0x7f882c009040 comp rx=0 tx=0).stop 2023-11-28T13:05:26.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.173+0000 7f883d133700 1 -- 172.21.15.111:0/1262504659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88380cdef0 msgr2=0x7f88380af310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:26.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.173+0000 7f883d133700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88380cdef0 0x7f88380af310 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f882800ebe0 tx=0x7f8828009e60 comp rx=0 tx=0).stop 2023-11-28T13:05:26.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.173+0000 7f883d133700 1 -- 172.21.15.111:0/1262504659 shutdown_connections 2023-11-28T13:05:26.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.173+0000 7f883d133700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f88380cfc90 0x7f88381f5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:26.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.173+0000 7f883d133700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f88240682f0 0x7f882406a7a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:26.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.173+0000 7f883d133700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f88380cc5a0 0x7f88380b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:26.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.173+0000 7f883d133700 1 --2- 172.21.15.111:0/1262504659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88380cdef0 0x7f88380af310 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:26.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.173+0000 7f883d133700 1 -- 172.21.15.111:0/1262504659 >> 172.21.15.111:0/1262504659 conn(0x7f8838098ee0 msgr2=0x7f88380acb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:26.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.173+0000 7f883d133700 1 -- 172.21.15.111:0/1262504659 shutdown_connections 2023-11-28T13:05:26.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:26.173+0000 7f883d133700 1 -- 172.21.15.111:0/1262504659 wait complete. 2023-11-28T13:05:26.186 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:05:29.497 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:05:29.500 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:05:31.187 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:05:31.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.317+0000 7fcb28571700 1 Processor -- start 2023-11-28T13:05:31.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.317+0000 7fcb28571700 1 -- start start 2023-11-28T13:05:31.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.318+0000 7fcb28571700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcb200b20f0 0x7fcb200b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:31.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.318+0000 7fcb28571700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb200b40c0 0x7fcb2006b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:31.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.318+0000 7fcb28571700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fcb2006b8c0 0x7fcb200ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:31.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.318+0000 7fcb28571700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb2006bd40 con 0x7fcb200b20f0 2023-11-28T13:05:31.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.318+0000 7fcb28571700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb200b5300 con 0x7fcb2006b8c0 2023-11-28T13:05:31.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.318+0000 7fcb28571700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200cfbb0 con 0x7fcb200b40c0 2023-11-28T13:05:31.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.318+0000 7fcb2630d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcb200b20f0 0x7fcb200b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:31.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.319+0000 7fcb2630d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcb200b20f0 0x7fcb200b3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38224/0 (socket says 172.21.15.111:38224) 2023-11-28T13:05:31.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.319+0000 7fcb2630d700 1 -- 172.21.15.111:0/1182144451 learned_addr learned my addr 172.21.15.111:0/1182144451 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:31.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.319+0000 7fcb26b0e700 1 --2- 172.21.15.111:0/1182144451 >> v2:172.21.15.149:3300/0 conn(0x7fcb2006b8c0 0x7fcb200ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:31.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.319+0000 7fcb25b0c700 1 --2- 172.21.15.111:0/1182144451 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb200b40c0 0x7fcb2006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:31.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.319+0000 7fcb2630d700 1 -- 172.21.15.111:0/1182144451 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb200b40c0 msgr2=0x7fcb2006b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:31.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.319+0000 7fcb2630d700 1 --2- 172.21.15.111:0/1182144451 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb200b40c0 0x7fcb2006b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:31.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.319+0000 7fcb2630d700 1 -- 172.21.15.111:0/1182144451 >> v2:172.21.15.149:3300/0 conn(0x7fcb2006b8c0 msgr2=0x7fcb200ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:31.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.319+0000 7fcb2630d700 1 --2- 172.21.15.111:0/1182144451 >> v2:172.21.15.149:3300/0 conn(0x7fcb2006b8c0 0x7fcb200ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:31.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.319+0000 7fcb2630d700 1 -- 172.21.15.111:0/1182144451 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200b46f0 con 0x7fcb200b20f0 2023-11-28T13:05:31.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.319+0000 7fcb2630d700 1 --2- 172.21.15.111:0/1182144451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b20f0 0x7fcb200b3b80 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fcb2006d1a0 tx=0x7fcb10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ae8ae6418adfa1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:31.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.319+0000 7fcb2530b700 1 -- 172.21.15.111:0/1182144451 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1000b870 con 0x7fcb200b20f0 2023-11-28T13:05:31.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.319+0000 7fcb2530b700 1 -- 172.21.15.111:0/1182144451 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1000b9d0 con 0x7fcb200b20f0 2023-11-28T13:05:31.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.319+0000 7fcb2530b700 1 -- 172.21.15.111:0/1182144451 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1000bd60 con 0x7fcb200b20f0 2023-11-28T13:05:31.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.320+0000 7fcb28571700 1 -- 172.21.15.111:0/1182144451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b20f0 msgr2=0x7fcb200b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:31.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.320+0000 7fcb28571700 1 --2- 172.21.15.111:0/1182144451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b20f0 0x7fcb200b3b80 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fcb2006d1a0 tx=0x7fcb10009bd0 comp rx=0 tx=0).stop 2023-11-28T13:05:31.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.320+0000 7fcb28571700 1 -- 172.21.15.111:0/1182144451 shutdown_connections 2023-11-28T13:05:31.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.320+0000 7fcb28571700 1 --2- 172.21.15.111:0/1182144451 >> v2:172.21.15.149:3300/0 conn(0x7fcb2006b8c0 0x7fcb200ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:31.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.320+0000 7fcb28571700 1 --2- 172.21.15.111:0/1182144451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b20f0 0x7fcb200b3b80 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:31.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.320+0000 7fcb28571700 1 --2- 172.21.15.111:0/1182144451 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb200b40c0 0x7fcb2006b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:31.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.320+0000 7fcb28571700 1 -- 172.21.15.111:0/1182144451 >> 172.21.15.111:0/1182144451 conn(0x7fcb2009a040 msgr2=0x7fcb200d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:31.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.320+0000 7fcb28571700 1 -- 172.21.15.111:0/1182144451 shutdown_connections 2023-11-28T13:05:31.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.320+0000 7fcb28571700 1 -- 172.21.15.111:0/1182144451 wait complete. 2023-11-28T13:05:31.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.321+0000 7fcb28571700 1 Processor -- start 2023-11-28T13:05:31.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.321+0000 7fcb28571700 1 -- start start 2023-11-28T13:05:31.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.322+0000 7fcb28571700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb2006b8c0 0x7fcb2015c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:31.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.322+0000 7fcb28571700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b20f0 0x7fcb2009c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:31.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.322+0000 7fcb28571700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb200b40c0 0x7fcb2009d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:31.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.322+0000 7fcb28571700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2009e1c0 con 0x7fcb200b20f0 2023-11-28T13:05:31.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.322+0000 7fcb28571700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2009df10 con 0x7fcb200b40c0 2023-11-28T13:05:31.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.322+0000 7fcb28571700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200cc550 con 0x7fcb2006b8c0 2023-11-28T13:05:31.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.322+0000 7fcb25b0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b20f0 0x7fcb2009c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:31.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.322+0000 7fcb25b0c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b20f0 0x7fcb2009c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38228/0 (socket says 172.21.15.111:38228) 2023-11-28T13:05:31.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.322+0000 7fcb25b0c700 1 -- 172.21.15.111:0/430465592 learned_addr learned my addr 172.21.15.111:0/430465592 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:31.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb26b0e700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb200b40c0 0x7fcb2009d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:31.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb2630d700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb2006b8c0 0x7fcb2015c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:31.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb25b0c700 1 -- 172.21.15.111:0/430465592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb2006b8c0 msgr2=0x7fcb2015c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:31.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb25b0c700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb2006b8c0 0x7fcb2015c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:31.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb25b0c700 1 -- 172.21.15.111:0/430465592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb200b40c0 msgr2=0x7fcb2009d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:31.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb25b0c700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb200b40c0 0x7fcb2009d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:31.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb25b0c700 1 -- 172.21.15.111:0/430465592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb10009550 con 0x7fcb200b20f0 2023-11-28T13:05:31.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb2630d700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb2006b8c0 0x7fcb2015c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:05:31.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb26b0e700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb200b40c0 0x7fcb2009d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:05:31.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb25b0c700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b20f0 0x7fcb2009c9e0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00e810 tx=0x7fcb1c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:31.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb16ffd700 1 -- 172.21.15.111:0/430465592 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c013610 con 0x7fcb200b20f0 2023-11-28T13:05:31.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb16ffd700 1 -- 172.21.15.111:0/430465592 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1c013770 con 0x7fcb200b20f0 2023-11-28T13:05:31.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb28571700 1 -- 172.21.15.111:0/430465592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb200b4cb0 con 0x7fcb200b20f0 2023-11-28T13:05:31.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb16ffd700 1 -- 172.21.15.111:0/430465592 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c013b20 con 0x7fcb200b20f0 2023-11-28T13:05:31.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.323+0000 7fcb28571700 1 -- 172.21.15.111:0/430465592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb200b49d0 con 0x7fcb200b20f0 2023-11-28T13:05:31.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.324+0000 7fcb28571700 1 -- 172.21.15.111:0/430465592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb200b4f90 con 0x7fcb200b20f0 2023-11-28T13:05:31.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.324+0000 7fcb16ffd700 1 -- 172.21.15.111:0/430465592 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcb1c023070 con 0x7fcb200b20f0 2023-11-28T13:05:31.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.325+0000 7fcb16ffd700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb040682f0 0x7fcb0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:31.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.325+0000 7fcb16ffd700 1 -- 172.21.15.111:0/430465592 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fcb1c085b20 con 0x7fcb200b20f0 2023-11-28T13:05:31.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.325+0000 7fcb2630d700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb040682f0 0x7fcb0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:31.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.325+0000 7fcb2630d700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb040682f0 0x7fcb0406a7a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009840 tx=0x7fcb10008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:31.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.329+0000 7fcb16ffd700 1 -- 172.21.15.111:0/430465592 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcb1c052180 con 0x7fcb200b20f0 2023-11-28T13:05:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.553+0000 7fcb28571700 1 -- 172.21.15.111:0/430465592 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcb20067170 con 0x7fcb200b20f0 2023-11-28T13:05:31.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.553+0000 7fcb16ffd700 1 -- 172.21.15.111:0/430465592 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7fcb1c0557a0 con 0x7fcb200b20f0 2023-11-28T13:05:31.555 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:05:31.555 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:05:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.555+0000 7fcb28571700 1 -- 172.21.15.111:0/430465592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb040682f0 msgr2=0x7fcb0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.555+0000 7fcb28571700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb040682f0 0x7fcb0406a7a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009840 tx=0x7fcb10008030 comp rx=0 tx=0).stop 2023-11-28T13:05:31.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.555+0000 7fcb28571700 1 -- 172.21.15.111:0/430465592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b20f0 msgr2=0x7fcb2009c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.555+0000 7fcb28571700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b20f0 0x7fcb2009c9e0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00e810 tx=0x7fcb1c00ebd0 comp rx=0 tx=0).stop 2023-11-28T13:05:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.555+0000 7fcb28571700 1 -- 172.21.15.111:0/430465592 shutdown_connections 2023-11-28T13:05:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.555+0000 7fcb28571700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb040682f0 0x7fcb0406a7a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.555+0000 7fcb28571700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb200b40c0 0x7fcb2009d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.555+0000 7fcb28571700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb2006b8c0 0x7fcb2015c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.555+0000 7fcb28571700 1 --2- 172.21.15.111:0/430465592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb200b20f0 0x7fcb2009c9e0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.555+0000 7fcb28571700 1 -- 172.21.15.111:0/430465592 >> 172.21.15.111:0/430465592 conn(0x7fcb2009a040 msgr2=0x7fcb2006f140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.556+0000 7fcb28571700 1 -- 172.21.15.111:0/430465592 shutdown_connections 2023-11-28T13:05:31.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:31.556+0000 7fcb28571700 1 -- 172.21.15.111:0/430465592 wait complete. 2023-11-28T13:05:31.568 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:05:36.570 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:05:36.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.701+0000 7ff7697d8700 1 Processor -- start 2023-11-28T13:05:36.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.701+0000 7ff7697d8700 1 -- start start 2023-11-28T13:05:36.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.702+0000 7ff7697d8700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7ff7640b8510 0x7ff76406b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:36.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.702+0000 7ff7697d8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff7640d0d60 0x7ff7640d1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:36.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.702+0000 7ff7697d8700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7640b88e0 con 0x7ff7640cd460 2023-11-28T13:05:36.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.702+0000 7ff7697d8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7640cdbd0 con 0x7ff7640d0d60 2023-11-28T13:05:36.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.702+0000 7ff7697d8700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7640d02d0 con 0x7ff7640b8510 2023-11-28T13:05:36.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.702+0000 7ff7637fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff7640d0d60 0x7ff7640d1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:36.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.702+0000 7ff7637fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff7640d0d60 0x7ff7640d1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38238/0 (socket says 172.21.15.111:38238) 2023-11-28T13:05:36.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.702+0000 7ff7637fe700 1 -- 172.21.15.111:0/3807474490 learned_addr learned my addr 172.21.15.111:0/3807474490 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:36.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.702+0000 7ff7637fe700 1 -- 172.21.15.111:0/3807474490 >> v2:172.21.15.149:3300/0 conn(0x7ff7640b8510 msgr2=0x7ff76406b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:36.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.702+0000 7ff762ffd700 1 --2- 172.21.15.111:0/3807474490 >> v2:172.21.15.149:3300/0 conn(0x7ff7640b8510 0x7ff76406b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:36.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.702+0000 7ff7637fe700 1 --2- 172.21.15.111:0/3807474490 >> v2:172.21.15.149:3300/0 conn(0x7ff7640b8510 0x7ff76406b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:36.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.702+0000 7ff7637fe700 1 -- 172.21.15.111:0/3807474490 >> v1:172.21.15.149:6789/0 conn(0x7ff7640cd460 legacy=0x7ff76406beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:36.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.702+0000 7ff7637fe700 1 -- 172.21.15.111:0/3807474490 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7640cd8f0 con 0x7ff7640d0d60 2023-11-28T13:05:36.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.703+0000 7ff7637fe700 1 --2- 172.21.15.111:0/3807474490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7640d0d60 0x7ff7640d1200 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7ff7640b5a50 tx=0x7ff75c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2304c3f70fa8b19e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:36.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.703+0000 7ff761ffb700 1 -- 172.21.15.111:0/3807474490 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c00e070 con 0x7ff7640d0d60 2023-11-28T13:05:36.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.703+0000 7ff761ffb700 1 -- 172.21.15.111:0/3807474490 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75c003a20 con 0x7ff7640d0d60 2023-11-28T13:05:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.703+0000 7ff761ffb700 1 -- 172.21.15.111:0/3807474490 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c003dd0 con 0x7ff7640d0d60 2023-11-28T13:05:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.703+0000 7ff7697d8700 1 -- 172.21.15.111:0/3807474490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7640d0d60 msgr2=0x7ff7640d1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.703+0000 7ff7697d8700 1 --2- 172.21.15.111:0/3807474490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7640d0d60 0x7ff7640d1200 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7ff7640b5a50 tx=0x7ff75c00b330 comp rx=0 tx=0).stop 2023-11-28T13:05:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.703+0000 7ff7697d8700 1 -- 172.21.15.111:0/3807474490 shutdown_connections 2023-11-28T13:05:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.703+0000 7ff7697d8700 1 --2- 172.21.15.111:0/3807474490 >> v2:172.21.15.149:3300/0 conn(0x7ff7640b8510 0x7ff76406b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.703+0000 7ff7697d8700 1 --2- 172.21.15.111:0/3807474490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7640d0d60 0x7ff7640d1200 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.703+0000 7ff7697d8700 1 -- 172.21.15.111:0/3807474490 >> 172.21.15.111:0/3807474490 conn(0x7ff7640719a0 msgr2=0x7ff764071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.703+0000 7ff7697d8700 1 -- 172.21.15.111:0/3807474490 shutdown_connections 2023-11-28T13:05:36.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.703+0000 7ff7697d8700 1 -- 172.21.15.111:0/3807474490 wait complete. 2023-11-28T13:05:36.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.705+0000 7ff7697d8700 1 Processor -- start 2023-11-28T13:05:36.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.705+0000 7ff7697d8700 1 -- start start 2023-11-28T13:05:36.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff7697d8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7640b8510 0x7ff7641591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:36.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff7697d8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7640cd460 0x7ff7640cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:36.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff7697d8700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7640d0d60 0x7ff76415aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:36.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff7697d8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7640cd190 con 0x7ff7640b8510 2023-11-28T13:05:36.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff7697d8700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7641f3020 con 0x7ff7640d0d60 2023-11-28T13:05:36.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff7697d8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7641f3300 con 0x7ff7640cd460 2023-11-28T13:05:36.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff762ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7640b8510 0x7ff7641591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:36.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff762ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7640b8510 0x7ff7641591e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38240/0 (socket says 172.21.15.111:38240) 2023-11-28T13:05:36.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff762ffd700 1 -- 172.21.15.111:0/1071602623 learned_addr learned my addr 172.21.15.111:0/1071602623 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:36.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff7637fe700 1 --2- 172.21.15.111:0/1071602623 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7640d0d60 0x7ff76415aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:36.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff762ffd700 1 -- 172.21.15.111:0/1071602623 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7640cd460 msgr2=0x7ff7640cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:36.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff762ffd700 1 --2- 172.21.15.111:0/1071602623 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7640cd460 0x7ff7640cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:36.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff762ffd700 1 -- 172.21.15.111:0/1071602623 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7640d0d60 msgr2=0x7ff76415aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:36.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff762ffd700 1 --2- 172.21.15.111:0/1071602623 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7640d0d60 0x7ff76415aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:36.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff762ffd700 1 -- 172.21.15.111:0/1071602623 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75c009b30 con 0x7ff7640b8510 2023-11-28T13:05:36.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff762ffd700 1 --2- 172.21.15.111:0/1071602623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7640b8510 0x7ff7641591e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7ff75400c7f0 tx=0x7ff75400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:36.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff74b7fe700 1 -- 172.21.15.111:0/1071602623 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754011800 con 0x7ff7640b8510 2023-11-28T13:05:36.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff74b7fe700 1 -- 172.21.15.111:0/1071602623 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff754011960 con 0x7ff7640b8510 2023-11-28T13:05:36.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff7697d8700 1 -- 172.21.15.111:0/1071602623 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7640cfff0 con 0x7ff7640b8510 2023-11-28T13:05:36.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff74b7fe700 1 -- 172.21.15.111:0/1071602623 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754011cf0 con 0x7ff7640b8510 2023-11-28T13:05:36.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.706+0000 7ff7697d8700 1 -- 172.21.15.111:0/1071602623 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7640cfd10 con 0x7ff7640b8510 2023-11-28T13:05:36.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.707+0000 7ff7697d8700 1 -- 172.21.15.111:0/1071602623 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7641f3ad0 con 0x7ff7640b8510 2023-11-28T13:05:36.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.708+0000 7ff74b7fe700 1 -- 172.21.15.111:0/1071602623 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff754021070 con 0x7ff7640b8510 2023-11-28T13:05:36.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.708+0000 7ff74b7fe700 1 --2- 172.21.15.111:0/1071602623 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7500682f0 0x7ff75006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:36.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.708+0000 7ff74b7fe700 1 -- 172.21.15.111:0/1071602623 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7ff754085d40 con 0x7ff7640b8510 2023-11-28T13:05:36.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.708+0000 7ff7627fc700 1 --2- 172.21.15.111:0/1071602623 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7500682f0 0x7ff75006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:36.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.709+0000 7ff7627fc700 1 --2- 172.21.15.111:0/1071602623 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7500682f0 0x7ff75006a7a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff7640cc470 tx=0x7ff758007a40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:36.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.712+0000 7ff74b7fe700 1 -- 172.21.15.111:0/1071602623 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff7540523a0 con 0x7ff7640b8510 2023-11-28T13:05:36.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.937+0000 7ff7697d8700 1 -- 172.21.15.111:0/1071602623 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff7641f3470 con 0x7ff7640b8510 2023-11-28T13:05:36.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.937+0000 7ff74b7fe700 1 -- 172.21.15.111:0/1071602623 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7ff75401f080 con 0x7ff7640b8510 2023-11-28T13:05:36.939 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:05:36.939 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:05:36.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.939+0000 7ff7697d8700 1 -- 172.21.15.111:0/1071602623 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7500682f0 msgr2=0x7ff75006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:36.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.939+0000 7ff7697d8700 1 --2- 172.21.15.111:0/1071602623 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7500682f0 0x7ff75006a7a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff7640cc470 tx=0x7ff758007a40 comp rx=0 tx=0).stop 2023-11-28T13:05:36.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.939+0000 7ff7697d8700 1 -- 172.21.15.111:0/1071602623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7640b8510 msgr2=0x7ff7641591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:36.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.939+0000 7ff7697d8700 1 --2- 172.21.15.111:0/1071602623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7640b8510 0x7ff7641591e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7ff75400c7f0 tx=0x7ff75400cbb0 comp rx=0 tx=0).stop 2023-11-28T13:05:36.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.939+0000 7ff7697d8700 1 -- 172.21.15.111:0/1071602623 shutdown_connections 2023-11-28T13:05:36.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.939+0000 7ff7697d8700 1 --2- 172.21.15.111:0/1071602623 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff7500682f0 0x7ff75006a7a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:36.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.939+0000 7ff7697d8700 1 --2- 172.21.15.111:0/1071602623 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff7640d0d60 0x7ff76415aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:36.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.939+0000 7ff7697d8700 1 --2- 172.21.15.111:0/1071602623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7640b8510 0x7ff7641591e0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:36.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.939+0000 7ff7697d8700 1 --2- 172.21.15.111:0/1071602623 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff7640cd460 0x7ff7640cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:36.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.939+0000 7ff7697d8700 1 -- 172.21.15.111:0/1071602623 >> 172.21.15.111:0/1071602623 conn(0x7ff7640719a0 msgr2=0x7ff7641e7280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:36.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.939+0000 7ff7697d8700 1 -- 172.21.15.111:0/1071602623 shutdown_connections 2023-11-28T13:05:36.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:36.940+0000 7ff7697d8700 1 -- 172.21.15.111:0/1071602623 wait complete. 2023-11-28T13:05:36.952 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:05:41.954 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:05:42.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.084+0000 7ff8987ab700 1 Processor -- start 2023-11-28T13:05:42.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.084+0000 7ff8987ab700 1 -- start start 2023-11-28T13:05:42.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff8987ab700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8900cdd90 0x7ff8900b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:42.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff8987ab700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8900ad420 con 0x7ff890159960 2023-11-28T13:05:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff8987ab700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8900ac810 con 0x7ff8900aecc0 2023-11-28T13:05:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff8987ab700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff890159d30 con 0x7ff8900cdd90 2023-11-28T13:05:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff896547700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7ff890159960 0x7ff8900cd680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39026/0 (socket says 172.21.15.111:39026) 2023-11-28T13:05:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff896547700 1 -- 172.21.15.111:0/1509666149 learned_addr learned my addr 172.21.15.111:0/1509666149 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:05:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895d46700 1 --2- 172.21.15.111:0/1509666149 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8900cdd90 0x7ff8900b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:42.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485619685 0 0) 0x7ff8900ad420 con 0x7ff890159960 2023-11-28T13:05:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff87c0035b0 con 0x7ff890159960 2023-11-28T13:05:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819294732 0 0) 0x7ff8900ac810 con 0x7ff8900aecc0 2023-11-28T13:05:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8900ad420 con 0x7ff8900aecc0 2023-11-28T13:05:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2719373178 0 0) 0x7ff87c0035b0 con 0x7ff890159960 2023-11-28T13:05:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8900cdd90 msgr2=0x7ff8900b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 --2- 172.21.15.111:0/1509666149 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8900cdd90 0x7ff8900b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:42.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 >> v1:172.21.15.149:6789/0 conn(0x7ff8900aecc0 legacy=0x7ff8900ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:42.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8900acaf0 con 0x7ff890159960 2023-11-28T13:05:42.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7ff88c0017f0 con 0x7ff890159960 2023-11-28T13:05:42.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 >> v1:172.21.15.111:6789/0 conn(0x7ff890159960 legacy=0x7ff8900cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:42.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 --2- 172.21.15.111:0/1509666149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff87c003bc0 0x7ff87c003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:42.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 --2- 172.21.15.111:0/1509666149 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff87c004850 0x7ff87c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:42.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 --2- 172.21.15.111:0/1509666149 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8900cdd90 0x7ff87c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:42.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff890159d30 con 0x7ff87c003bc0 2023-11-28T13:05:42.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c00baf0 con 0x7ff87c004850 2023-11-28T13:05:42.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c00bcb0 con 0x7ff8900cdd90 2023-11-28T13:05:42.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.085+0000 7ff896547700 1 --2- 172.21.15.111:0/1509666149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff87c003bc0 0x7ff87c003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:42.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff895d46700 1 --2- 172.21.15.111:0/1509666149 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff87c004850 0x7ff87c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:42.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff896d48700 1 --2- 172.21.15.111:0/1509666149 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8900cdd90 0x7ff87c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:42.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff896547700 1 -- 172.21.15.111:0/1509666149 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8900cdd90 msgr2=0x7ff87c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:42.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff896547700 1 --2- 172.21.15.111:0/1509666149 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8900cdd90 0x7ff87c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:42.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff896547700 1 -- 172.21.15.111:0/1509666149 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff87c004850 msgr2=0x7ff87c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:42.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff896547700 1 --2- 172.21.15.111:0/1509666149 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff87c004850 0x7ff87c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:42.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff896547700 1 -- 172.21.15.111:0/1509666149 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c0035b0 con 0x7ff87c003bc0 2023-11-28T13:05:42.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff896d48700 1 --2- 172.21.15.111:0/1509666149 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff8900cdd90 0x7ff87c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:05:42.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff896547700 1 --2- 172.21.15.111:0/1509666149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff87c003bc0 0x7ff87c003fb0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7ff88c007cd0 tx=0x7ff87c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ce3178fcef0432 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:42.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c00cd50 con 0x7ff87c003bc0 2023-11-28T13:05:42.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88c001c70 con 0x7ff87c003bc0 2023-11-28T13:05:42.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff895545700 1 -- 172.21.15.111:0/1509666149 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c007310 con 0x7ff87c003bc0 2023-11-28T13:05:42.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff8987ab700 1 -- 172.21.15.111:0/1509666149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff87c003bc0 msgr2=0x7ff87c003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:42.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff8987ab700 1 --2- 172.21.15.111:0/1509666149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff87c003bc0 0x7ff87c003fb0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7ff88c007cd0 tx=0x7ff87c005880 comp rx=0 tx=0).stop 2023-11-28T13:05:42.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.086+0000 7ff896d48700 1 -- 172.21.15.111:0/1509666149 reap_dead start 2023-11-28T13:05:42.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.087+0000 7ff8987ab700 1 -- 172.21.15.111:0/1509666149 shutdown_connections 2023-11-28T13:05:42.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.087+0000 7ff8987ab700 1 -- 172.21.15.111:0/1509666149 >> 172.21.15.111:0/1509666149 conn(0x7ff890098ee0 msgr2=0x7ff89006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:42.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.087+0000 7ff8987ab700 1 -- 172.21.15.111:0/1509666149 shutdown_connections 2023-11-28T13:05:42.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.087+0000 7ff8987ab700 1 -- 172.21.15.111:0/1509666149 wait complete. 2023-11-28T13:05:42.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.088+0000 7ff8987ab700 1 Processor -- start 2023-11-28T13:05:42.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.088+0000 7ff8987ab700 1 -- start start 2023-11-28T13:05:42.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff8987ab700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff87c003bc0 0x7ff8901f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:42.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff8987ab700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8900b2970 0x7ff89006de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:42.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff8987ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8900b2de0 0x7ff89006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:42.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff8987ab700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8901f5a90 con 0x7ff8900b2de0 2023-11-28T13:05:42.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff8987ab700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8901537e0 con 0x7ff8900b2970 2023-11-28T13:05:42.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff8987ab700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8900b59a0 con 0x7ff87c003bc0 2023-11-28T13:05:42.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff896d48700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8900b2de0 0x7ff89006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:42.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff896d48700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8900b2de0 0x7ff89006e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38264/0 (socket says 172.21.15.111:38264) 2023-11-28T13:05:42.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff896d48700 1 -- 172.21.15.111:0/309352395 learned_addr learned my addr 172.21.15.111:0/309352395 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:42.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff895d46700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8900b2970 0x7ff89006de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:42.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff896547700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff87c003bc0 0x7ff8901f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:42.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff896d48700 1 -- 172.21.15.111:0/309352395 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff87c003bc0 msgr2=0x7ff8901f5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:42.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff896d48700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff87c003bc0 0x7ff8901f5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:42.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff896d48700 1 -- 172.21.15.111:0/309352395 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8900b2970 msgr2=0x7ff89006de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:42.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff896d48700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8900b2970 0x7ff89006de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:42.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff896d48700 1 -- 172.21.15.111:0/309352395 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8900ad0b0 con 0x7ff8900b2de0 2023-11-28T13:05:42.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff896d48700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8900b2de0 0x7ff89006e350 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7ff888001a50 tx=0x7ff88800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:42.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff882ffd700 1 -- 172.21.15.111:0/309352395 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888003c10 con 0x7ff8900b2de0 2023-11-28T13:05:42.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff882ffd700 1 -- 172.21.15.111:0/309352395 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff888003d90 con 0x7ff8900b2de0 2023-11-28T13:05:42.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff8987ab700 1 -- 172.21.15.111:0/309352395 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8900acdd0 con 0x7ff8900b2de0 2023-11-28T13:05:42.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff882ffd700 1 -- 172.21.15.111:0/309352395 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888005310 con 0x7ff8900b2de0 2023-11-28T13:05:42.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.089+0000 7ff8987ab700 1 -- 172.21.15.111:0/309352395 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8900b1fd0 con 0x7ff8900b2de0 2023-11-28T13:05:42.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.090+0000 7ff8987ab700 1 -- 172.21.15.111:0/309352395 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff890067170 con 0x7ff8900b2de0 2023-11-28T13:05:42.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.091+0000 7ff882ffd700 1 -- 172.21.15.111:0/309352395 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff888005470 con 0x7ff8900b2de0 2023-11-28T13:05:42.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.091+0000 7ff882ffd700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8740682f0 0x7ff87406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:42.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.091+0000 7ff882ffd700 1 -- 172.21.15.111:0/309352395 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7ff888085f20 con 0x7ff8900b2de0 2023-11-28T13:05:42.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.091+0000 7ff896547700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8740682f0 0x7ff87406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:42.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.092+0000 7ff896547700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8740682f0 0x7ff87406a7a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff88c013040 tx=0x7ff88c007f90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:42.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.096+0000 7ff882ffd700 1 -- 172.21.15.111:0/309352395 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff888052600 con 0x7ff8900b2de0 2023-11-28T13:05:42.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.320+0000 7ff8987ab700 1 -- 172.21.15.111:0/309352395 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff8900cdb70 con 0x7ff8900b2de0 2023-11-28T13:05:42.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.321+0000 7ff882ffd700 1 -- 172.21.15.111:0/309352395 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7ff888055c20 con 0x7ff8900b2de0 2023-11-28T13:05:42.323 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:05:42.324 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:05:42.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.322+0000 7ff8987ab700 1 -- 172.21.15.111:0/309352395 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8740682f0 msgr2=0x7ff87406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:42.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.323+0000 7ff8987ab700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8740682f0 0x7ff87406a7a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff88c013040 tx=0x7ff88c007f90 comp rx=0 tx=0).stop 2023-11-28T13:05:42.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.323+0000 7ff8987ab700 1 -- 172.21.15.111:0/309352395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8900b2de0 msgr2=0x7ff89006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:42.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.323+0000 7ff8987ab700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8900b2de0 0x7ff89006e350 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7ff888001a50 tx=0x7ff88800ee90 comp rx=0 tx=0).stop 2023-11-28T13:05:42.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.323+0000 7ff8987ab700 1 -- 172.21.15.111:0/309352395 shutdown_connections 2023-11-28T13:05:42.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.323+0000 7ff8987ab700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8900b2de0 0x7ff89006e350 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:42.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.323+0000 7ff8987ab700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7ff87c003bc0 0x7ff8901f5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:42.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.323+0000 7ff8987ab700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7ff8740682f0 0x7ff87406a7a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:42.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.323+0000 7ff8987ab700 1 --2- 172.21.15.111:0/309352395 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7ff8900b2970 0x7ff89006de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:42.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.323+0000 7ff8987ab700 1 -- 172.21.15.111:0/309352395 >> 172.21.15.111:0/309352395 conn(0x7ff890098ee0 msgr2=0x7ff8900ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:42.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.323+0000 7ff8987ab700 1 -- 172.21.15.111:0/309352395 shutdown_connections 2023-11-28T13:05:42.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:42.323+0000 7ff8987ab700 1 -- 172.21.15.111:0/309352395 wait complete. 2023-11-28T13:05:42.335 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:05:47.336 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:05:47.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.466+0000 7f1a73f2d700 1 Processor -- start 2023-11-28T13:05:47.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.466+0000 7f1a73f2d700 1 -- start start 2023-11-28T13:05:47.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a73f2d700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f1a6c15a5d0 0x7f1a6c0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:47.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a73f2d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1a6c0b0160 0x7f1a6c0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:47.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a73f2d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c0ae8b0 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a73f2d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6c0cc1b0 con 0x7f1a6c0b0160 2023-11-28T13:05:47.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a73f2d700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6c0cc390 con 0x7f1a6c15a5d0 2023-11-28T13:05:47.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a714c8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1a6c0b0160 0x7f1a6c0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:47.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a724ca700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1a6c0cb2d0 0x7f1a6c15e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39050/0 (socket says 172.21.15.111:39050) 2023-11-28T13:05:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a724ca700 1 -- 172.21.15.111:0/3334833571 learned_addr learned my addr 172.21.15.111:0/3334833571 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:05:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a71cc9700 1 --2- 172.21.15.111:0/3334833571 >> v2:172.21.15.149:3300/0 conn(0x7f1a6c15a5d0 0x7f1a6c0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a714c8700 1 -- 172.21.15.111:0/3334833571 >> v2:172.21.15.149:3300/0 conn(0x7f1a6c15a5d0 msgr2=0x7f1a6c0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a714c8700 1 --2- 172.21.15.111:0/3334833571 >> v2:172.21.15.149:3300/0 conn(0x7f1a6c15a5d0 0x7f1a6c0ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a714c8700 1 -- 172.21.15.111:0/3334833571 >> v1:172.21.15.111:6789/0 conn(0x7f1a6c0cb2d0 legacy=0x7f1a6c15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a714c8700 1 -- 172.21.15.111:0/3334833571 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c0cc670 con 0x7f1a6c0b0160 2023-11-28T13:05:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.467+0000 7f1a70cc7700 1 -- 172.21.15.111:0/3334833571 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764418369 0 0) 0x7f1a6c0ae8b0 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.468+0000 7f1a714c8700 1 --2- 172.21.15.111:0/3334833571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a6c0b0160 0x7f1a6c0ced90 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c067d30 tx=0x7f1a68009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212b8d795ffc754c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.468+0000 7f1a70cc7700 1 -- 172.21.15.111:0/3334833571 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a68011830 con 0x7f1a6c0b0160 2023-11-28T13:05:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.468+0000 7f1a70cc7700 1 -- 172.21.15.111:0/3334833571 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a68011990 con 0x7f1a6c0b0160 2023-11-28T13:05:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.468+0000 7f1a70cc7700 1 -- 172.21.15.111:0/3334833571 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a68011d20 con 0x7f1a6c0b0160 2023-11-28T13:05:47.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.468+0000 7f1a73f2d700 1 -- 172.21.15.111:0/3334833571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a6c0b0160 msgr2=0x7f1a6c0ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:47.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.468+0000 7f1a73f2d700 1 --2- 172.21.15.111:0/3334833571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a6c0b0160 0x7f1a6c0ced90 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c067d30 tx=0x7f1a68009b20 comp rx=0 tx=0).stop 2023-11-28T13:05:47.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.468+0000 7f1a73f2d700 1 -- 172.21.15.111:0/3334833571 shutdown_connections 2023-11-28T13:05:47.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.468+0000 7f1a73f2d700 1 --2- 172.21.15.111:0/3334833571 >> v2:172.21.15.149:3300/0 conn(0x7f1a6c15a5d0 0x7f1a6c0ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:47.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.468+0000 7f1a73f2d700 1 --2- 172.21.15.111:0/3334833571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a6c0b0160 0x7f1a6c0ced90 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:47.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.468+0000 7f1a73f2d700 1 -- 172.21.15.111:0/3334833571 >> 172.21.15.111:0/3334833571 conn(0x7f1a6c09a040 msgr2=0x7f1a6c0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:47.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.468+0000 7f1a73f2d700 1 -- 172.21.15.111:0/3334833571 shutdown_connections 2023-11-28T13:05:47.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.468+0000 7f1a73f2d700 1 -- 172.21.15.111:0/3334833571 wait complete. 2023-11-28T13:05:47.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.470+0000 7f1a73f2d700 1 Processor -- start 2023-11-28T13:05:47.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.470+0000 7f1a73f2d700 1 -- start start 2023-11-28T13:05:47.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.470+0000 7f1a73f2d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a6c0b0160 0x7f1a6c152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:47.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.470+0000 7f1a73f2d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a6c0cb2d0 0x7f1a6c1f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:47.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.470+0000 7f1a73f2d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1a6c15a5d0 0x7f1a6c0b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:47.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.470+0000 7f1a73f2d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c0ae1e0 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.470+0000 7f1a73f2d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c154240 con 0x7f1a6c0b0160 2023-11-28T13:05:47.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.470+0000 7f1a73f2d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c0ae4a0 con 0x7f1a6c15a5d0 2023-11-28T13:05:47.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a714c8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a6c0cb2d0 0x7f1a6c1f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:47.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a714c8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a6c0cb2d0 0x7f1a6c1f3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38274/0 (socket says 172.21.15.111:38274) 2023-11-28T13:05:47.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a714c8700 1 -- 172.21.15.111:0/1273524300 learned_addr learned my addr 172.21.15.111:0/1273524300 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:47.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a714c8700 1 -- 172.21.15.111:0/1273524300 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1a6c15a5d0 msgr2=0x7f1a6c0b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:47.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a714c8700 1 --2- 172.21.15.111:0/1273524300 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1a6c15a5d0 0x7f1a6c0b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:47.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a714c8700 1 -- 172.21.15.111:0/1273524300 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a6c0b0160 msgr2=0x7f1a6c152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:47.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a71cc9700 1 --2- 172.21.15.111:0/1273524300 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a6c0b0160 0x7f1a6c152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:47.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a714c8700 1 --2- 172.21.15.111:0/1273524300 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a6c0b0160 0x7f1a6c152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:47.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a714c8700 1 -- 172.21.15.111:0/1273524300 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a680094a0 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a714c8700 1 --2- 172.21.15.111:0/1273524300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a6c0cb2d0 0x7f1a6c1f3ab0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f1a68014040 tx=0x7f1a68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:47.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a627fc700 1 -- 172.21.15.111:0/1273524300 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a680038a0 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a627fc700 1 -- 172.21.15.111:0/1273524300 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6801e070 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a73f2d700 1 -- 172.21.15.111:0/1273524300 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6c0ccc30 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a627fc700 1 -- 172.21.15.111:0/1273524300 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a68003c20 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.471+0000 7f1a73f2d700 1 -- 172.21.15.111:0/1273524300 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c0cc950 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.472+0000 7f1a73f2d700 1 -- 172.21.15.111:0/1273524300 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6c0ccf10 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.473+0000 7f1a627fc700 1 -- 172.21.15.111:0/1273524300 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1a6800f050 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.473+0000 7f1a627fc700 1 --2- 172.21.15.111:0/1273524300 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1a500682f0 0x7f1a5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:47.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.473+0000 7f1a627fc700 1 -- 172.21.15.111:0/1273524300 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f1a68086dc0 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.473+0000 7f1a71cc9700 1 --2- 172.21.15.111:0/1273524300 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1a500682f0 0x7f1a5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:47.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.473+0000 7f1a71cc9700 1 --2- 172.21.15.111:0/1273524300 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1a500682f0 0x7f1a5006a7a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009a90 tx=0x7f1a5c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:47.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.477+0000 7f1a627fc700 1 -- 172.21.15.111:0/1273524300 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1a680534a0 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.703+0000 7f1a73f2d700 1 -- 172.21.15.111:0/1273524300 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1a6c067170 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.703+0000 7f1a627fc700 1 -- 172.21.15.111:0/1273524300 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f1a680150c0 con 0x7f1a6c0cb2d0 2023-11-28T13:05:47.705 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:05:47.705 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:05:47.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.705+0000 7f1a73f2d700 1 -- 172.21.15.111:0/1273524300 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1a500682f0 msgr2=0x7f1a5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:47.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.705+0000 7f1a73f2d700 1 --2- 172.21.15.111:0/1273524300 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1a500682f0 0x7f1a5006a7a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009a90 tx=0x7f1a5c008040 comp rx=0 tx=0).stop 2023-11-28T13:05:47.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.705+0000 7f1a73f2d700 1 -- 172.21.15.111:0/1273524300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a6c0cb2d0 msgr2=0x7f1a6c1f3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:47.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.705+0000 7f1a73f2d700 1 --2- 172.21.15.111:0/1273524300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a6c0cb2d0 0x7f1a6c1f3ab0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f1a68014040 tx=0x7f1a68004750 comp rx=0 tx=0).stop 2023-11-28T13:05:47.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.706+0000 7f1a73f2d700 1 -- 172.21.15.111:0/1273524300 shutdown_connections 2023-11-28T13:05:47.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.706+0000 7f1a73f2d700 1 --2- 172.21.15.111:0/1273524300 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1a6c15a5d0 0x7f1a6c0b0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:47.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.706+0000 7f1a73f2d700 1 --2- 172.21.15.111:0/1273524300 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1a500682f0 0x7f1a5006a7a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:47.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.706+0000 7f1a73f2d700 1 --2- 172.21.15.111:0/1273524300 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1a6c0b0160 0x7f1a6c152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:47.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.706+0000 7f1a73f2d700 1 --2- 172.21.15.111:0/1273524300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a6c0cb2d0 0x7f1a6c1f3ab0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:47.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.706+0000 7f1a73f2d700 1 -- 172.21.15.111:0/1273524300 >> 172.21.15.111:0/1273524300 conn(0x7f1a6c09a040 msgr2=0x7f1a6c0cd330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:47.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.706+0000 7f1a73f2d700 1 -- 172.21.15.111:0/1273524300 shutdown_connections 2023-11-28T13:05:47.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:47.706+0000 7f1a73f2d700 1 -- 172.21.15.111:0/1273524300 wait complete. 2023-11-28T13:05:47.719 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:05:52.720 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:05:52.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.850+0000 7f9e78822700 1 Processor -- start 2023-11-28T13:05:52.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.850+0000 7f9e78822700 1 -- start start 2023-11-28T13:05:52.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e78822700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740cfc90 0x7f9e740d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:52.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e78822700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e740d05a0 con 0x7f9e740cdef0 2023-11-28T13:05:52.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e78822700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e740ca260 con 0x7f9e740cc5a0 2023-11-28T13:05:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e78822700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740ccaa0 con 0x7f9e740cfc90 2023-11-28T13:05:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e72d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f9e740cdef0 0x7f9e740ca6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39064/0 (socket says 172.21.15.111:39064) 2023-11-28T13:05:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e72d9d700 1 -- 172.21.15.111:0/1780270498 learned_addr learned my addr 172.21.15.111:0/1780270498 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:05:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611624937 0 0) 0x7f9e6c001490 con 0x7f9e740cdef0 2023-11-28T13:05:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e5c0035b0 con 0x7f9e740cdef0 2023-11-28T13:05:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e7259c700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740cfc90 0x7f9e740d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 18237786 0 0) 0x7f9e5c0035b0 con 0x7f9e740cdef0 2023-11-28T13:05:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740cfc90 msgr2=0x7f9e740d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e7159a700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740cfc90 0x7f9e740d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 >> v1:172.21.15.149:6789/0 conn(0x7f9e740cc5a0 legacy=0x7f9e740cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e740ccd80 con 0x7f9e740cdef0 2023-11-28T13:05:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f9e6c001b80 con 0x7f9e740cdef0 2023-11-28T13:05:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.851+0000 7f9e7259c700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740cfc90 0x7f9e740d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:05:52.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 >> v1:172.21.15.111:6789/0 conn(0x7f9e740cdef0 legacy=0x7f9e740ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7159a700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e5c003a30 0x7f9e5c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7159a700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740cfc90 0x7f9e5c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7159a700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e5c004d90 0x7f9e5c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740ccaa0 con 0x7f9e5c003a30 2023-11-28T13:05:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c00baf0 con 0x7f9e5c004d90 2023-11-28T13:05:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c00bcb0 con 0x7f9e740cfc90 2023-11-28T13:05:52.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7259c700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e5c003a30 0x7f9e5c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e71d9b700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740cfc90 0x7f9e5c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e72d9d700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e5c004d90 0x7f9e5c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7259c700 1 -- 172.21.15.111:0/1780270498 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740cfc90 msgr2=0x7f9e5c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7259c700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740cfc90 0x7f9e5c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7259c700 1 -- 172.21.15.111:0/1780270498 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e5c004d90 msgr2=0x7f9e5c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:05:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7259c700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e5c004d90 0x7f9e5c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7259c700 1 -- 172.21.15.111:0/1780270498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c0035b0 con 0x7f9e5c003a30 2023-11-28T13:05:52.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e72d9d700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e5c004d90 0x7f9e5c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:05:52.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7259c700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e5c003a30 0x7f9e5c003e40 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009950 tx=0x7f9e5c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4be989b4732760a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:52.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6400c070 con 0x7f9e5c003a30 2023-11-28T13:05:52.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e64013610 con 0x7f9e5c003a30 2023-11-28T13:05:52.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.852+0000 7f9e7159a700 1 -- 172.21.15.111:0/1780270498 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e640139a0 con 0x7f9e5c003a30 2023-11-28T13:05:52.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.853+0000 7f9e78822700 1 -- 172.21.15.111:0/1780270498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e5c003a30 msgr2=0x7f9e5c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:52.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.853+0000 7f9e78822700 1 --2- 172.21.15.111:0/1780270498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e5c003a30 0x7f9e5c003e40 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009950 tx=0x7f9e5c005880 comp rx=0 tx=0).stop 2023-11-28T13:05:52.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.853+0000 7f9e72d9d700 1 -- 172.21.15.111:0/1780270498 reap_dead start 2023-11-28T13:05:52.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.853+0000 7f9e78822700 1 -- 172.21.15.111:0/1780270498 shutdown_connections 2023-11-28T13:05:52.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.853+0000 7f9e78822700 1 -- 172.21.15.111:0/1780270498 >> 172.21.15.111:0/1780270498 conn(0x7f9e74098ee0 msgr2=0x7f9e7406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:52.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.853+0000 7f9e78822700 1 -- 172.21.15.111:0/1780270498 shutdown_connections 2023-11-28T13:05:52.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.853+0000 7f9e78822700 1 -- 172.21.15.111:0/1780270498 wait complete. 2023-11-28T13:05:52.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.854+0000 7f9e78822700 1 Processor -- start 2023-11-28T13:05:52.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.854+0000 7f9e78822700 1 -- start start 2023-11-28T13:05:52.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e78822700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e5c003a30 0x7f9e740adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:52.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e78822700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740c5bf0 0x7f9e740c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:52.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e78822700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e7409cbf0 0x7f9e7409d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:52.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e78822700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e74148640 con 0x7f9e7409cbf0 2023-11-28T13:05:52.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e78822700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740c65a0 con 0x7f9e5c003a30 2023-11-28T13:05:52.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e78822700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e74148850 con 0x7f9e740c5bf0 2023-11-28T13:05:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e72d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e7409cbf0 0x7f9e7409d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e72d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e7409cbf0 0x7f9e7409d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38298/0 (socket says 172.21.15.111:38298) 2023-11-28T13:05:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e72d9d700 1 -- 172.21.15.111:0/1555619148 learned_addr learned my addr 172.21.15.111:0/1555619148 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e7259c700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e5c003a30 0x7f9e740adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e71d9b700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740c5bf0 0x7f9e740c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e72d9d700 1 -- 172.21.15.111:0/1555619148 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740c5bf0 msgr2=0x7f9e740c6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e72d9d700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740c5bf0 0x7f9e740c6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e72d9d700 1 -- 172.21.15.111:0/1555619148 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e5c003a30 msgr2=0x7f9e740adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:52.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e72d9d700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e5c003a30 0x7f9e740adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:52.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.855+0000 7f9e72d9d700 1 -- 172.21.15.111:0/1555619148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e740cd340 con 0x7f9e7409cbf0 2023-11-28T13:05:52.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.856+0000 7f9e72d9d700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e7409cbf0 0x7f9e7409d060 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f9e74066980 tx=0x7f9e6c00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:52.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.856+0000 7f9e62ffd700 1 -- 172.21.15.111:0/1555619148 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c00eb80 con 0x7f9e7409cbf0 2023-11-28T13:05:52.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.856+0000 7f9e62ffd700 1 -- 172.21.15.111:0/1555619148 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c00ece0 con 0x7f9e7409cbf0 2023-11-28T13:05:52.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.856+0000 7f9e78822700 1 -- 172.21.15.111:0/1555619148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e740cd060 con 0x7f9e7409cbf0 2023-11-28T13:05:52.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.856+0000 7f9e62ffd700 1 -- 172.21.15.111:0/1555619148 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c013660 con 0x7f9e7409cbf0 2023-11-28T13:05:52.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.856+0000 7f9e78822700 1 -- 172.21.15.111:0/1555619148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e74143280 con 0x7f9e7409cbf0 2023-11-28T13:05:52.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.857+0000 7f9e78822700 1 -- 172.21.15.111:0/1555619148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e74067170 con 0x7f9e7409cbf0 2023-11-28T13:05:52.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.857+0000 7f9e62ffd700 1 -- 172.21.15.111:0/1555619148 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9e6c0137c0 con 0x7f9e7409cbf0 2023-11-28T13:05:52.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.858+0000 7f9e62ffd700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9e54068320 0x7f9e5406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:52.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.858+0000 7f9e62ffd700 1 -- 172.21.15.111:0/1555619148 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f9e6c086170 con 0x7f9e7409cbf0 2023-11-28T13:05:52.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.858+0000 7f9e7259c700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9e54068320 0x7f9e5406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:52.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.858+0000 7f9e7259c700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9e54068320 0x7f9e5406a7d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9e64000c00 tx=0x7f9e6401e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:52.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:52.864+0000 7f9e62ffd700 1 -- 172.21.15.111:0/1555619148 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9e6c052850 con 0x7f9e7409cbf0 2023-11-28T13:05:53.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.088+0000 7f9e78822700 1 -- 172.21.15.111:0/1555619148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9e740d05a0 con 0x7f9e7409cbf0 2023-11-28T13:05:53.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.088+0000 7f9e62ffd700 1 -- 172.21.15.111:0/1555619148 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f9e740d05a0 con 0x7f9e7409cbf0 2023-11-28T13:05:53.090 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:05:53.091 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:05:53.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.090+0000 7f9e78822700 1 -- 172.21.15.111:0/1555619148 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9e54068320 msgr2=0x7f9e5406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:53.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.090+0000 7f9e78822700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9e54068320 0x7f9e5406a7d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9e64000c00 tx=0x7f9e6401e040 comp rx=0 tx=0).stop 2023-11-28T13:05:53.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.090+0000 7f9e78822700 1 -- 172.21.15.111:0/1555619148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e7409cbf0 msgr2=0x7f9e7409d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:53.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.090+0000 7f9e78822700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e7409cbf0 0x7f9e7409d060 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f9e74066980 tx=0x7f9e6c00e440 comp rx=0 tx=0).stop 2023-11-28T13:05:53.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.090+0000 7f9e78822700 1 -- 172.21.15.111:0/1555619148 shutdown_connections 2023-11-28T13:05:53.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.090+0000 7f9e78822700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e7409cbf0 0x7f9e7409d060 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:53.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.090+0000 7f9e78822700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f9e54068320 0x7f9e5406a7d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:53.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.090+0000 7f9e78822700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9e5c003a30 0x7f9e740adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:53.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.090+0000 7f9e78822700 1 --2- 172.21.15.111:0/1555619148 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9e740c5bf0 0x7f9e740c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:53.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.090+0000 7f9e78822700 1 -- 172.21.15.111:0/1555619148 >> 172.21.15.111:0/1555619148 conn(0x7f9e74098ee0 msgr2=0x7f9e7406dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:53.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.090+0000 7f9e78822700 1 -- 172.21.15.111:0/1555619148 shutdown_connections 2023-11-28T13:05:53.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:53.090+0000 7f9e78822700 1 -- 172.21.15.111:0/1555619148 wait complete. 2023-11-28T13:05:53.103 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:05:58.105 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:05:58.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.235+0000 7f41c3c65700 1 Processor -- start 2023-11-28T13:05:58.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.236+0000 7f41c3c65700 1 -- start start 2023-11-28T13:05:58.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.237+0000 7f41c3c65700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f41bc15a5d0 0x7f41bc0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:58.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.237+0000 7f41c3c65700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41bc0ae8b0 con 0x7f41bc0cb2d0 2023-11-28T13:05:58.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.237+0000 7f41c3c65700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41bc15ef40 con 0x7f41bc0b0160 2023-11-28T13:05:58.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.237+0000 7f41c3c65700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41bc0cc1b0 con 0x7f41bc15a5d0 2023-11-28T13:05:58.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.237+0000 7f41c2202700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f41bc0cb2d0 0x7f41bc15e830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39084/0 (socket says 172.21.15.111:39084) 2023-11-28T13:05:58.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.237+0000 7f41c2202700 1 -- 172.21.15.111:0/1634196134 learned_addr learned my addr 172.21.15.111:0/1634196134 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:05:58.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.237+0000 7f41c1a01700 1 --2- 172.21.15.111:0/1634196134 >> v2:172.21.15.111:3300/0 conn(0x7f41bc15a5d0 0x7f41bc0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.237+0000 7f41c09ff700 1 -- 172.21.15.111:0/1634196134 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665338623 0 0) 0x7f41bc0ae8b0 con 0x7f41bc0cb2d0 2023-11-28T13:05:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.237+0000 7f41c09ff700 1 -- 172.21.15.111:0/1634196134 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a80035b0 con 0x7f41bc0cb2d0 2023-11-28T13:05:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.237+0000 7f41c1a01700 1 -- 172.21.15.111:0/1634196134 >> v1:172.21.15.149:6789/0 conn(0x7f41bc0b0160 legacy=0x7f41bc15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.237+0000 7f41c1a01700 1 -- 172.21.15.111:0/1634196134 >> v1:172.21.15.111:6789/0 conn(0x7f41bc0cb2d0 legacy=0x7f41bc15e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.237+0000 7f41c1a01700 1 -- 172.21.15.111:0/1634196134 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc0cc390 con 0x7f41bc15a5d0 2023-11-28T13:05:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.238+0000 7f41c1a01700 1 --2- 172.21.15.111:0/1634196134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41bc15a5d0 0x7f41bc0ce850 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f41b8009830 tx=0x7f41b8009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a45cb97af77e1d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.238+0000 7f41c09ff700 1 -- 172.21.15.111:0/1634196134 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b8011830 con 0x7f41bc15a5d0 2023-11-28T13:05:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.238+0000 7f41c09ff700 1 -- 172.21.15.111:0/1634196134 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b8011990 con 0x7f41bc15a5d0 2023-11-28T13:05:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.238+0000 7f41c09ff700 1 -- 172.21.15.111:0/1634196134 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b8011d40 con 0x7f41bc15a5d0 2023-11-28T13:05:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.238+0000 7f41c3c65700 1 -- 172.21.15.111:0/1634196134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41bc15a5d0 msgr2=0x7f41bc0ce850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.238+0000 7f41c3c65700 1 --2- 172.21.15.111:0/1634196134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41bc15a5d0 0x7f41bc0ce850 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f41b8009830 tx=0x7f41b8009b40 comp rx=0 tx=0).stop 2023-11-28T13:05:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.238+0000 7f41c3c65700 1 -- 172.21.15.111:0/1634196134 shutdown_connections 2023-11-28T13:05:58.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.238+0000 7f41c3c65700 1 --2- 172.21.15.111:0/1634196134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41bc15a5d0 0x7f41bc0ce850 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:58.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.238+0000 7f41c3c65700 1 -- 172.21.15.111:0/1634196134 >> 172.21.15.111:0/1634196134 conn(0x7f41bc09a040 msgr2=0x7f41bc0d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:58.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.239+0000 7f41c3c65700 1 -- 172.21.15.111:0/1634196134 shutdown_connections 2023-11-28T13:05:58.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.239+0000 7f41c3c65700 1 -- 172.21.15.111:0/1634196134 wait complete. 2023-11-28T13:05:58.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.240+0000 7f41c3c65700 1 Processor -- start 2023-11-28T13:05:58.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.240+0000 7f41c3c65700 1 -- start start 2023-11-28T13:05:58.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.240+0000 7f41c3c65700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41bc0b0160 0x7f41bc1f1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:58.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.240+0000 7f41c3c65700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41bc0cb2d0 0x7f41bc0b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:58.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.240+0000 7f41c3c65700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41bc15a5d0 0x7f41bc0b5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:58.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.240+0000 7f41c3c65700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0b3e20 con 0x7f41bc15a5d0 2023-11-28T13:05:58.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.240+0000 7f41c3c65700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0b4960 con 0x7f41bc0cb2d0 2023-11-28T13:05:58.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.240+0000 7f41c3c65700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc0b5870 con 0x7f41bc0b0160 2023-11-28T13:05:58.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c2202700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41bc15a5d0 0x7f41bc0b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:58.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c2202700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41bc15a5d0 0x7f41bc0b5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38310/0 (socket says 172.21.15.111:38310) 2023-11-28T13:05:58.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c2202700 1 -- 172.21.15.111:0/3875981375 learned_addr learned my addr 172.21.15.111:0/3875981375 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:05:58.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c1200700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41bc0cb2d0 0x7f41bc0b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:58.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c1a01700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41bc0b0160 0x7f41bc1f1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:58.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c2202700 1 -- 172.21.15.111:0/3875981375 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41bc0b0160 msgr2=0x7f41bc1f1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:58.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c2202700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41bc0b0160 0x7f41bc1f1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:58.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c2202700 1 -- 172.21.15.111:0/3875981375 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41bc0cb2d0 msgr2=0x7f41bc0b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:58.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c2202700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41bc0cb2d0 0x7f41bc0b4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:58.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c2202700 1 -- 172.21.15.111:0/3875981375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b8009440 con 0x7f41bc15a5d0 2023-11-28T13:05:58.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c1200700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41bc0cb2d0 0x7f41bc0b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:05:58.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c1a01700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41bc0b0160 0x7f41bc1f1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:05:58.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c2202700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41bc15a5d0 0x7f41bc0b5330 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f41b400cb40 tx=0x7f41b400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:58.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41b27fc700 1 -- 172.21.15.111:0/3875981375 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b400ab50 con 0x7f41bc15a5d0 2023-11-28T13:05:58.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41b27fc700 1 -- 172.21.15.111:0/3875981375 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b400acb0 con 0x7f41bc15a5d0 2023-11-28T13:05:58.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41b27fc700 1 -- 172.21.15.111:0/3875981375 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b4012620 con 0x7f41bc15a5d0 2023-11-28T13:05:58.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c3c65700 1 -- 172.21.15.111:0/3875981375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41bc0cc950 con 0x7f41bc15a5d0 2023-11-28T13:05:58.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.241+0000 7f41c3c65700 1 -- 172.21.15.111:0/3875981375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41bc0cc670 con 0x7f41bc15a5d0 2023-11-28T13:05:58.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.242+0000 7f41c3c65700 1 -- 172.21.15.111:0/3875981375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41bc0ccc30 con 0x7f41bc15a5d0 2023-11-28T13:05:58.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.243+0000 7f41b27fc700 1 -- 172.21.15.111:0/3875981375 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f41b4019030 con 0x7f41bc15a5d0 2023-11-28T13:05:58.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.243+0000 7f41b27fc700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41a00682f0 0x7f41a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:05:58.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.243+0000 7f41c1a01700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41a00682f0 0x7f41a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:05:58.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.243+0000 7f41b27fc700 1 -- 172.21.15.111:0/3875981375 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f41b4085ae0 con 0x7f41bc15a5d0 2023-11-28T13:05:58.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.243+0000 7f41c1a01700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41a00682f0 0x7f41a006a7a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f41bc09ebf0 tx=0x7f41b801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:05:58.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.247+0000 7f41b27fc700 1 -- 172.21.15.111:0/3875981375 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f41b40521c0 con 0x7f41bc15a5d0 2023-11-28T13:05:58.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.470+0000 7f41c3c65700 1 -- 172.21.15.111:0/3875981375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f41bc067170 con 0x7f41bc15a5d0 2023-11-28T13:05:58.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.470+0000 7f41b27fc700 1 -- 172.21.15.111:0/3875981375 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f41b40557e0 con 0x7f41bc15a5d0 2023-11-28T13:05:58.472 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:05:58.472 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:05:58.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.472+0000 7f41c3c65700 1 -- 172.21.15.111:0/3875981375 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41a00682f0 msgr2=0x7f41a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:58.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.472+0000 7f41c3c65700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41a00682f0 0x7f41a006a7a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f41bc09ebf0 tx=0x7f41b801c040 comp rx=0 tx=0).stop 2023-11-28T13:05:58.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.472+0000 7f41c3c65700 1 -- 172.21.15.111:0/3875981375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41bc15a5d0 msgr2=0x7f41bc0b5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:05:58.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.472+0000 7f41c3c65700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41bc15a5d0 0x7f41bc0b5330 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f41b400cb40 tx=0x7f41b400cf00 comp rx=0 tx=0).stop 2023-11-28T13:05:58.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.472+0000 7f41c3c65700 1 -- 172.21.15.111:0/3875981375 shutdown_connections 2023-11-28T13:05:58.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.472+0000 7f41c3c65700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41bc15a5d0 0x7f41bc0b5330 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:58.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.472+0000 7f41c3c65700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f41bc0b0160 0x7f41bc1f1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:58.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.472+0000 7f41c3c65700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f41a00682f0 0x7f41a006a7a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:58.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.472+0000 7f41c3c65700 1 --2- 172.21.15.111:0/3875981375 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f41bc0cb2d0 0x7f41bc0b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:05:58.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.472+0000 7f41c3c65700 1 -- 172.21.15.111:0/3875981375 >> 172.21.15.111:0/3875981375 conn(0x7f41bc09a040 msgr2=0x7f41bc0ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:05:58.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.472+0000 7f41c3c65700 1 -- 172.21.15.111:0/3875981375 shutdown_connections 2023-11-28T13:05:58.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:05:58.472+0000 7f41c3c65700 1 -- 172.21.15.111:0/3875981375 wait complete. 2023-11-28T13:05:58.485 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:05:59.561 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:05:59.565 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:06:03.486 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:06:03.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.618+0000 7f4ca8bb4700 1 Processor -- start 2023-11-28T13:06:03.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.618+0000 7f4ca8bb4700 1 -- start start 2023-11-28T13:06:03.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.619+0000 7f4ca8bb4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4ca40cf010 0x7f4ca40cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:03.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.619+0000 7f4ca8bb4700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f4ca40cb380 0x7f4ca40b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:03.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.619+0000 7f4ca8bb4700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca40cc7b0 con 0x7f4ca40b2220 2023-11-28T13:06:03.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.619+0000 7f4ca8bb4700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca408ae90 con 0x7f4ca40cf010 2023-11-28T13:06:03.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.619+0000 7f4ca8bb4700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca40b25f0 con 0x7f4ca40cb380 2023-11-28T13:06:03.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.619+0000 7f4ca1d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4ca40cf010 0x7f4ca40cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:03.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.619+0000 7f4ca259c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4ca40b2220 0x7f4ca40ce900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39094/0 (socket says 172.21.15.111:39094) 2023-11-28T13:06:03.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.619+0000 7f4ca259c700 1 -- 172.21.15.111:0/1162449960 learned_addr learned my addr 172.21.15.111:0/1162449960 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:03.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.620+0000 7f4ca1d9b700 1 -- 172.21.15.111:0/1162449960 >> v2:172.21.15.149:3300/0 conn(0x7f4ca40cb380 msgr2=0x7f4ca40b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:03.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.620+0000 7f4ca1d9b700 1 --2- 172.21.15.111:0/1162449960 >> v2:172.21.15.149:3300/0 conn(0x7f4ca40cb380 0x7f4ca40b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:03.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.620+0000 7f4ca1d9b700 1 -- 172.21.15.111:0/1162449960 >> v1:172.21.15.111:6789/0 conn(0x7f4ca40b2220 legacy=0x7f4ca40ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:03.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.620+0000 7f4ca159a700 1 -- 172.21.15.111:0/1162449960 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545604365 0 0) 0x7f4ca40cc7b0 con 0x7f4ca40b2220 2023-11-28T13:06:03.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.620+0000 7f4ca1d9b700 1 -- 172.21.15.111:0/1162449960 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca40cca10 con 0x7f4ca40cf010 2023-11-28T13:06:03.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.620+0000 7f4ca1d9b700 1 --2- 172.21.15.111:0/1162449960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca40cf010 0x7f4ca40cc270 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f4ca40c7d90 tx=0x7f4c94009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d054e6bdae901b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:03.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.620+0000 7f4ca159a700 1 -- 172.21.15.111:0/1162449960 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c94011830 con 0x7f4ca40cf010 2023-11-28T13:06:03.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.620+0000 7f4ca159a700 1 -- 172.21.15.111:0/1162449960 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c94011990 con 0x7f4ca40cf010 2023-11-28T13:06:03.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.620+0000 7f4ca159a700 1 -- 172.21.15.111:0/1162449960 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c94011d20 con 0x7f4ca40cf010 2023-11-28T13:06:03.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.621+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/1162449960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca40cf010 msgr2=0x7f4ca40cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:03.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.621+0000 7f4ca8bb4700 1 --2- 172.21.15.111:0/1162449960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca40cf010 0x7f4ca40cc270 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f4ca40c7d90 tx=0x7f4c94009b20 comp rx=0 tx=0).stop 2023-11-28T13:06:03.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.621+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/1162449960 shutdown_connections 2023-11-28T13:06:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.621+0000 7f4ca8bb4700 1 --2- 172.21.15.111:0/1162449960 >> v2:172.21.15.149:3300/0 conn(0x7f4ca40cb380 0x7f4ca40b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.621+0000 7f4ca8bb4700 1 --2- 172.21.15.111:0/1162449960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca40cf010 0x7f4ca40cc270 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.621+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/1162449960 >> 172.21.15.111:0/1162449960 conn(0x7f4ca409a040 msgr2=0x7f4ca40d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.621+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/1162449960 shutdown_connections 2023-11-28T13:06:03.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.621+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/1162449960 wait complete. 2023-11-28T13:06:03.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.622+0000 7f4ca8bb4700 1 Processor -- start 2023-11-28T13:06:03.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.622+0000 7f4ca8bb4700 1 -- start start 2023-11-28T13:06:03.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca8bb4700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ca40b2220 0x7f4ca40c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:03.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca8bb4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ca40cb380 0x7f4ca40c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:03.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca8bb4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca40cf010 0x7f4ca4158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:03.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca8bb4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40b4f40 con 0x7f4ca40cf010 2023-11-28T13:06:03.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca8bb4700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40b4610 con 0x7f4ca40b2220 2023-11-28T13:06:03.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca8bb4700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40b48f0 con 0x7f4ca40cb380 2023-11-28T13:06:03.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca2d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca40cf010 0x7f4ca4158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:03.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca2d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca40cf010 0x7f4ca4158660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38326/0 (socket says 172.21.15.111:38326) 2023-11-28T13:06:03.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca2d9d700 1 -- 172.21.15.111:0/2683806830 learned_addr learned my addr 172.21.15.111:0/2683806830 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:03.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca259c700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ca40b2220 0x7f4ca40c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:03.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca1d9b700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ca40cb380 0x7f4ca40c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:03.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca2d9d700 1 -- 172.21.15.111:0/2683806830 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ca40cb380 msgr2=0x7f4ca40c6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:03.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca2d9d700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ca40cb380 0x7f4ca40c6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:03.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca2d9d700 1 -- 172.21.15.111:0/2683806830 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ca40b2220 msgr2=0x7f4ca40c7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:03.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca2d9d700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ca40b2220 0x7f4ca40c7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:03.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca2d9d700 1 -- 172.21.15.111:0/2683806830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c940094a0 con 0x7f4ca40cf010 2023-11-28T13:06:03.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4ca2d9d700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca40cf010 0x7f4ca4158660 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00b970 tx=0x7f4c9c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:03.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.623+0000 7f4c92ffd700 1 -- 172.21.15.111:0/2683806830 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c00c9b0 con 0x7f4ca40cf010 2023-11-28T13:06:03.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.624+0000 7f4c92ffd700 1 -- 172.21.15.111:0/2683806830 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c00cb30 con 0x7f4ca40cf010 2023-11-28T13:06:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.624+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/2683806830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca40ccfd0 con 0x7f4ca40cf010 2023-11-28T13:06:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.624+0000 7f4c92ffd700 1 -- 172.21.15.111:0/2683806830 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c014440 con 0x7f4ca40cf010 2023-11-28T13:06:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.624+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/2683806830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca40cccf0 con 0x7f4ca40cf010 2023-11-28T13:06:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.624+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/2683806830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca40cd2b0 con 0x7f4ca40cf010 2023-11-28T13:06:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.625+0000 7f4c92ffd700 1 -- 172.21.15.111:0/2683806830 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4c9c01b030 con 0x7f4ca40cf010 2023-11-28T13:06:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.625+0000 7f4c92ffd700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4c840682f0 0x7f4c8406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.625+0000 7f4c92ffd700 1 -- 172.21.15.111:0/2683806830 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f4c9c006e40 con 0x7f4ca40cf010 2023-11-28T13:06:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.625+0000 7f4ca259c700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4c840682f0 0x7f4c8406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.626+0000 7f4ca259c700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4c840682f0 0x7f4c8406a7a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4c98000f60 tx=0x7f4c98007be0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:03.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.630+0000 7f4c92ffd700 1 -- 172.21.15.111:0/2683806830 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4c9c052160 con 0x7f4ca40cf010 2023-11-28T13:06:03.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.853+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/2683806830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4ca4067170 con 0x7f4ca40cf010 2023-11-28T13:06:03.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.853+0000 7f4c92ffd700 1 -- 172.21.15.111:0/2683806830 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f4c9c0190c0 con 0x7f4ca40cf010 2023-11-28T13:06:03.854 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:03.854 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:06:03.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.855+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/2683806830 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4c840682f0 msgr2=0x7f4c8406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:03.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.855+0000 7f4ca8bb4700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4c840682f0 0x7f4c8406a7a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4c98000f60 tx=0x7f4c98007be0 comp rx=0 tx=0).stop 2023-11-28T13:06:03.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.855+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/2683806830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca40cf010 msgr2=0x7f4ca4158660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:03.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.855+0000 7f4ca8bb4700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca40cf010 0x7f4ca4158660 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00b970 tx=0x7f4c9c00bd30 comp rx=0 tx=0).stop 2023-11-28T13:06:03.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.855+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/2683806830 shutdown_connections 2023-11-28T13:06:03.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.855+0000 7f4ca8bb4700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca40cf010 0x7f4ca4158660 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:03.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.855+0000 7f4ca8bb4700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f4c840682f0 0x7f4c8406a7a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:03.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.855+0000 7f4ca8bb4700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f4ca40b2220 0x7f4ca40c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:03.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.855+0000 7f4ca8bb4700 1 --2- 172.21.15.111:0/2683806830 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f4ca40cb380 0x7f4ca40c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:03.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.855+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/2683806830 >> 172.21.15.111:0/2683806830 conn(0x7f4ca409a040 msgr2=0x7f4ca40d15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:03.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.855+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/2683806830 shutdown_connections 2023-11-28T13:06:03.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:03.855+0000 7f4ca8bb4700 1 -- 172.21.15.111:0/2683806830 wait complete. 2023-11-28T13:06:03.867 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=200 retry_count=0)... 2023-11-28T13:06:08.869 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T13:06:09.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.163+0000 7f2fca0a0700 1 Processor -- start 2023-11-28T13:06:09.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.163+0000 7f2fca0a0700 1 -- start start 2023-11-28T13:06:09.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fca0a0700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f2fc41596e0 0x7f2fc415cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fca0a0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc415de70 con 0x7f2fc4159310 2023-11-28T13:06:09.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fca0a0700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc40ced10 con 0x7f2fc415d510 2023-11-28T13:06:09.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fca0a0700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc415d980 con 0x7f2fc41596e0 2023-11-28T13:06:09.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc37fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2fc4159310 0x7f2fc40ce1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39108/0 (socket says 172.21.15.111:39108) 2023-11-28T13:06:09.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc37fe700 1 -- 172.21.15.111:0/1868514294 learned_addr learned my addr 172.21.15.111:0/1868514294 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:09.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc2ffd700 1 --2- 172.21.15.111:0/1868514294 >> v2:172.21.15.149:3300/0 conn(0x7f2fc41596e0 0x7f2fc415cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107504720 0 0) 0x7f2fc415de70 con 0x7f2fc4159310 2023-11-28T13:06:09.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fac0035b0 con 0x7f2fc4159310 2023-11-28T13:06:09.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1472096269 0 0) 0x7f2fac0035b0 con 0x7f2fc4159310 2023-11-28T13:06:09.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 >> v2:172.21.15.149:3300/0 conn(0x7f2fc41596e0 msgr2=0x7f2fc415cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:09.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 --2- 172.21.15.111:0/1868514294 >> v2:172.21.15.149:3300/0 conn(0x7f2fc41596e0 0x7f2fc415cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 >> v1:172.21.15.149:6789/0 conn(0x7f2fc415d510 legacy=0x7f2fc40cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc40ceff0 con 0x7f2fc4159310 2023-11-28T13:06:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f2fb4001670 con 0x7f2fc4159310 2023-11-28T13:06:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 >> v1:172.21.15.111:6789/0 conn(0x7f2fc4159310 legacy=0x7f2fc40ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 --2- 172.21.15.111:0/1868514294 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac003a00 0x7f2fac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 --2- 172.21.15.111:0/1868514294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fac004850 0x7f2fac004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 --2- 172.21.15.111:0/1868514294 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fac005200 0x7f2fac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc415d980 con 0x7f2fac004850 2023-11-28T13:06:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac00c080 con 0x7f2fac005200 2023-11-28T13:06:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.164+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac00c240 con 0x7f2fac003a00 2023-11-28T13:06:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.165+0000 7f2fc2ffd700 1 --2- 172.21.15.111:0/1868514294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fac004850 0x7f2fac004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.165+0000 7f2fc3fff700 1 --2- 172.21.15.111:0/1868514294 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fac005200 0x7f2fac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.165+0000 7f2fc37fe700 1 --2- 172.21.15.111:0/1868514294 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac003a00 0x7f2fac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.165+0000 7f2fc2ffd700 1 -- 172.21.15.111:0/1868514294 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac003a00 msgr2=0x7f2fac003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.165+0000 7f2fc2ffd700 1 --2- 172.21.15.111:0/1868514294 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac003a00 0x7f2fac003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.165+0000 7f2fc2ffd700 1 -- 172.21.15.111:0/1868514294 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fac005200 msgr2=0x7f2fac00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.165+0000 7f2fc2ffd700 1 --2- 172.21.15.111:0/1868514294 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fac005200 0x7f2fac00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.165+0000 7f2fc3fff700 1 -- 172.21.15.111:0/1868514294 reap_dead start 2023-11-28T13:06:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.165+0000 7f2fc2ffd700 1 -- 172.21.15.111:0/1868514294 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac0035b0 con 0x7f2fac004850 2023-11-28T13:06:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.165+0000 7f2fc2ffd700 1 --2- 172.21.15.111:0/1868514294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fac004850 0x7f2fac004cc0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40af5f0 tx=0x7f2fac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bcd61c34266c230 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.165+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb800e6c0 con 0x7f2fac004850 2023-11-28T13:06:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.165+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb800e820 con 0x7f2fac004850 2023-11-28T13:06:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.165+0000 7f2fc27fc700 1 -- 172.21.15.111:0/1868514294 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb800eb90 con 0x7f2fac004850 2023-11-28T13:06:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.166+0000 7f2fca0a0700 1 -- 172.21.15.111:0/1868514294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fac004850 msgr2=0x7f2fac004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.166+0000 7f2fca0a0700 1 --2- 172.21.15.111:0/1868514294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fac004850 0x7f2fac004cc0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40af5f0 tx=0x7f2fac005880 comp rx=0 tx=0).stop 2023-11-28T13:06:09.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.166+0000 7f2fca0a0700 1 -- 172.21.15.111:0/1868514294 shutdown_connections 2023-11-28T13:06:09.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.166+0000 7f2fca0a0700 1 --2- 172.21.15.111:0/1868514294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fac004850 0x7f2fac004cc0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.166+0000 7f2fca0a0700 1 -- 172.21.15.111:0/1868514294 >> 172.21.15.111:0/1868514294 conn(0x7f2fc406db10 msgr2=0x7f2fc406df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:09.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.166+0000 7f2fca0a0700 1 -- 172.21.15.111:0/1868514294 shutdown_connections 2023-11-28T13:06:09.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.166+0000 7f2fca0a0700 1 -- 172.21.15.111:0/1868514294 wait complete. 2023-11-28T13:06:09.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.167+0000 7f2fca0a0700 1 Processor -- start 2023-11-28T13:06:09.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.167+0000 7f2fca0a0700 1 -- start start 2023-11-28T13:06:09.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fca0a0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac004850 0x7f2fc40b4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fca0a0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fc40c5ca0 0x7f2fc4153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fca0a0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fc40b51d0 0x7f2fc406a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fca0a0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40c4c90 con 0x7f2fc40b51d0 2023-11-28T13:06:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fca0a0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc4154220 con 0x7f2fc40c5ca0 2023-11-28T13:06:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fca0a0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40c4ee0 con 0x7f2fac004850 2023-11-28T13:06:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fc3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fc40b51d0 0x7f2fc406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fc3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fc40b51d0 0x7f2fc406a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38344/0 (socket says 172.21.15.111:38344) 2023-11-28T13:06:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fc3fff700 1 -- 172.21.15.111:0/3886548574 learned_addr learned my addr 172.21.15.111:0/3886548574 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fc2ffd700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fc40c5ca0 0x7f2fc4153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fc37fe700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac004850 0x7f2fc40b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fc3fff700 1 -- 172.21.15.111:0/3886548574 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac004850 msgr2=0x7f2fc40b4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fc3fff700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac004850 0x7f2fc40b4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fc3fff700 1 -- 172.21.15.111:0/3886548574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fc40c5ca0 msgr2=0x7f2fc4153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:09.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fc3fff700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fc40c5ca0 0x7f2fc4153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fc3fff700 1 -- 172.21.15.111:0/3886548574 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc40cb0f0 con 0x7f2fc40b51d0 2023-11-28T13:06:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fc3fff700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fc40b51d0 0x7f2fc406a270 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40cd640 tx=0x7f2fbc00ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.168+0000 7f2fabfff700 1 -- 172.21.15.111:0/3886548574 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc003cd0 con 0x7f2fc40b51d0 2023-11-28T13:06:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.169+0000 7f2fabfff700 1 -- 172.21.15.111:0/3886548574 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc003e70 con 0x7f2fc40b51d0 2023-11-28T13:06:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.169+0000 7f2fca0a0700 1 -- 172.21.15.111:0/3886548574 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc40cf2d0 con 0x7f2fc40b51d0 2023-11-28T13:06:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.169+0000 7f2fabfff700 1 -- 172.21.15.111:0/3886548574 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc0052f0 con 0x7f2fc40b51d0 2023-11-28T13:06:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.169+0000 7f2fca0a0700 1 -- 172.21.15.111:0/3886548574 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc40cb3d0 con 0x7f2fc40b51d0 2023-11-28T13:06:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.170+0000 7f2fca0a0700 1 -- 172.21.15.111:0/3886548574 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc40c6070 con 0x7f2fc40b51d0 2023-11-28T13:06:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.170+0000 7f2fabfff700 1 -- 172.21.15.111:0/3886548574 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2fbc005450 con 0x7f2fc40b51d0 2023-11-28T13:06:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.170+0000 7f2fabfff700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2fb00682f0 0x7f2fb006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.170+0000 7f2fabfff700 1 -- 172.21.15.111:0/3886548574 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f2fbc086090 con 0x7f2fc40b51d0 2023-11-28T13:06:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.170+0000 7f2fc37fe700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2fb00682f0 0x7f2fb006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.171+0000 7f2fc37fe700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2fb00682f0 0x7f2fb006a7a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4001050 tx=0x7f2fb4000f80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:09.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.175+0000 7f2fabfff700 1 -- 172.21.15.111:0/3886548574 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2fbc052710 con 0x7f2fc40b51d0 2023-11-28T13:06:09.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.397+0000 7f2fca0a0700 1 -- 172.21.15.111:0/3886548574 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2fc4067170 con 0x7f2fc40b51d0 2023-11-28T13:06:09.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.397+0000 7f2fabfff700 1 -- 172.21.15.111:0/3886548574 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f2fbc055d30 con 0x7f2fc40b51d0 2023-11-28T13:06:09.398 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:09.398 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2023-11-28T13:06:09.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.399+0000 7f2fca0a0700 1 -- 172.21.15.111:0/3886548574 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2fb00682f0 msgr2=0x7f2fb006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:09.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.399+0000 7f2fca0a0700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2fb00682f0 0x7f2fb006a7a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4001050 tx=0x7f2fb4000f80 comp rx=0 tx=0).stop 2023-11-28T13:06:09.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.399+0000 7f2fca0a0700 1 -- 172.21.15.111:0/3886548574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fc40b51d0 msgr2=0x7f2fc406a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:09.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.399+0000 7f2fca0a0700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fc40b51d0 0x7f2fc406a270 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40cd640 tx=0x7f2fbc00ff00 comp rx=0 tx=0).stop 2023-11-28T13:06:09.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.399+0000 7f2fca0a0700 1 -- 172.21.15.111:0/3886548574 shutdown_connections 2023-11-28T13:06:09.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.399+0000 7f2fca0a0700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fc40b51d0 0x7f2fc406a270 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.399+0000 7f2fca0a0700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac004850 0x7f2fc40b4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.399+0000 7f2fca0a0700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2fb00682f0 0x7f2fb006a7a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.399+0000 7f2fca0a0700 1 --2- 172.21.15.111:0/3886548574 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fc40c5ca0 0x7f2fc4153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.399+0000 7f2fca0a0700 1 -- 172.21.15.111:0/3886548574 >> 172.21.15.111:0/3886548574 conn(0x7f2fc406db10 msgr2=0x7f2fc4156900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:09.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.400+0000 7f2fca0a0700 1 -- 172.21.15.111:0/3886548574 shutdown_connections 2023-11-28T13:06:09.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.400+0000 7f2fca0a0700 1 -- 172.21.15.111:0/3886548574 wait complete. 2023-11-28T13:06:09.412 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T13:06:09.412 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T13:06:09.447 DEBUG:teuthology.orchestra.run.smithi149:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T13:06:09.492 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T13:06:09.492 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:06:09.492 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:06:09.510 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:06:09.510 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:06:09.567 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:06:09.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.726+0000 7fa6f7af0700 1 Processor -- start 2023-11-28T13:06:09.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.727+0000 7fa6f7af0700 1 -- start start 2023-11-28T13:06:09.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.727+0000 7fa6f7af0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa6f01594f0 0x7fa6f015a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.727+0000 7fa6f7af0700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fa6f00712b0 0x7fa6f0071720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.727+0000 7fa6f7af0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa6f015cf50 0x7fa6f015d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.727+0000 7fa6f7af0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f015a1c0 con 0x7fa6f015cf50 2023-11-28T13:06:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.727+0000 7fa6f7af0700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f00725c0 con 0x7fa6f00712b0 2023-11-28T13:06:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.727+0000 7fa6f7af0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f0071c60 con 0x7fa6f01594f0 2023-11-28T13:06:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.727+0000 7fa6f608d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa6f015cf50 0x7fa6f015d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f608d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa6f015cf50 0x7fa6f015d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38354/0 (socket says 172.21.15.111:38354) 2023-11-28T13:06:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f608d700 1 -- 172.21.15.111:0/480798537 learned_addr learned my addr 172.21.15.111:0/480798537 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f508b700 1 --2- 172.21.15.111:0/480798537 >> v2:172.21.15.149:3300/0 conn(0x7fa6f00712b0 0x7fa6f0071720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f588c700 1 --2- 172.21.15.111:0/480798537 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa6f01594f0 0x7fa6f015a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f608d700 1 -- 172.21.15.111:0/480798537 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa6f01594f0 msgr2=0x7fa6f015a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f608d700 1 --2- 172.21.15.111:0/480798537 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa6f01594f0 0x7fa6f015a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f608d700 1 -- 172.21.15.111:0/480798537 >> v2:172.21.15.149:3300/0 conn(0x7fa6f00712b0 msgr2=0x7fa6f0071720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f608d700 1 --2- 172.21.15.111:0/480798537 >> v2:172.21.15.149:3300/0 conn(0x7fa6f00712b0 0x7fa6f0071720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f608d700 1 -- 172.21.15.111:0/480798537 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f01598c0 con 0x7fa6f015cf50 2023-11-28T13:06:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f588c700 1 --2- 172.21.15.111:0/480798537 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa6f01594f0 0x7fa6f015a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f608d700 1 --2- 172.21.15.111:0/480798537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f015cf50 0x7fa6f015d3d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0066750 tx=0x7fa6e800d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e10b5b55dd2201c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f488a700 1 -- 172.21.15.111:0/480798537 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8010070 con 0x7fa6f015cf50 2023-11-28T13:06:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f488a700 1 -- 172.21.15.111:0/480798537 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e800dcd0 con 0x7fa6f015cf50 2023-11-28T13:06:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.728+0000 7fa6f488a700 1 -- 172.21.15.111:0/480798537 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e800b310 con 0x7fa6f015cf50 2023-11-28T13:06:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.729+0000 7fa6f7af0700 1 -- 172.21.15.111:0/480798537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f015cf50 msgr2=0x7fa6f015d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.729+0000 7fa6f7af0700 1 --2- 172.21.15.111:0/480798537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f015cf50 0x7fa6f015d3d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0066750 tx=0x7fa6e800d460 comp rx=0 tx=0).stop 2023-11-28T13:06:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.729+0000 7fa6f7af0700 1 -- 172.21.15.111:0/480798537 shutdown_connections 2023-11-28T13:06:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.729+0000 7fa6f7af0700 1 --2- 172.21.15.111:0/480798537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f015cf50 0x7fa6f015d3d0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.729+0000 7fa6f7af0700 1 --2- 172.21.15.111:0/480798537 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa6f01594f0 0x7fa6f015a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.729+0000 7fa6f7af0700 1 --2- 172.21.15.111:0/480798537 >> v2:172.21.15.149:3300/0 conn(0x7fa6f00712b0 0x7fa6f0071720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.729+0000 7fa6f7af0700 1 -- 172.21.15.111:0/480798537 >> 172.21.15.111:0/480798537 conn(0x7fa6f00cf2e0 msgr2=0x7fa6f00cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:09.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.729+0000 7fa6f7af0700 1 -- 172.21.15.111:0/480798537 shutdown_connections 2023-11-28T13:06:09.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.729+0000 7fa6f7af0700 1 -- 172.21.15.111:0/480798537 wait complete. 2023-11-28T13:06:09.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.730+0000 7fa6f7af0700 1 Processor -- start 2023-11-28T13:06:09.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.730+0000 7fa6f7af0700 1 -- start start 2023-11-28T13:06:09.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f7af0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa6f00712b0 0x7fa6f00b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f7af0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f01594f0 0x7fa6f01472b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f7af0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa6f015cf50 0x7fa6f01f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f7af0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f01ef7b0 con 0x7fa6f01594f0 2023-11-28T13:06:09.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f7af0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f00b3160 con 0x7fa6f00712b0 2023-11-28T13:06:09.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f7af0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f00b3400 con 0x7fa6f015cf50 2023-11-28T13:06:09.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f508b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f01594f0 0x7fa6f01472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f508b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f01594f0 0x7fa6f01472b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38358/0 (socket says 172.21.15.111:38358) 2023-11-28T13:06:09.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f508b700 1 -- 172.21.15.111:0/2039196712 learned_addr learned my addr 172.21.15.111:0/2039196712 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:09.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f608d700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa6f015cf50 0x7fa6f01f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f588c700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa6f00712b0 0x7fa6f00b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f508b700 1 -- 172.21.15.111:0/2039196712 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa6f015cf50 msgr2=0x7fa6f01f5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f508b700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa6f015cf50 0x7fa6f01f5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f508b700 1 -- 172.21.15.111:0/2039196712 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa6f00712b0 msgr2=0x7fa6f00b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f508b700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa6f00712b0 0x7fa6f00b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f508b700 1 -- 172.21.15.111:0/2039196712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e800bd40 con 0x7fa6f01594f0 2023-11-28T13:06:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f608d700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa6f015cf50 0x7fa6f01f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f508b700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f01594f0 0x7fa6f01472b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00e870 tx=0x7fa6ec00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6e67fc700 1 -- 172.21.15.111:0/2039196712 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec00c640 con 0x7fa6f01594f0 2023-11-28T13:06:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6e67fc700 1 -- 172.21.15.111:0/2039196712 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec00c7a0 con 0x7fa6f01594f0 2023-11-28T13:06:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f7af0700 1 -- 172.21.15.111:0/2039196712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f015d910 con 0x7fa6f01594f0 2023-11-28T13:06:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6e67fc700 1 -- 172.21.15.111:0/2039196712 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec00cb30 con 0x7fa6f01594f0 2023-11-28T13:06:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.731+0000 7fa6f7af0700 1 -- 172.21.15.111:0/2039196712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f015de40 con 0x7fa6f01594f0 2023-11-28T13:06:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.732+0000 7fa6f7af0700 1 -- 172.21.15.111:0/2039196712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f015db60 con 0x7fa6f01594f0 2023-11-28T13:06:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.733+0000 7fa6e67fc700 1 -- 172.21.15.111:0/2039196712 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa6ec019030 con 0x7fa6f01594f0 2023-11-28T13:06:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.733+0000 7fa6e67fc700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa6d40682f0 0x7fa6d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.733+0000 7fa6e67fc700 1 -- 172.21.15.111:0/2039196712 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fa6ec085c30 con 0x7fa6f01594f0 2023-11-28T13:06:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.733+0000 7fa6f588c700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa6d40682f0 0x7fa6d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.733+0000 7fa6f588c700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa6d40682f0 0x7fa6d406a7a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009930 tx=0x7fa6e0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:09.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.737+0000 7fa6e67fc700 1 -- 172.21.15.111:0/2039196712 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa6ec052310 con 0x7fa6f01594f0 2023-11-28T13:06:09.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.922+0000 7fa6f7af0700 1 -- 172.21.15.111:0/2039196712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6f006bd30 con 0x7fa6f01594f0 2023-11-28T13:06:09.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.924+0000 7fa6e67fc700 1 -- 172.21.15.111:0/2039196712 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+18328 (secure 0 0 0) 0x7fa6ec055930 con 0x7fa6f01594f0 2023-11-28T13:06:09.924 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:09.924 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":69,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:02:42.754611+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:01:51.934388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-28T13:01:53.219817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.3","mappings":[{"from":6,"to":0}]},{"pgid":"11.20","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:06:09.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.926+0000 7fa6f7af0700 1 -- 172.21.15.111:0/2039196712 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa6d40682f0 msgr2=0x7fa6d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:09.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.926+0000 7fa6f7af0700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa6d40682f0 0x7fa6d406a7a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009930 tx=0x7fa6e0008040 comp rx=0 tx=0).stop 2023-11-28T13:06:09.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.926+0000 7fa6f7af0700 1 -- 172.21.15.111:0/2039196712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f01594f0 msgr2=0x7fa6f01472b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:09.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.926+0000 7fa6f7af0700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f01594f0 0x7fa6f01472b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00e870 tx=0x7fa6ec00ec30 comp rx=0 tx=0).stop 2023-11-28T13:06:09.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.926+0000 7fa6f7af0700 1 -- 172.21.15.111:0/2039196712 shutdown_connections 2023-11-28T13:06:09.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.926+0000 7fa6f7af0700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fa6f015cf50 0x7fa6f01f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.926+0000 7fa6f7af0700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fa6d40682f0 0x7fa6d406a7a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.926+0000 7fa6f7af0700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fa6f00712b0 0x7fa6f00b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.926+0000 7fa6f7af0700 1 --2- 172.21.15.111:0/2039196712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6f01594f0 0x7fa6f01472b0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:09.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.926+0000 7fa6f7af0700 1 -- 172.21.15.111:0/2039196712 >> 172.21.15.111:0/2039196712 conn(0x7fa6f00cf2e0 msgr2=0x7fa6f015a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:09.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.926+0000 7fa6f7af0700 1 -- 172.21.15.111:0/2039196712 shutdown_connections 2023-11-28T13:06:09.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:09.926+0000 7fa6f7af0700 1 -- 172.21.15.111:0/2039196712 wait complete. 2023-11-28T13:06:09.938 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T13:01:51.934388+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T13:01:53.219817+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T13:06:09.939 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T13:06:10.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.062+0000 7f1ca4609700 1 Processor -- start 2023-11-28T13:06:10.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.062+0000 7f1ca4609700 1 -- start start 2023-11-28T13:06:10.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca4609700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1c9c0b15c0 0x7f1c9c0ac670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca4609700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f1c9c0b0250 0x7f1c9c0b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca4609700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c9c0b0be0 con 0x7f1c9c15a9e0 2023-11-28T13:06:10.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca4609700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c9c0af650 con 0x7f1c9c0b0250 2023-11-28T13:06:10.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca4609700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c0cb6d0 con 0x7f1c9c0b15c0 2023-11-28T13:06:10.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca1ba4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1c9c0b15c0 0x7f1c9c0ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca2ba6700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f1c9c0b0250 0x7f1c9c0b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca2ba6700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f1c9c0b0250 0x7f1c9c0b06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:56268/0 (socket says 172.21.15.111:56268) 2023-11-28T13:06:10.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca1ba4700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1c9c0b15c0 0x7f1c9c0ac670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:60114/0 (socket says 172.21.15.111:60114) 2023-11-28T13:06:10.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca2ba6700 1 -- 172.21.15.111:0/3276988637 learned_addr learned my addr 172.21.15.111:0/3276988637 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:10.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca23a5700 1 --1- 172.21.15.111:0/3276988637 >> v1:172.21.15.149:6789/0 conn(0x7f1c9c15a9e0 0x7f1c9c0caef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:47684/0 (socket says 172.21.15.111:47684) 2023-11-28T13:06:10.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca1ba4700 1 -- 172.21.15.111:0/3276988637 >> v2:172.21.15.149:3300/0 conn(0x7f1c9c0b0250 msgr2=0x7f1c9c0b06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:10.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca1ba4700 1 --2- 172.21.15.111:0/3276988637 >> v2:172.21.15.149:3300/0 conn(0x7f1c9c0b0250 0x7f1c9c0b06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca1ba4700 1 -- 172.21.15.111:0/3276988637 >> v1:172.21.15.149:6789/0 conn(0x7f1c9c15a9e0 legacy=0x7f1c9c0caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca1ba4700 1 -- 172.21.15.111:0/3276988637 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9c0cb9b0 con 0x7f1c9c0b15c0 2023-11-28T13:06:10.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.063+0000 7f1ca2ba6700 1 --2- 172.21.15.111:0/3276988637 >> v2:172.21.15.149:3300/0 conn(0x7f1c9c0b0250 0x7f1c9c0b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:06:10.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.064+0000 7f1ca1ba4700 1 --2- 172.21.15.111:0/3276988637 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1c9c0b15c0 0x7f1c9c0ac670 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c009840 tx=0x7f1c8c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a83a5f0dff865dcc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:10.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.064+0000 7f1ca13a3700 1 -- 172.21.15.111:0/3276988637 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c011830 con 0x7f1c9c0b15c0 2023-11-28T13:06:10.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.064+0000 7f1ca13a3700 1 -- 172.21.15.111:0/3276988637 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8c011990 con 0x7f1c9c0b15c0 2023-11-28T13:06:10.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.064+0000 7f1ca13a3700 1 -- 172.21.15.111:0/3276988637 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c011d20 con 0x7f1c9c0b15c0 2023-11-28T13:06:10.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.064+0000 7f1ca4609700 1 -- 172.21.15.111:0/3276988637 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1c9c0b15c0 msgr2=0x7f1c9c0ac670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.064+0000 7f1ca4609700 1 --2- 172.21.15.111:0/3276988637 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1c9c0b15c0 0x7f1c9c0ac670 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c009840 tx=0x7f1c8c009b50 comp rx=0 tx=0).stop 2023-11-28T13:06:10.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.064+0000 7f1ca4609700 1 -- 172.21.15.111:0/3276988637 shutdown_connections 2023-11-28T13:06:10.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.064+0000 7f1ca4609700 1 --2- 172.21.15.111:0/3276988637 >> v2:172.21.15.149:3300/0 conn(0x7f1c9c0b0250 0x7f1c9c0b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.064+0000 7f1ca4609700 1 --2- 172.21.15.111:0/3276988637 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1c9c0b15c0 0x7f1c9c0ac670 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.064+0000 7f1ca4609700 1 -- 172.21.15.111:0/3276988637 >> 172.21.15.111:0/3276988637 conn(0x7f1c9c098ee0 msgr2=0x7f1c9c06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:10.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.064+0000 7f1ca4609700 1 -- 172.21.15.111:0/3276988637 shutdown_connections 2023-11-28T13:06:10.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.064+0000 7f1ca4609700 1 -- 172.21.15.111:0/3276988637 wait complete. 2023-11-28T13:06:10.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.066+0000 7f1ca4609700 1 Processor -- start 2023-11-28T13:06:10.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.066+0000 7f1ca4609700 1 -- start start 2023-11-28T13:06:10.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.066+0000 7f1ca4609700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c9c0b0250 0x7f1c9c1518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.066+0000 7f1ca4609700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1c9c0b15c0 0x7f1c9c15ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.066+0000 7f1ca4609700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1c9c15a9e0 0x7f1c9c15c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.066+0000 7f1ca4609700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c0c71b0 con 0x7f1c9c0b0250 2023-11-28T13:06:10.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.066+0000 7f1ca4609700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c151e20 con 0x7f1c9c15a9e0 2023-11-28T13:06:10.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.066+0000 7f1ca4609700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c0c6b80 con 0x7f1c9c0b15c0 2023-11-28T13:06:10.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.066+0000 7f1ca23a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c9c0b0250 0x7f1c9c1518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.066+0000 7f1ca23a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c9c0b0250 0x7f1c9c1518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38370/0 (socket says 172.21.15.111:38370) 2023-11-28T13:06:10.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.066+0000 7f1ca23a5700 1 -- 172.21.15.111:0/1718336168 learned_addr learned my addr 172.21.15.111:0/1718336168 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:10.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.067+0000 7f1ca1ba4700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1c9c0b15c0 0x7f1c9c15ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.067+0000 7f1ca2ba6700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1c9c15a9e0 0x7f1c9c15c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.067+0000 7f1ca23a5700 1 -- 172.21.15.111:0/1718336168 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1c9c0b15c0 msgr2=0x7f1c9c15ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.067+0000 7f1ca23a5700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1c9c0b15c0 0x7f1c9c15ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.067+0000 7f1ca23a5700 1 -- 172.21.15.111:0/1718336168 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1c9c15a9e0 msgr2=0x7f1c9c15c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.067+0000 7f1ca23a5700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1c9c15a9e0 0x7f1c9c15c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.067+0000 7f1ca23a5700 1 -- 172.21.15.111:0/1718336168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c94009b30 con 0x7f1c9c0b0250 2023-11-28T13:06:10.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.067+0000 7f1ca23a5700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c9c0b0250 0x7f1c9c1518e0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800c460 tx=0x7f1c9800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:10.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.067+0000 7f1c92ffd700 1 -- 172.21.15.111:0/1718336168 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98005d00 con 0x7f1c9c0b0250 2023-11-28T13:06:10.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.067+0000 7f1c92ffd700 1 -- 172.21.15.111:0/1718336168 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c98005e60 con 0x7f1c9c0b0250 2023-11-28T13:06:10.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.067+0000 7f1ca4609700 1 -- 172.21.15.111:0/1718336168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8c0094a0 con 0x7f1c9c0b0250 2023-11-28T13:06:10.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.067+0000 7f1c92ffd700 1 -- 172.21.15.111:0/1718336168 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98007310 con 0x7f1c9c0b0250 2023-11-28T13:06:10.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.067+0000 7f1ca4609700 1 -- 172.21.15.111:0/1718336168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c9c0cbf70 con 0x7f1c9c0b0250 2023-11-28T13:06:10.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.068+0000 7f1ca4609700 1 -- 172.21.15.111:0/1718336168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c9c0cbc90 con 0x7f1c9c0b0250 2023-11-28T13:06:10.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.068+0000 7f1c92ffd700 1 -- 172.21.15.111:0/1718336168 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1c98019030 con 0x7f1c9c0b0250 2023-11-28T13:06:10.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.069+0000 7f1c92ffd700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1c800682f0 0x7f1c8006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.069+0000 7f1c92ffd700 1 -- 172.21.15.111:0/1718336168 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f1c98084fa0 con 0x7f1c9c0b0250 2023-11-28T13:06:10.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.069+0000 7f1ca1ba4700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1c800682f0 0x7f1c8006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.069+0000 7f1ca1ba4700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1c800682f0 0x7f1c8006a7a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0090b0 tx=0x7f1c8c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:10.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.073+0000 7f1c92ffd700 1 -- 172.21.15.111:0/1718336168 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1c98051680 con 0x7f1c9c0b0250 2023-11-28T13:06:10.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.255+0000 7f1ca4609700 1 -- 172.21.15.111:0/1718336168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1c9c0cc250 con 0x7f1c9c0b0250 2023-11-28T13:06:10.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.255+0000 7f1c92ffd700 1 -- 172.21.15.111:0/1718336168 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1c98017020 con 0x7f1c9c0b0250 2023-11-28T13:06:10.256 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T13:06:10.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.257+0000 7f1ca4609700 1 -- 172.21.15.111:0/1718336168 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1c800682f0 msgr2=0x7f1c8006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.257+0000 7f1ca4609700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1c800682f0 0x7f1c8006a7a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0090b0 tx=0x7f1c8c008030 comp rx=0 tx=0).stop 2023-11-28T13:06:10.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.257+0000 7f1ca4609700 1 -- 172.21.15.111:0/1718336168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c9c0b0250 msgr2=0x7f1c9c1518e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.257+0000 7f1ca4609700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c9c0b0250 0x7f1c9c1518e0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800c460 tx=0x7f1c9800c820 comp rx=0 tx=0).stop 2023-11-28T13:06:10.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.257+0000 7f1ca4609700 1 -- 172.21.15.111:0/1718336168 shutdown_connections 2023-11-28T13:06:10.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.257+0000 7f1ca4609700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1c800682f0 0x7f1c8006a7a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.257+0000 7f1ca4609700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1c9c15a9e0 0x7f1c9c15c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.257+0000 7f1ca4609700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c9c0b0250 0x7f1c9c1518e0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.257+0000 7f1ca4609700 1 --2- 172.21.15.111:0/1718336168 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1c9c0b15c0 0x7f1c9c15ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.257+0000 7f1ca4609700 1 -- 172.21.15.111:0/1718336168 >> 172.21.15.111:0/1718336168 conn(0x7f1c9c098ee0 msgr2=0x7f1c9c0cfad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:10.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.258+0000 7f1ca4609700 1 -- 172.21.15.111:0/1718336168 shutdown_connections 2023-11-28T13:06:10.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.258+0000 7f1ca4609700 1 -- 172.21.15.111:0/1718336168 wait complete. 2023-11-28T13:06:10.270 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T13:06:10.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.394+0000 7fc3f63e8700 1 Processor -- start 2023-11-28T13:06:10.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.394+0000 7fc3f63e8700 1 -- start start 2023-11-28T13:06:10.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.395+0000 7fc3f63e8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3f006a980 0x7fc3f00ad710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.395+0000 7fc3f63e8700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f015aa70 con 0x7fc3f00af160 2023-11-28T13:06:10.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.395+0000 7fc3f63e8700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f00ad280 con 0x7fc3f015a6a0 2023-11-28T13:06:10.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.395+0000 7fc3f63e8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f00ac950 con 0x7fc3f006a980 2023-11-28T13:06:10.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.395+0000 7fc3f4985700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc3f00af160 0x7fc3f00adc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39160/0 (socket says 172.21.15.111:39160) 2023-11-28T13:06:10.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.395+0000 7fc3f4985700 1 -- 172.21.15.111:0/4024050541 learned_addr learned my addr 172.21.15.111:0/4024050541 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:10.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3ef7fe700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3f006a980 0x7fc3f00ad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896795092 0 0) 0x7fc3e8001490 con 0x7fc3f00af160 2023-11-28T13:06:10.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d80035b0 con 0x7fc3f00af160 2023-11-28T13:06:10.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577456269 0 0) 0x7fc3f00ad280 con 0x7fc3f015a6a0 2023-11-28T13:06:10.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e8001490 con 0x7fc3f015a6a0 2023-11-28T13:06:10.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1955399457 0 0) 0x7fc3d80035b0 con 0x7fc3f00af160 2023-11-28T13:06:10.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3f006a980 msgr2=0x7fc3f00ad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:10.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3f006a980 0x7fc3f00ad710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 >> v1:172.21.15.149:6789/0 conn(0x7fc3f015a6a0 legacy=0x7fc3f006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f00acc30 con 0x7fc3f00af160 2023-11-28T13:06:10.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fc3e8001b80 con 0x7fc3f00af160 2023-11-28T13:06:10.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 >> v1:172.21.15.111:6789/0 conn(0x7fc3f00af160 legacy=0x7fc3f00adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3f006a980 0x7fc3d8003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3d8004120 0x7fc3d8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc3d8004d90 0x7fc3d8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f00ac950 con 0x7fc3d8004120 2023-11-28T13:06:10.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d800bba0 con 0x7fc3d8004d90 2023-11-28T13:06:10.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d800bd30 con 0x7fc3f006a980 2023-11-28T13:06:10.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3ef7fe700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3d8004120 0x7fc3d8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3f4985700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc3d8004d90 0x7fc3d8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3effff700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3f006a980 0x7fc3d8003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3ef7fe700 1 -- 172.21.15.111:0/4024050541 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3f006a980 msgr2=0x7fc3d8003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:10.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3ef7fe700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3f006a980 0x7fc3d8003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3ef7fe700 1 -- 172.21.15.111:0/4024050541 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc3d8004d90 msgr2=0x7fc3d8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:10.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3ef7fe700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc3d8004d90 0x7fc3d8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3ef7fe700 1 -- 172.21.15.111:0/4024050541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d80035b0 con 0x7fc3d8004120 2023-11-28T13:06:10.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3f4985700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc3d8004d90 0x7fc3d8005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:10.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.396+0000 7fc3effff700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3f006a980 0x7fc3d8003bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:10.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.397+0000 7fc3ef7fe700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3d8004120 0x7fc3d8004850 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0009b30 tx=0x7fc3d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e1bd1d4f07276f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:10.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.397+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e000e6c0 con 0x7fc3d8004120 2023-11-28T13:06:10.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.397+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e000e820 con 0x7fc3d8004120 2023-11-28T13:06:10.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.397+0000 7fc3eeffd700 1 -- 172.21.15.111:0/4024050541 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e000eb90 con 0x7fc3d8004120 2023-11-28T13:06:10.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.397+0000 7fc3f63e8700 1 -- 172.21.15.111:0/4024050541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3d8004120 msgr2=0x7fc3d8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.397+0000 7fc3f63e8700 1 --2- 172.21.15.111:0/4024050541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3d8004120 0x7fc3d8004850 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0009b30 tx=0x7fc3d8005880 comp rx=0 tx=0).stop 2023-11-28T13:06:10.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.397+0000 7fc3f4985700 1 -- 172.21.15.111:0/4024050541 reap_dead start 2023-11-28T13:06:10.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.397+0000 7fc3f63e8700 1 -- 172.21.15.111:0/4024050541 shutdown_connections 2023-11-28T13:06:10.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.397+0000 7fc3f63e8700 1 -- 172.21.15.111:0/4024050541 >> 172.21.15.111:0/4024050541 conn(0x7fc3f0098ee0 msgr2=0x7fc3f006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:10.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.397+0000 7fc3f63e8700 1 -- 172.21.15.111:0/4024050541 shutdown_connections 2023-11-28T13:06:10.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.397+0000 7fc3f63e8700 1 -- 172.21.15.111:0/4024050541 wait complete. 2023-11-28T13:06:10.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.398+0000 7fc3f63e8700 1 Processor -- start 2023-11-28T13:06:10.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.399+0000 7fc3f63e8700 1 -- start start 2023-11-28T13:06:10.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.399+0000 7fc3f63e8700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3d8004120 0x7fc3f00cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.399+0000 7fc3f63e8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f00c5960 0x7fc3f00cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.399+0000 7fc3f63e8700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc3f014b740 0x7fc3f006ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.399+0000 7fc3f63e8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f00cce30 con 0x7fc3f00c5960 2023-11-28T13:06:10.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.399+0000 7fc3f63e8700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f00c54d0 con 0x7fc3f014b740 2023-11-28T13:06:10.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.399+0000 7fc3f63e8700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f00c5dd0 con 0x7fc3d8004120 2023-11-28T13:06:10.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.399+0000 7fc3ef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f00c5960 0x7fc3f00cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.399+0000 7fc3ef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f00c5960 0x7fc3f00cc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38392/0 (socket says 172.21.15.111:38392) 2023-11-28T13:06:10.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.399+0000 7fc3ef7fe700 1 -- 172.21.15.111:0/2780103987 learned_addr learned my addr 172.21.15.111:0/2780103987 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:10.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.400+0000 7fc3f4985700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc3f014b740 0x7fc3f006ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.400+0000 7fc3effff700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3d8004120 0x7fc3f00cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.400+0000 7fc3ef7fe700 1 -- 172.21.15.111:0/2780103987 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3d8004120 msgr2=0x7fc3f00cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.400+0000 7fc3ef7fe700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3d8004120 0x7fc3f00cb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.400+0000 7fc3ef7fe700 1 -- 172.21.15.111:0/2780103987 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc3f014b740 msgr2=0x7fc3f006ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.400+0000 7fc3ef7fe700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc3f014b740 0x7fc3f006ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.400+0000 7fc3ef7fe700 1 -- 172.21.15.111:0/2780103987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f00992e0 con 0x7fc3f00c5960 2023-11-28T13:06:10.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.400+0000 7fc3ef7fe700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f00c5960 0x7fc3f00cc8f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000e4b0 tx=0x7fc3e00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:10.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.400+0000 7fc3ecff9700 1 -- 172.21.15.111:0/2780103987 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e000e6c0 con 0x7fc3f00c5960 2023-11-28T13:06:10.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.400+0000 7fc3ecff9700 1 -- 172.21.15.111:0/2780103987 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e0003ad0 con 0x7fc3f00c5960 2023-11-28T13:06:10.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.400+0000 7fc3f63e8700 1 -- 172.21.15.111:0/2780103987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f00acf10 con 0x7fc3f00c5960 2023-11-28T13:06:10.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.400+0000 7fc3ecff9700 1 -- 172.21.15.111:0/2780103987 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0003e80 con 0x7fc3f00c5960 2023-11-28T13:06:10.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.400+0000 7fc3f63e8700 1 -- 172.21.15.111:0/2780103987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f00aec10 con 0x7fc3f00c5960 2023-11-28T13:06:10.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.401+0000 7fc3f63e8700 1 -- 172.21.15.111:0/2780103987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f0067170 con 0x7fc3f00c5960 2023-11-28T13:06:10.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.401+0000 7fc3ecff9700 1 -- 172.21.15.111:0/2780103987 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc3e0025070 con 0x7fc3f00c5960 2023-11-28T13:06:10.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.402+0000 7fc3ecff9700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc3dc0682f0 0x7fc3dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.402+0000 7fc3ecff9700 1 -- 172.21.15.111:0/2780103987 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fc3e00875f0 con 0x7fc3f00c5960 2023-11-28T13:06:10.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.402+0000 7fc3effff700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc3dc0682f0 0x7fc3dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.402+0000 7fc3effff700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc3dc0682f0 0x7fc3dc06a7a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4000c00 tx=0x7fc3e4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:10.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.406+0000 7fc3ecff9700 1 -- 172.21.15.111:0/2780103987 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc3e0053cd0 con 0x7fc3f00c5960 2023-11-28T13:06:10.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.587+0000 7fc3f63e8700 1 -- 172.21.15.111:0/2780103987 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc3f006a270 con 0x7fc3f00c5960 2023-11-28T13:06:10.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.588+0000 7fc3ecff9700 1 -- 172.21.15.111:0/2780103987 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v69) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc3e00572f0 con 0x7fc3f00c5960 2023-11-28T13:06:10.589 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T13:06:10.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.590+0000 7fc3f63e8700 1 -- 172.21.15.111:0/2780103987 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc3dc0682f0 msgr2=0x7fc3dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.590+0000 7fc3f63e8700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc3dc0682f0 0x7fc3dc06a7a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4000c00 tx=0x7fc3e4009040 comp rx=0 tx=0).stop 2023-11-28T13:06:10.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.590+0000 7fc3f63e8700 1 -- 172.21.15.111:0/2780103987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f00c5960 msgr2=0x7fc3f00cc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.590+0000 7fc3f63e8700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f00c5960 0x7fc3f00cc8f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000e4b0 tx=0x7fc3e00047c0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.590+0000 7fc3f63e8700 1 -- 172.21.15.111:0/2780103987 shutdown_connections 2023-11-28T13:06:10.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.590+0000 7fc3f63e8700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc3dc0682f0 0x7fc3dc06a7a0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.590+0000 7fc3f63e8700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc3f014b740 0x7fc3f006ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.590+0000 7fc3f63e8700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc3d8004120 0x7fc3f00cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.590+0000 7fc3f63e8700 1 --2- 172.21.15.111:0/2780103987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc3f00c5960 0x7fc3f00cc8f0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.590+0000 7fc3f63e8700 1 -- 172.21.15.111:0/2780103987 >> 172.21.15.111:0/2780103987 conn(0x7fc3f0098ee0 msgr2=0x7fc3f00b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:10.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.590+0000 7fc3f63e8700 1 -- 172.21.15.111:0/2780103987 shutdown_connections 2023-11-28T13:06:10.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.590+0000 7fc3f63e8700 1 -- 172.21.15.111:0/2780103987 wait complete. 2023-11-28T13:06:10.603 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T13:06:10.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.725+0000 7f95aa42c700 1 Processor -- start 2023-11-28T13:06:10.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.725+0000 7f95aa42c700 1 -- start start 2023-11-28T13:06:10.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.726+0000 7f95aa42c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f95a40b0d70 0x7f95a41604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.726+0000 7f95aa42c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95a40b1140 0x7f95a40ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.726+0000 7f95aa42c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f95a40b15b0 0x7f95a40cc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.726+0000 7f95aa42c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a415cd90 con 0x7f95a40b0d70 2023-11-28T13:06:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.726+0000 7f95aa42c700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a41613e0 con 0x7f95a40b15b0 2023-11-28T13:06:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.726+0000 7f95aa42c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a4160b40 con 0x7f95a40b1140 2023-11-28T13:06:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.726+0000 7f95a3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f95a40b0d70 0x7f95a41604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.726+0000 7f95a3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f95a40b0d70 0x7f95a41604b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38398/0 (socket says 172.21.15.111:38398) 2023-11-28T13:06:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.726+0000 7f95a3fff700 1 -- 172.21.15.111:0/3092487163 learned_addr learned my addr 172.21.15.111:0/3092487163 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.727+0000 7f95a37fe700 1 --2- 172.21.15.111:0/3092487163 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95a40b1140 0x7f95a40ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.727+0000 7f95a89c9700 1 --2- 172.21.15.111:0/3092487163 >> v2:172.21.15.149:3300/0 conn(0x7f95a40b15b0 0x7f95a40cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.727+0000 7f95a3fff700 1 -- 172.21.15.111:0/3092487163 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95a40b1140 msgr2=0x7f95a40ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.727+0000 7f95a3fff700 1 --2- 172.21.15.111:0/3092487163 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95a40b1140 0x7f95a40ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.727+0000 7f95a3fff700 1 -- 172.21.15.111:0/3092487163 >> v2:172.21.15.149:3300/0 conn(0x7f95a40b15b0 msgr2=0x7f95a40cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.727+0000 7f95a3fff700 1 --2- 172.21.15.111:0/3092487163 >> v2:172.21.15.149:3300/0 conn(0x7f95a40b15b0 0x7f95a40cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.727+0000 7f95a3fff700 1 -- 172.21.15.111:0/3092487163 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a4160e20 con 0x7f95a40b0d70 2023-11-28T13:06:10.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.727+0000 7f95a37fe700 1 --2- 172.21.15.111:0/3092487163 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95a40b1140 0x7f95a40ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:10.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.727+0000 7f95a89c9700 1 --2- 172.21.15.111:0/3092487163 >> v2:172.21.15.149:3300/0 conn(0x7f95a40b15b0 0x7f95a40cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:10.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.727+0000 7f95a3fff700 1 --2- 172.21.15.111:0/3092487163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95a40b0d70 0x7f95a41604b0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f95a40cfb10 tx=0x7f9598009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43cceb1d758d9ce server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:10.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.727+0000 7f95a2ffd700 1 -- 172.21.15.111:0/3092487163 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9598005e30 con 0x7f95a40b0d70 2023-11-28T13:06:10.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.727+0000 7f95a2ffd700 1 -- 172.21.15.111:0/3092487163 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9598004030 con 0x7f95a40b0d70 2023-11-28T13:06:10.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.727+0000 7f95a2ffd700 1 -- 172.21.15.111:0/3092487163 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95980043e0 con 0x7f95a40b0d70 2023-11-28T13:06:10.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.728+0000 7f95aa42c700 1 -- 172.21.15.111:0/3092487163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95a40b0d70 msgr2=0x7f95a41604b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.728+0000 7f95aa42c700 1 --2- 172.21.15.111:0/3092487163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95a40b0d70 0x7f95a41604b0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f95a40cfb10 tx=0x7f9598009bd0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.728+0000 7f95aa42c700 1 -- 172.21.15.111:0/3092487163 shutdown_connections 2023-11-28T13:06:10.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.728+0000 7f95aa42c700 1 --2- 172.21.15.111:0/3092487163 >> v2:172.21.15.149:3300/0 conn(0x7f95a40b15b0 0x7f95a40cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.728+0000 7f95aa42c700 1 --2- 172.21.15.111:0/3092487163 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95a40b0d70 0x7f95a41604b0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.728+0000 7f95aa42c700 1 --2- 172.21.15.111:0/3092487163 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95a40b1140 0x7f95a40ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.728+0000 7f95aa42c700 1 -- 172.21.15.111:0/3092487163 >> 172.21.15.111:0/3092487163 conn(0x7f95a4098ee0 msgr2=0x7f95a406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.728+0000 7f95aa42c700 1 -- 172.21.15.111:0/3092487163 shutdown_connections 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.728+0000 7f95aa42c700 1 -- 172.21.15.111:0/3092487163 wait complete. 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.729+0000 7f95aa42c700 1 Processor -- start 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.729+0000 7f95aa42c700 1 -- start start 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95aa42c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f95a40b0d70 0x7f95a41594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95aa42c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95a40b1140 0x7f95a40b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95aa42c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95a40b15b0 0x7f95a415acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95aa42c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a40b3e80 con 0x7f95a40b15b0 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95aa42c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a40b3270 con 0x7f95a40b0d70 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95aa42c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a40b3550 con 0x7f95a40b1140 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a89c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95a40b15b0 0x7f95a415acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a89c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95a40b15b0 0x7f95a415acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38408/0 (socket says 172.21.15.111:38408) 2023-11-28T13:06:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a89c9700 1 -- 172.21.15.111:0/344022488 learned_addr learned my addr 172.21.15.111:0/344022488 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a37fe700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95a40b1140 0x7f95a40b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a3fff700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f95a40b0d70 0x7f95a41594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a89c9700 1 -- 172.21.15.111:0/344022488 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95a40b1140 msgr2=0x7f95a40b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a89c9700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95a40b1140 0x7f95a40b2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a89c9700 1 -- 172.21.15.111:0/344022488 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f95a40b0d70 msgr2=0x7f95a41594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a89c9700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f95a40b0d70 0x7f95a41594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a89c9700 1 -- 172.21.15.111:0/344022488 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9598009500 con 0x7f95a40b15b0 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a89c9700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95a40b15b0 0x7f95a415acf0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f959c00cd10 tx=0x7f959c00a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a0ff9700 1 -- 172.21.15.111:0/344022488 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c003a20 con 0x7f95a40b15b0 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a0ff9700 1 -- 172.21.15.111:0/344022488 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959c003ba0 con 0x7f95a40b15b0 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95aa42c700 1 -- 172.21.15.111:0/344022488 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a40af340 con 0x7f95a40b15b0 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95a0ff9700 1 -- 172.21.15.111:0/344022488 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c004020 con 0x7f95a40b15b0 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.730+0000 7f95aa42c700 1 -- 172.21.15.111:0/344022488 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a4161100 con 0x7f95a40b15b0 2023-11-28T13:06:10.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.731+0000 7f95aa42c700 1 -- 172.21.15.111:0/344022488 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a40cec00 con 0x7f95a40b15b0 2023-11-28T13:06:10.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.732+0000 7f95a0ff9700 1 -- 172.21.15.111:0/344022488 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f959c01f070 con 0x7f95a40b15b0 2023-11-28T13:06:10.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.732+0000 7f95a0ff9700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95900682f0 0x7f959006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:10.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.732+0000 7f95a0ff9700 1 -- 172.21.15.111:0/344022488 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f959c086450 con 0x7f95a40b15b0 2023-11-28T13:06:10.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.732+0000 7f95a3fff700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95900682f0 0x7f959006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:10.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.733+0000 7f95a3fff700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95900682f0 0x7f959006a7a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f95a40cfb10 tx=0x7f9598008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:10.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.736+0000 7f95a0ff9700 1 -- 172.21.15.111:0/344022488 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f959c052a40 con 0x7f95a40b15b0 2023-11-28T13:06:10.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.917+0000 7f95aa42c700 1 -- 172.21.15.111:0/344022488 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f95a4067170 con 0x7f95a40b15b0 2023-11-28T13:06:10.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.917+0000 7f95a0ff9700 1 -- 172.21.15.111:0/344022488 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f959c056060 con 0x7f95a40b15b0 2023-11-28T13:06:10.918 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T13:06:10.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.919+0000 7f95aa42c700 1 -- 172.21.15.111:0/344022488 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95900682f0 msgr2=0x7f959006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.919+0000 7f95aa42c700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95900682f0 0x7f959006a7a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f95a40cfb10 tx=0x7f9598008070 comp rx=0 tx=0).stop 2023-11-28T13:06:10.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.919+0000 7f95aa42c700 1 -- 172.21.15.111:0/344022488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95a40b15b0 msgr2=0x7f95a415acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:10.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.919+0000 7f95aa42c700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95a40b15b0 0x7f95a415acf0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f959c00cd10 tx=0x7f959c00a560 comp rx=0 tx=0).stop 2023-11-28T13:06:10.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.919+0000 7f95aa42c700 1 -- 172.21.15.111:0/344022488 shutdown_connections 2023-11-28T13:06:10.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.919+0000 7f95aa42c700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95a40b15b0 0x7f95a415acf0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.919+0000 7f95aa42c700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f95900682f0 0x7f959006a7a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.919+0000 7f95aa42c700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f95a40b0d70 0x7f95a41594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.919+0000 7f95aa42c700 1 --2- 172.21.15.111:0/344022488 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f95a40b1140 0x7f95a40b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:10.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.919+0000 7f95aa42c700 1 -- 172.21.15.111:0/344022488 >> 172.21.15.111:0/344022488 conn(0x7f95a4098ee0 msgr2=0x7f95a406f380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:10.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.919+0000 7f95aa42c700 1 -- 172.21.15.111:0/344022488 shutdown_connections 2023-11-28T13:06:10.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:10.919+0000 7f95aa42c700 1 -- 172.21.15.111:0/344022488 wait complete. 2023-11-28T13:06:10.932 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T13:06:11.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.054+0000 7f65e1512700 1 Processor -- start 2023-11-28T13:06:11.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.054+0000 7f65e1512700 1 -- start start 2023-11-28T13:06:11.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65e1512700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f65dc0b8510 0x7f65dc06b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65e1512700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f65dc0cd460 0x7f65dc06beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65e1512700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f65dc06c3f0 0x7f65dc15cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65e1512700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65dc0cdbd0 con 0x7f65dc0cd460 2023-11-28T13:06:11.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65e1512700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65dc0d02d0 con 0x7f65dc0b8510 2023-11-28T13:06:11.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65e1512700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc0b88e0 con 0x7f65dc06c3f0 2023-11-28T13:06:11.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65da7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f65dc0cd460 0x7f65dc06beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65da7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f65dc0cd460 0x7f65dc06beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38412/0 (socket says 172.21.15.111:38412) 2023-11-28T13:06:11.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65da7fc700 1 -- 172.21.15.111:0/1785938521 learned_addr learned my addr 172.21.15.111:0/1785938521 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:11.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65daffd700 1 --2- 172.21.15.111:0/1785938521 >> v2:172.21.15.149:3300/0 conn(0x7f65dc0b8510 0x7f65dc06b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65db7fe700 1 --2- 172.21.15.111:0/1785938521 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f65dc06c3f0 0x7f65dc15cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65da7fc700 1 -- 172.21.15.111:0/1785938521 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f65dc06c3f0 msgr2=0x7f65dc15cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:11.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65da7fc700 1 --2- 172.21.15.111:0/1785938521 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f65dc06c3f0 0x7f65dc15cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65da7fc700 1 -- 172.21.15.111:0/1785938521 >> v2:172.21.15.149:3300/0 conn(0x7f65dc0b8510 msgr2=0x7f65dc06b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:11.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65da7fc700 1 --2- 172.21.15.111:0/1785938521 >> v2:172.21.15.149:3300/0 conn(0x7f65dc0b8510 0x7f65dc06b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65da7fc700 1 -- 172.21.15.111:0/1785938521 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc0cd8f0 con 0x7f65dc0cd460 2023-11-28T13:06:11.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65daffd700 1 --2- 172.21.15.111:0/1785938521 >> v2:172.21.15.149:3300/0 conn(0x7f65dc0b8510 0x7f65dc06b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:11.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65db7fe700 1 --2- 172.21.15.111:0/1785938521 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f65dc06c3f0 0x7f65dc15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:11.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.055+0000 7f65da7fc700 1 --2- 172.21.15.111:0/1785938521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65dc0cd460 0x7f65dc06beb0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009920 tx=0x7f65cc009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d02c64cce4e888 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:11.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.056+0000 7f65d9ffb700 1 -- 172.21.15.111:0/1785938521 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00b9e0 con 0x7f65dc0cd460 2023-11-28T13:06:11.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.056+0000 7f65d9ffb700 1 -- 172.21.15.111:0/1785938521 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc00bb40 con 0x7f65dc0cd460 2023-11-28T13:06:11.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.056+0000 7f65d9ffb700 1 -- 172.21.15.111:0/1785938521 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc009e70 con 0x7f65dc0cd460 2023-11-28T13:06:11.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.056+0000 7f65e1512700 1 -- 172.21.15.111:0/1785938521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65dc0cd460 msgr2=0x7f65dc06beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.056+0000 7f65e1512700 1 --2- 172.21.15.111:0/1785938521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65dc0cd460 0x7f65dc06beb0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009920 tx=0x7f65cc009c30 comp rx=0 tx=0).stop 2023-11-28T13:06:11.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.056+0000 7f65e1512700 1 -- 172.21.15.111:0/1785938521 shutdown_connections 2023-11-28T13:06:11.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.056+0000 7f65e1512700 1 --2- 172.21.15.111:0/1785938521 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f65dc06c3f0 0x7f65dc15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.056+0000 7f65e1512700 1 --2- 172.21.15.111:0/1785938521 >> v2:172.21.15.149:3300/0 conn(0x7f65dc0b8510 0x7f65dc06b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.056+0000 7f65e1512700 1 --2- 172.21.15.111:0/1785938521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65dc0cd460 0x7f65dc06beb0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.056+0000 7f65e1512700 1 -- 172.21.15.111:0/1785938521 >> 172.21.15.111:0/1785938521 conn(0x7f65dc0719a0 msgr2=0x7f65dc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:11.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.056+0000 7f65e1512700 1 -- 172.21.15.111:0/1785938521 shutdown_connections 2023-11-28T13:06:11.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.056+0000 7f65e1512700 1 -- 172.21.15.111:0/1785938521 wait complete. 2023-11-28T13:06:11.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.058+0000 7f65e1512700 1 Processor -- start 2023-11-28T13:06:11.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.058+0000 7f65e1512700 1 -- start start 2023-11-28T13:06:11.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.058+0000 7f65e1512700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f65dc06c3f0 0x7f65dc1e68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.058+0000 7f65e1512700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65dc0b8510 0x7f65dc1f24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.058+0000 7f65e1512700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65dc0cd460 0x7f65dc157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.058+0000 7f65e1512700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc1f2a20 con 0x7f65dc0cd460 2023-11-28T13:06:11.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.058+0000 7f65e1512700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc1e6fa0 con 0x7f65dc0b8510 2023-11-28T13:06:11.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.058+0000 7f65e1512700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc1e1fe0 con 0x7f65dc06c3f0 2023-11-28T13:06:11.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.058+0000 7f65db7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65dc0cd460 0x7f65dc157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.058+0000 7f65db7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65dc0cd460 0x7f65dc157610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38420/0 (socket says 172.21.15.111:38420) 2023-11-28T13:06:11.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.058+0000 7f65db7fe700 1 -- 172.21.15.111:0/2211128209 learned_addr learned my addr 172.21.15.111:0/2211128209 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:11.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65da7fc700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65dc0b8510 0x7f65dc1f24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65daffd700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f65dc06c3f0 0x7f65dc1e68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65db7fe700 1 -- 172.21.15.111:0/2211128209 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f65dc06c3f0 msgr2=0x7f65dc1e68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65db7fe700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f65dc06c3f0 0x7f65dc1e68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65db7fe700 1 -- 172.21.15.111:0/2211128209 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65dc0b8510 msgr2=0x7f65dc1f24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65db7fe700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65dc0b8510 0x7f65dc1f24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65db7fe700 1 -- 172.21.15.111:0/2211128209 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc009550 con 0x7f65dc0cd460 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65daffd700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f65dc06c3f0 0x7f65dc1e68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65db7fe700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65dc0cd460 0x7f65dc157610 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f65d400cef0 tx=0x7f65d400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65c37fe700 1 -- 172.21.15.111:0/2211128209 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d400add0 con 0x7f65dc0cd460 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65c37fe700 1 -- 172.21.15.111:0/2211128209 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d4015440 con 0x7f65dc0cd460 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65e1512700 1 -- 172.21.15.111:0/2211128209 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65dc0cfff0 con 0x7f65dc0cd460 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65c37fe700 1 -- 172.21.15.111:0/2211128209 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d40157f0 con 0x7f65dc0cd460 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.059+0000 7f65e1512700 1 -- 172.21.15.111:0/2211128209 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65dc0cfd10 con 0x7f65dc0cd460 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.060+0000 7f65e1512700 1 -- 172.21.15.111:0/2211128209 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65dc067880 con 0x7f65dc0cd460 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.060+0000 7f65c37fe700 1 -- 172.21.15.111:0/2211128209 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f65d401c030 con 0x7f65dc0cd460 2023-11-28T13:06:11.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.061+0000 7f65c37fe700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f65c80682f0 0x7f65c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.061+0000 7f65c37fe700 1 -- 172.21.15.111:0/2211128209 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f65d4086210 con 0x7f65dc0cd460 2023-11-28T13:06:11.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.061+0000 7f65daffd700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f65c80682f0 0x7f65c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.061+0000 7f65daffd700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f65c80682f0 0x7f65c806a7a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f65d00089b0 tx=0x7f65d0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:11.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.065+0000 7f65c37fe700 1 -- 172.21.15.111:0/2211128209 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f65d4052890 con 0x7f65dc0cd460 2023-11-28T13:06:11.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.246+0000 7f65e1512700 1 -- 172.21.15.111:0/2211128209 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f65dc0d2940 con 0x7f65dc0cd460 2023-11-28T13:06:11.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.246+0000 7f65c37fe700 1 -- 172.21.15.111:0/2211128209 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f65d401a020 con 0x7f65dc0cd460 2023-11-28T13:06:11.247 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T13:06:11.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.248+0000 7f65e1512700 1 -- 172.21.15.111:0/2211128209 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f65c80682f0 msgr2=0x7f65c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.248+0000 7f65e1512700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f65c80682f0 0x7f65c806a7a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f65d00089b0 tx=0x7f65d0008040 comp rx=0 tx=0).stop 2023-11-28T13:06:11.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.248+0000 7f65e1512700 1 -- 172.21.15.111:0/2211128209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65dc0cd460 msgr2=0x7f65dc157610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.248+0000 7f65e1512700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65dc0cd460 0x7f65dc157610 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f65d400cef0 tx=0x7f65d400a340 comp rx=0 tx=0).stop 2023-11-28T13:06:11.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.248+0000 7f65e1512700 1 -- 172.21.15.111:0/2211128209 shutdown_connections 2023-11-28T13:06:11.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.248+0000 7f65e1512700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65dc0cd460 0x7f65dc157610 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.248+0000 7f65e1512700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f65dc06c3f0 0x7f65dc1e68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.248+0000 7f65e1512700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f65c80682f0 0x7f65c806a7a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.248+0000 7f65e1512700 1 --2- 172.21.15.111:0/2211128209 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f65dc0b8510 0x7f65dc1f24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.248+0000 7f65e1512700 1 -- 172.21.15.111:0/2211128209 >> 172.21.15.111:0/2211128209 conn(0x7f65dc0719a0 msgr2=0x7f65dc0b5290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:11.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.248+0000 7f65e1512700 1 -- 172.21.15.111:0/2211128209 shutdown_connections 2023-11-28T13:06:11.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.248+0000 7f65e1512700 1 -- 172.21.15.111:0/2211128209 wait complete. 2023-11-28T13:06:11.261 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T13:06:11.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.381+0000 7f8424cf0700 1 Processor -- start 2023-11-28T13:06:11.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.381+0000 7f8424cf0700 1 -- start start 2023-11-28T13:06:11.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.382+0000 7f8424cf0700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f84200b2350 0x7f84200d0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.382+0000 7f8424cf0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f842015eae0 0x7f84200cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.382+0000 7f8424cf0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84200b18b0 con 0x7f84200b11f0 2023-11-28T13:06:11.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.382+0000 7f8424cf0700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f842015ef60 con 0x7f84200b2350 2023-11-28T13:06:11.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.382+0000 7f8424cf0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84200b1fa0 con 0x7f842015eae0 2023-11-28T13:06:11.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.382+0000 7f841e59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f84200b11f0 0x7f842015e3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39202/0 (socket says 172.21.15.111:39202) 2023-11-28T13:06:11.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.382+0000 7f841e59c700 1 -- 172.21.15.111:0/3589381671 learned_addr learned my addr 172.21.15.111:0/3589381671 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:11.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.382+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017009997 0 0) 0x7f84200b18b0 con 0x7f84200b11f0 2023-11-28T13:06:11.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.382+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84080035b0 con 0x7f84200b11f0 2023-11-28T13:06:11.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.382+0000 7f841ed9d700 1 --2- 172.21.15.111:0/3589381671 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f842015eae0 0x7f84200cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.382+0000 7f841dd9b700 1 --2- 172.21.15.111:0/3589381671 >> v2:172.21.15.149:3300/0 conn(0x7f84200b2350 0x7f84200d0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1409782922 0 0) 0x7f84080035b0 con 0x7f84200b11f0 2023-11-28T13:06:11.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f842015eae0 msgr2=0x7f84200cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:11.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 --2- 172.21.15.111:0/3589381671 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f842015eae0 0x7f84200cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 >> v2:172.21.15.149:3300/0 conn(0x7f84200b2350 msgr2=0x7f84200d0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:11.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 --2- 172.21.15.111:0/3589381671 >> v2:172.21.15.149:3300/0 conn(0x7f84200b2350 0x7f84200d0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84200b1b00 con 0x7f84200b11f0 2023-11-28T13:06:11.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f8414001670 con 0x7f84200b11f0 2023-11-28T13:06:11.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 >> v1:172.21.15.111:6789/0 conn(0x7f84200b11f0 legacy=0x7f842015e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 --2- 172.21.15.111:0/3589381671 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8408003a00 0x7f8408003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 --2- 172.21.15.111:0/3589381671 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f842015eae0 0x7f8408004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 --2- 172.21.15.111:0/3589381671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8408004d90 0x7f8408005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842015ef60 con 0x7f8408004d90 2023-11-28T13:06:11.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84200b1fa0 con 0x7f8408003a00 2023-11-28T13:06:11.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840800bac0 con 0x7f842015eae0 2023-11-28T13:06:11.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841dd9b700 1 --2- 172.21.15.111:0/3589381671 >> v2:172.21.15.149:3300/0 conn(0x7f84200b2350 0x7f84200d0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:11.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841ed9d700 1 --2- 172.21.15.111:0/3589381671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8408004d90 0x7f8408005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841e59c700 1 --2- 172.21.15.111:0/3589381671 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8408003a00 0x7f8408003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841dd9b700 1 --2- 172.21.15.111:0/3589381671 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f842015eae0 0x7f8408004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841ed9d700 1 -- 172.21.15.111:0/3589381671 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f842015eae0 msgr2=0x7f8408004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:11.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841ed9d700 1 --2- 172.21.15.111:0/3589381671 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f842015eae0 0x7f8408004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841ed9d700 1 -- 172.21.15.111:0/3589381671 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8408003a00 msgr2=0x7f8408003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:11.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841ed9d700 1 --2- 172.21.15.111:0/3589381671 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8408003a00 0x7f8408003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841ed9d700 1 -- 172.21.15.111:0/3589381671 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84080035b0 con 0x7f8408004d90 2023-11-28T13:06:11.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841e59c700 1 --2- 172.21.15.111:0/3589381671 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8408003a00 0x7f8408003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:11.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841ed9d700 1 --2- 172.21.15.111:0/3589381671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8408004d90 0x7f8408005200 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f8418007360 tx=0x7f8408005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9db67e4bf99a727 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:11.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8418007e80 con 0x7f8408004d90 2023-11-28T13:06:11.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8418014400 con 0x7f8408004d90 2023-11-28T13:06:11.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.383+0000 7f841d59a700 1 -- 172.21.15.111:0/3589381671 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8418014790 con 0x7f8408004d90 2023-11-28T13:06:11.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.384+0000 7f8424cf0700 1 -- 172.21.15.111:0/3589381671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8408004d90 msgr2=0x7f8408005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.384+0000 7f8424cf0700 1 --2- 172.21.15.111:0/3589381671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8408004d90 0x7f8408005200 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f8418007360 tx=0x7f8408005880 comp rx=0 tx=0).stop 2023-11-28T13:06:11.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.384+0000 7f841ed9d700 1 -- 172.21.15.111:0/3589381671 reap_dead start 2023-11-28T13:06:11.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.384+0000 7f8424cf0700 1 -- 172.21.15.111:0/3589381671 shutdown_connections 2023-11-28T13:06:11.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.384+0000 7f8424cf0700 1 -- 172.21.15.111:0/3589381671 >> 172.21.15.111:0/3589381671 conn(0x7f842009a080 msgr2=0x7f84200d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:11.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.384+0000 7f8424cf0700 1 -- 172.21.15.111:0/3589381671 shutdown_connections 2023-11-28T13:06:11.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.384+0000 7f8424cf0700 1 -- 172.21.15.111:0/3589381671 wait complete. 2023-11-28T13:06:11.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.385+0000 7f8424cf0700 1 Processor -- start 2023-11-28T13:06:11.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.385+0000 7f8424cf0700 1 -- start start 2023-11-28T13:06:11.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f8424cf0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8408004d90 0x7f84200c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f8424cf0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842015b8d0 0x7f842015bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f8424cf0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f84200b3be0 0x7f84200b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f8424cf0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84200b4590 con 0x7f842015b8d0 2023-11-28T13:06:11.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f8424cf0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842015c400 con 0x7f84200b3be0 2023-11-28T13:06:11.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f8424cf0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f842015c6e0 con 0x7f8408004d90 2023-11-28T13:06:11.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f841dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842015b8d0 0x7f842015bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f841dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842015b8d0 0x7f842015bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38438/0 (socket says 172.21.15.111:38438) 2023-11-28T13:06:11.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f841dd9b700 1 -- 172.21.15.111:0/3142348017 learned_addr learned my addr 172.21.15.111:0/3142348017 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:11.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f841dd9b700 1 -- 172.21.15.111:0/3142348017 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8408004d90 msgr2=0x7f84200c74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f841dd9b700 1 --2- 172.21.15.111:0/3142348017 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8408004d90 0x7f84200c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f841dd9b700 1 -- 172.21.15.111:0/3142348017 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f84200b3be0 msgr2=0x7f84200b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f841dd9b700 1 --2- 172.21.15.111:0/3142348017 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f84200b3be0 0x7f84200b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.386+0000 7f841dd9b700 1 -- 172.21.15.111:0/3142348017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84200d1c70 con 0x7f842015b8d0 2023-11-28T13:06:11.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.387+0000 7f841dd9b700 1 --2- 172.21.15.111:0/3142348017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842015b8d0 0x7f842015bd40 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f8410010890 tx=0x7f8410010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:11.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.387+0000 7f840effd700 1 -- 172.21.15.111:0/3142348017 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841000e7e0 con 0x7f842015b8d0 2023-11-28T13:06:11.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.387+0000 7f840effd700 1 -- 172.21.15.111:0/3142348017 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841000c050 con 0x7f842015b8d0 2023-11-28T13:06:11.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.387+0000 7f8424cf0700 1 -- 172.21.15.111:0/3142348017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842015f1e0 con 0x7f842015b8d0 2023-11-28T13:06:11.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.387+0000 7f840effd700 1 -- 172.21.15.111:0/3142348017 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841000eb80 con 0x7f842015b8d0 2023-11-28T13:06:11.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.387+0000 7f8424cf0700 1 -- 172.21.15.111:0/3142348017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84200b15c0 con 0x7f842015b8d0 2023-11-28T13:06:11.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.388+0000 7f8424cf0700 1 -- 172.21.15.111:0/3142348017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8420142590 con 0x7f842015b8d0 2023-11-28T13:06:11.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.388+0000 7f840effd700 1 -- 172.21.15.111:0/3142348017 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8410026070 con 0x7f842015b8d0 2023-11-28T13:06:11.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.388+0000 7f840effd700 1 --2- 172.21.15.111:0/3142348017 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f84000682f0 0x7f840006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.388+0000 7f840effd700 1 -- 172.21.15.111:0/3142348017 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f8410086620 con 0x7f842015b8d0 2023-11-28T13:06:11.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.388+0000 7f841e59c700 1 --2- 172.21.15.111:0/3142348017 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f84000682f0 0x7f840006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.389+0000 7f841e59c700 1 --2- 172.21.15.111:0/3142348017 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f84000682f0 0x7f840006a7a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8414000f60 tx=0x7f8414001c40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:11.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.393+0000 7f840effd700 1 -- 172.21.15.111:0/3142348017 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8410052d00 con 0x7f842015b8d0 2023-11-28T13:06:11.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.591+0000 7f8424cf0700 1 -- 172.21.15.111:0/3142348017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8420072850 con 0x7f842015b8d0 2023-11-28T13:06:11.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.591+0000 7f840effd700 1 -- 172.21.15.111:0/3142348017 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v69) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8410027350 con 0x7f842015b8d0 2023-11-28T13:06:11.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.593+0000 7f8424cf0700 1 -- 172.21.15.111:0/3142348017 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f84000682f0 msgr2=0x7f840006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.593+0000 7f8424cf0700 1 --2- 172.21.15.111:0/3142348017 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f84000682f0 0x7f840006a7a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8414000f60 tx=0x7f8414001c40 comp rx=0 tx=0).stop 2023-11-28T13:06:11.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.593+0000 7f8424cf0700 1 -- 172.21.15.111:0/3142348017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842015b8d0 msgr2=0x7f842015bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.593+0000 7f8424cf0700 1 --2- 172.21.15.111:0/3142348017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842015b8d0 0x7f842015bd40 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f8410010890 tx=0x7f8410010c50 comp rx=0 tx=0).stop 2023-11-28T13:06:11.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.593+0000 7f8424cf0700 1 -- 172.21.15.111:0/3142348017 shutdown_connections 2023-11-28T13:06:11.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.593+0000 7f8424cf0700 1 --2- 172.21.15.111:0/3142348017 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f84000682f0 0x7f840006a7a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.593+0000 7f8424cf0700 1 --2- 172.21.15.111:0/3142348017 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f84200b3be0 0x7f84200b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.593+0000 7f8424cf0700 1 --2- 172.21.15.111:0/3142348017 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8408004d90 0x7f84200c74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.593+0000 7f8424cf0700 1 --2- 172.21.15.111:0/3142348017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842015b8d0 0x7f842015bd40 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.593+0000 7f8424cf0700 1 -- 172.21.15.111:0/3142348017 >> 172.21.15.111:0/3142348017 conn(0x7f842009a080 msgr2=0x7f84201586f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:11.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.593+0000 7f8424cf0700 1 -- 172.21.15.111:0/3142348017 shutdown_connections 2023-11-28T13:06:11.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.593+0000 7f8424cf0700 1 -- 172.21.15.111:0/3142348017 wait complete. 2023-11-28T13:06:11.595 INFO:teuthology.orchestra.run.smithi111.stderr:listed 0 entries 2023-11-28T13:06:11.605 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-28T13:06:11.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:11.606 DEBUG:teuthology.orchestra.run.smithi111:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-28T13:06:11.666 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T13:06:11.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:11.666 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T13:06:11.693 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T13:06:11.693 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:06:11.713 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8026740k,nr_inodes=2006685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16040 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=1609048k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-28T13:06:11.713 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:06:11.714 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:06:11.750 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284268k,nr_inodes=4071067,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18839 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-28T13:06:11.751 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T13:06:11.751 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:06:11.771 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8026740k,nr_inodes=2006685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16040 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=1609048k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-28T13:06:11.771 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:06:11.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.911+0000 7fcb96e64700 1 Processor -- start 2023-11-28T13:06:11.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.911+0000 7fcb96e64700 1 -- start start 2023-11-28T13:06:11.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.912+0000 7fcb96e64700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fcb901604b0 0x7fcb900ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.912+0000 7fcb96e64700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcb900cc550 0x7fcb900cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.912+0000 7fcb96e64700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9015cd90 con 0x7fcb900b0d70 2023-11-28T13:06:11.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.912+0000 7fcb96e64700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb90160950 con 0x7fcb900cc550 2023-11-28T13:06:11.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.912+0000 7fcb96e64700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb900cced0 con 0x7fcb901604b0 2023-11-28T13:06:11.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.912+0000 7fcb8ffff700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fcb901604b0 0x7fcb900ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.913+0000 7fcb94c00700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7fcb900b0d70 0x7fcb900b1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:47766/0 (socket says 172.21.15.111:47766) 2023-11-28T13:06:11.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.913+0000 7fcb8ffff700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fcb901604b0 0x7fcb900ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:56348/0 (socket says 172.21.15.111:56348) 2023-11-28T13:06:11.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.913+0000 7fcb94c00700 1 -- 172.21.15.111:0/985377088 learned_addr learned my addr 172.21.15.111:0/985377088 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:11.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.913+0000 7fcb95401700 1 --2- 172.21.15.111:0/985377088 >> v2:172.21.15.111:3300/0 conn(0x7fcb900cc550 0x7fcb900cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.913+0000 7fcb95401700 1 -- 172.21.15.111:0/985377088 >> v2:172.21.15.149:3300/0 conn(0x7fcb901604b0 msgr2=0x7fcb900ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:11.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.913+0000 7fcb95401700 1 --2- 172.21.15.111:0/985377088 >> v2:172.21.15.149:3300/0 conn(0x7fcb901604b0 0x7fcb900ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.913+0000 7fcb95401700 1 -- 172.21.15.111:0/985377088 >> v1:172.21.15.149:6789/0 conn(0x7fcb900b0d70 legacy=0x7fcb900b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.913+0000 7fcb95401700 1 -- 172.21.15.111:0/985377088 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb901613e0 con 0x7fcb900cc550 2023-11-28T13:06:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.913+0000 7fcb8ffff700 1 --2- 172.21.15.111:0/985377088 >> v2:172.21.15.149:3300/0 conn(0x7fcb901604b0 0x7fcb900ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:06:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.913+0000 7fcb95401700 1 --2- 172.21.15.111:0/985377088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb900cc550 0x7fcb900cc990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fcb9006b310 tx=0x7fcb8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfba5c29890b981f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.913+0000 7fcb8f7fe700 1 -- 172.21.15.111:0/985377088 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8800e070 con 0x7fcb900cc550 2023-11-28T13:06:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.913+0000 7fcb8f7fe700 1 -- 172.21.15.111:0/985377088 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8800bb20 con 0x7fcb900cc550 2023-11-28T13:06:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.913+0000 7fcb8f7fe700 1 -- 172.21.15.111:0/985377088 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88012430 con 0x7fcb900cc550 2023-11-28T13:06:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.914+0000 7fcb96e64700 1 -- 172.21.15.111:0/985377088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb900cc550 msgr2=0x7fcb900cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.914+0000 7fcb96e64700 1 --2- 172.21.15.111:0/985377088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb900cc550 0x7fcb900cc990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fcb9006b310 tx=0x7fcb8800b330 comp rx=0 tx=0).stop 2023-11-28T13:06:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.914+0000 7fcb96e64700 1 -- 172.21.15.111:0/985377088 shutdown_connections 2023-11-28T13:06:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.914+0000 7fcb96e64700 1 --2- 172.21.15.111:0/985377088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb900cc550 0x7fcb900cc990 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.914+0000 7fcb96e64700 1 --2- 172.21.15.111:0/985377088 >> v2:172.21.15.149:3300/0 conn(0x7fcb901604b0 0x7fcb900ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.914+0000 7fcb96e64700 1 -- 172.21.15.111:0/985377088 >> 172.21.15.111:0/985377088 conn(0x7fcb90098ee0 msgr2=0x7fcb9006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.914+0000 7fcb96e64700 1 -- 172.21.15.111:0/985377088 shutdown_connections 2023-11-28T13:06:11.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.914+0000 7fcb96e64700 1 -- 172.21.15.111:0/985377088 wait complete. 2023-11-28T13:06:11.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.915+0000 7fcb96e64700 1 Processor -- start 2023-11-28T13:06:11.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.915+0000 7fcb96e64700 1 -- start start 2023-11-28T13:06:11.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb96e64700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb900b0d70 0x7fcb901f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb96e64700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb900cc550 0x7fcb9006de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb96e64700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb901604b0 0x7fcb90157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb96e64700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb901f2d00 con 0x7fcb900b0d70 2023-11-28T13:06:11.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb96e64700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb901f2fe0 con 0x7fcb901604b0 2023-11-28T13:06:11.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb96e64700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb900c3660 con 0x7fcb900cc550 2023-11-28T13:06:11.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb94c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb900b0d70 0x7fcb901f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb94c00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb900b0d70 0x7fcb901f2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38452/0 (socket says 172.21.15.111:38452) 2023-11-28T13:06:11.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb94c00700 1 -- 172.21.15.111:0/2805125364 learned_addr learned my addr 172.21.15.111:0/2805125364 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:11.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb95401700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb901604b0 0x7fcb90157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb8ffff700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb900cc550 0x7fcb9006de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb94c00700 1 -- 172.21.15.111:0/2805125364 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb900cc550 msgr2=0x7fcb9006de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb94c00700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb900cc550 0x7fcb9006de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb94c00700 1 -- 172.21.15.111:0/2805125364 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb901604b0 msgr2=0x7fcb90157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:11.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb94c00700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb901604b0 0x7fcb90157e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:11.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb94c00700 1 -- 172.21.15.111:0/2805125364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c0094a0 con 0x7fcb900b0d70 2023-11-28T13:06:11.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.916+0000 7fcb94c00700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb900b0d70 0x7fcb901f2480 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400c500 tx=0x7fcb8400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:11.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.917+0000 7fcb8d7fa700 1 -- 172.21.15.111:0/2805125364 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb84005dc0 con 0x7fcb900b0d70 2023-11-28T13:06:11.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.917+0000 7fcb8d7fa700 1 -- 172.21.15.111:0/2805125364 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb840070d0 con 0x7fcb900b0d70 2023-11-28T13:06:11.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.917+0000 7fcb96e64700 1 -- 172.21.15.111:0/2805125364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb88009a70 con 0x7fcb900b0d70 2023-11-28T13:06:11.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.917+0000 7fcb8d7fa700 1 -- 172.21.15.111:0/2805125364 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb84007460 con 0x7fcb900b0d70 2023-11-28T13:06:11.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.917+0000 7fcb96e64700 1 -- 172.21.15.111:0/2805125364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb90160e20 con 0x7fcb900b0d70 2023-11-28T13:06:11.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.918+0000 7fcb96e64700 1 -- 172.21.15.111:0/2805125364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb90160b40 con 0x7fcb900b0d70 2023-11-28T13:06:11.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.918+0000 7fcb8d7fa700 1 -- 172.21.15.111:0/2805125364 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcb84019030 con 0x7fcb900b0d70 2023-11-28T13:06:11.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.918+0000 7fcb8d7fa700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb740682f0 0x7fcb7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:11.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.918+0000 7fcb8d7fa700 1 -- 172.21.15.111:0/2805125364 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fcb84084ff0 con 0x7fcb900b0d70 2023-11-28T13:06:11.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.918+0000 7fcb8ffff700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb740682f0 0x7fcb7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:11.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.919+0000 7fcb8ffff700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb740682f0 0x7fcb7406a7a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcb901f59d0 tx=0x7fcb7c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:11.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:11.922+0000 7fcb8d7fa700 1 -- 172.21.15.111:0/2805125364 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcb840516d0 con 0x7fcb900b0d70 2023-11-28T13:06:12.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.149+0000 7fcb96e64700 1 -- 172.21.15.111:0/2805125364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb90161100 con 0x7fcb900b0d70 2023-11-28T13:06:12.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.150+0000 7fcb8d7fa700 1 -- 172.21.15.111:0/2805125364 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4659 (secure 0 0 0) 0x7fcb84017090 con 0x7fcb900b0d70 2023-11-28T13:06:12.150 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:12.150 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/1668572397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1668572397},{"type":"v1","addr":"172.21.15.149:6834","nonce":1668572397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5755,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1129092621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1129092621},{"type":"v1","addr":"172.21.15.149:6835","nonce":1129092621}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/904588136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":904588136},{"type":"v1","addr":"172.21.15.111:6836","nonce":904588136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:01:54.257638+0000","modified":"2023-11-28T13:02:06.634463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.111:6837/1897250522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1897250522},{"type":"v1","addr":"172.21.15.111:6837","nonce":1897250522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T13:06:12.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.152+0000 7fcb96e64700 1 -- 172.21.15.111:0/2805125364 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb740682f0 msgr2=0x7fcb7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.152+0000 7fcb96e64700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb740682f0 0x7fcb7406a7a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcb901f59d0 tx=0x7fcb7c008040 comp rx=0 tx=0).stop 2023-11-28T13:06:12.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.152+0000 7fcb96e64700 1 -- 172.21.15.111:0/2805125364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb900b0d70 msgr2=0x7fcb901f2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.152+0000 7fcb96e64700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb900b0d70 0x7fcb901f2480 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400c500 tx=0x7fcb8400c8c0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.152+0000 7fcb96e64700 1 -- 172.21.15.111:0/2805125364 shutdown_connections 2023-11-28T13:06:12.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.152+0000 7fcb96e64700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb740682f0 0x7fcb7406a7a0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.152+0000 7fcb96e64700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb901604b0 0x7fcb90157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.152+0000 7fcb96e64700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb900b0d70 0x7fcb901f2480 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.152+0000 7fcb96e64700 1 --2- 172.21.15.111:0/2805125364 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb900cc550 0x7fcb9006de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.152+0000 7fcb96e64700 1 -- 172.21.15.111:0/2805125364 >> 172.21.15.111:0/2805125364 conn(0x7fcb90098ee0 msgr2=0x7fcb9015c230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:12.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.152+0000 7fcb96e64700 1 -- 172.21.15.111:0/2805125364 shutdown_connections 2023-11-28T13:06:12.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.152+0000 7fcb96e64700 1 -- 172.21.15.111:0/2805125364 wait complete. 2023-11-28T13:06:12.154 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2023-11-28T13:06:12.164 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:06:12.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.282+0000 7f97f70a9700 1 Processor -- start 2023-11-28T13:06:12.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.282+0000 7f97f70a9700 1 -- start start 2023-11-28T13:06:12.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.283+0000 7f97f70a9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f97f015a6a0 0x7f97f006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.283+0000 7f97f70a9700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f97f00af160 0x7f97f006a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.283+0000 7f97f70a9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97f00cfb30 0x7f97f00cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.283+0000 7f97f70a9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f015aa70 con 0x7f97f015a6a0 2023-11-28T13:06:12.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.283+0000 7f97f70a9700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f00ad280 con 0x7f97f00af160 2023-11-28T13:06:12.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.283+0000 7f97f70a9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f00ac950 con 0x7f97f00cfb30 2023-11-28T13:06:12.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.283+0000 7f97f4e45700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f97f015a6a0 0x7f97f006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.283+0000 7f97f4e45700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f97f015a6a0 0x7f97f006a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38460/0 (socket says 172.21.15.111:38460) 2023-11-28T13:06:12.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.283+0000 7f97f4e45700 1 -- 172.21.15.111:0/2627054279 learned_addr learned my addr 172.21.15.111:0/2627054279 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:12.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97effff700 1 --2- 172.21.15.111:0/2627054279 >> v2:172.21.15.149:3300/0 conn(0x7f97f00af160 0x7f97f006a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97f5646700 1 --2- 172.21.15.111:0/2627054279 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97f00cfb30 0x7f97f00cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97f4e45700 1 -- 172.21.15.111:0/2627054279 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97f00cfb30 msgr2=0x7f97f00cffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:12.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97f4e45700 1 --2- 172.21.15.111:0/2627054279 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97f00cfb30 0x7f97f00cffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97f4e45700 1 -- 172.21.15.111:0/2627054279 >> v2:172.21.15.149:3300/0 conn(0x7f97f00af160 msgr2=0x7f97f006a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:12.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97f4e45700 1 --2- 172.21.15.111:0/2627054279 >> v2:172.21.15.149:3300/0 conn(0x7f97f00af160 0x7f97f006a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97f4e45700 1 -- 172.21.15.111:0/2627054279 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f00acc30 con 0x7f97f015a6a0 2023-11-28T13:06:12.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97f5646700 1 --2- 172.21.15.111:0/2627054279 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97f00cfb30 0x7f97f00cffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:12.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97f4e45700 1 --2- 172.21.15.111:0/2627054279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97f015a6a0 0x7f97f006a270 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f97f00c3710 tx=0x7f97e4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1197b5c737dfb438 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:12.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97ef7fe700 1 -- 172.21.15.111:0/2627054279 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e400b870 con 0x7f97f015a6a0 2023-11-28T13:06:12.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97ef7fe700 1 -- 172.21.15.111:0/2627054279 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e400b9d0 con 0x7f97f015a6a0 2023-11-28T13:06:12.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97ef7fe700 1 -- 172.21.15.111:0/2627054279 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e400bd60 con 0x7f97f015a6a0 2023-11-28T13:06:12.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97f70a9700 1 -- 172.21.15.111:0/2627054279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97f015a6a0 msgr2=0x7f97f006a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.284+0000 7f97f70a9700 1 --2- 172.21.15.111:0/2627054279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97f015a6a0 0x7f97f006a270 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f97f00c3710 tx=0x7f97e4009bd0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.285+0000 7f97f70a9700 1 -- 172.21.15.111:0/2627054279 shutdown_connections 2023-11-28T13:06:12.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.285+0000 7f97f70a9700 1 --2- 172.21.15.111:0/2627054279 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97f00cfb30 0x7f97f00cffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.285+0000 7f97f70a9700 1 --2- 172.21.15.111:0/2627054279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97f015a6a0 0x7f97f006a270 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.285+0000 7f97f70a9700 1 --2- 172.21.15.111:0/2627054279 >> v2:172.21.15.149:3300/0 conn(0x7f97f00af160 0x7f97f006a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.285+0000 7f97f70a9700 1 -- 172.21.15.111:0/2627054279 >> 172.21.15.111:0/2627054279 conn(0x7f97f0098ee0 msgr2=0x7f97f006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:12.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.285+0000 7f97f70a9700 1 -- 172.21.15.111:0/2627054279 shutdown_connections 2023-11-28T13:06:12.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.285+0000 7f97f70a9700 1 -- 172.21.15.111:0/2627054279 wait complete. 2023-11-28T13:06:12.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.286+0000 7f97f70a9700 1 Processor -- start 2023-11-28T13:06:12.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.286+0000 7f97f70a9700 1 -- start start 2023-11-28T13:06:12.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97f70a9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97f00af160 0x7f97f0151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97f70a9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97f00cfb30 0x7f97f0151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97f70a9700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97f015a6a0 0x7f97f00c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97f70a9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f015bff0 con 0x7f97f00cfb30 2023-11-28T13:06:12.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97f70a9700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f015d5a0 con 0x7f97f015a6a0 2023-11-28T13:06:12.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97f70a9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f015b6c0 con 0x7f97f00af160 2023-11-28T13:06:12.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97f00cfb30 0x7f97f0151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97f00cfb30 0x7f97f0151730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38468/0 (socket says 172.21.15.111:38468) 2023-11-28T13:06:12.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97effff700 1 -- 172.21.15.111:0/1852834157 learned_addr learned my addr 172.21.15.111:0/1852834157 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:12.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97f5646700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97f015a6a0 0x7f97f00c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97f4e45700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97f00af160 0x7f97f0151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97effff700 1 -- 172.21.15.111:0/1852834157 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97f00af160 msgr2=0x7f97f0151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97effff700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97f00af160 0x7f97f0151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97effff700 1 -- 172.21.15.111:0/1852834157 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97f015a6a0 msgr2=0x7f97f00c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97effff700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97f015a6a0 0x7f97f00c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97effff700 1 -- 172.21.15.111:0/1852834157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e4009550 con 0x7f97f00cfb30 2023-11-28T13:06:12.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97effff700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97f00cfb30 0x7f97f0151730 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00e810 tx=0x7f97dc00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:12.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97ed7fa700 1 -- 172.21.15.111:0/1852834157 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc00c070 con 0x7f97f00cfb30 2023-11-28T13:06:12.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97ed7fa700 1 -- 172.21.15.111:0/1852834157 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc013610 con 0x7f97f00cfb30 2023-11-28T13:06:12.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97f70a9700 1 -- 172.21.15.111:0/1852834157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f00992e0 con 0x7f97f00cfb30 2023-11-28T13:06:12.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97ed7fa700 1 -- 172.21.15.111:0/1852834157 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc0139a0 con 0x7f97f00cfb30 2023-11-28T13:06:12.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.287+0000 7f97f70a9700 1 -- 172.21.15.111:0/1852834157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f00acf10 con 0x7f97f00cfb30 2023-11-28T13:06:12.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.288+0000 7f97f70a9700 1 -- 172.21.15.111:0/1852834157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f00aec10 con 0x7f97f00cfb30 2023-11-28T13:06:12.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.289+0000 7f97ed7fa700 1 -- 172.21.15.111:0/1852834157 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f97dc023070 con 0x7f97f00cfb30 2023-11-28T13:06:12.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.289+0000 7f97ed7fa700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f97d40682f0 0x7f97d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.289+0000 7f97ed7fa700 1 -- 172.21.15.111:0/1852834157 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f97dc085a20 con 0x7f97f00cfb30 2023-11-28T13:06:12.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.289+0000 7f97f4e45700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f97d40682f0 0x7f97d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.290+0000 7f97f4e45700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f97d40682f0 0x7f97d406a7a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f97e4014040 tx=0x7f97e4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:12.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.293+0000 7f97ed7fa700 1 -- 172.21.15.111:0/1852834157 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f97dc052100 con 0x7f97f00cfb30 2023-11-28T13:06:12.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.480+0000 7f97f70a9700 1 -- 172.21.15.111:0/1852834157 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97f0067170 con 0x7f97f00cfb30 2023-11-28T13:06:12.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.482+0000 7f97ed7fa700 1 -- 172.21.15.111:0/1852834157 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+18328 (secure 0 0 0) 0x7f97dc055720 con 0x7f97f00cfb30 2023-11-28T13:06:12.483 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:12.483 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":69,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:02:42.754611+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:01:51.934388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-28T13:01:53.219817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.3","mappings":[{"from":6,"to":0}]},{"pgid":"11.20","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:06:12.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.483+0000 7f97f70a9700 1 -- 172.21.15.111:0/1852834157 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f97d40682f0 msgr2=0x7f97d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.483+0000 7f97f70a9700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f97d40682f0 0x7f97d406a7a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f97e4014040 tx=0x7f97e4008030 comp rx=0 tx=0).stop 2023-11-28T13:06:12.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.484+0000 7f97f70a9700 1 -- 172.21.15.111:0/1852834157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97f00cfb30 msgr2=0x7f97f0151730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.484+0000 7f97f70a9700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97f00cfb30 0x7f97f0151730 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00e810 tx=0x7f97dc00eb20 comp rx=0 tx=0).stop 2023-11-28T13:06:12.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.484+0000 7f97f70a9700 1 -- 172.21.15.111:0/1852834157 shutdown_connections 2023-11-28T13:06:12.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.484+0000 7f97f70a9700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f97d40682f0 0x7f97d406a7a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.484+0000 7f97f70a9700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f97f015a6a0 0x7f97f00c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.484+0000 7f97f70a9700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f97f00af160 0x7f97f0151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.484+0000 7f97f70a9700 1 --2- 172.21.15.111:0/1852834157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97f00cfb30 0x7f97f0151730 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.484+0000 7f97f70a9700 1 -- 172.21.15.111:0/1852834157 >> 172.21.15.111:0/1852834157 conn(0x7f97f0098ee0 msgr2=0x7f97f006a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:12.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.484+0000 7f97f70a9700 1 -- 172.21.15.111:0/1852834157 shutdown_connections 2023-11-28T13:06:12.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.484+0000 7f97f70a9700 1 -- 172.21.15.111:0/1852834157 wait complete. 2023-11-28T13:06:12.496 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T12:58:46.376939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-28T12:58:48.865894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T13:01:51.934388+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T13:01:53.219817+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T13:06:12.497 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T13:06:12.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.619+0000 7f6d0593f700 1 Processor -- start 2023-11-28T13:06:12.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.619+0000 7f6d0593f700 1 -- start start 2023-11-28T13:06:12.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.620+0000 7f6d0593f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6d000cd460 0x7f6d0006c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.620+0000 7f6d0593f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d000b88e0 con 0x7f6d000b8510 2023-11-28T13:06:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.620+0000 7f6d0593f700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d000cd8f0 con 0x7f6d000d0d60 2023-11-28T13:06:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.620+0000 7f6d0593f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d000cdbd0 con 0x7f6d000cd460 2023-11-28T13:06:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.620+0000 7f6cfe7fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6d000cd460 0x7f6d0006c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.620+0000 7f6cfeffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f6d000b8510 0x7f6d0006b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39252/0 (socket says 172.21.15.111:39252) 2023-11-28T13:06:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.620+0000 7f6cfeffd700 1 -- 172.21.15.111:0/4140695314 learned_addr learned my addr 172.21.15.111:0/4140695314 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.620+0000 7f6cfe7fc700 1 -- 172.21.15.111:0/4140695314 >> v1:172.21.15.149:6789/0 conn(0x7f6d000d0d60 legacy=0x7f6d000d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.620+0000 7f6cfe7fc700 1 -- 172.21.15.111:0/4140695314 >> v1:172.21.15.111:6789/0 conn(0x7f6d000b8510 legacy=0x7f6d0006b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.620+0000 7f6cfdffb700 1 -- 172.21.15.111:0/4140695314 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808192775 0 0) 0x7f6d000b88e0 con 0x7f6d000b8510 2023-11-28T13:06:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.620+0000 7f6cfe7fc700 1 -- 172.21.15.111:0/4140695314 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d000cfd10 con 0x7f6d000cd460 2023-11-28T13:06:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.620+0000 7f6cfe7fc700 1 --2- 172.21.15.111:0/4140695314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d000cd460 0x7f6d0006c080 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009840 tx=0x7f6cf0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8779d39f0083df78 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.621+0000 7f6cfdffb700 1 -- 172.21.15.111:0/4140695314 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0011830 con 0x7f6d000cd460 2023-11-28T13:06:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.621+0000 7f6cfdffb700 1 -- 172.21.15.111:0/4140695314 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf0011990 con 0x7f6d000cd460 2023-11-28T13:06:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.621+0000 7f6cfdffb700 1 -- 172.21.15.111:0/4140695314 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0011d20 con 0x7f6d000cd460 2023-11-28T13:06:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.621+0000 7f6d0593f700 1 -- 172.21.15.111:0/4140695314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d000cd460 msgr2=0x7f6d0006c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.621+0000 7f6d0593f700 1 --2- 172.21.15.111:0/4140695314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d000cd460 0x7f6d0006c080 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009840 tx=0x7f6cf0009b50 comp rx=0 tx=0).stop 2023-11-28T13:06:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.621+0000 7f6d0593f700 1 -- 172.21.15.111:0/4140695314 shutdown_connections 2023-11-28T13:06:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.621+0000 7f6d0593f700 1 --2- 172.21.15.111:0/4140695314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d000cd460 0x7f6d0006c080 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.621+0000 7f6d0593f700 1 -- 172.21.15.111:0/4140695314 >> 172.21.15.111:0/4140695314 conn(0x7f6d000719a0 msgr2=0x7f6d00071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.621+0000 7f6d0593f700 1 -- 172.21.15.111:0/4140695314 shutdown_connections 2023-11-28T13:06:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.621+0000 7f6d0593f700 1 -- 172.21.15.111:0/4140695314 wait complete. 2023-11-28T13:06:12.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.622+0000 7f6d0593f700 1 Processor -- start 2023-11-28T13:06:12.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.623+0000 7f6d0593f700 1 -- start start 2023-11-28T13:06:12.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.623+0000 7f6d0593f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6d000b8510 0x7f6d000b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.623+0000 7f6d0593f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d000cd460 0x7f6d001e2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.623+0000 7f6d0593f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d000d0d60 0x7f6d0014de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.623+0000 7f6d0593f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d001e2ef0 con 0x7f6d000d0d60 2023-11-28T13:06:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.623+0000 7f6d0593f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d001e31d0 con 0x7f6d000cd460 2023-11-28T13:06:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.623+0000 7f6d0593f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d001e34b0 con 0x7f6d000b8510 2023-11-28T13:06:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.623+0000 7f6cff7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d000d0d60 0x7f6d0014de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.623+0000 7f6cff7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d000d0d60 0x7f6d0014de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38484/0 (socket says 172.21.15.111:38484) 2023-11-28T13:06:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.623+0000 7f6cff7fe700 1 -- 172.21.15.111:0/3917038074 learned_addr learned my addr 172.21.15.111:0/3917038074 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.623+0000 7f6cfeffd700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6d000b8510 0x7f6d000b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.624+0000 7f6cfe7fc700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d000cd460 0x7f6d001e2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.624+0000 7f6cff7fe700 1 -- 172.21.15.111:0/3917038074 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6d000b8510 msgr2=0x7f6d000b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.624+0000 7f6cff7fe700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6d000b8510 0x7f6d000b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.624+0000 7f6cff7fe700 1 -- 172.21.15.111:0/3917038074 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d000cd460 msgr2=0x7f6d001e2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.624+0000 7f6cff7fe700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d000cd460 0x7f6d001e2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.624+0000 7f6cff7fe700 1 -- 172.21.15.111:0/3917038074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf00094a0 con 0x7f6d000d0d60 2023-11-28T13:06:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.624+0000 7f6cff7fe700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d000d0d60 0x7f6d0014de50 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800cb50 tx=0x7f6cf800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.624+0000 7f6ce7fff700 1 -- 172.21.15.111:0/3917038074 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8003c00 con 0x7f6d000d0d60 2023-11-28T13:06:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.624+0000 7f6ce7fff700 1 -- 172.21.15.111:0/3917038074 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf8003d60 con 0x7f6d000d0d60 2023-11-28T13:06:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.624+0000 7f6d0593f700 1 -- 172.21.15.111:0/3917038074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d000d29c0 con 0x7f6d000d0d60 2023-11-28T13:06:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.624+0000 7f6ce7fff700 1 -- 172.21.15.111:0/3917038074 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf800aa20 con 0x7f6d000d0d60 2023-11-28T13:06:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.624+0000 7f6d0593f700 1 -- 172.21.15.111:0/3917038074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d000cfff0 con 0x7f6d000d0d60 2023-11-28T13:06:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.625+0000 7f6d0593f700 1 -- 172.21.15.111:0/3917038074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d000d2b50 con 0x7f6d000d0d60 2023-11-28T13:06:12.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.626+0000 7f6ce7fff700 1 -- 172.21.15.111:0/3917038074 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6cf8010050 con 0x7f6d000d0d60 2023-11-28T13:06:12.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.626+0000 7f6ce7fff700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6cec0682f0 0x7f6cec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.626+0000 7f6ce7fff700 1 -- 172.21.15.111:0/3917038074 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f6cf8085d50 con 0x7f6d000d0d60 2023-11-28T13:06:12.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.626+0000 7f6cfeffd700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6cec0682f0 0x7f6cec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.626+0000 7f6cfeffd700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6cec0682f0 0x7f6cec06a7a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4007b10 tx=0x7f6cf4007a60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:12.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.630+0000 7f6ce7fff700 1 -- 172.21.15.111:0/3917038074 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6cf80523e0 con 0x7f6d000d0d60 2023-11-28T13:06:12.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.811+0000 7f6d0593f700 1 -- 172.21.15.111:0/3917038074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6d00067880 con 0x7f6d000d0d60 2023-11-28T13:06:12.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.811+0000 7f6ce7fff700 1 -- 172.21.15.111:0/3917038074 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6cf8055a00 con 0x7f6d000d0d60 2023-11-28T13:06:12.813 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-11-28T13:06:12.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.813+0000 7f6d0593f700 1 -- 172.21.15.111:0/3917038074 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6cec0682f0 msgr2=0x7f6cec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.813+0000 7f6d0593f700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6cec0682f0 0x7f6cec06a7a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4007b10 tx=0x7f6cf4007a60 comp rx=0 tx=0).stop 2023-11-28T13:06:12.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.813+0000 7f6d0593f700 1 -- 172.21.15.111:0/3917038074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d000d0d60 msgr2=0x7f6d0014de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.813+0000 7f6d0593f700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d000d0d60 0x7f6d0014de50 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800cb50 tx=0x7f6cf800ce60 comp rx=0 tx=0).stop 2023-11-28T13:06:12.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.813+0000 7f6d0593f700 1 -- 172.21.15.111:0/3917038074 shutdown_connections 2023-11-28T13:06:12.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.813+0000 7f6d0593f700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6d000d0d60 0x7f6d0014de50 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.813+0000 7f6d0593f700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6d000b8510 0x7f6d000b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.813+0000 7f6d0593f700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6cec0682f0 0x7f6cec06a7a0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.813+0000 7f6d0593f700 1 --2- 172.21.15.111:0/3917038074 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6d000cd460 0x7f6d001e2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.813+0000 7f6d0593f700 1 -- 172.21.15.111:0/3917038074 >> 172.21.15.111:0/3917038074 conn(0x7f6d000719a0 msgr2=0x7f6d000b6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:12.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.814+0000 7f6d0593f700 1 -- 172.21.15.111:0/3917038074 shutdown_connections 2023-11-28T13:06:12.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.814+0000 7f6d0593f700 1 -- 172.21.15.111:0/3917038074 wait complete. 2023-11-28T13:06:12.827 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-11-28T13:06:12.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.951+0000 7f39ab0cf700 1 Processor -- start 2023-11-28T13:06:12.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.951+0000 7f39ab0cf700 1 -- start start 2023-11-28T13:06:12.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.951+0000 7f39ab0cf700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39a4159960 0x7f39a40b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.951+0000 7f39ab0cf700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39a40aecc0 0x7f39a40cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.951+0000 7f39ab0cf700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a40ad420 con 0x7f39a40cdbc0 2023-11-28T13:06:12.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.951+0000 7f39ab0cf700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a40ce040 con 0x7f39a4159960 2023-11-28T13:06:12.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.951+0000 7f39ab0cf700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4159d30 con 0x7f39a40aecc0 2023-11-28T13:06:12.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.952+0000 7f39a8e6b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39a4159960 0x7f39a40b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.952+0000 7f39a966c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f39a40cdbc0 0x7f39a40afa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39270/0 (socket says 172.21.15.111:39270) 2023-11-28T13:06:12.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.952+0000 7f39a966c700 1 -- 172.21.15.111:0/2411644442 learned_addr learned my addr 172.21.15.111:0/2411644442 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:12.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.952+0000 7f39a8e6b700 1 --2- 172.21.15.111:0/2411644442 >> v2:172.21.15.111:3300/0 conn(0x7f39a4159960 0x7f39a40b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38488/0 (socket says 172.21.15.111:38488) 2023-11-28T13:06:12.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.952+0000 7f39a3fff700 1 --2- 172.21.15.111:0/2411644442 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39a40aecc0 0x7f39a40cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.952+0000 7f39a8e6b700 1 -- 172.21.15.111:0/2411644442 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39a40aecc0 msgr2=0x7f39a40cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:12.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.952+0000 7f39a8e6b700 1 --2- 172.21.15.111:0/2411644442 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39a40aecc0 0x7f39a40cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.952+0000 7f39a8e6b700 1 -- 172.21.15.111:0/2411644442 >> v1:172.21.15.111:6789/0 conn(0x7f39a40cdbc0 legacy=0x7f39a40afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.952+0000 7f39a8e6b700 1 -- 172.21.15.111:0/2411644442 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a40ac810 con 0x7f39a4159960 2023-11-28T13:06:12.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.952+0000 7f39a8e6b700 1 --2- 172.21.15.111:0/2411644442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a4159960 0x7f39a40b1fd0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f3998009820 tx=0x7f3998009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2521b642fe9e60 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:12.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.952+0000 7f39a37fe700 1 -- 172.21.15.111:0/2411644442 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998005e30 con 0x7f39a4159960 2023-11-28T13:06:12.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.952+0000 7f39a37fe700 1 -- 172.21.15.111:0/2411644442 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3998006030 con 0x7f39a4159960 2023-11-28T13:06:12.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.952+0000 7f39a37fe700 1 -- 172.21.15.111:0/2411644442 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39980063e0 con 0x7f39a4159960 2023-11-28T13:06:12.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.953+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2411644442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a4159960 msgr2=0x7f39a40b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.953+0000 7f39ab0cf700 1 --2- 172.21.15.111:0/2411644442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a4159960 0x7f39a40b1fd0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f3998009820 tx=0x7f3998009b30 comp rx=0 tx=0).stop 2023-11-28T13:06:12.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.953+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2411644442 shutdown_connections 2023-11-28T13:06:12.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.953+0000 7f39ab0cf700 1 --2- 172.21.15.111:0/2411644442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a4159960 0x7f39a40b1fd0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.953+0000 7f39ab0cf700 1 --2- 172.21.15.111:0/2411644442 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39a40aecc0 0x7f39a40cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.953+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2411644442 >> 172.21.15.111:0/2411644442 conn(0x7f39a4098ee0 msgr2=0x7f39a406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:12.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.953+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2411644442 shutdown_connections 2023-11-28T13:06:12.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.953+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2411644442 wait complete. 2023-11-28T13:06:12.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.954+0000 7f39ab0cf700 1 Processor -- start 2023-11-28T13:06:12.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.954+0000 7f39ab0cf700 1 -- start start 2023-11-28T13:06:12.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39ab0cf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a40aecc0 0x7f39a415aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39ab0cf700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39a40cdbc0 0x7f39a41f5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39ab0cf700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f39a4159960 0x7f39a4152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39ab0cf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4157a90 con 0x7f39a40aecc0 2023-11-28T13:06:12.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39ab0cf700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a41f5fc0 con 0x7f39a4159960 2023-11-28T13:06:12.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39ab0cf700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a41f62a0 con 0x7f39a40cdbc0 2023-11-28T13:06:12.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39a8e6b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a40aecc0 0x7f39a415aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39a8e6b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a40aecc0 0x7f39a415aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38494/0 (socket says 172.21.15.111:38494) 2023-11-28T13:06:12.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39a8e6b700 1 -- 172.21.15.111:0/2803781671 learned_addr learned my addr 172.21.15.111:0/2803781671 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:12.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39a966c700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f39a4159960 0x7f39a4152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39a3fff700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39a40cdbc0 0x7f39a41f5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39a8e6b700 1 -- 172.21.15.111:0/2803781671 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39a40cdbc0 msgr2=0x7f39a41f5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39a8e6b700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39a40cdbc0 0x7f39a41f5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39a8e6b700 1 -- 172.21.15.111:0/2803781671 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f39a4159960 msgr2=0x7f39a4152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:12.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39a8e6b700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f39a4159960 0x7f39a4152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:12.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39a8e6b700 1 -- 172.21.15.111:0/2803781671 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3998009430 con 0x7f39a40aecc0 2023-11-28T13:06:12.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.955+0000 7f39a966c700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f39a4159960 0x7f39a4152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:12.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.956+0000 7f39a8e6b700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a40aecc0 0x7f39a415aed0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f3998009b70 tx=0x7f3998004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:12.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.956+0000 7f39a17fa700 1 -- 172.21.15.111:0/2803781671 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998004040 con 0x7f39a40aecc0 2023-11-28T13:06:12.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.956+0000 7f39a17fa700 1 -- 172.21.15.111:0/2803781671 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3998020070 con 0x7f39a40aecc0 2023-11-28T13:06:12.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.956+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2803781671 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a40acdd0 con 0x7f39a40aecc0 2023-11-28T13:06:12.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.956+0000 7f39a17fa700 1 -- 172.21.15.111:0/2803781671 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3998004400 con 0x7f39a40aecc0 2023-11-28T13:06:12.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.956+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2803781671 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a40acaf0 con 0x7f39a40aecc0 2023-11-28T13:06:12.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.957+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2803781671 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a40ad0b0 con 0x7f39a40aecc0 2023-11-28T13:06:12.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.957+0000 7f39a17fa700 1 -- 172.21.15.111:0/2803781671 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f399800f050 con 0x7f39a40aecc0 2023-11-28T13:06:12.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.957+0000 7f39a17fa700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f39880682f0 0x7f398806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:12.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.957+0000 7f39a17fa700 1 -- 172.21.15.111:0/2803781671 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f3998087600 con 0x7f39a40aecc0 2023-11-28T13:06:12.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.957+0000 7f39a3fff700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f39880682f0 0x7f398806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:12.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.958+0000 7f39a3fff700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f39880682f0 0x7f398806a7a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f39a40cf870 tx=0x7f3990008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:12.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:12.962+0000 7f39a17fa700 1 -- 172.21.15.111:0/2803781671 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3998053ce0 con 0x7f39a40aecc0 2023-11-28T13:06:13.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.145+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2803781671 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f39a406c4f0 con 0x7f39a40aecc0 2023-11-28T13:06:13.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.145+0000 7f39a17fa700 1 -- 172.21.15.111:0/2803781671 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v69) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3998057300 con 0x7f39a40aecc0 2023-11-28T13:06:13.146 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 8 2023-11-28T13:06:13.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.147+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2803781671 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f39880682f0 msgr2=0x7f398806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.147+0000 7f39ab0cf700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f39880682f0 0x7f398806a7a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f39a40cf870 tx=0x7f3990008040 comp rx=0 tx=0).stop 2023-11-28T13:06:13.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.147+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2803781671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a40aecc0 msgr2=0x7f39a415aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.147+0000 7f39ab0cf700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a40aecc0 0x7f39a415aed0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f3998009b70 tx=0x7f3998004750 comp rx=0 tx=0).stop 2023-11-28T13:06:13.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.147+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2803781671 shutdown_connections 2023-11-28T13:06:13.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.147+0000 7f39ab0cf700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f39880682f0 0x7f398806a7a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.147+0000 7f39ab0cf700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f39a4159960 0x7f39a4152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.147+0000 7f39ab0cf700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39a40aecc0 0x7f39a415aed0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.147+0000 7f39ab0cf700 1 --2- 172.21.15.111:0/2803781671 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f39a40cdbc0 0x7f39a41f5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.147+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2803781671 >> 172.21.15.111:0/2803781671 conn(0x7f39a4098ee0 msgr2=0x7f39a4156590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:13.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.147+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2803781671 shutdown_connections 2023-11-28T13:06:13.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.147+0000 7f39ab0cf700 1 -- 172.21.15.111:0/2803781671 wait complete. 2023-11-28T13:06:13.160 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T13:06:13.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.281+0000 7fc41d341700 1 Processor -- start 2023-11-28T13:06:13.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.281+0000 7fc41d341700 1 -- start start 2023-11-28T13:06:13.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.281+0000 7fc41d341700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc4180cd460 0x7fc41806c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.281+0000 7fc41d341700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc4180d0d60 0x7fc4180d1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.281+0000 7fc41d341700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4180b88e0 con 0x7fc4180b8510 2023-11-28T13:06:13.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.281+0000 7fc41d341700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4180cdbd0 con 0x7fc4180cd460 2023-11-28T13:06:13.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.281+0000 7fc41d341700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4180d02d0 con 0x7fc4180d0d60 2023-11-28T13:06:13.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.282+0000 7fc4167fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc4180cd460 0x7fc41806c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.282+0000 7fc416ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc4180b8510 0x7fc41806b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39280/0 (socket says 172.21.15.111:39280) 2023-11-28T13:06:13.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.282+0000 7fc416ffd700 1 -- 172.21.15.111:0/1768901112 learned_addr learned my addr 172.21.15.111:0/1768901112 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:13.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.282+0000 7fc4177fe700 1 --2- 172.21.15.111:0/1768901112 >> v2:172.21.15.149:3300/0 conn(0x7fc4180d0d60 0x7fc4180d1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.282+0000 7fc4167fc700 1 -- 172.21.15.111:0/1768901112 >> v2:172.21.15.149:3300/0 conn(0x7fc4180d0d60 msgr2=0x7fc4180d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:13.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.282+0000 7fc4167fc700 1 --2- 172.21.15.111:0/1768901112 >> v2:172.21.15.149:3300/0 conn(0x7fc4180d0d60 0x7fc4180d1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.282+0000 7fc4167fc700 1 -- 172.21.15.111:0/1768901112 >> v1:172.21.15.111:6789/0 conn(0x7fc4180b8510 legacy=0x7fc41806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.282+0000 7fc4167fc700 1 -- 172.21.15.111:0/1768901112 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4180cd8f0 con 0x7fc4180cd460 2023-11-28T13:06:13.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.282+0000 7fc415ffb700 1 -- 172.21.15.111:0/1768901112 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089254837 0 0) 0x7fc4180b88e0 con 0x7fc4180b8510 2023-11-28T13:06:13.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.282+0000 7fc4167fc700 1 --2- 172.21.15.111:0/1768901112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180cd460 0x7fc41806c080 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fc4180ce8c0 tx=0x7fc408009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83b5ffa2349b814 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:13.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.282+0000 7fc415ffb700 1 -- 172.21.15.111:0/1768901112 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408011830 con 0x7fc4180cd460 2023-11-28T13:06:13.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.282+0000 7fc415ffb700 1 -- 172.21.15.111:0/1768901112 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc408011990 con 0x7fc4180cd460 2023-11-28T13:06:13.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.282+0000 7fc415ffb700 1 -- 172.21.15.111:0/1768901112 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408011d20 con 0x7fc4180cd460 2023-11-28T13:06:13.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.283+0000 7fc41d341700 1 -- 172.21.15.111:0/1768901112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180cd460 msgr2=0x7fc41806c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.283+0000 7fc41d341700 1 --2- 172.21.15.111:0/1768901112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180cd460 0x7fc41806c080 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fc4180ce8c0 tx=0x7fc408009b50 comp rx=0 tx=0).stop 2023-11-28T13:06:13.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.283+0000 7fc41d341700 1 -- 172.21.15.111:0/1768901112 shutdown_connections 2023-11-28T13:06:13.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.283+0000 7fc41d341700 1 --2- 172.21.15.111:0/1768901112 >> v2:172.21.15.149:3300/0 conn(0x7fc4180d0d60 0x7fc4180d1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.283+0000 7fc41d341700 1 --2- 172.21.15.111:0/1768901112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180cd460 0x7fc41806c080 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.283+0000 7fc41d341700 1 -- 172.21.15.111:0/1768901112 >> 172.21.15.111:0/1768901112 conn(0x7fc4180719a0 msgr2=0x7fc418071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:13.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.283+0000 7fc41d341700 1 -- 172.21.15.111:0/1768901112 shutdown_connections 2023-11-28T13:06:13.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.283+0000 7fc41d341700 1 -- 172.21.15.111:0/1768901112 wait complete. 2023-11-28T13:06:13.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.284+0000 7fc41d341700 1 Processor -- start 2023-11-28T13:06:13.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.284+0000 7fc41d341700 1 -- start start 2023-11-28T13:06:13.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc41d341700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4180b8510 0x7fc41815a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc41d341700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180cd460 0x7fc4181f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc41d341700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4180d0d60 0x7fc4180cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc41d341700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41815a7d0 con 0x7fc4180cd460 2023-11-28T13:06:13.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc41d341700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4180cb8d0 con 0x7fc4180d0d60 2023-11-28T13:06:13.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc41d341700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4180cbbb0 con 0x7fc4180b8510 2023-11-28T13:06:13.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc4167fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180cd460 0x7fc4181f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc4167fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180cd460 0x7fc4181f2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38510/0 (socket says 172.21.15.111:38510) 2023-11-28T13:06:13.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc4167fc700 1 -- 172.21.15.111:0/2475408512 learned_addr learned my addr 172.21.15.111:0/2475408512 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:13.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc4177fe700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4180d0d60 0x7fc4180cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc416ffd700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4180b8510 0x7fc41815a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc4167fc700 1 -- 172.21.15.111:0/2475408512 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4180b8510 msgr2=0x7fc41815a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc4167fc700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4180b8510 0x7fc41815a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc4167fc700 1 -- 172.21.15.111:0/2475408512 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4180d0d60 msgr2=0x7fc4180cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc4167fc700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4180d0d60 0x7fc4180cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc4167fc700 1 -- 172.21.15.111:0/2475408512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4080094a0 con 0x7fc4180cd460 2023-11-28T13:06:13.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc4167fc700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180cd460 0x7fc4181f2430 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fc4080090b0 tx=0x7fc408004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:13.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.285+0000 7fc3ff7fe700 1 -- 172.21.15.111:0/2475408512 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4080038a0 con 0x7fc4180cd460 2023-11-28T13:06:13.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.286+0000 7fc3ff7fe700 1 -- 172.21.15.111:0/2475408512 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40801f070 con 0x7fc4180cd460 2023-11-28T13:06:13.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.286+0000 7fc3ff7fe700 1 -- 172.21.15.111:0/2475408512 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc408003c40 con 0x7fc4180cd460 2023-11-28T13:06:13.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.286+0000 7fc41d341700 1 -- 172.21.15.111:0/2475408512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4180cfff0 con 0x7fc4180cd460 2023-11-28T13:06:13.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.286+0000 7fc41d341700 1 -- 172.21.15.111:0/2475408512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4180cfd10 con 0x7fc4180cd460 2023-11-28T13:06:13.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.287+0000 7fc41d341700 1 -- 172.21.15.111:0/2475408512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc418067880 con 0x7fc4180cd460 2023-11-28T13:06:13.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.287+0000 7fc3ff7fe700 1 -- 172.21.15.111:0/2475408512 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc40800f050 con 0x7fc4180cd460 2023-11-28T13:06:13.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.287+0000 7fc3ff7fe700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4040682f0 0x7fc40406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.288+0000 7fc3ff7fe700 1 -- 172.21.15.111:0/2475408512 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fc408021070 con 0x7fc4180cd460 2023-11-28T13:06:13.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.288+0000 7fc416ffd700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4040682f0 0x7fc40406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.288+0000 7fc416ffd700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4040682f0 0x7fc40406a7a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc40c000f60 tx=0x7fc40c007be0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:13.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.292+0000 7fc3ff7fe700 1 -- 172.21.15.111:0/2475408512 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc408053fc0 con 0x7fc4180cd460 2023-11-28T13:06:13.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.472+0000 7fc41d341700 1 -- 172.21.15.111:0/2475408512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc4180d2940 con 0x7fc4180cd460 2023-11-28T13:06:13.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.473+0000 7fc3ff7fe700 1 -- 172.21.15.111:0/2475408512 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc40800b3e0 con 0x7fc4180cd460 2023-11-28T13:06:13.474 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-11-28T13:06:13.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.475+0000 7fc41d341700 1 -- 172.21.15.111:0/2475408512 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4040682f0 msgr2=0x7fc40406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.475+0000 7fc41d341700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4040682f0 0x7fc40406a7a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc40c000f60 tx=0x7fc40c007be0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.475+0000 7fc41d341700 1 -- 172.21.15.111:0/2475408512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180cd460 msgr2=0x7fc4181f2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.475+0000 7fc41d341700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180cd460 0x7fc4181f2430 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fc4080090b0 tx=0x7fc408004750 comp rx=0 tx=0).stop 2023-11-28T13:06:13.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.475+0000 7fc41d341700 1 -- 172.21.15.111:0/2475408512 shutdown_connections 2023-11-28T13:06:13.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.475+0000 7fc41d341700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc4040682f0 0x7fc40406a7a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.475+0000 7fc41d341700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc4180d0d60 0x7fc4180cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.475+0000 7fc41d341700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc4180b8510 0x7fc41815a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.475+0000 7fc41d341700 1 --2- 172.21.15.111:0/2475408512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180cd460 0x7fc4181f2430 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.475+0000 7fc41d341700 1 -- 172.21.15.111:0/2475408512 >> 172.21.15.111:0/2475408512 conn(0x7fc4180719a0 msgr2=0x7fc4180b3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:13.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.475+0000 7fc41d341700 1 -- 172.21.15.111:0/2475408512 shutdown_connections 2023-11-28T13:06:13.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.475+0000 7fc41d341700 1 -- 172.21.15.111:0/2475408512 wait complete. 2023-11-28T13:06:13.488 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T13:06:13.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.609+0000 7faea4c33700 1 Processor -- start 2023-11-28T13:06:13.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.610+0000 7faea4c33700 1 -- start start 2023-11-28T13:06:13.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.610+0000 7faea4c33700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faea00b0d70 0x7faea01604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.610+0000 7faea4c33700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faea00b15b0 0x7faea00cc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.610+0000 7faea4c33700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea015cd90 con 0x7faea00b1140 2023-11-28T13:06:13.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.610+0000 7faea4c33700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea00cedd0 con 0x7faea00b15b0 2023-11-28T13:06:13.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.610+0000 7faea4c33700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea01613e0 con 0x7faea00b0d70 2023-11-28T13:06:13.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.611+0000 7fae9ed9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faea00b15b0 0x7faea00cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.611+0000 7fae9ed9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faea00b15b0 0x7faea00cc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38520/0 (socket says 172.21.15.111:38520) 2023-11-28T13:06:13.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.611+0000 7fae9ed9d700 1 -- 172.21.15.111:0/3166081787 learned_addr learned my addr 172.21.15.111:0/3166081787 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:13.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.611+0000 7fae9e59c700 1 --2- 172.21.15.111:0/3166081787 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faea00b0d70 0x7faea01604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.611+0000 7fae9ed9d700 1 -- 172.21.15.111:0/3166081787 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faea00b0d70 msgr2=0x7faea01604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:13.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.611+0000 7fae9ed9d700 1 --2- 172.21.15.111:0/3166081787 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faea00b0d70 0x7faea01604b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.611+0000 7fae9ed9d700 1 -- 172.21.15.111:0/3166081787 >> v1:172.21.15.149:6789/0 conn(0x7faea00b1140 legacy=0x7faea00ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:13.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.611+0000 7fae9ed9d700 1 -- 172.21.15.111:0/3166081787 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea0160b40 con 0x7faea00b15b0 2023-11-28T13:06:13.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.611+0000 7fae9ed9d700 1 --2- 172.21.15.111:0/3166081787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea00b15b0 0x7faea00cc550 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fae98009e50 tx=0x7fae9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88336a75ae593db7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:13.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.611+0000 7fae9d59a700 1 -- 172.21.15.111:0/3166081787 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9800e070 con 0x7faea00b15b0 2023-11-28T13:06:13.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.611+0000 7fae9d59a700 1 -- 172.21.15.111:0/3166081787 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae98003a20 con 0x7faea00b15b0 2023-11-28T13:06:13.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.611+0000 7fae9d59a700 1 -- 172.21.15.111:0/3166081787 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae98003dd0 con 0x7faea00b15b0 2023-11-28T13:06:13.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.612+0000 7faea4c33700 1 -- 172.21.15.111:0/3166081787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea00b15b0 msgr2=0x7faea00cc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.612+0000 7faea4c33700 1 --2- 172.21.15.111:0/3166081787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea00b15b0 0x7faea00cc550 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fae98009e50 tx=0x7fae9800b330 comp rx=0 tx=0).stop 2023-11-28T13:06:13.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.612+0000 7faea4c33700 1 -- 172.21.15.111:0/3166081787 shutdown_connections 2023-11-28T13:06:13.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.612+0000 7faea4c33700 1 --2- 172.21.15.111:0/3166081787 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faea00b0d70 0x7faea01604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.612+0000 7faea4c33700 1 --2- 172.21.15.111:0/3166081787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea00b15b0 0x7faea00cc550 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.612+0000 7faea4c33700 1 -- 172.21.15.111:0/3166081787 >> 172.21.15.111:0/3166081787 conn(0x7faea0098ee0 msgr2=0x7faea006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:13.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.612+0000 7faea4c33700 1 -- 172.21.15.111:0/3166081787 shutdown_connections 2023-11-28T13:06:13.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.612+0000 7faea4c33700 1 -- 172.21.15.111:0/3166081787 wait complete. 2023-11-28T13:06:13.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.613+0000 7faea4c33700 1 Processor -- start 2023-11-28T13:06:13.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.613+0000 7faea4c33700 1 -- start start 2023-11-28T13:06:13.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7faea4c33700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faea00b0d70 0x7faea015b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7faea4c33700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faea00b1140 0x7faea01e7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7faea4c33700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea00b15b0 0x7faea00b5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7faea4c33700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea01e6fc0 con 0x7faea00b15b0 2023-11-28T13:06:13.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7faea4c33700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea01e63b0 con 0x7faea00b0d70 2023-11-28T13:06:13.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7faea4c33700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea01e6690 con 0x7faea00b1140 2023-11-28T13:06:13.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7fae9ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea00b15b0 0x7faea00b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7fae9ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea00b15b0 0x7faea00b5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38526/0 (socket says 172.21.15.111:38526) 2023-11-28T13:06:13.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7fae9ed9d700 1 -- 172.21.15.111:0/967018374 learned_addr learned my addr 172.21.15.111:0/967018374 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:13.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7fae9e59c700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faea00b0d70 0x7faea015b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7fae9dd9b700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faea00b1140 0x7faea01e7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7fae9ed9d700 1 -- 172.21.15.111:0/967018374 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faea00b1140 msgr2=0x7faea01e7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7fae9ed9d700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faea00b1140 0x7faea01e7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7fae9ed9d700 1 -- 172.21.15.111:0/967018374 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faea00b0d70 msgr2=0x7faea015b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7fae9ed9d700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faea00b0d70 0x7faea015b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.614+0000 7fae9ed9d700 1 -- 172.21.15.111:0/967018374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae98009b30 con 0x7faea00b15b0 2023-11-28T13:06:13.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.615+0000 7fae9ed9d700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea00b15b0 0x7faea00b5060 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fae98015040 tx=0x7fae98004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:13.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.615+0000 7fae8effd700 1 -- 172.21.15.111:0/967018374 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae98004440 con 0x7faea00b15b0 2023-11-28T13:06:13.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.615+0000 7fae8effd700 1 -- 172.21.15.111:0/967018374 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae980045c0 con 0x7faea00b15b0 2023-11-28T13:06:13.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.615+0000 7fae8effd700 1 -- 172.21.15.111:0/967018374 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae98004970 con 0x7faea00b15b0 2023-11-28T13:06:13.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.615+0000 7faea4c33700 1 -- 172.21.15.111:0/967018374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea0161100 con 0x7faea00b15b0 2023-11-28T13:06:13.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.615+0000 7faea4c33700 1 -- 172.21.15.111:0/967018374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea0160e20 con 0x7faea00b15b0 2023-11-28T13:06:13.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.616+0000 7faea4c33700 1 -- 172.21.15.111:0/967018374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea00af340 con 0x7faea00b15b0 2023-11-28T13:06:13.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.616+0000 7fae8effd700 1 -- 172.21.15.111:0/967018374 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fae98022030 con 0x7faea00b15b0 2023-11-28T13:06:13.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.616+0000 7fae8effd700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fae800682f0 0x7fae8006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.616+0000 7fae8effd700 1 -- 172.21.15.111:0/967018374 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fae98087090 con 0x7faea00b15b0 2023-11-28T13:06:13.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.616+0000 7fae9e59c700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fae800682f0 0x7fae8006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.617+0000 7fae9e59c700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fae800682f0 0x7fae8006a7a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fae9400aaa0 tx=0x7fae94008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:13.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.621+0000 7fae8effd700 1 -- 172.21.15.111:0/967018374 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae98053680 con 0x7faea00b15b0 2023-11-28T13:06:13.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.801+0000 7faea4c33700 1 -- 172.21.15.111:0/967018374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faea015cd90 con 0x7faea00b15b0 2023-11-28T13:06:13.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.802+0000 7fae8effd700 1 -- 172.21.15.111:0/967018374 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7faea015cd90 con 0x7faea00b15b0 2023-11-28T13:06:13.803 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-11-28T13:06:13.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.804+0000 7faea4c33700 1 -- 172.21.15.111:0/967018374 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fae800682f0 msgr2=0x7fae8006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.804+0000 7faea4c33700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fae800682f0 0x7fae8006a7a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fae9400aaa0 tx=0x7fae94008040 comp rx=0 tx=0).stop 2023-11-28T13:06:13.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.804+0000 7faea4c33700 1 -- 172.21.15.111:0/967018374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea00b15b0 msgr2=0x7faea00b5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.804+0000 7faea4c33700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea00b15b0 0x7faea00b5060 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fae98015040 tx=0x7fae98004ad0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.804+0000 7faea4c33700 1 -- 172.21.15.111:0/967018374 shutdown_connections 2023-11-28T13:06:13.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.804+0000 7faea4c33700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faea00b15b0 0x7faea00b5060 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.804+0000 7faea4c33700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fae800682f0 0x7fae8006a7a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.804+0000 7faea4c33700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faea00b0d70 0x7faea015b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.804+0000 7faea4c33700 1 --2- 172.21.15.111:0/967018374 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faea00b1140 0x7faea01e7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.804+0000 7faea4c33700 1 -- 172.21.15.111:0/967018374 >> 172.21.15.111:0/967018374 conn(0x7faea0098ee0 msgr2=0x7faea006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:13.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.804+0000 7faea4c33700 1 -- 172.21.15.111:0/967018374 shutdown_connections 2023-11-28T13:06:13.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.804+0000 7faea4c33700 1 -- 172.21.15.111:0/967018374 wait complete. 2023-11-28T13:06:13.817 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:06:13.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.938+0000 7f2067981700 1 Processor -- start 2023-11-28T13:06:13.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.938+0000 7f2067981700 1 -- start start 2023-11-28T13:06:13.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2067981700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f20601596e0 0x7f206015cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2067981700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f206015d510 0x7f20600cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2067981700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206015de70 con 0x7f2060159310 2023-11-28T13:06:13.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2067981700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f206015d980 con 0x7f20601596e0 2023-11-28T13:06:13.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2067981700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20600ced10 con 0x7f206015d510 2023-11-28T13:06:13.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2064f1c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f20601596e0 0x7f206015cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2065f1e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f206015d510 0x7f20600cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2064f1c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f20601596e0 0x7f206015cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:56432/0 (socket says 172.21.15.111:56432) 2023-11-28T13:06:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2064f1c700 1 -- 172.21.15.111:0/356554832 learned_addr learned my addr 172.21.15.111:0/356554832 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f206571d700 1 --1- 172.21.15.111:0/356554832 >> v1:172.21.15.149:6789/0 conn(0x7f2060159310 0x7f20600ce1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:47850/0 (socket says 172.21.15.111:47850) 2023-11-28T13:06:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2064f1c700 1 -- 172.21.15.111:0/356554832 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f206015d510 msgr2=0x7f20600cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2064f1c700 1 --2- 172.21.15.111:0/356554832 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f206015d510 0x7f20600cb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2064f1c700 1 -- 172.21.15.111:0/356554832 >> v1:172.21.15.149:6789/0 conn(0x7f2060159310 legacy=0x7f20600ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2064f1c700 1 -- 172.21.15.111:0/356554832 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20600ceff0 con 0x7f20601596e0 2023-11-28T13:06:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.939+0000 7f2057fff700 1 -- 172.21.15.111:0/356554832 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182419653 0 0) 0x7f206015de70 con 0x7f2060159310 2023-11-28T13:06:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.940+0000 7f2064f1c700 1 --2- 172.21.15.111:0/356554832 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f20601596e0 0x7f206015cfd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f20500098f0 tx=0x7f2050009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc9445b4228af66d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:13.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.940+0000 7f2057fff700 1 -- 172.21.15.111:0/356554832 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050011830 con 0x7f20601596e0 2023-11-28T13:06:13.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.940+0000 7f2057fff700 1 -- 172.21.15.111:0/356554832 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2050011990 con 0x7f20601596e0 2023-11-28T13:06:13.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.940+0000 7f2057fff700 1 -- 172.21.15.111:0/356554832 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050011d20 con 0x7f20601596e0 2023-11-28T13:06:13.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.940+0000 7f2067981700 1 -- 172.21.15.111:0/356554832 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f20601596e0 msgr2=0x7f206015cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.940+0000 7f2067981700 1 --2- 172.21.15.111:0/356554832 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f20601596e0 0x7f206015cfd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f20500098f0 tx=0x7f2050009c00 comp rx=0 tx=0).stop 2023-11-28T13:06:13.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.940+0000 7f2067981700 1 -- 172.21.15.111:0/356554832 shutdown_connections 2023-11-28T13:06:13.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.940+0000 7f2067981700 1 --2- 172.21.15.111:0/356554832 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f206015d510 0x7f20600cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.940+0000 7f2067981700 1 --2- 172.21.15.111:0/356554832 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f20601596e0 0x7f206015cfd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.940+0000 7f2067981700 1 -- 172.21.15.111:0/356554832 >> 172.21.15.111:0/356554832 conn(0x7f206006db10 msgr2=0x7f206006df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:13.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.940+0000 7f2067981700 1 -- 172.21.15.111:0/356554832 shutdown_connections 2023-11-28T13:06:13.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.940+0000 7f2067981700 1 -- 172.21.15.111:0/356554832 wait complete. 2023-11-28T13:06:13.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.942+0000 7f2067981700 1 Processor -- start 2023-11-28T13:06:13.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.942+0000 7f2067981700 1 -- start start 2023-11-28T13:06:13.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.942+0000 7f2067981700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2060159310 0x7f2060152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.942+0000 7f2067981700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f20601596e0 0x7f2060152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.942+0000 7f2067981700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f206015d510 0x7f206015a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.942+0000 7f2067981700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20601527c0 con 0x7f206015d510 2023-11-28T13:06:13.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.942+0000 7f2067981700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206015a2c0 con 0x7f2060159310 2023-11-28T13:06:13.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.942+0000 7f2067981700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206015c760 con 0x7f20601596e0 2023-11-28T13:06:13.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.942+0000 7f2065f1e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f206015d510 0x7f206015a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.942+0000 7f2065f1e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f206015d510 0x7f206015a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38540/0 (socket says 172.21.15.111:38540) 2023-11-28T13:06:13.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2065f1e700 1 -- 172.21.15.111:0/3376350998 learned_addr learned my addr 172.21.15.111:0/3376350998 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:13.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f206571d700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2060159310 0x7f2060152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2064f1c700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f20601596e0 0x7f2060152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2065f1e700 1 -- 172.21.15.111:0/3376350998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f20601596e0 msgr2=0x7f2060152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2065f1e700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f20601596e0 0x7f2060152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2065f1e700 1 -- 172.21.15.111:0/3376350998 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2060159310 msgr2=0x7f2060152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:13.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2065f1e700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2060159310 0x7f2060152930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:13.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2065f1e700 1 -- 172.21.15.111:0/3376350998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2050009550 con 0x7f206015d510 2023-11-28T13:06:13.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f206571d700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2060159310 0x7f2060152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:13.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2064f1c700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f20601596e0 0x7f2060152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:13.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2065f1e700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f206015d510 0x7f206015a750 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f205800efc0 tx=0x7f205800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:13.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2055ffb700 1 -- 172.21.15.111:0/3376350998 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205800cd40 con 0x7f206015d510 2023-11-28T13:06:13.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2055ffb700 1 -- 172.21.15.111:0/3376350998 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205800f040 con 0x7f206015d510 2023-11-28T13:06:13.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2067981700 1 -- 172.21.15.111:0/3376350998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20600cb0f0 con 0x7f206015d510 2023-11-28T13:06:13.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2055ffb700 1 -- 172.21.15.111:0/3376350998 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20580146c0 con 0x7f206015d510 2023-11-28T13:06:13.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.943+0000 7f2067981700 1 -- 172.21.15.111:0/3376350998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20600cf2d0 con 0x7f206015d510 2023-11-28T13:06:13.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.944+0000 7f2067981700 1 -- 172.21.15.111:0/3376350998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20600cb3d0 con 0x7f206015d510 2023-11-28T13:06:13.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.945+0000 7f2055ffb700 1 -- 172.21.15.111:0/3376350998 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2058014820 con 0x7f206015d510 2023-11-28T13:06:13.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.945+0000 7f2055ffb700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f20440682f0 0x7f204406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:13.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.945+0000 7f2055ffb700 1 -- 172.21.15.111:0/3376350998 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f2058086880 con 0x7f206015d510 2023-11-28T13:06:13.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.945+0000 7f206571d700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f20440682f0 0x7f204406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:13.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.945+0000 7f206571d700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f20440682f0 0x7f204406a7a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f205c000f60 tx=0x7f205c007d50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:13.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:13.949+0000 7f2055ffb700 1 -- 172.21.15.111:0/3376350998 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2058052f60 con 0x7f206015d510 2023-11-28T13:06:14.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.174+0000 7f2067981700 1 -- 172.21.15.111:0/3376350998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2060159ab0 con 0x7f206015d510 2023-11-28T13:06:14.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.174+0000 7f2055ffb700 1 -- 172.21.15.111:0/3376350998 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4659 (secure 0 0 0) 0x7f205800cea0 con 0x7f206015d510 2023-11-28T13:06:14.176 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:14.176 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/1668572397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1668572397},{"type":"v1","addr":"172.21.15.149:6834","nonce":1668572397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5755,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1129092621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1129092621},{"type":"v1","addr":"172.21.15.149:6835","nonce":1129092621}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/904588136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":904588136},{"type":"v1","addr":"172.21.15.111:6836","nonce":904588136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:01:54.257638+0000","modified":"2023-11-28T13:02:06.634463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.111:6837/1897250522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1897250522},{"type":"v1","addr":"172.21.15.111:6837","nonce":1897250522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T13:06:14.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.176+0000 7f2067981700 1 -- 172.21.15.111:0/3376350998 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f20440682f0 msgr2=0x7f204406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.176+0000 7f2067981700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f20440682f0 0x7f204406a7a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f205c000f60 tx=0x7f205c007d50 comp rx=0 tx=0).stop 2023-11-28T13:06:14.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.176+0000 7f2067981700 1 -- 172.21.15.111:0/3376350998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f206015d510 msgr2=0x7f206015a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.176+0000 7f2067981700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f206015d510 0x7f206015a750 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f205800efc0 tx=0x7f205800c3a0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.177+0000 7f2067981700 1 -- 172.21.15.111:0/3376350998 shutdown_connections 2023-11-28T13:06:14.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.177+0000 7f2067981700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f206015d510 0x7f206015a750 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.177+0000 7f2067981700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f20440682f0 0x7f204406a7a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.177+0000 7f2067981700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2060159310 0x7f2060152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.177+0000 7f2067981700 1 --2- 172.21.15.111:0/3376350998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f20601596e0 0x7f2060152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.177+0000 7f2067981700 1 -- 172.21.15.111:0/3376350998 >> 172.21.15.111:0/3376350998 conn(0x7f206006db10 msgr2=0x7f2060155eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:14.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.177+0000 7f2067981700 1 -- 172.21.15.111:0/3376350998 shutdown_connections 2023-11-28T13:06:14.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.177+0000 7f2067981700 1 -- 172.21.15.111:0/3376350998 wait complete. 2023-11-28T13:06:14.182 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2023-11-28T13:06:14.190 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:06:14.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.308+0000 7fe5ea6c2700 1 Processor -- start 2023-11-28T13:06:14.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.308+0000 7fe5ea6c2700 1 -- start start 2023-11-28T13:06:14.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.308+0000 7fe5ea6c2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe5e40ad980 0x7fe5e40d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.308+0000 7fe5ea6c2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e40b0980 0x7fe5e40b0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.308+0000 7fe5ea6c2700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e40b0210 con 0x7fe5e40b1330 2023-11-28T13:06:14.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.308+0000 7fe5ea6c2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e40ce6b0 con 0x7fe5e40ad980 2023-11-28T13:06:14.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.308+0000 7fe5ea6c2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e40b0080 con 0x7fe5e40b0980 2023-11-28T13:06:14.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.308+0000 7fe5e3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe5e40ad980 0x7fe5e40d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:14.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.308+0000 7fe5e8c5f700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe5e40b1330 0x7fe5e40add50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39326/0 (socket says 172.21.15.111:39326) 2023-11-28T13:06:14.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.308+0000 7fe5e8c5f700 1 -- 172.21.15.111:0/3223719497 learned_addr learned my addr 172.21.15.111:0/3223719497 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:14.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.309+0000 7fe5e37fe700 1 --2- 172.21.15.111:0/3223719497 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e40b0980 0x7fe5e40b0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:14.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.309+0000 7fe5e3fff700 1 -- 172.21.15.111:0/3223719497 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e40b0980 msgr2=0x7fe5e40b0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:14.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.309+0000 7fe5e3fff700 1 --2- 172.21.15.111:0/3223719497 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e40b0980 0x7fe5e40b0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.309+0000 7fe5e3fff700 1 -- 172.21.15.111:0/3223719497 >> v1:172.21.15.111:6789/0 conn(0x7fe5e40b1330 legacy=0x7fe5e40add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.309+0000 7fe5e3fff700 1 -- 172.21.15.111:0/3223719497 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e40b04f0 con 0x7fe5e40ad980 2023-11-28T13:06:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.309+0000 7fe5e2ffd700 1 -- 172.21.15.111:0/3223719497 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303324150 0 0) 0x7fe5e40b0210 con 0x7fe5e40b1330 2023-11-28T13:06:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.309+0000 7fe5e37fe700 1 --2- 172.21.15.111:0/3223719497 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e40b0980 0x7fe5e40b0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.309+0000 7fe5e3fff700 1 --2- 172.21.15.111:0/3223719497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e40ad980 0x7fe5e40d06d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8009840 tx=0x7fe5d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60bf1c37c2b03a06 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.309+0000 7fe5e2ffd700 1 -- 172.21.15.111:0/3223719497 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8011830 con 0x7fe5e40ad980 2023-11-28T13:06:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.309+0000 7fe5e2ffd700 1 -- 172.21.15.111:0/3223719497 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d8011990 con 0x7fe5e40ad980 2023-11-28T13:06:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.309+0000 7fe5e2ffd700 1 -- 172.21.15.111:0/3223719497 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8011d20 con 0x7fe5e40ad980 2023-11-28T13:06:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.310+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/3223719497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e40ad980 msgr2=0x7fe5e40d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.310+0000 7fe5ea6c2700 1 --2- 172.21.15.111:0/3223719497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e40ad980 0x7fe5e40d06d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8009840 tx=0x7fe5d8009b50 comp rx=0 tx=0).stop 2023-11-28T13:06:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.310+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/3223719497 shutdown_connections 2023-11-28T13:06:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.310+0000 7fe5ea6c2700 1 --2- 172.21.15.111:0/3223719497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e40ad980 0x7fe5e40d06d0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.310+0000 7fe5ea6c2700 1 --2- 172.21.15.111:0/3223719497 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e40b0980 0x7fe5e40b0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.310+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/3223719497 >> 172.21.15.111:0/3223719497 conn(0x7fe5e409a020 msgr2=0x7fe5e40d0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:14.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.310+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/3223719497 shutdown_connections 2023-11-28T13:06:14.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.310+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/3223719497 wait complete. 2023-11-28T13:06:14.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.311+0000 7fe5ea6c2700 1 Processor -- start 2023-11-28T13:06:14.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.311+0000 7fe5ea6c2700 1 -- start start 2023-11-28T13:06:14.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5ea6c2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e40ad980 0x7fe5e40c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5ea6c2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e40b0980 0x7fe5e414f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5ea6c2700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5e40b1330 0x7fe5e4150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5ea6c2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e41594b0 con 0x7fe5e40b0980 2023-11-28T13:06:14.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5ea6c2700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e41591d0 con 0x7fe5e40b1330 2023-11-28T13:06:14.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5ea6c2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e41e2f90 con 0x7fe5e40ad980 2023-11-28T13:06:14.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e40b0980 0x7fe5e414f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:14.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e40b0980 0x7fe5e414f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38552/0 (socket says 172.21.15.111:38552) 2023-11-28T13:06:14.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e37fe700 1 -- 172.21.15.111:0/2468954258 learned_addr learned my addr 172.21.15.111:0/2468954258 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:14.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e3fff700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e40ad980 0x7fe5e40c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e8c5f700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5e40b1330 0x7fe5e4150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e37fe700 1 -- 172.21.15.111:0/2468954258 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e40ad980 msgr2=0x7fe5e40c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e37fe700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e40ad980 0x7fe5e40c5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e37fe700 1 -- 172.21.15.111:0/2468954258 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5e40b1330 msgr2=0x7fe5e4150920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e37fe700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5e40b1330 0x7fe5e4150920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e37fe700 1 -- 172.21.15.111:0/2468954258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d80094a0 con 0x7fe5e40b0980 2023-11-28T13:06:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e3fff700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e40ad980 0x7fe5e40c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e8c5f700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5e40b1330 0x7fe5e4150920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e37fe700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e40b0980 0x7fe5e414f500 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400c750 tx=0x7fe5d400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e0ff9700 1 -- 172.21.15.111:0/2468954258 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4011800 con 0x7fe5e40b0980 2023-11-28T13:06:14.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e0ff9700 1 -- 172.21.15.111:0/2468954258 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d4011960 con 0x7fe5e40b0980 2023-11-28T13:06:14.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/2468954258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e40af940 con 0x7fe5e40b0980 2023-11-28T13:06:14.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.312+0000 7fe5e0ff9700 1 -- 172.21.15.111:0/2468954258 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4011cd0 con 0x7fe5e40b0980 2023-11-28T13:06:14.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.313+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/2468954258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e40af660 con 0x7fe5e40b0980 2023-11-28T13:06:14.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.314+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/2468954258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e40afcc0 con 0x7fe5e40b0980 2023-11-28T13:06:14.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.314+0000 7fe5e0ff9700 1 -- 172.21.15.111:0/2468954258 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe5d4020070 con 0x7fe5e40b0980 2023-11-28T13:06:14.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.314+0000 7fe5e0ff9700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5d00682f0 0x7fe5d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.314+0000 7fe5e0ff9700 1 -- 172.21.15.111:0/2468954258 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7fe5d4085cb0 con 0x7fe5e40b0980 2023-11-28T13:06:14.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.314+0000 7fe5e3fff700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5d00682f0 0x7fe5d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:14.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.315+0000 7fe5e3fff700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5d00682f0 0x7fe5d006a7a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80090b0 tx=0x7fe5d8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:14.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.319+0000 7fe5e0ff9700 1 -- 172.21.15.111:0/2468954258 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe5d4052390 con 0x7fe5e40b0980 2023-11-28T13:06:14.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.504+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/2468954258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5e4067170 con 0x7fe5e40b0980 2023-11-28T13:06:14.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.505+0000 7fe5e0ff9700 1 -- 172.21.15.111:0/2468954258 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+18328 (secure 0 0 0) 0x7fe5d401e020 con 0x7fe5e40b0980 2023-11-28T13:06:14.507 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:14.507 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":69,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:02:42.754611+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:01:51.934388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-28T13:01:53.219817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.3","mappings":[{"from":6,"to":0}]},{"pgid":"11.20","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:06:14.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.507+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/2468954258 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5d00682f0 msgr2=0x7fe5d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.507+0000 7fe5ea6c2700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5d00682f0 0x7fe5d006a7a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80090b0 tx=0x7fe5d8008030 comp rx=0 tx=0).stop 2023-11-28T13:06:14.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.507+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/2468954258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e40b0980 msgr2=0x7fe5e414f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.507+0000 7fe5ea6c2700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e40b0980 0x7fe5e414f500 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400c750 tx=0x7fe5d400cb10 comp rx=0 tx=0).stop 2023-11-28T13:06:14.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.507+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/2468954258 shutdown_connections 2023-11-28T13:06:14.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.507+0000 7fe5ea6c2700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe5d00682f0 0x7fe5d006a7a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.507+0000 7fe5ea6c2700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe5e40b1330 0x7fe5e4150920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.507+0000 7fe5ea6c2700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe5e40ad980 0x7fe5e40c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.507+0000 7fe5ea6c2700 1 --2- 172.21.15.111:0/2468954258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5e40b0980 0x7fe5e414f500 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.507+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/2468954258 >> 172.21.15.111:0/2468954258 conn(0x7fe5e409a020 msgr2=0x7fe5e41535d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:14.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.507+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/2468954258 shutdown_connections 2023-11-28T13:06:14.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.508+0000 7fe5ea6c2700 1 -- 172.21.15.111:0/2468954258 wait complete. 2023-11-28T13:06:14.521 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T13:06:14.521 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:06:14.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.639+0000 7f3a3ec4f700 1 Processor -- start 2023-11-28T13:06:14.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.640+0000 7f3a3ec4f700 1 -- start start 2023-11-28T13:06:14.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.640+0000 7f3a3ec4f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a381604b0 0x7f3a380ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.640+0000 7f3a3ec4f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3815cd90 con 0x7f3a380cc550 2023-11-28T13:06:14.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.640+0000 7f3a3ec4f700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a381613e0 con 0x7f3a380b0d70 2023-11-28T13:06:14.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.640+0000 7f3a3ec4f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38160920 con 0x7f3a381604b0 2023-11-28T13:06:14.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.640+0000 7f3a3d1ec700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3a380cc550 0x7f3a380af860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39340/0 (socket says 172.21.15.111:39340) 2023-11-28T13:06:14.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.640+0000 7f3a3d1ec700 1 -- 172.21.15.111:0/4242959314 learned_addr learned my addr 172.21.15.111:0/4242959314 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:14.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999108413 0 0) 0x7f3a30001490 con 0x7f3a380cc550 2023-11-28T13:06:14.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a37fff700 1 --2- 172.21.15.111:0/4242959314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a381604b0 0x7f3a380ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:14.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a280035b0 con 0x7f3a380cc550 2023-11-28T13:06:14.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1266236472 0 0) 0x7f3a280035b0 con 0x7f3a380cc550 2023-11-28T13:06:14.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a381604b0 msgr2=0x7f3a380ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:14.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 --2- 172.21.15.111:0/4242959314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a381604b0 0x7f3a380ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 >> v1:172.21.15.149:6789/0 conn(0x7f3a380b0d70 legacy=0x7f3a380b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a38160b40 con 0x7f3a380cc550 2023-11-28T13:06:14.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f3a30001b80 con 0x7f3a380cc550 2023-11-28T13:06:14.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 >> v1:172.21.15.111:6789/0 conn(0x7f3a380cc550 legacy=0x7f3a380af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 --2- 172.21.15.111:0/4242959314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a381604b0 0x7f3a28003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 --2- 172.21.15.111:0/4242959314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28003f70 0x7f3a28004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 --2- 172.21.15.111:0/4242959314 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a28004d90 0x7f3a28005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38160920 con 0x7f3a28003f70 2023-11-28T13:06:14.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2800ba40 con 0x7f3a28004d90 2023-11-28T13:06:14.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2800bc00 con 0x7f3a381604b0 2023-11-28T13:06:14.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a37fff700 1 --2- 172.21.15.111:0/4242959314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28003f70 0x7f3a28004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:14.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a3c9eb700 1 --2- 172.21.15.111:0/4242959314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a381604b0 0x7f3a28003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:14.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a3d1ec700 1 --2- 172.21.15.111:0/4242959314 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a28004d90 0x7f3a28005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:14.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a37fff700 1 -- 172.21.15.111:0/4242959314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a381604b0 msgr2=0x7f3a28003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:14.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a37fff700 1 --2- 172.21.15.111:0/4242959314 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a381604b0 0x7f3a28003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a37fff700 1 -- 172.21.15.111:0/4242959314 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a28004d90 msgr2=0x7f3a28005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:14.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a37fff700 1 --2- 172.21.15.111:0/4242959314 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a28004d90 0x7f3a28005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a37fff700 1 -- 172.21.15.111:0/4242959314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a280035b0 con 0x7f3a28003f70 2023-11-28T13:06:14.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.641+0000 7f3a3d1ec700 1 --2- 172.21.15.111:0/4242959314 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a28004d90 0x7f3a28005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:14.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.642+0000 7f3a37fff700 1 --2- 172.21.15.111:0/4242959314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28003f70 0x7f3a28004850 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009b60 tx=0x7f3a28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1b9804a8c4f05e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:14.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.642+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2400e6c0 con 0x7f3a28003f70 2023-11-28T13:06:14.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.642+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2400e820 con 0x7f3a28003f70 2023-11-28T13:06:14.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.642+0000 7f3a377fe700 1 -- 172.21.15.111:0/4242959314 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2400eb90 con 0x7f3a28003f70 2023-11-28T13:06:14.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.642+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/4242959314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28003f70 msgr2=0x7f3a28004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.642+0000 7f3a3ec4f700 1 --2- 172.21.15.111:0/4242959314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28003f70 0x7f3a28004850 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009b60 tx=0x7f3a28005880 comp rx=0 tx=0).stop 2023-11-28T13:06:14.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.642+0000 7f3a3d1ec700 1 -- 172.21.15.111:0/4242959314 reap_dead start 2023-11-28T13:06:14.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.642+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/4242959314 shutdown_connections 2023-11-28T13:06:14.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.642+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/4242959314 >> 172.21.15.111:0/4242959314 conn(0x7f3a38098ee0 msgr2=0x7f3a3806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:14.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.642+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/4242959314 shutdown_connections 2023-11-28T13:06:14.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.642+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/4242959314 wait complete. 2023-11-28T13:06:14.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.644+0000 7f3a3ec4f700 1 Processor -- start 2023-11-28T13:06:14.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.644+0000 7f3a3ec4f700 1 -- start start 2023-11-28T13:06:14.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.644+0000 7f3a3ec4f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28003f70 0x7f3a3815e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.644+0000 7f3a3ec4f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a380c6b80 0x7f3a380c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.644+0000 7f3a3ec4f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a38155b90 0x7f3a380ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.644+0000 7f3a3ec4f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a380c70f0 con 0x7f3a28003f70 2023-11-28T13:06:14.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.644+0000 7f3a3ec4f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a380c73d0 con 0x7f3a38155b90 2023-11-28T13:06:14.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.644+0000 7f3a3ec4f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a380c76b0 con 0x7f3a380c6b80 2023-11-28T13:06:14.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.644+0000 7f3a3c9eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28003f70 0x7f3a3815e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:14.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.644+0000 7f3a3c9eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28003f70 0x7f3a3815e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38570/0 (socket says 172.21.15.111:38570) 2023-11-28T13:06:14.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.644+0000 7f3a3c9eb700 1 -- 172.21.15.111:0/962337442 learned_addr learned my addr 172.21.15.111:0/962337442 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:14.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.645+0000 7f3a3c9eb700 1 -- 172.21.15.111:0/962337442 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a380c6b80 msgr2=0x7f3a380c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.645+0000 7f3a3c9eb700 1 --2- 172.21.15.111:0/962337442 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a380c6b80 0x7f3a380c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.645+0000 7f3a3c9eb700 1 -- 172.21.15.111:0/962337442 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a38155b90 msgr2=0x7f3a380ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.645+0000 7f3a3c9eb700 1 --2- 172.21.15.111:0/962337442 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a38155b90 0x7f3a380ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.645+0000 7f3a3c9eb700 1 -- 172.21.15.111:0/962337442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a38161100 con 0x7f3a28003f70 2023-11-28T13:06:14.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.645+0000 7f3a3c9eb700 1 --2- 172.21.15.111:0/962337442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28003f70 0x7f3a3815e9f0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00f7f0 tx=0x7f3a2c00fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:14.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.645+0000 7f3a357fa700 1 -- 172.21.15.111:0/962337442 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c0052e0 con 0x7f3a28003f70 2023-11-28T13:06:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.645+0000 7f3a357fa700 1 -- 172.21.15.111:0/962337442 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2c005460 con 0x7f3a28003f70 2023-11-28T13:06:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.645+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/962337442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a38160e20 con 0x7f3a28003f70 2023-11-28T13:06:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.645+0000 7f3a357fa700 1 -- 172.21.15.111:0/962337442 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c003910 con 0x7f3a28003f70 2023-11-28T13:06:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.645+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/962337442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a380af340 con 0x7f3a28003f70 2023-11-28T13:06:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.646+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/962337442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a38067170 con 0x7f3a28003f70 2023-11-28T13:06:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.646+0000 7f3a357fa700 1 -- 172.21.15.111:0/962337442 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3a2c003a70 con 0x7f3a28003f70 2023-11-28T13:06:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.647+0000 7f3a357fa700 1 --2- 172.21.15.111:0/962337442 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3a1c0682f0 0x7f3a1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.647+0000 7f3a357fa700 1 -- 172.21.15.111:0/962337442 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f3a2c085e90 con 0x7f3a28003f70 2023-11-28T13:06:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.647+0000 7f3a37fff700 1 --2- 172.21.15.111:0/962337442 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3a1c0682f0 0x7f3a1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.647+0000 7f3a37fff700 1 --2- 172.21.15.111:0/962337442 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3a1c0682f0 0x7f3a1c06a7a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3a3806f9b0 tx=0x7f3a24013000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:14.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.651+0000 7f3a357fa700 1 -- 172.21.15.111:0/962337442 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3a2c052570 con 0x7f3a28003f70 2023-11-28T13:06:14.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.878+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/962337442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a3815c820 con 0x7f3a28003f70 2023-11-28T13:06:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.879+0000 7f3a357fa700 1 -- 172.21.15.111:0/962337442 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4659 (secure 0 0 0) 0x7f3a2c055b90 con 0x7f3a28003f70 2023-11-28T13:06:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:14.880 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/1668572397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1668572397},{"type":"v1","addr":"172.21.15.149:6834","nonce":1668572397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5755,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1129092621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1129092621},{"type":"v1","addr":"172.21.15.149:6835","nonce":1129092621}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/904588136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":904588136},{"type":"v1","addr":"172.21.15.111:6836","nonce":904588136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:01:54.257638+0000","modified":"2023-11-28T13:02:06.634463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.111:6837/1897250522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1897250522},{"type":"v1","addr":"172.21.15.111:6837","nonce":1897250522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T13:06:14.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.880+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/962337442 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3a1c0682f0 msgr2=0x7f3a1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.880+0000 7f3a3ec4f700 1 --2- 172.21.15.111:0/962337442 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3a1c0682f0 0x7f3a1c06a7a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3a3806f9b0 tx=0x7f3a24013000 comp rx=0 tx=0).stop 2023-11-28T13:06:14.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.881+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/962337442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28003f70 msgr2=0x7f3a3815e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:14.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.881+0000 7f3a3ec4f700 1 --2- 172.21.15.111:0/962337442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28003f70 0x7f3a3815e9f0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c00f7f0 tx=0x7f3a2c00fbb0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.881+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/962337442 shutdown_connections 2023-11-28T13:06:14.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.881+0000 7f3a3ec4f700 1 --2- 172.21.15.111:0/962337442 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3a1c0682f0 0x7f3a1c06a7a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.881+0000 7f3a3ec4f700 1 --2- 172.21.15.111:0/962337442 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3a38155b90 0x7f3a380ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.881+0000 7f3a3ec4f700 1 --2- 172.21.15.111:0/962337442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a28003f70 0x7f3a3815e9f0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.881+0000 7f3a3ec4f700 1 --2- 172.21.15.111:0/962337442 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3a380c6b80 0x7f3a380c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:14.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.881+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/962337442 >> 172.21.15.111:0/962337442 conn(0x7f3a38098ee0 msgr2=0x7f3a3806e450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:14.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.881+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/962337442 shutdown_connections 2023-11-28T13:06:14.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:14.881+0000 7f3a3ec4f700 1 -- 172.21.15.111:0/962337442 wait complete. 2023-11-28T13:06:14.884 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2023-11-28T13:06:14.894 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:06:15.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.013+0000 7f3d7b747700 1 Processor -- start 2023-11-28T13:06:15.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.013+0000 7f3d7b747700 1 -- start start 2023-11-28T13:06:15.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.014+0000 7f3d7b747700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f3d740af160 0x7f3d740ade20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.014+0000 7f3d7b747700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7415aa70 con 0x7f3d7406a980 2023-11-28T13:06:15.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.014+0000 7f3d7b747700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d740ad280 con 0x7f3d7415a6a0 2023-11-28T13:06:15.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.014+0000 7f3d7b747700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d740ac950 con 0x7f3d740af160 2023-11-28T13:06:15.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.014+0000 7f3d78ce2700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3d7406a980 0x7f3d740ad710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39356/0 (socket says 172.21.15.111:39356) 2023-11-28T13:06:15.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.014+0000 7f3d78ce2700 1 -- 172.21.15.111:0/644049342 learned_addr learned my addr 172.21.15.111:0/644049342 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:15.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952667084 0 0) 0x7f3d70001110 con 0x7f3d7406a980 2023-11-28T13:06:15.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d600035b0 con 0x7f3d7406a980 2023-11-28T13:06:15.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d79ce4700 1 --2- 172.21.15.111:0/644049342 >> v2:172.21.15.149:3300/0 conn(0x7f3d740af160 0x7f3d740ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 203890925 0 0) 0x7f3d600035b0 con 0x7f3d7406a980 2023-11-28T13:06:15.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 >> v2:172.21.15.149:3300/0 conn(0x7f3d740af160 msgr2=0x7f3d740ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:15.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 --2- 172.21.15.111:0/644049342 >> v2:172.21.15.149:3300/0 conn(0x7f3d740af160 0x7f3d740ade20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 >> v1:172.21.15.149:6789/0 conn(0x7f3d7415a6a0 legacy=0x7f3d7406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d740acc30 con 0x7f3d7406a980 2023-11-28T13:06:15.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f3d70001800 con 0x7f3d7406a980 2023-11-28T13:06:15.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 >> v1:172.21.15.111:6789/0 conn(0x7f3d7406a980 legacy=0x7f3d740ad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 --2- 172.21.15.111:0/644049342 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d60003a00 0x7f3d60003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 --2- 172.21.15.111:0/644049342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d60004850 0x7f3d60004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 --2- 172.21.15.111:0/644049342 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d60005320 0x7f3d6000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740ac950 con 0x7f3d60004850 2023-11-28T13:06:15.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6000c1e0 con 0x7f3d60003a00 2023-11-28T13:06:15.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6000c3a0 con 0x7f3d60005320 2023-11-28T13:06:15.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d78ce2700 1 --2- 172.21.15.111:0/644049342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d60004850 0x7f3d60004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d794e3700 1 --2- 172.21.15.111:0/644049342 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d60003a00 0x7f3d60003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d79ce4700 1 --2- 172.21.15.111:0/644049342 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d60005320 0x7f3d6000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d78ce2700 1 -- 172.21.15.111:0/644049342 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d60005320 msgr2=0x7f3d6000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:15.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d78ce2700 1 --2- 172.21.15.111:0/644049342 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d60005320 0x7f3d6000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d78ce2700 1 -- 172.21.15.111:0/644049342 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d60003a00 msgr2=0x7f3d60003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:15.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d78ce2700 1 --2- 172.21.15.111:0/644049342 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d60003a00 0x7f3d60003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d78ce2700 1 -- 172.21.15.111:0/644049342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d600035b0 con 0x7f3d60004850 2023-11-28T13:06:15.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d79ce4700 1 -- 172.21.15.111:0/644049342 reap_dead start 2023-11-28T13:06:15.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.015+0000 7f3d78ce2700 1 --2- 172.21.15.111:0/644049342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d60004850 0x7f3d60004cc0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f3d7000c210 tx=0x7f3d60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c58aeae33073fad server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:15.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.016+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70007db0 con 0x7f3d60004850 2023-11-28T13:06:15.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.016+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7000cdd0 con 0x7f3d60004850 2023-11-28T13:06:15.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.016+0000 7f3d6bfff700 1 -- 172.21.15.111:0/644049342 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70005ba0 con 0x7f3d60004850 2023-11-28T13:06:15.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.016+0000 7f3d7b747700 1 -- 172.21.15.111:0/644049342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d60004850 msgr2=0x7f3d60004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.016+0000 7f3d7b747700 1 --2- 172.21.15.111:0/644049342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d60004850 0x7f3d60004cc0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f3d7000c210 tx=0x7f3d60005880 comp rx=0 tx=0).stop 2023-11-28T13:06:15.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.016+0000 7f3d7b747700 1 -- 172.21.15.111:0/644049342 shutdown_connections 2023-11-28T13:06:15.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.016+0000 7f3d7b747700 1 --2- 172.21.15.111:0/644049342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d60004850 0x7f3d60004cc0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.016+0000 7f3d7b747700 1 -- 172.21.15.111:0/644049342 >> 172.21.15.111:0/644049342 conn(0x7f3d74098ee0 msgr2=0x7f3d7406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:15.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.016+0000 7f3d7b747700 1 -- 172.21.15.111:0/644049342 shutdown_connections 2023-11-28T13:06:15.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.016+0000 7f3d7b747700 1 -- 172.21.15.111:0/644049342 wait complete. 2023-11-28T13:06:15.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.018+0000 7f3d7b747700 1 Processor -- start 2023-11-28T13:06:15.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.018+0000 7f3d7b747700 1 -- start start 2023-11-28T13:06:15.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d7b747700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d60004850 0x7f3d7414e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d7b747700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d740cbf30 0x7f3d740cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d7b747700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d741f3530 0x7f3d741f39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d7b747700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740cc8e0 con 0x7f3d741f3530 2023-11-28T13:06:15.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d7b747700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d741521e0 con 0x7f3d740cbf30 2023-11-28T13:06:15.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d7b747700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d741f3ee0 con 0x7f3d60004850 2023-11-28T13:06:15.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d79ce4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d741f3530 0x7f3d741f39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d79ce4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d741f3530 0x7f3d741f39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38594/0 (socket says 172.21.15.111:38594) 2023-11-28T13:06:15.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d79ce4700 1 -- 172.21.15.111:0/1655626674 learned_addr learned my addr 172.21.15.111:0/1655626674 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:15.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d78ce2700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d740cbf30 0x7f3d740cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d794e3700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d60004850 0x7f3d7414e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d79ce4700 1 -- 172.21.15.111:0/1655626674 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d60004850 msgr2=0x7f3d7414e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d79ce4700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d60004850 0x7f3d7414e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d79ce4700 1 -- 172.21.15.111:0/1655626674 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d740cbf30 msgr2=0x7f3d740cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d79ce4700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d740cbf30 0x7f3d740cc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d79ce4700 1 -- 172.21.15.111:0/1655626674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d740992e0 con 0x7f3d741f3530 2023-11-28T13:06:15.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d78ce2700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d740cbf30 0x7f3d740cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:15.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d794e3700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d60004850 0x7f3d7414e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:15.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d79ce4700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d741f3530 0x7f3d741f39a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c0045e0 tx=0x7f3d6c0049a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:15.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.019+0000 7f3d69ffb700 1 -- 172.21.15.111:0/1655626674 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c007bb0 con 0x7f3d741f3530 2023-11-28T13:06:15.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.020+0000 7f3d69ffb700 1 -- 172.21.15.111:0/1655626674 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6c00f050 con 0x7f3d741f3530 2023-11-28T13:06:15.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.020+0000 7f3d7b747700 1 -- 172.21.15.111:0/1655626674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d740acf10 con 0x7f3d741f3530 2023-11-28T13:06:15.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.020+0000 7f3d69ffb700 1 -- 172.21.15.111:0/1655626674 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c014410 con 0x7f3d741f3530 2023-11-28T13:06:15.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.020+0000 7f3d7b747700 1 -- 172.21.15.111:0/1655626674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d740aec10 con 0x7f3d741f3530 2023-11-28T13:06:15.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.021+0000 7f3d7b747700 1 -- 172.21.15.111:0/1655626674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d74067170 con 0x7f3d741f3530 2023-11-28T13:06:15.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.021+0000 7f3d69ffb700 1 -- 172.21.15.111:0/1655626674 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d6c014570 con 0x7f3d741f3530 2023-11-28T13:06:15.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.021+0000 7f3d69ffb700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d580682f0 0x7f3d5806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.022+0000 7f3d69ffb700 1 -- 172.21.15.111:0/1655626674 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f3d6c086230 con 0x7f3d741f3530 2023-11-28T13:06:15.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.022+0000 7f3d794e3700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d580682f0 0x7f3d5806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.022+0000 7f3d794e3700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d580682f0 0x7f3d5806a7a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3d64000c00 tx=0x7f3d64009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:15.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.026+0000 7f3d69ffb700 1 -- 172.21.15.111:0/1655626674 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d6c0528b0 con 0x7f3d741f3530 2023-11-28T13:06:15.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.248+0000 7f3d7b747700 1 -- 172.21.15.111:0/1655626674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d741f4110 con 0x7f3d741f3530 2023-11-28T13:06:15.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.249+0000 7f3d69ffb700 1 -- 172.21.15.111:0/1655626674 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4659 (secure 0 0 0) 0x7f3d6c055ed0 con 0x7f3d741f3530 2023-11-28T13:06:15.251 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:15.251 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/1668572397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1668572397},{"type":"v1","addr":"172.21.15.149:6834","nonce":1668572397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5755,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1129092621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1129092621},{"type":"v1","addr":"172.21.15.149:6835","nonce":1129092621}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/904588136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":904588136},{"type":"v1","addr":"172.21.15.111:6836","nonce":904588136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T13:01:54.257638+0000","modified":"2023-11-28T13:02:06.634463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.111:6837/1897250522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1897250522},{"type":"v1","addr":"172.21.15.111:6837","nonce":1897250522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-11-28T13:06:15.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.251+0000 7f3d7b747700 1 -- 172.21.15.111:0/1655626674 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d580682f0 msgr2=0x7f3d5806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.251+0000 7f3d7b747700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d580682f0 0x7f3d5806a7a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3d64000c00 tx=0x7f3d64009040 comp rx=0 tx=0).stop 2023-11-28T13:06:15.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.251+0000 7f3d7b747700 1 -- 172.21.15.111:0/1655626674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d741f3530 msgr2=0x7f3d741f39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.251+0000 7f3d7b747700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d741f3530 0x7f3d741f39a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c0045e0 tx=0x7f3d6c0049a0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.251+0000 7f3d7b747700 1 -- 172.21.15.111:0/1655626674 shutdown_connections 2023-11-28T13:06:15.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.251+0000 7f3d7b747700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d741f3530 0x7f3d741f39a0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.251+0000 7f3d7b747700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f3d60004850 0x7f3d7414e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.251+0000 7f3d7b747700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f3d580682f0 0x7f3d5806a7a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.251+0000 7f3d7b747700 1 --2- 172.21.15.111:0/1655626674 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f3d740cbf30 0x7f3d740cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.251+0000 7f3d7b747700 1 -- 172.21.15.111:0/1655626674 >> 172.21.15.111:0/1655626674 conn(0x7f3d74098ee0 msgr2=0x7f3d74159630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:15.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.251+0000 7f3d7b747700 1 -- 172.21.15.111:0/1655626674 shutdown_connections 2023-11-28T13:06:15.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.251+0000 7f3d7b747700 1 -- 172.21.15.111:0/1655626674 wait complete. 2023-11-28T13:06:15.255 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2023-11-28T13:06:15.264 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:06:15.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.382+0000 7f8b4d19f700 1 Processor -- start 2023-11-28T13:06:15.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.382+0000 7f8b4d19f700 1 -- start start 2023-11-28T13:06:15.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b4d19f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b481604b0 0x7f8b480ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b4d19f700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f8b480cc550 0x7f8b480cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b4d19f700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b4815cd90 con 0x7f8b480b0d70 2023-11-28T13:06:15.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b4d19f700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b480ccf10 con 0x7f8b480cc550 2023-11-28T13:06:15.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b4d19f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b48160920 con 0x7f8b481604b0 2023-11-28T13:06:15.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b4759e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f8b480cc550 0x7f8b480cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b4659c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b481604b0 0x7f8b480ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b46d9d700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f8b480b0d70 0x7f8b480b1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:47918/0 (socket says 172.21.15.111:47918) 2023-11-28T13:06:15.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b46d9d700 1 -- 172.21.15.111:0/56562824 learned_addr learned my addr 172.21.15.111:0/56562824 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:15.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b4659c700 1 -- 172.21.15.111:0/56562824 >> v2:172.21.15.149:3300/0 conn(0x7f8b480cc550 msgr2=0x7f8b480cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:15.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b4659c700 1 --2- 172.21.15.111:0/56562824 >> v2:172.21.15.149:3300/0 conn(0x7f8b480cc550 0x7f8b480cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b4659c700 1 -- 172.21.15.111:0/56562824 >> v1:172.21.15.149:6789/0 conn(0x7f8b480b0d70 legacy=0x7f8b480b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b4659c700 1 -- 172.21.15.111:0/56562824 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b481613e0 con 0x7f8b481604b0 2023-11-28T13:06:15.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.383+0000 7f8b4759e700 1 --2- 172.21.15.111:0/56562824 >> v2:172.21.15.149:3300/0 conn(0x7f8b480cc550 0x7f8b480cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:06:15.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.384+0000 7f8b4659c700 1 --2- 172.21.15.111:0/56562824 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b481604b0 0x7f8b480ce6c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b38009840 tx=0x7f8b38009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36576aae70dcdcfb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:15.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.384+0000 7f8b45d9b700 1 -- 172.21.15.111:0/56562824 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b38011830 con 0x7f8b481604b0 2023-11-28T13:06:15.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.384+0000 7f8b45d9b700 1 -- 172.21.15.111:0/56562824 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b38011990 con 0x7f8b481604b0 2023-11-28T13:06:15.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.384+0000 7f8b45d9b700 1 -- 172.21.15.111:0/56562824 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b38011d20 con 0x7f8b481604b0 2023-11-28T13:06:15.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.384+0000 7f8b4d19f700 1 -- 172.21.15.111:0/56562824 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b481604b0 msgr2=0x7f8b480ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.384+0000 7f8b4d19f700 1 --2- 172.21.15.111:0/56562824 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b481604b0 0x7f8b480ce6c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b38009840 tx=0x7f8b38009b50 comp rx=0 tx=0).stop 2023-11-28T13:06:15.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.384+0000 7f8b4d19f700 1 -- 172.21.15.111:0/56562824 shutdown_connections 2023-11-28T13:06:15.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.384+0000 7f8b4d19f700 1 --2- 172.21.15.111:0/56562824 >> v2:172.21.15.149:3300/0 conn(0x7f8b480cc550 0x7f8b480cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.384+0000 7f8b4d19f700 1 --2- 172.21.15.111:0/56562824 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b481604b0 0x7f8b480ce6c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.384+0000 7f8b4d19f700 1 -- 172.21.15.111:0/56562824 >> 172.21.15.111:0/56562824 conn(0x7f8b48098ee0 msgr2=0x7f8b4806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:15.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.384+0000 7f8b4d19f700 1 -- 172.21.15.111:0/56562824 shutdown_connections 2023-11-28T13:06:15.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.384+0000 7f8b4d19f700 1 -- 172.21.15.111:0/56562824 wait complete. 2023-11-28T13:06:15.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.386+0000 7f8b4d19f700 1 Processor -- start 2023-11-28T13:06:15.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.386+0000 7f8b4d19f700 1 -- start start 2023-11-28T13:06:15.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b4d19f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b480b0d70 0x7f8b480c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b4d19f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b480cc550 0x7f8b480c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b4d19f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b481604b0 0x7f8b480b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b4d19f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480b58e0 con 0x7f8b480b0d70 2023-11-28T13:06:15.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b4d19f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b481f20f0 con 0x7f8b480cc550 2023-11-28T13:06:15.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b4d19f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480c5470 con 0x7f8b481604b0 2023-11-28T13:06:15.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b46d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b480b0d70 0x7f8b480c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b46d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b480b0d70 0x7f8b480c4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38604/0 (socket says 172.21.15.111:38604) 2023-11-28T13:06:15.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b46d9d700 1 -- 172.21.15.111:0/685367315 learned_addr learned my addr 172.21.15.111:0/685367315 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:15.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b4759e700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b481604b0 0x7f8b480b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b4659c700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b480cc550 0x7f8b480c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b46d9d700 1 -- 172.21.15.111:0/685367315 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b481604b0 msgr2=0x7f8b480b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b46d9d700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b481604b0 0x7f8b480b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b46d9d700 1 -- 172.21.15.111:0/685367315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b480cc550 msgr2=0x7f8b480c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b46d9d700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b480cc550 0x7f8b480c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b46d9d700 1 -- 172.21.15.111:0/685367315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b40009a70 con 0x7f8b480b0d70 2023-11-28T13:06:15.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b46d9d700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b480b0d70 0x7f8b480c4f30 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00c520 tx=0x7f8b3c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:15.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.387+0000 7f8b2f7fe700 1 -- 172.21.15.111:0/685367315 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c005dc0 con 0x7f8b480b0d70 2023-11-28T13:06:15.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.388+0000 7f8b2f7fe700 1 -- 172.21.15.111:0/685367315 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c0070d0 con 0x7f8b480b0d70 2023-11-28T13:06:15.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.388+0000 7f8b4d19f700 1 -- 172.21.15.111:0/685367315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b380094a0 con 0x7f8b480b0d70 2023-11-28T13:06:15.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.388+0000 7f8b2f7fe700 1 -- 172.21.15.111:0/685367315 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c007460 con 0x7f8b480b0d70 2023-11-28T13:06:15.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.388+0000 7f8b4d19f700 1 -- 172.21.15.111:0/685367315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b48160e20 con 0x7f8b480b0d70 2023-11-28T13:06:15.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.389+0000 7f8b4d19f700 1 -- 172.21.15.111:0/685367315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b48160b40 con 0x7f8b480b0d70 2023-11-28T13:06:15.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.389+0000 7f8b2f7fe700 1 -- 172.21.15.111:0/685367315 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8b3c019030 con 0x7f8b480b0d70 2023-11-28T13:06:15.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.389+0000 7f8b2f7fe700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8b340682f0 0x7f8b3406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.389+0000 7f8b2f7fe700 1 -- 172.21.15.111:0/685367315 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f8b3c084f20 con 0x7f8b480b0d70 2023-11-28T13:06:15.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.389+0000 7f8b4659c700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8b340682f0 0x7f8b3406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.390+0000 7f8b4659c700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8b340682f0 0x7f8b3406a7a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8b38000c00 tx=0x7f8b3801c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:15.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.394+0000 7f8b2f7fe700 1 -- 172.21.15.111:0/685367315 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b3c051600 con 0x7f8b480b0d70 2023-11-28T13:06:15.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.576+0000 7f8b4d19f700 1 -- 172.21.15.111:0/685367315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b48161100 con 0x7f8b480b0d70 2023-11-28T13:06:15.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.577+0000 7f8b2f7fe700 1 -- 172.21.15.111:0/685367315 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+18328 (secure 0 0 0) 0x7f8b3c017090 con 0x7f8b480b0d70 2023-11-28T13:06:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:15.579 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":69,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:02:42.754611+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-11-28T13:01:51.934388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-11-28T13:01:53.219817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.3","mappings":[{"from":6,"to":0}]},{"pgid":"11.20","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:06:15.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.579+0000 7f8b4d19f700 1 -- 172.21.15.111:0/685367315 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8b340682f0 msgr2=0x7f8b3406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.579+0000 7f8b4d19f700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8b340682f0 0x7f8b3406a7a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8b38000c00 tx=0x7f8b3801c040 comp rx=0 tx=0).stop 2023-11-28T13:06:15.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.579+0000 7f8b4d19f700 1 -- 172.21.15.111:0/685367315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b480b0d70 msgr2=0x7f8b480c4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.579+0000 7f8b4d19f700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b480b0d70 0x7f8b480c4f30 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00c520 tx=0x7f8b3c00c830 comp rx=0 tx=0).stop 2023-11-28T13:06:15.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.579+0000 7f8b4d19f700 1 -- 172.21.15.111:0/685367315 shutdown_connections 2023-11-28T13:06:15.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.579+0000 7f8b4d19f700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8b481604b0 0x7f8b480b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.579+0000 7f8b4d19f700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b480b0d70 0x7f8b480c4f30 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.579+0000 7f8b4d19f700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8b340682f0 0x7f8b3406a7a0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.579+0000 7f8b4d19f700 1 --2- 172.21.15.111:0/685367315 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8b480cc550 0x7f8b480c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.579+0000 7f8b4d19f700 1 -- 172.21.15.111:0/685367315 >> 172.21.15.111:0/685367315 conn(0x7f8b48098ee0 msgr2=0x7f8b480ae680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:15.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.579+0000 7f8b4d19f700 1 -- 172.21.15.111:0/685367315 shutdown_connections 2023-11-28T13:06:15.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.579+0000 7f8b4d19f700 1 -- 172.21.15.111:0/685367315 wait complete. 2023-11-28T13:06:15.593 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T13:06:15.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.712+0000 7f721ad4e700 1 Processor -- start 2023-11-28T13:06:15.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.712+0000 7f721ad4e700 1 -- start start 2023-11-28T13:06:15.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.712+0000 7f721ad4e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f721415a6e0 0x7f721406b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.712+0000 7f721ad4e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72140ca210 0x7f72140ca680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.712+0000 7f721ad4e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72140cc330 0x7f72140cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.712+0000 7f721ad4e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721406ba40 con 0x7f72140cc330 2023-11-28T13:06:15.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.712+0000 7f721ad4e700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f721415aab0 con 0x7f721415a6e0 2023-11-28T13:06:15.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.712+0000 7f721ad4e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721406a480 con 0x7f72140ca210 2023-11-28T13:06:15.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f72192eb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72140cc330 0x7f72140cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f72192eb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72140cc330 0x7f72140cc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38616/0 (socket says 172.21.15.111:38616) 2023-11-28T13:06:15.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f72192eb700 1 -- 172.21.15.111:0/453441602 learned_addr learned my addr 172.21.15.111:0/453441602 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:15.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f7213fff700 1 --2- 172.21.15.111:0/453441602 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72140ca210 0x7f72140ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f7218aea700 1 --2- 172.21.15.111:0/453441602 >> v2:172.21.15.149:3300/0 conn(0x7f721415a6e0 0x7f721406b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f72192eb700 1 -- 172.21.15.111:0/453441602 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72140ca210 msgr2=0x7f72140ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:15.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f72192eb700 1 --2- 172.21.15.111:0/453441602 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72140ca210 0x7f72140ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f72192eb700 1 -- 172.21.15.111:0/453441602 >> v2:172.21.15.149:3300/0 conn(0x7f721415a6e0 msgr2=0x7f721406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f72192eb700 1 --2- 172.21.15.111:0/453441602 >> v2:172.21.15.149:3300/0 conn(0x7f721415a6e0 0x7f721406b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f72192eb700 1 -- 172.21.15.111:0/453441602 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721406a740 con 0x7f72140cc330 2023-11-28T13:06:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f72192eb700 1 --2- 172.21.15.111:0/453441602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72140cc330 0x7f72140cc7b0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f720c009e50 tx=0x7f720c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2e2e50a70ef01a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f72137fe700 1 -- 172.21.15.111:0/453441602 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c00e070 con 0x7f72140cc330 2023-11-28T13:06:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f72137fe700 1 -- 172.21.15.111:0/453441602 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720c00b980 con 0x7f72140cc330 2023-11-28T13:06:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.713+0000 7f72137fe700 1 -- 172.21.15.111:0/453441602 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c00bd30 con 0x7f72140cc330 2023-11-28T13:06:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.714+0000 7f721ad4e700 1 -- 172.21.15.111:0/453441602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72140cc330 msgr2=0x7f72140cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.714+0000 7f721ad4e700 1 --2- 172.21.15.111:0/453441602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72140cc330 0x7f72140cc7b0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f720c009e50 tx=0x7f720c00b330 comp rx=0 tx=0).stop 2023-11-28T13:06:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.714+0000 7f721ad4e700 1 -- 172.21.15.111:0/453441602 shutdown_connections 2023-11-28T13:06:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.714+0000 7f721ad4e700 1 --2- 172.21.15.111:0/453441602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72140cc330 0x7f72140cc7b0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.714+0000 7f721ad4e700 1 --2- 172.21.15.111:0/453441602 >> v2:172.21.15.149:3300/0 conn(0x7f721415a6e0 0x7f721406b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.714+0000 7f721ad4e700 1 --2- 172.21.15.111:0/453441602 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72140ca210 0x7f72140ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.714+0000 7f721ad4e700 1 -- 172.21.15.111:0/453441602 >> 172.21.15.111:0/453441602 conn(0x7f721406db10 msgr2=0x7f721406df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:15.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.714+0000 7f721ad4e700 1 -- 172.21.15.111:0/453441602 shutdown_connections 2023-11-28T13:06:15.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.714+0000 7f721ad4e700 1 -- 172.21.15.111:0/453441602 wait complete. 2023-11-28T13:06:15.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.715+0000 7f721ad4e700 1 Processor -- start 2023-11-28T13:06:15.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.715+0000 7f721ad4e700 1 -- start start 2023-11-28T13:06:15.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f721ad4e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72140ca210 0x7f72140c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f721ad4e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72140cc330 0x7f72140b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f721ad4e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f721415a6e0 0x7f7214150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f721ad4e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72140c52d0 con 0x7f721415a6e0 2023-11-28T13:06:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f721ad4e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72140b2a90 con 0x7f72140ca210 2023-11-28T13:06:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f721ad4e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72140af600 con 0x7f72140cc330 2023-11-28T13:06:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f72192eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f721415a6e0 0x7f7214150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f72192eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f721415a6e0 0x7f7214150e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38622/0 (socket says 172.21.15.111:38622) 2023-11-28T13:06:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f72192eb700 1 -- 172.21.15.111:0/2786609081 learned_addr learned my addr 172.21.15.111:0/2786609081 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f7213fff700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72140cc330 0x7f72140b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f7218aea700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72140ca210 0x7f72140c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f72192eb700 1 -- 172.21.15.111:0/2786609081 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72140cc330 msgr2=0x7f72140b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f72192eb700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72140cc330 0x7f72140b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f72192eb700 1 -- 172.21.15.111:0/2786609081 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72140ca210 msgr2=0x7f72140c5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f72192eb700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72140ca210 0x7f72140c5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f72192eb700 1 -- 172.21.15.111:0/2786609081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720c009b30 con 0x7f721415a6e0 2023-11-28T13:06:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f7218aea700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72140ca210 0x7f72140c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f72192eb700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f721415a6e0 0x7f7214150e00 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f720c00b730 tx=0x7f720c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f72117fa700 1 -- 172.21.15.111:0/2786609081 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c004450 con 0x7f721415a6e0 2023-11-28T13:06:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f72117fa700 1 -- 172.21.15.111:0/2786609081 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720c0045b0 con 0x7f721415a6e0 2023-11-28T13:06:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f72117fa700 1 -- 172.21.15.111:0/2786609081 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c004960 con 0x7f721415a6e0 2023-11-28T13:06:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f721ad4e700 1 -- 172.21.15.111:0/2786609081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721406ad00 con 0x7f721415a6e0 2023-11-28T13:06:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.716+0000 7f721ad4e700 1 -- 172.21.15.111:0/2786609081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721406aa20 con 0x7f721415a6e0 2023-11-28T13:06:15.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.717+0000 7f721ad4e700 1 -- 172.21.15.111:0/2786609081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721406afe0 con 0x7f721415a6e0 2023-11-28T13:06:15.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.718+0000 7f72117fa700 1 -- 172.21.15.111:0/2786609081 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f720c02b070 con 0x7f721415a6e0 2023-11-28T13:06:15.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.718+0000 7f72117fa700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f71f80682f0 0x7f71f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:15.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.718+0000 7f7218aea700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f71f80682f0 0x7f71f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:15.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.718+0000 7f72117fa700 1 -- 172.21.15.111:0/2786609081 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6440+0+0 (secure 0 0 0) 0x7f720c0874e0 con 0x7f721415a6e0 2023-11-28T13:06:15.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.719+0000 7f7218aea700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f71f80682f0 0x7f71f806a7a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f720800a930 tx=0x7f7208008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:15.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.723+0000 7f72117fa700 1 -- 172.21.15.111:0/2786609081 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f720c053bc0 con 0x7f721415a6e0 2023-11-28T13:06:15.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:15.950+0000 7f721ad4e700 1 -- 172.21.15.111:0/2786609081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7214067170 con 0x7f721415a6e0 2023-11-28T13:06:16.916 INFO:tasks.ceph.mon.a.smithi111.stderr:2023-11-28T13:06:16.915+0000 7fb379352700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T13:06:16.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:16.926+0000 7f72117fa700 1 -- 172.21.15.111:0/2786609081 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7f720c0571e0 con 0x7f721415a6e0 2023-11-28T13:06:16.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:16.928+0000 7f721ad4e700 1 -- 172.21.15.111:0/2786609081 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f71f80682f0 msgr2=0x7f71f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:16.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:16.928+0000 7f721ad4e700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f71f80682f0 0x7f71f806a7a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f720800a930 tx=0x7f7208008040 comp rx=0 tx=0).stop 2023-11-28T13:06:16.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:16.928+0000 7f721ad4e700 1 -- 172.21.15.111:0/2786609081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f721415a6e0 msgr2=0x7f7214150e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:16.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:16.928+0000 7f721ad4e700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f721415a6e0 0x7f7214150e00 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f720c00b730 tx=0x7f720c004ad0 comp rx=0 tx=0).stop 2023-11-28T13:06:16.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:16.929+0000 7f721ad4e700 1 -- 172.21.15.111:0/2786609081 shutdown_connections 2023-11-28T13:06:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:16.929+0000 7f721ad4e700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f721415a6e0 0x7f7214150e00 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:16.929+0000 7f721ad4e700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f71f80682f0 0x7f71f806a7a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:16.929+0000 7f721ad4e700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f72140ca210 0x7f72140c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:16.929+0000 7f721ad4e700 1 --2- 172.21.15.111:0/2786609081 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f72140cc330 0x7f72140b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:16.929+0000 7f721ad4e700 1 -- 172.21.15.111:0/2786609081 >> 172.21.15.111:0/2786609081 conn(0x7f721406db10 msgr2=0x7f7214157280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:16.929+0000 7f721ad4e700 1 -- 172.21.15.111:0/2786609081 shutdown_connections 2023-11-28T13:06:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:16.929+0000 7f721ad4e700 1 -- 172.21.15.111:0/2786609081 wait complete. 2023-11-28T13:06:16.931 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T13:06:16.945 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T13:06:16.967 INFO:tasks.ceph.mds.c.smithi111.stdout:starting mds.c at 2023-11-28T13:06:17.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.073+0000 7f089fc8e700 1 Processor -- start 2023-11-28T13:06:17.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.073+0000 7f089fc8e700 1 -- start start 2023-11-28T13:06:17.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.074+0000 7f089fc8e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f089815a5d0 0x7f08980ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.074+0000 7f089fc8e700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f08980cb2d0 0x7f08980ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.074+0000 7f089fc8e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08980ae8b0 con 0x7f08980b0160 2023-11-28T13:06:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.074+0000 7f089fc8e700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08980cc1b0 con 0x7f08980cb2d0 2023-11-28T13:06:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.074+0000 7f089fc8e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08980cc390 con 0x7f089815a5d0 2023-11-28T13:06:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.074+0000 7f089d229700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f08980b0160 0x7f089815e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39416/0 (socket says 172.21.15.111:39416) 2023-11-28T13:06:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.074+0000 7f089d229700 1 -- 172.21.15.111:0/3784007030 learned_addr learned my addr 172.21.15.111:0/3784007030 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.074+0000 7f089e22b700 1 --2- 172.21.15.111:0/3784007030 >> v2:172.21.15.149:3300/0 conn(0x7f08980cb2d0 0x7f08980ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.074+0000 7f089da2a700 1 --2- 172.21.15.111:0/3784007030 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f089815a5d0 0x7f08980ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.074+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807879358 0 0) 0x7f08980ae8b0 con 0x7f08980b0160 2023-11-28T13:06:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.074+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08840035b0 con 0x7f08980b0160 2023-11-28T13:06:17.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3745485258 0 0) 0x7f08840035b0 con 0x7f08980b0160 2023-11-28T13:06:17.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f089815a5d0 msgr2=0x7f08980ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:17.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 --2- 172.21.15.111:0/3784007030 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f089815a5d0 0x7f08980ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:17.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 >> v2:172.21.15.149:3300/0 conn(0x7f08980cb2d0 msgr2=0x7f08980ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:17.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 --2- 172.21.15.111:0/3784007030 >> v2:172.21.15.149:3300/0 conn(0x7f08980cb2d0 0x7f08980ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:17.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08980cc670 con 0x7f08980b0160 2023-11-28T13:06:17.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f0894001670 con 0x7f08980b0160 2023-11-28T13:06:17.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 >> v1:172.21.15.111:6789/0 conn(0x7f08980b0160 legacy=0x7f089815e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:17.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 --2- 172.21.15.111:0/3784007030 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f089815a5d0 0x7f0884003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:17.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 --2- 172.21.15.111:0/3784007030 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0884003f80 0x7f0884004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:17.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 --2- 172.21.15.111:0/3784007030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0884004d90 0x7f0884005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:17.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08980cc1b0 con 0x7f0884004d90 2023-11-28T13:06:17.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08980cc390 con 0x7f0884003f80 2023-11-28T13:06:17.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088400ba10 con 0x7f089815a5d0 2023-11-28T13:06:17.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089e22b700 1 --2- 172.21.15.111:0/3784007030 >> v2:172.21.15.149:3300/0 conn(0x7f08980cb2d0 0x7f08980ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:06:17.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089e22b700 1 --2- 172.21.15.111:0/3784007030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0884004d90 0x7f0884005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:17.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089da2a700 1 --2- 172.21.15.111:0/3784007030 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f089815a5d0 0x7f0884003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:17.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089d229700 1 --2- 172.21.15.111:0/3784007030 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0884003f80 0x7f0884004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:17.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089e22b700 1 -- 172.21.15.111:0/3784007030 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f089815a5d0 msgr2=0x7f0884003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:17.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089e22b700 1 --2- 172.21.15.111:0/3784007030 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f089815a5d0 0x7f0884003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:17.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089e22b700 1 -- 172.21.15.111:0/3784007030 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0884003f80 msgr2=0x7f0884004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:17.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089e22b700 1 --2- 172.21.15.111:0/3784007030 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f0884003f80 0x7f0884004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:17.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089e22b700 1 -- 172.21.15.111:0/3784007030 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08840035b0 con 0x7f0884004d90 2023-11-28T13:06:17.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089e22b700 1 --2- 172.21.15.111:0/3784007030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0884004d90 0x7f0884005200 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f089000e320 tx=0x7f0884005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0cba3cf2fcd62c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:17.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089000ede0 con 0x7f0884004d90 2023-11-28T13:06:17.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089000f040 con 0x7f0884004d90 2023-11-28T13:06:17.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.075+0000 7f089ca28700 1 -- 172.21.15.111:0/3784007030 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890003c40 con 0x7f0884004d90 2023-11-28T13:06:17.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.076+0000 7f089fc8e700 1 -- 172.21.15.111:0/3784007030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0884004d90 msgr2=0x7f0884005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:17.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.076+0000 7f089fc8e700 1 --2- 172.21.15.111:0/3784007030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0884004d90 0x7f0884005200 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f089000e320 tx=0x7f0884005880 comp rx=0 tx=0).stop 2023-11-28T13:06:17.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.076+0000 7f089e22b700 1 -- 172.21.15.111:0/3784007030 reap_dead start 2023-11-28T13:06:17.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.076+0000 7f089fc8e700 1 -- 172.21.15.111:0/3784007030 shutdown_connections 2023-11-28T13:06:17.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.076+0000 7f089fc8e700 1 -- 172.21.15.111:0/3784007030 >> 172.21.15.111:0/3784007030 conn(0x7f089809a040 msgr2=0x7f08980d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:17.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.076+0000 7f089fc8e700 1 -- 172.21.15.111:0/3784007030 shutdown_connections 2023-11-28T13:06:17.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.076+0000 7f089fc8e700 1 -- 172.21.15.111:0/3784007030 wait complete. 2023-11-28T13:06:17.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.077+0000 7f089fc8e700 1 Processor -- start 2023-11-28T13:06:17.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.077+0000 7f089fc8e700 1 -- start start 2023-11-28T13:06:17.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089fc8e700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08980b66b0 0x7f08981e3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:17.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089fc8e700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f08981f5d70 0x7f08980b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:17.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089fc8e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08980b59e0 0x7f08981e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:17.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089fc8e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981f62e0 con 0x7f08980b59e0 2023-11-28T13:06:17.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089fc8e700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981f65c0 con 0x7f08981f5d70 2023-11-28T13:06:17.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089fc8e700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08981f68a0 con 0x7f08980b66b0 2023-11-28T13:06:17.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089e22b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08980b59e0 0x7f08981e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:17.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089e22b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08980b59e0 0x7f08981e4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38654/0 (socket says 172.21.15.111:38654) 2023-11-28T13:06:17.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089e22b700 1 -- 172.21.15.111:0/3870159614 learned_addr learned my addr 172.21.15.111:0/3870159614 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:17.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089d229700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f08981f5d70 0x7f08980b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:17.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089da2a700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08980b66b0 0x7f08981e3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:17.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089e22b700 1 -- 172.21.15.111:0/3870159614 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08980b66b0 msgr2=0x7f08981e3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:17.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089e22b700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08980b66b0 0x7f08981e3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:17.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089e22b700 1 -- 172.21.15.111:0/3870159614 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f08981f5d70 msgr2=0x7f08980b54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:17.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089e22b700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f08981f5d70 0x7f08980b54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:17.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.078+0000 7f089e22b700 1 -- 172.21.15.111:0/3870159614 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08980ccc30 con 0x7f08980b59e0 2023-11-28T13:06:17.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.079+0000 7f089e22b700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08980b59e0 0x7f08981e4b00 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f089000e2f0 tx=0x7f0890009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:17.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.079+0000 7f088e7fc700 1 -- 172.21.15.111:0/3870159614 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890004440 con 0x7f08980b59e0 2023-11-28T13:06:17.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.079+0000 7f088e7fc700 1 -- 172.21.15.111:0/3870159614 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0890022070 con 0x7f08980b59e0 2023-11-28T13:06:17.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.079+0000 7f088e7fc700 1 -- 172.21.15.111:0/3870159614 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890004800 con 0x7f08980b59e0 2023-11-28T13:06:17.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.079+0000 7f089fc8e700 1 -- 172.21.15.111:0/3870159614 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08980cc950 con 0x7f08980b59e0 2023-11-28T13:06:17.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.079+0000 7f089fc8e700 1 -- 172.21.15.111:0/3870159614 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08980ccf10 con 0x7f08980b59e0 2023-11-28T13:06:17.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.080+0000 7f089fc8e700 1 -- 172.21.15.111:0/3870159614 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0898067170 con 0x7f08980b59e0 2023-11-28T13:06:17.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.080+0000 7f088e7fc700 1 -- 172.21.15.111:0/3870159614 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f089001a020 con 0x7f08980b59e0 2023-11-28T13:06:17.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.080+0000 7f088e7fc700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f087c0682f0 0x7f087c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:17.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.081+0000 7f088e7fc700 1 -- 172.21.15.111:0/3870159614 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6526+0+0 (secure 0 0 0) 0x7f0890024070 con 0x7f08980b59e0 2023-11-28T13:06:17.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.081+0000 7f089da2a700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f087c0682f0 0x7f087c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:17.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.081+0000 7f089da2a700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f087c0682f0 0x7f087c06a7a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0888009c40 tx=0x7f08880091c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:17.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.085+0000 7f088e7fc700 1 -- 172.21.15.111:0/3870159614 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0890054170 con 0x7f08980b59e0 2023-11-28T13:06:17.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.312+0000 7f089fc8e700 1 -- 172.21.15.111:0/3870159614 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f08980b2680 con 0x7f08980b59e0 2023-11-28T13:06:17.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.962+0000 7f088e7fc700 1 -- 172.21.15.111:0/3870159614 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0890057790 con 0x7f08980b59e0 2023-11-28T13:06:17.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.965+0000 7f089fc8e700 1 -- 172.21.15.111:0/3870159614 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f087c0682f0 msgr2=0x7f087c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:17.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.965+0000 7f089fc8e700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f087c0682f0 0x7f087c06a7a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0888009c40 tx=0x7f08880091c0 comp rx=0 tx=0).stop 2023-11-28T13:06:17.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.965+0000 7f089fc8e700 1 -- 172.21.15.111:0/3870159614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08980b59e0 msgr2=0x7f08981e4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:17.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.965+0000 7f089fc8e700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08980b59e0 0x7f08981e4b00 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f089000e2f0 tx=0x7f0890009750 comp rx=0 tx=0).stop 2023-11-28T13:06:17.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.965+0000 7f089fc8e700 1 -- 172.21.15.111:0/3870159614 shutdown_connections 2023-11-28T13:06:17.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.965+0000 7f089fc8e700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08980b59e0 0x7f08981e4b00 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:17.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.965+0000 7f089fc8e700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f08980b66b0 0x7f08981e3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:17.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.965+0000 7f089fc8e700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f087c0682f0 0x7f087c06a7a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:17.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.965+0000 7f089fc8e700 1 --2- 172.21.15.111:0/3870159614 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f08981f5d70 0x7f08980b54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:17.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.965+0000 7f089fc8e700 1 -- 172.21.15.111:0/3870159614 >> 172.21.15.111:0/3870159614 conn(0x7f089809a040 msgr2=0x7f08980af9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:17.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.965+0000 7f089fc8e700 1 -- 172.21.15.111:0/3870159614 shutdown_connections 2023-11-28T13:06:17.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:17.965+0000 7f089fc8e700 1 -- 172.21.15.111:0/3870159614 wait complete. 2023-11-28T13:06:17.981 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T13:06:18.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.151+0000 7fe187f92700 1 Processor -- start 2023-11-28T13:06:18.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.151+0000 7fe187f92700 1 -- start start 2023-11-28T13:06:18.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.152+0000 7fe187f92700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe1800b0d70 0x7fe1801604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:18.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.152+0000 7fe187f92700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18015cd90 con 0x7fe1800b1140 2023-11-28T13:06:18.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.152+0000 7fe187f92700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1801613e0 con 0x7fe1800cc550 2023-11-28T13:06:18.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.152+0000 7fe187f92700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1800cedd0 con 0x7fe1800b0d70 2023-11-28T13:06:18.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.152+0000 7fe18552d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe1800b1140 0x7fe1800ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39438/0 (socket says 172.21.15.111:39438) 2023-11-28T13:06:18.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.152+0000 7fe18552d700 1 -- 172.21.15.111:0/118681438 learned_addr learned my addr 172.21.15.111:0/118681438 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:18.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595528568 0 0) 0x7fe17c001110 con 0x7fe1800b1140 2023-11-28T13:06:18.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe16c0035b0 con 0x7fe1800b1140 2023-11-28T13:06:18.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe185d2e700 1 --2- 172.21.15.111:0/118681438 >> v2:172.21.15.149:3300/0 conn(0x7fe1800b0d70 0x7fe1801604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:18.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2667070277 0 0) 0x7fe16c0035b0 con 0x7fe1800b1140 2023-11-28T13:06:18.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 >> v2:172.21.15.149:3300/0 conn(0x7fe1800b0d70 msgr2=0x7fe1801604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:18.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 --2- 172.21.15.111:0/118681438 >> v2:172.21.15.149:3300/0 conn(0x7fe1800b0d70 0x7fe1801604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:18.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 >> v1:172.21.15.149:6789/0 conn(0x7fe1800cc550 legacy=0x7fe1800af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:18.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe180160b40 con 0x7fe1800b1140 2023-11-28T13:06:18.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fe17c001800 con 0x7fe1800b1140 2023-11-28T13:06:18.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 >> v1:172.21.15.111:6789/0 conn(0x7fe1800b1140 legacy=0x7fe1800ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:18.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 --2- 172.21.15.111:0/118681438 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe16c003a00 0x7fe16c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:18.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 --2- 172.21.15.111:0/118681438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe16c004850 0x7fe16c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:18.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 --2- 172.21.15.111:0/118681438 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe16c005200 0x7fe16c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:18.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800cedd0 con 0x7fe16c004850 2023-11-28T13:06:18.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c00c190 con 0x7fe16c005200 2023-11-28T13:06:18.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c00c350 con 0x7fe16c003a00 2023-11-28T13:06:18.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe185d2e700 1 --2- 172.21.15.111:0/118681438 >> v2:172.21.15.149:3300/0 conn(0x7fe1800b0d70 0x7fe1801604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:18.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe18552d700 1 --2- 172.21.15.111:0/118681438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe16c004850 0x7fe16c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:18.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe18652f700 1 --2- 172.21.15.111:0/118681438 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe16c005200 0x7fe16c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:18.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe185d2e700 1 --2- 172.21.15.111:0/118681438 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe16c003a00 0x7fe16c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:18.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe18552d700 1 -- 172.21.15.111:0/118681438 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe16c003a00 msgr2=0x7fe16c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:18.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe18552d700 1 --2- 172.21.15.111:0/118681438 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe16c003a00 0x7fe16c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe18552d700 1 -- 172.21.15.111:0/118681438 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe16c005200 msgr2=0x7fe16c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe18552d700 1 --2- 172.21.15.111:0/118681438 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe16c005200 0x7fe16c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe18552d700 1 -- 172.21.15.111:0/118681438 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c0035b0 con 0x7fe16c004850 2023-11-28T13:06:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe18652f700 1 -- 172.21.15.111:0/118681438 reap_dead start 2023-11-28T13:06:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe185d2e700 1 --2- 172.21.15.111:0/118681438 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe16c003a00 0x7fe16c003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.153+0000 7fe18552d700 1 --2- 172.21.15.111:0/118681438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe16c004850 0x7fe16c004cc0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00c210 tx=0x7fe16c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb3cd8378271600 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.154+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c007db0 con 0x7fe16c004850 2023-11-28T13:06:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.154+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17c00cdd0 con 0x7fe16c004850 2023-11-28T13:06:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.154+0000 7fe184d2c700 1 -- 172.21.15.111:0/118681438 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c005ba0 con 0x7fe16c004850 2023-11-28T13:06:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.154+0000 7fe187f92700 1 -- 172.21.15.111:0/118681438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe16c004850 msgr2=0x7fe16c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.154+0000 7fe187f92700 1 --2- 172.21.15.111:0/118681438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe16c004850 0x7fe16c004cc0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00c210 tx=0x7fe16c005880 comp rx=0 tx=0).stop 2023-11-28T13:06:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.154+0000 7fe187f92700 1 -- 172.21.15.111:0/118681438 shutdown_connections 2023-11-28T13:06:18.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.154+0000 7fe187f92700 1 --2- 172.21.15.111:0/118681438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe16c004850 0x7fe16c004cc0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:18.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.154+0000 7fe187f92700 1 -- 172.21.15.111:0/118681438 >> 172.21.15.111:0/118681438 conn(0x7fe180098ee0 msgr2=0x7fe18006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:18.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.154+0000 7fe187f92700 1 -- 172.21.15.111:0/118681438 shutdown_connections 2023-11-28T13:06:18.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.154+0000 7fe187f92700 1 -- 172.21.15.111:0/118681438 wait complete. 2023-11-28T13:06:18.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.155+0000 7fe187f92700 1 Processor -- start 2023-11-28T13:06:18.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.155+0000 7fe187f92700 1 -- start start 2023-11-28T13:06:18.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe187f92700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe1800ad930 0x7fe18015e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:18.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe187f92700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1800add00 0x7fe1800b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:18.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe187f92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18015bef0 0x7fe18015c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:18.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe187f92700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800b3dc0 con 0x7fe18015bef0 2023-11-28T13:06:18.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe187f92700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18015c8d0 con 0x7fe1800ad930 2023-11-28T13:06:18.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe187f92700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800b5830 con 0x7fe1800add00 2023-11-28T13:06:18.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe18652f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18015bef0 0x7fe18015c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:18.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe18652f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18015bef0 0x7fe18015c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38674/0 (socket says 172.21.15.111:38674) 2023-11-28T13:06:18.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe18652f700 1 -- 172.21.15.111:0/1427921863 learned_addr learned my addr 172.21.15.111:0/1427921863 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:18.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe185d2e700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe1800ad930 0x7fe18015e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:18.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe18552d700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1800add00 0x7fe1800b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:18.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe18652f700 1 -- 172.21.15.111:0/1427921863 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1800add00 msgr2=0x7fe1800b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:18.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe18652f700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1800add00 0x7fe1800b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:18.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe18652f700 1 -- 172.21.15.111:0/1427921863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe1800ad930 msgr2=0x7fe18015e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:18.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe18652f700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe1800ad930 0x7fe18015e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:18.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe18652f700 1 -- 172.21.15.111:0/1427921863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe180161100 con 0x7fe18015bef0 2023-11-28T13:06:18.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.156+0000 7fe185d2e700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe1800ad930 0x7fe18015e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:18.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.157+0000 7fe18652f700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18015bef0 0x7fe18015c390 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fe1800b1ee0 tx=0x7fe17800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:18.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.157+0000 7fe1767fc700 1 -- 172.21.15.111:0/1427921863 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178003c10 con 0x7fe18015bef0 2023-11-28T13:06:18.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.157+0000 7fe1767fc700 1 -- 172.21.15.111:0/1427921863 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe178003d90 con 0x7fe18015bef0 2023-11-28T13:06:18.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.157+0000 7fe187f92700 1 -- 172.21.15.111:0/1427921863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe180160e20 con 0x7fe18015bef0 2023-11-28T13:06:18.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.157+0000 7fe1767fc700 1 -- 172.21.15.111:0/1427921863 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178005310 con 0x7fe18015bef0 2023-11-28T13:06:18.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.157+0000 7fe187f92700 1 -- 172.21.15.111:0/1427921863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1800af340 con 0x7fe18015bef0 2023-11-28T13:06:18.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.158+0000 7fe187f92700 1 -- 172.21.15.111:0/1427921863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe180067170 con 0x7fe18015bef0 2023-11-28T13:06:18.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.158+0000 7fe1767fc700 1 -- 172.21.15.111:0/1427921863 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe178005470 con 0x7fe18015bef0 2023-11-28T13:06:18.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.158+0000 7fe1767fc700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe1640682f0 0x7fe16406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:18.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.159+0000 7fe1767fc700 1 -- 172.21.15.111:0/1427921863 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6526+0+0 (secure 0 0 0) 0x7fe178085f50 con 0x7fe18015bef0 2023-11-28T13:06:18.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.159+0000 7fe185d2e700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe1640682f0 0x7fe16406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:18.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.159+0000 7fe185d2e700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe1640682f0 0x7fe16406a7a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe18004fdd0 tx=0x7fe170008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:18.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.163+0000 7fe1767fc700 1 -- 172.21.15.111:0/1427921863 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe1780525e0 con 0x7fe18015bef0 2023-11-28T13:06:18.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.348+0000 7fe187f92700 1 -- 172.21.15.111:0/1427921863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe18006d9a0 con 0x7fe18015bef0 2023-11-28T13:06:18.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.974+0000 7fe1767fc700 1 -- 172.21.15.111:0/1427921863 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v71) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe178055c00 con 0x7fe18015bef0 2023-11-28T13:06:18.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.977+0000 7fe187f92700 1 -- 172.21.15.111:0/1427921863 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe1640682f0 msgr2=0x7fe16406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:18.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.977+0000 7fe187f92700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe1640682f0 0x7fe16406a7a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe18004fdd0 tx=0x7fe170008040 comp rx=0 tx=0).stop 2023-11-28T13:06:18.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.977+0000 7fe187f92700 1 -- 172.21.15.111:0/1427921863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18015bef0 msgr2=0x7fe18015c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:18.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.977+0000 7fe187f92700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18015bef0 0x7fe18015c390 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fe1800b1ee0 tx=0x7fe17800ee00 comp rx=0 tx=0).stop 2023-11-28T13:06:18.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.977+0000 7fe187f92700 1 -- 172.21.15.111:0/1427921863 shutdown_connections 2023-11-28T13:06:18.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.977+0000 7fe187f92700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe18015bef0 0x7fe18015c390 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:18.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.977+0000 7fe187f92700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe1640682f0 0x7fe16406a7a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:18.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.977+0000 7fe187f92700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe1800ad930 0x7fe18015e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:18.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.977+0000 7fe187f92700 1 --2- 172.21.15.111:0/1427921863 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe1800add00 0x7fe1800b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:18.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.977+0000 7fe187f92700 1 -- 172.21.15.111:0/1427921863 >> 172.21.15.111:0/1427921863 conn(0x7fe180098ee0 msgr2=0x7fe18006dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:18.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.978+0000 7fe187f92700 1 -- 172.21.15.111:0/1427921863 shutdown_connections 2023-11-28T13:06:18.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:18.978+0000 7fe187f92700 1 -- 172.21.15.111:0/1427921863 wait complete. 2023-11-28T13:06:18.980 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-11-28T13:06:18.992 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T13:06:19.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.115+0000 7fbe361e2700 1 Processor -- start 2023-11-28T13:06:19.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.116+0000 7fbe361e2700 1 -- start start 2023-11-28T13:06:19.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.116+0000 7fbe361e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbe3015c650 0x7fbe300b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:19.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.116+0000 7fbe361e2700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe300af8a0 con 0x7fbe3006a980 2023-11-28T13:06:19.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.116+0000 7fbe361e2700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe300ca400 con 0x7fbe300afb10 2023-11-28T13:06:19.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.116+0000 7fbe361e2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe300b00d0 con 0x7fbe3015c650 2023-11-28T13:06:19.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.116+0000 7fbe2f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbe3015c650 0x7fbe300b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:19.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.116+0000 7fbe2ffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fbe3006a980 0x7fbe300cb1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39458/0 (socket says 172.21.15.111:39458) 2023-11-28T13:06:19.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.116+0000 7fbe2ffff700 1 -- 172.21.15.111:0/433615477 learned_addr learned my addr 172.21.15.111:0/433615477 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:19.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.117+0000 7fbe2f7fe700 1 -- 172.21.15.111:0/433615477 >> v1:172.21.15.149:6789/0 conn(0x7fbe300afb10 legacy=0x7fbe3006a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:19.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.117+0000 7fbe2f7fe700 1 -- 172.21.15.111:0/433615477 >> v1:172.21.15.111:6789/0 conn(0x7fbe3006a980 legacy=0x7fbe300cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:19.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.117+0000 7fbe2f7fe700 1 -- 172.21.15.111:0/433615477 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe300ca6e0 con 0x7fbe3015c650 2023-11-28T13:06:19.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.117+0000 7fbe2f7fe700 1 --2- 172.21.15.111:0/433615477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe3015c650 0x7fbe300b0c60 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009840 tx=0x7fbe24009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7994b7c641f5c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:19.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.117+0000 7fbe2e7fc700 1 -- 172.21.15.111:0/433615477 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe24004230 con 0x7fbe3015c650 2023-11-28T13:06:19.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.117+0000 7fbe2e7fc700 1 -- 172.21.15.111:0/433615477 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe24004390 con 0x7fbe3015c650 2023-11-28T13:06:19.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.117+0000 7fbe2e7fc700 1 -- 172.21.15.111:0/433615477 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe240036a0 con 0x7fbe3015c650 2023-11-28T13:06:19.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.117+0000 7fbe361e2700 1 -- 172.21.15.111:0/433615477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe3015c650 msgr2=0x7fbe300b0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:19.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.117+0000 7fbe361e2700 1 --2- 172.21.15.111:0/433615477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe3015c650 0x7fbe300b0c60 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009840 tx=0x7fbe24009b50 comp rx=0 tx=0).stop 2023-11-28T13:06:19.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.118+0000 7fbe361e2700 1 -- 172.21.15.111:0/433615477 shutdown_connections 2023-11-28T13:06:19.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.118+0000 7fbe361e2700 1 --2- 172.21.15.111:0/433615477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe3015c650 0x7fbe300b0c60 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:19.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.118+0000 7fbe361e2700 1 -- 172.21.15.111:0/433615477 >> 172.21.15.111:0/433615477 conn(0x7fbe30098ee0 msgr2=0x7fbe3006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:19.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.118+0000 7fbe361e2700 1 -- 172.21.15.111:0/433615477 shutdown_connections 2023-11-28T13:06:19.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.118+0000 7fbe361e2700 1 -- 172.21.15.111:0/433615477 wait complete. 2023-11-28T13:06:19.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.119+0000 7fbe361e2700 1 Processor -- start 2023-11-28T13:06:19.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.119+0000 7fbe361e2700 1 -- start start 2023-11-28T13:06:19.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.120+0000 7fbe361e2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbe3006a980 0x7fbe30157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:19.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.120+0000 7fbe361e2700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbe300afb10 0x7fbe300ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:19.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.120+0000 7fbe361e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe3015c650 0x7fbe300c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:19.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.120+0000 7fbe361e2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300ad870 con 0x7fbe3015c650 2023-11-28T13:06:19.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.120+0000 7fbe361e2700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300acc60 con 0x7fbe300afb10 2023-11-28T13:06:19.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.120+0000 7fbe361e2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300acf40 con 0x7fbe3006a980 2023-11-28T13:06:19.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.120+0000 7fbe2ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe3015c650 0x7fbe300c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:19.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.120+0000 7fbe2ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe3015c650 0x7fbe300c6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38686/0 (socket says 172.21.15.111:38686) 2023-11-28T13:06:19.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.120+0000 7fbe2ffff700 1 -- 172.21.15.111:0/1494922162 learned_addr learned my addr 172.21.15.111:0/1494922162 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:19.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe2effd700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbe300afb10 0x7fbe300ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:19.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe2f7fe700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbe3006a980 0x7fbe30157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:19.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe2ffff700 1 -- 172.21.15.111:0/1494922162 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbe3006a980 msgr2=0x7fbe30157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:19.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe2ffff700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbe3006a980 0x7fbe30157b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:19.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe2ffff700 1 -- 172.21.15.111:0/1494922162 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbe300afb10 msgr2=0x7fbe300ae070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:19.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe2ffff700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbe300afb10 0x7fbe300ae070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:19.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe2ffff700 1 -- 172.21.15.111:0/1494922162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe240094a0 con 0x7fbe3015c650 2023-11-28T13:06:19.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe2f7fe700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbe3006a980 0x7fbe30157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:19.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe2effd700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbe300afb10 0x7fbe300ae070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:19.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe2ffff700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe3015c650 0x7fbe300c6ce0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800ca80 tx=0x7fbe2800ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:19.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe17fff700 1 -- 172.21.15.111:0/1494922162 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2800aaa0 con 0x7fbe3015c650 2023-11-28T13:06:19.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe17fff700 1 -- 172.21.15.111:0/1494922162 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2800ac00 con 0x7fbe3015c650 2023-11-28T13:06:19.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe361e2700 1 -- 172.21.15.111:0/1494922162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe300caca0 con 0x7fbe3015c650 2023-11-28T13:06:19.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe17fff700 1 -- 172.21.15.111:0/1494922162 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe28012400 con 0x7fbe3015c650 2023-11-28T13:06:19.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.121+0000 7fbe361e2700 1 -- 172.21.15.111:0/1494922162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe300ca9c0 con 0x7fbe3015c650 2023-11-28T13:06:19.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.122+0000 7fbe361e2700 1 -- 172.21.15.111:0/1494922162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe3015ca20 con 0x7fbe3015c650 2023-11-28T13:06:19.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.122+0000 7fbe17fff700 1 -- 172.21.15.111:0/1494922162 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbe28019030 con 0x7fbe3015c650 2023-11-28T13:06:19.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.123+0000 7fbe17fff700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbe1c0682f0 0x7fbe1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:19.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.123+0000 7fbe2f7fe700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbe1c0682f0 0x7fbe1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:19.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.123+0000 7fbe17fff700 1 -- 172.21.15.111:0/1494922162 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6088+0+0 (secure 0 0 0) 0x7fbe280856d0 con 0x7fbe3015c650 2023-11-28T13:06:19.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.123+0000 7fbe2f7fe700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbe1c0682f0 0x7fbe1c06a7a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009b90 tx=0x7fbe24008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:19.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.127+0000 7fbe17fff700 1 -- 172.21.15.111:0/1494922162 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbe2804b5e0 con 0x7fbe3015c650 2023-11-28T13:06:19.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.310+0000 7fbe361e2700 1 -- 172.21.15.111:0/1494922162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbe30067170 con 0x7fbe3015c650 2023-11-28T13:06:19.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.973+0000 7fbe17fff700 1 -- 172.21.15.111:0/1494922162 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v72) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbe28051fc0 con 0x7fbe3015c650 2023-11-28T13:06:19.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.977+0000 7fbe361e2700 1 -- 172.21.15.111:0/1494922162 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbe1c0682f0 msgr2=0x7fbe1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:19.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.977+0000 7fbe361e2700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbe1c0682f0 0x7fbe1c06a7a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009b90 tx=0x7fbe24008030 comp rx=0 tx=0).stop 2023-11-28T13:06:19.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.977+0000 7fbe361e2700 1 -- 172.21.15.111:0/1494922162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe3015c650 msgr2=0x7fbe300c6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:19.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.977+0000 7fbe361e2700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe3015c650 0x7fbe300c6ce0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800ca80 tx=0x7fbe2800ce40 comp rx=0 tx=0).stop 2023-11-28T13:06:19.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.977+0000 7fbe361e2700 1 -- 172.21.15.111:0/1494922162 shutdown_connections 2023-11-28T13:06:19.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.977+0000 7fbe361e2700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe3015c650 0x7fbe300c6ce0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:19.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.977+0000 7fbe361e2700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fbe3006a980 0x7fbe30157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:19.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.977+0000 7fbe361e2700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fbe1c0682f0 0x7fbe1c06a7a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:19.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.977+0000 7fbe361e2700 1 --2- 172.21.15.111:0/1494922162 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fbe300afb10 0x7fbe300ae070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:19.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.977+0000 7fbe361e2700 1 -- 172.21.15.111:0/1494922162 >> 172.21.15.111:0/1494922162 conn(0x7fbe30098ee0 msgr2=0x7fbe300ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:19.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.978+0000 7fbe361e2700 1 -- 172.21.15.111:0/1494922162 shutdown_connections 2023-11-28T13:06:19.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:19.978+0000 7fbe361e2700 1 -- 172.21.15.111:0/1494922162 wait complete. 2023-11-28T13:06:19.980 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-11-28T13:06:19.992 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T13:06:20.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.115+0000 7fe88b74d700 1 Processor -- start 2023-11-28T13:06:20.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.116+0000 7fe88b74d700 1 -- start start 2023-11-28T13:06:20.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.116+0000 7fe88b74d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8840cef60 0x7fe8840ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.116+0000 7fe88b74d700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fe8840b0160 0x7fe8840adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.116+0000 7fe88b74d700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8840cc390 con 0x7fe88415a5d0 2023-11-28T13:06:20.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.116+0000 7fe88b74d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8840cc1b0 con 0x7fe8840cef60 2023-11-28T13:06:20.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.116+0000 7fe88b74d700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8840cc670 con 0x7fe8840b0160 2023-11-28T13:06:20.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.117+0000 7fe888ce8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8840cef60 0x7fe8840ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:20.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.117+0000 7fe888ce8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8840cef60 0x7fe8840ad8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38690/0 (socket says 172.21.15.111:38690) 2023-11-28T13:06:20.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.117+0000 7fe888ce8700 1 -- 172.21.15.111:0/521647270 learned_addr learned my addr 172.21.15.111:0/521647270 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:20.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.117+0000 7fe888ce8700 1 -- 172.21.15.111:0/521647270 >> v2:172.21.15.149:3300/0 conn(0x7fe8840b0160 msgr2=0x7fe8840adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:20.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.117+0000 7fe888ce8700 1 --2- 172.21.15.111:0/521647270 >> v2:172.21.15.149:3300/0 conn(0x7fe8840b0160 0x7fe8840adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.117+0000 7fe888ce8700 1 -- 172.21.15.111:0/521647270 >> v1:172.21.15.149:6789/0 conn(0x7fe88415a5d0 legacy=0x7fe8840ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:20.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.117+0000 7fe888ce8700 1 -- 172.21.15.111:0/521647270 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8840cc950 con 0x7fe8840cef60 2023-11-28T13:06:20.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.117+0000 7fe888ce8700 1 --2- 172.21.15.111:0/521647270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840cef60 0x7fe8840ad8a0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe88408a070 tx=0x7fe880009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb794c66c32b9844 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:20.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.117+0000 7fe87bfff700 1 -- 172.21.15.111:0/521647270 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880004230 con 0x7fe8840cef60 2023-11-28T13:06:20.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.117+0000 7fe87bfff700 1 -- 172.21.15.111:0/521647270 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe880004390 con 0x7fe8840cef60 2023-11-28T13:06:20.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.117+0000 7fe87bfff700 1 -- 172.21.15.111:0/521647270 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8800036a0 con 0x7fe8840cef60 2023-11-28T13:06:20.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.118+0000 7fe88b74d700 1 -- 172.21.15.111:0/521647270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840cef60 msgr2=0x7fe8840ad8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:20.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.118+0000 7fe88b74d700 1 --2- 172.21.15.111:0/521647270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840cef60 0x7fe8840ad8a0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe88408a070 tx=0x7fe880009bd0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.118+0000 7fe88b74d700 1 -- 172.21.15.111:0/521647270 shutdown_connections 2023-11-28T13:06:20.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.118+0000 7fe88b74d700 1 --2- 172.21.15.111:0/521647270 >> v2:172.21.15.149:3300/0 conn(0x7fe8840b0160 0x7fe8840adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.118+0000 7fe88b74d700 1 --2- 172.21.15.111:0/521647270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840cef60 0x7fe8840ad8a0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.118+0000 7fe88b74d700 1 -- 172.21.15.111:0/521647270 >> 172.21.15.111:0/521647270 conn(0x7fe88409a040 msgr2=0x7fe8840d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:20.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.118+0000 7fe88b74d700 1 -- 172.21.15.111:0/521647270 shutdown_connections 2023-11-28T13:06:20.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.118+0000 7fe88b74d700 1 -- 172.21.15.111:0/521647270 wait complete. 2023-11-28T13:06:20.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.119+0000 7fe88b74d700 1 Processor -- start 2023-11-28T13:06:20.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.119+0000 7fe88b74d700 1 -- start start 2023-11-28T13:06:20.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe88b74d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8840b0160 0x7fe8840b23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe88b74d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840cef60 0x7fe8841559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe88b74d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe88415a5d0 0x7fe884155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe88b74d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8841f15e0 con 0x7fe8840cef60 2023-11-28T13:06:20.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe88b74d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8841f09d0 con 0x7fe8840b0160 2023-11-28T13:06:20.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe88b74d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8841f0cb0 con 0x7fe88415a5d0 2023-11-28T13:06:20.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe888ce8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840cef60 0x7fe8841559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:20.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe888ce8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840cef60 0x7fe8841559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38696/0 (socket says 172.21.15.111:38696) 2023-11-28T13:06:20.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe888ce8700 1 -- 172.21.15.111:0/3645654790 learned_addr learned my addr 172.21.15.111:0/3645654790 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:20.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe888ce8700 1 -- 172.21.15.111:0/3645654790 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe88415a5d0 msgr2=0x7fe884155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:20.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe888ce8700 1 --2- 172.21.15.111:0/3645654790 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe88415a5d0 0x7fe884155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe888ce8700 1 -- 172.21.15.111:0/3645654790 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8840b0160 msgr2=0x7fe8840b23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:20.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe888ce8700 1 --2- 172.21.15.111:0/3645654790 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8840b0160 0x7fe8840b23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe888ce8700 1 -- 172.21.15.111:0/3645654790 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe880009550 con 0x7fe8840cef60 2023-11-28T13:06:20.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe888ce8700 1 --2- 172.21.15.111:0/3645654790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840cef60 0x7fe8841559a0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe880004000 tx=0x7fe880004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:20.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe879ffb700 1 -- 172.21.15.111:0/3645654790 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880003dc0 con 0x7fe8840cef60 2023-11-28T13:06:20.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe879ffb700 1 -- 172.21.15.111:0/3645654790 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88001f070 con 0x7fe8840cef60 2023-11-28T13:06:20.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe88b74d700 1 -- 172.21.15.111:0/3645654790 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8840ccf10 con 0x7fe8840cef60 2023-11-28T13:06:20.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe879ffb700 1 -- 172.21.15.111:0/3645654790 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880011870 con 0x7fe8840cef60 2023-11-28T13:06:20.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.120+0000 7fe88b74d700 1 -- 172.21.15.111:0/3645654790 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8840ccc30 con 0x7fe8840cef60 2023-11-28T13:06:20.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.121+0000 7fe88b74d700 1 -- 172.21.15.111:0/3645654790 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe88415a9a0 con 0x7fe8840cef60 2023-11-28T13:06:20.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.122+0000 7fe879ffb700 1 -- 172.21.15.111:0/3645654790 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe880017020 con 0x7fe8840cef60 2023-11-28T13:06:20.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.122+0000 7fe879ffb700 1 --2- 172.21.15.111:0/3645654790 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8680682f0 0x7fe86806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.122+0000 7fe879ffb700 1 -- 172.21.15.111:0/3645654790 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fe8800872d0 con 0x7fe8840cef60 2023-11-28T13:06:20.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.122+0000 7fe8894e9700 1 --2- 172.21.15.111:0/3645654790 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8680682f0 0x7fe86806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:20.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.123+0000 7fe8894e9700 1 --2- 172.21.15.111:0/3645654790 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8680682f0 0x7fe86806a7a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe8740096d0 tx=0x7fe874006c20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:20.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.127+0000 7fe879ffb700 1 -- 172.21.15.111:0/3645654790 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe880053cf0 con 0x7fe8840cef60 2023-11-28T13:06:20.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.311+0000 7fe88b74d700 1 -- 172.21.15.111:0/3645654790 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe8840cb0b0 con 0x7fe8840cef60 2023-11-28T13:06:20.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.315+0000 7fe879ffb700 1 -- 172.21.15.111:0/3645654790 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe880057310 con 0x7fe8840cef60 2023-11-28T13:06:20.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.317+0000 7fe88b74d700 1 -- 172.21.15.111:0/3645654790 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8680682f0 msgr2=0x7fe86806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:20.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.317+0000 7fe88b74d700 1 --2- 172.21.15.111:0/3645654790 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8680682f0 0x7fe86806a7a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe8740096d0 tx=0x7fe874006c20 comp rx=0 tx=0).stop 2023-11-28T13:06:20.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.317+0000 7fe88b74d700 1 -- 172.21.15.111:0/3645654790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840cef60 msgr2=0x7fe8841559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:20.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.317+0000 7fe88b74d700 1 --2- 172.21.15.111:0/3645654790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840cef60 0x7fe8841559a0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe880004000 tx=0x7fe880004750 comp rx=0 tx=0).stop 2023-11-28T13:06:20.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.317+0000 7fe88b74d700 1 -- 172.21.15.111:0/3645654790 shutdown_connections 2023-11-28T13:06:20.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.317+0000 7fe88b74d700 1 --2- 172.21.15.111:0/3645654790 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe88415a5d0 0x7fe884155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.317+0000 7fe88b74d700 1 --2- 172.21.15.111:0/3645654790 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe8680682f0 0x7fe86806a7a0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.317+0000 7fe88b74d700 1 --2- 172.21.15.111:0/3645654790 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe8840b0160 0x7fe8840b23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.317+0000 7fe88b74d700 1 --2- 172.21.15.111:0/3645654790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8840cef60 0x7fe8841559a0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.317+0000 7fe88b74d700 1 -- 172.21.15.111:0/3645654790 >> 172.21.15.111:0/3645654790 conn(0x7fe88409a040 msgr2=0x7fe8840ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:20.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.317+0000 7fe88b74d700 1 -- 172.21.15.111:0/3645654790 shutdown_connections 2023-11-28T13:06:20.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.317+0000 7fe88b74d700 1 -- 172.21.15.111:0/3645654790 wait complete. 2023-11-28T13:06:20.331 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2023-11-28T13:06:20.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.449+0000 7fcb4c1c0700 1 Processor -- start 2023-11-28T13:06:20.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.449+0000 7fcb4c1c0700 1 -- start start 2023-11-28T13:06:20.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.450+0000 7fcb4c1c0700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fcb440b0d70 0x7fcb441604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.450+0000 7fcb4c1c0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b1140 0x7fcb440ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.450+0000 7fcb4c1c0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4415cd90 con 0x7fcb440b15b0 2023-11-28T13:06:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.450+0000 7fcb4c1c0700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb441613e0 con 0x7fcb440b0d70 2023-11-28T13:06:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.450+0000 7fcb4c1c0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb44160b40 con 0x7fcb440b1140 2023-11-28T13:06:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.450+0000 7fcb4a75d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcb440b15b0 0x7fcb440cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39484/0 (socket says 172.21.15.111:39484) 2023-11-28T13:06:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.450+0000 7fcb4a75d700 1 -- 172.21.15.111:0/1644549008 learned_addr learned my addr 172.21.15.111:0/1644549008 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.450+0000 7fcb49f5c700 1 --2- 172.21.15.111:0/1644549008 >> v2:172.21.15.149:3300/0 conn(0x7fcb440b0d70 0x7fcb441604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.450+0000 7fcb4975b700 1 --2- 172.21.15.111:0/1644549008 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b1140 0x7fcb440ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906252581 0 0) 0x7fcb3c001490 con 0x7fcb440b15b0 2023-11-28T13:06:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb300035b0 con 0x7fcb440b15b0 2023-11-28T13:06:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 889658979 0 0) 0x7fcb300035b0 con 0x7fcb440b15b0 2023-11-28T13:06:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b1140 msgr2=0x7fcb440ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 --2- 172.21.15.111:0/1644549008 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b1140 0x7fcb440ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 >> v2:172.21.15.149:3300/0 conn(0x7fcb440b0d70 msgr2=0x7fcb441604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 --2- 172.21.15.111:0/1644549008 >> v2:172.21.15.149:3300/0 conn(0x7fcb440b0d70 0x7fcb441604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb44160e20 con 0x7fcb440b15b0 2023-11-28T13:06:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fcb3c001b80 con 0x7fcb440b15b0 2023-11-28T13:06:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 >> v1:172.21.15.111:6789/0 conn(0x7fcb440b15b0 legacy=0x7fcb440cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 --2- 172.21.15.111:0/1644549008 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b1140 0x7fcb30003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 --2- 172.21.15.111:0/1644549008 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb30003f80 0x7fcb30004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 --2- 172.21.15.111:0/1644549008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb30004d90 0x7fcb30005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb441613e0 con 0x7fcb30004d90 2023-11-28T13:06:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb44160b40 con 0x7fcb30003f80 2023-11-28T13:06:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3000ba10 con 0x7fcb440b1140 2023-11-28T13:06:20.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb4a75d700 1 --2- 172.21.15.111:0/1644549008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb30004d90 0x7fcb30005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:20.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb49f5c700 1 --2- 172.21.15.111:0/1644549008 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b1140 0x7fcb30003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:20.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb4975b700 1 --2- 172.21.15.111:0/1644549008 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb30003f80 0x7fcb30004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:20.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb4a75d700 1 -- 172.21.15.111:0/1644549008 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b1140 msgr2=0x7fcb30003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:20.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb4a75d700 1 --2- 172.21.15.111:0/1644549008 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b1140 0x7fcb30003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb4a75d700 1 -- 172.21.15.111:0/1644549008 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb30003f80 msgr2=0x7fcb30004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:20.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb4a75d700 1 --2- 172.21.15.111:0/1644549008 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb30003f80 0x7fcb30004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb4a75d700 1 -- 172.21.15.111:0/1644549008 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb300035b0 con 0x7fcb30004d90 2023-11-28T13:06:20.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.451+0000 7fcb4a75d700 1 --2- 172.21.15.111:0/1644549008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb30004d90 0x7fcb30005200 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fcb44070330 tx=0x7fcb30005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3664656d191f16c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:20.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.452+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c00a4c0 con 0x7fcb30004d90 2023-11-28T13:06:20.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.452+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3c012600 con 0x7fcb30004d90 2023-11-28T13:06:20.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.452+0000 7fcb48f5a700 1 -- 172.21.15.111:0/1644549008 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c0129b0 con 0x7fcb30004d90 2023-11-28T13:06:20.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.452+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1644549008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb30004d90 msgr2=0x7fcb30005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:20.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.452+0000 7fcb4c1c0700 1 --2- 172.21.15.111:0/1644549008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb30004d90 0x7fcb30005200 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fcb44070330 tx=0x7fcb30005880 comp rx=0 tx=0).stop 2023-11-28T13:06:20.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.452+0000 7fcb4a75d700 1 -- 172.21.15.111:0/1644549008 reap_dead start 2023-11-28T13:06:20.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.452+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1644549008 shutdown_connections 2023-11-28T13:06:20.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.452+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1644549008 >> 172.21.15.111:0/1644549008 conn(0x7fcb44098ee0 msgr2=0x7fcb4406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:20.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.452+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1644549008 shutdown_connections 2023-11-28T13:06:20.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.452+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1644549008 wait complete. 2023-11-28T13:06:20.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.453+0000 7fcb4c1c0700 1 Processor -- start 2023-11-28T13:06:20.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.453+0000 7fcb4c1c0700 1 -- start start 2023-11-28T13:06:20.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb4c1c0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb44159410 0x7fcb441f2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb4c1c0700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b2c10 0x7fcb440ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb4c1c0700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb440adc00 0x7fcb441e6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb4c1c0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440ae0a0 con 0x7fcb44159410 2023-11-28T13:06:20.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb4c1c0700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb441e7360 con 0x7fcb440adc00 2023-11-28T13:06:20.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb4c1c0700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440b3440 con 0x7fcb440b2c10 2023-11-28T13:06:20.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb49f5c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb44159410 0x7fcb441f2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:20.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb49f5c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb44159410 0x7fcb441f2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38714/0 (socket says 172.21.15.111:38714) 2023-11-28T13:06:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb49f5c700 1 -- 172.21.15.111:0/1645240968 learned_addr learned my addr 172.21.15.111:0/1645240968 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb4a75d700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb440adc00 0x7fcb441e6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb4975b700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b2c10 0x7fcb440ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb49f5c700 1 -- 172.21.15.111:0/1645240968 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b2c10 msgr2=0x7fcb440ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb49f5c700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b2c10 0x7fcb440ad6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb49f5c700 1 -- 172.21.15.111:0/1645240968 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb440adc00 msgr2=0x7fcb441e6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb49f5c700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb440adc00 0x7fcb441e6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb49f5c700 1 -- 172.21.15.111:0/1645240968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb440af340 con 0x7fcb44159410 2023-11-28T13:06:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.454+0000 7fcb4975b700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b2c10 0x7fcb440ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.455+0000 7fcb49f5c700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb44159410 0x7fcb441f2e80 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fcb34011a70 tx=0x7fcb34011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.455+0000 7fcb3a7fc700 1 -- 172.21.15.111:0/1645240968 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34005a20 con 0x7fcb44159410 2023-11-28T13:06:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.455+0000 7fcb3a7fc700 1 -- 172.21.15.111:0/1645240968 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb34005b80 con 0x7fcb44159410 2023-11-28T13:06:20.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.455+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1645240968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb44161100 con 0x7fcb44159410 2023-11-28T13:06:20.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.455+0000 7fcb3a7fc700 1 -- 172.21.15.111:0/1645240968 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3400f9a0 con 0x7fcb44159410 2023-11-28T13:06:20.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.455+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1645240968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb440ccc60 con 0x7fcb44159410 2023-11-28T13:06:20.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.456+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1645240968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb44067170 con 0x7fcb44159410 2023-11-28T13:06:20.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.456+0000 7fcb3a7fc700 1 -- 172.21.15.111:0/1645240968 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcb34024070 con 0x7fcb44159410 2023-11-28T13:06:20.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.457+0000 7fcb3a7fc700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb280682f0 0x7fcb2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:20.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.457+0000 7fcb3a7fc700 1 -- 172.21.15.111:0/1645240968 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fcb34085db0 con 0x7fcb44159410 2023-11-28T13:06:20.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.457+0000 7fcb4975b700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb280682f0 0x7fcb2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:20.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.457+0000 7fcb4975b700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb280682f0 0x7fcb2806a7a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009470 tx=0x7fcb40008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:20.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.461+0000 7fcb3a7fc700 1 -- 172.21.15.111:0/1645240968 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcb340527d0 con 0x7fcb44159410 2023-11-28T13:06:20.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.722+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1645240968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) v1 -- 0x7fcb4406d9a0 con 0x7fcb44159410 2023-11-28T13:06:20.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.976+0000 7fcb3a7fc700 1 -- 172.21.15.111:0/1645240968 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v330) v1 ==== 169+0+0 (secure 0 0 0) 0x7fcb34022020 con 0x7fcb44159410 2023-11-28T13:06:20.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.979+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1645240968 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb280682f0 msgr2=0x7fcb2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:20.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.979+0000 7fcb4c1c0700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb280682f0 0x7fcb2806a7a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009470 tx=0x7fcb40008040 comp rx=0 tx=0).stop 2023-11-28T13:06:20.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.979+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1645240968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb44159410 msgr2=0x7fcb441f2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:20.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.979+0000 7fcb4c1c0700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb44159410 0x7fcb441f2e80 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fcb34011a70 tx=0x7fcb34011e30 comp rx=0 tx=0).stop 2023-11-28T13:06:20.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.979+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1645240968 shutdown_connections 2023-11-28T13:06:20.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.979+0000 7fcb4c1c0700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fcb280682f0 0x7fcb2806a7a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.979+0000 7fcb4c1c0700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fcb440adc00 0x7fcb441e6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.979+0000 7fcb4c1c0700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb44159410 0x7fcb441f2e80 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.979+0000 7fcb4c1c0700 1 --2- 172.21.15.111:0/1645240968 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fcb440b2c10 0x7fcb440ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:20.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.979+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1645240968 >> 172.21.15.111:0/1645240968 conn(0x7fcb44098ee0 msgr2=0x7fcb440ca2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:20.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.979+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1645240968 shutdown_connections 2023-11-28T13:06:20.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:20.979+0000 7fcb4c1c0700 1 -- 172.21.15.111:0/1645240968 wait complete. 2023-11-28T13:06:20.993 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ERROR 2023-11-28T13:06:20.993 INFO:tasks.cephfs_test_runner: 2023-11-28T13:06:20.993 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-28T13:06:20.994 INFO:tasks.cephfs_test_runner:ERROR: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-28T13:06:20.994 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared 2023-11-28T13:06:20.994 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-28T13:06:20.994 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-28T13:06:20.994 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_3ae99413970476692dfab1fe38dadb056700a61d/qa/tasks/cephfs/test_forward_scrub.py", line 500, in test_health_status_after_backtrace_repair 2023-11-28T13:06:20.994 INFO:tasks.cephfs_test_runner: self.wait_until_true(lambda: self._check_for_presence_of_MDS_damage(), timeout=200) 2023-11-28T13:06:20.994 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_3ae99413970476692dfab1fe38dadb056700a61d/qa/tasks/ceph_test_case.py", line 212, in wait_until_true 2023-11-28T13:06:20.994 INFO:tasks.cephfs_test_runner: raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count)) 2023-11-28T13:06:20.995 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 200s and 0 retries 2023-11-28T13:06:20.995 INFO:tasks.cephfs_test_runner: 2023-11-28T13:06:20.995 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-28T13:06:20.995 INFO:tasks.cephfs_test_runner:Ran 3 tests in 397.500s 2023-11-28T13:06:20.995 INFO:tasks.cephfs_test_runner: 2023-11-28T13:06:20.995 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2023-11-28T13:06:20.995 INFO:tasks.cephfs_test_runner: 2023-11-28T13:06:20.996 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-28T13:06:20.996 INFO:tasks.cephfs_test_runner:ERROR: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-28T13:06:20.996 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared 2023-11-28T13:06:20.996 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-28T13:06:20.996 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-28T13:06:20.997 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_3ae99413970476692dfab1fe38dadb056700a61d/qa/tasks/cephfs/test_forward_scrub.py", line 500, in test_health_status_after_backtrace_repair 2023-11-28T13:06:20.997 INFO:tasks.cephfs_test_runner: self.wait_until_true(lambda: self._check_for_presence_of_MDS_damage(), timeout=200) 2023-11-28T13:06:20.997 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_neesingh-rh_ceph_3ae99413970476692dfab1fe38dadb056700a61d/qa/tasks/ceph_test_case.py", line 212, in wait_until_true 2023-11-28T13:06:20.997 INFO:tasks.cephfs_test_runner: raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count)) 2023-11-28T13:06:20.997 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 200s and 0 retries 2023-11-28T13:06:20.997 INFO:tasks.cephfs_test_runner: 2023-11-28T13:06:20.998 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_neesingh-rh_ceph_3ae99413970476692dfab1fe38dadb056700a61d/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-28T13:06:21.085 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7bf12345d9af4830bef79119ff249606 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_neesingh-rh_ceph_3ae99413970476692dfab1fe38dadb056700a61d/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-11-28T13:06:21.124 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-11-28T13:06:21.133 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-11-28T13:06:21.141 INFO:tasks.kclient:Unmounting kernel clients... 2023-11-28T13:06:21.141 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:06:21.142 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:06:21.162 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284268k,nr_inodes=4071067,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18839 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-28T13:06:21.162 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-11-28T13:06:21.162 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:06:21.219 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284268k,nr_inodes=4071067,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18839 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260552k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-28T13:06:21.219 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T13:06:21.219 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:06:21.240 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8026740k,nr_inodes=2006685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16040 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=1609048k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-28T13:06:21.240 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2023-11-28T13:06:21.240 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T13:06:21.301 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8026740k,nr_inodes=2006685,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=42,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16040 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=1609048k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-28T13:06:21.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:21.301 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-11-28T13:06:21.317 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T13:06:21.318 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T13:06:21.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:21.318 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T13:06:21.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:21.398 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T13:06:21.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:21.475 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link delete ceph-brx 2023-11-28T13:06:21.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:21.630 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-11-28T13:06:21.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:21.647 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link delete ceph-brx 2023-11-28T13:06:21.717 INFO:teuthology.orchestra.run.smithi111.stderr:Cannot find device "ceph-brx" 2023-11-28T13:06:21.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T13:06:21.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:21.719 DEBUG:teuthology.orchestra.run.smithi149:> ip netns list 2023-11-28T13:06:21.739 INFO:teuthology.orchestra.run.smithi149.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-28T13:06:21.739 INFO:teuthology.orchestra.run.smithi149.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-28T13:06:21.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:21.740 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T13:06:21.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:21.819 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T13:06:21.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:21.900 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip link delete ceph-brx 2023-11-28T13:06:22.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:22.032 DEBUG:teuthology.orchestra.run.smithi149:> ip netns list 2023-11-28T13:06:22.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T13:06:22.050 DEBUG:teuthology.orchestra.run.smithi149:> sudo ip link delete ceph-brx 2023-11-28T13:06:22.120 INFO:teuthology.orchestra.run.smithi149.stderr:Cannot find device "ceph-brx" 2023-11-28T13:06:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T13:06:22.122 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-28T13:06:22.131 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-28T13:06:22.131 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:06:22.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.251+0000 7efecd056700 1 Processor -- start 2023-11-28T13:06:22.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.251+0000 7efecd056700 1 -- start start 2023-11-28T13:06:22.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.251+0000 7efecd056700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efec8159080 0x7efec80ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.251+0000 7efecd056700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec806d870 con 0x7efec8071020 2023-11-28T13:06:22.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.251+0000 7efecd056700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec806cc60 con 0x7efec8158cb0 2023-11-28T13:06:22.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.251+0000 7efecd056700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec806cf40 con 0x7efec8159080 2023-11-28T13:06:22.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec759e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efec8071020 0x7efec80cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39504/0 (socket says 172.21.15.111:39504) 2023-11-28T13:06:22.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec759e700 1 -- 172.21.15.111:0/2594142317 learned_addr learned my addr 172.21.15.111:0/2594142317 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:22.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10046103 0 0) 0x7efec806d870 con 0x7efec8071020 2023-11-28T13:06:22.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb00046c0 con 0x7efec8071020 2023-11-28T13:06:22.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1436087456 0 0) 0x7efeb00046c0 con 0x7efec8071020 2023-11-28T13:06:22.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efec8159080 msgr2=0x7efec80ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:22.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 --2- 172.21.15.111:0/2594142317 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efec8159080 0x7efec80ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 >> v1:172.21.15.149:6789/0 conn(0x7efec8158cb0 legacy=0x7efec8070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:22.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec806d220 con 0x7efec8071020 2023-11-28T13:06:22.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7efec00019f0 con 0x7efec8071020 2023-11-28T13:06:22.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 >> v1:172.21.15.111:6789/0 conn(0x7efec8071020 legacy=0x7efec80cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 --2- 172.21.15.111:0/2594142317 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efeb0004b40 0x7efeb0004f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 --2- 172.21.15.111:0/2594142317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeb0005960 0x7efeb0005dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 --2- 172.21.15.111:0/2594142317 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efeb0006310 0x7efeb000cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec806cf40 con 0x7efeb0005960 2023-11-28T13:06:22.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb000d1d0 con 0x7efeb0006310 2023-11-28T13:06:22.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb000d390 con 0x7efeb0004b40 2023-11-28T13:06:22.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec659c700 1 --2- 172.21.15.111:0/2594142317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeb0005960 0x7efeb0005dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec6d9d700 1 --2- 172.21.15.111:0/2594142317 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efeb0004b40 0x7efeb0004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.252+0000 7efec759e700 1 --2- 172.21.15.111:0/2594142317 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efeb0006310 0x7efeb000cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efec659c700 1 -- 172.21.15.111:0/2594142317 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efeb0004b40 msgr2=0x7efeb0004f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:22.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efec659c700 1 --2- 172.21.15.111:0/2594142317 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efeb0004b40 0x7efeb0004f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efec659c700 1 -- 172.21.15.111:0/2594142317 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efeb0006310 msgr2=0x7efeb000cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:22.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efec659c700 1 --2- 172.21.15.111:0/2594142317 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efeb0006310 0x7efeb000cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efec659c700 1 -- 172.21.15.111:0/2594142317 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb00046c0 con 0x7efeb0005960 2023-11-28T13:06:22.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efec659c700 1 --2- 172.21.15.111:0/2594142317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeb0005960 0x7efeb0005dd0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7efeb8009910 tx=0x7efeb0006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a9374f903a0f83 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:22.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb8004230 con 0x7efeb0005960 2023-11-28T13:06:22.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb80043b0 con 0x7efeb0005960 2023-11-28T13:06:22.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efec5d9b700 1 -- 172.21.15.111:0/2594142317 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb80038c0 con 0x7efeb0005960 2023-11-28T13:06:22.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efecd056700 1 -- 172.21.15.111:0/2594142317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeb0005960 msgr2=0x7efeb0005dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efecd056700 1 --2- 172.21.15.111:0/2594142317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efeb0005960 0x7efeb0005dd0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7efeb8009910 tx=0x7efeb0006990 comp rx=0 tx=0).stop 2023-11-28T13:06:22.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efec759e700 1 -- 172.21.15.111:0/2594142317 reap_dead start 2023-11-28T13:06:22.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efecd056700 1 -- 172.21.15.111:0/2594142317 shutdown_connections 2023-11-28T13:06:22.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efecd056700 1 -- 172.21.15.111:0/2594142317 >> 172.21.15.111:0/2594142317 conn(0x7efec80cb490 msgr2=0x7efec80cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:22.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efecd056700 1 -- 172.21.15.111:0/2594142317 shutdown_connections 2023-11-28T13:06:22.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.253+0000 7efecd056700 1 -- 172.21.15.111:0/2594142317 wait complete. 2023-11-28T13:06:22.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.255+0000 7efecd056700 1 Processor -- start 2023-11-28T13:06:22.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.255+0000 7efecd056700 1 -- start start 2023-11-28T13:06:22.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efecd056700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efeb0005960 0x7efec8146820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efecd056700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efec81471c0 0x7efec81438d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efecd056700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efec8143e10 0x7efec8145250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efecd056700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec81442b0 con 0x7efec8143e10 2023-11-28T13:06:22.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efecd056700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec8144520 con 0x7efeb0005960 2023-11-28T13:06:22.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efecd056700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec81594a0 con 0x7efec81471c0 2023-11-28T13:06:22.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efec759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efec8143e10 0x7efec8145250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efec759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efec8143e10 0x7efec8145250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38738/0 (socket says 172.21.15.111:38738) 2023-11-28T13:06:22.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efec759e700 1 -- 172.21.15.111:0/3031573879 learned_addr learned my addr 172.21.15.111:0/3031573879 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:22.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efec659c700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efec81471c0 0x7efec81438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efec6d9d700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efeb0005960 0x7efec8146820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efec759e700 1 -- 172.21.15.111:0/3031573879 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efec81471c0 msgr2=0x7efec81438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efec759e700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efec81471c0 0x7efec81438d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efec759e700 1 -- 172.21.15.111:0/3031573879 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efeb0005960 msgr2=0x7efec8146820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efec759e700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efeb0005960 0x7efec8146820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efec759e700 1 -- 172.21.15.111:0/3031573879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec8071b20 con 0x7efec8143e10 2023-11-28T13:06:22.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efec659c700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efec81471c0 0x7efec81438d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:22.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efec6d9d700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efeb0005960 0x7efec8146820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efec759e700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efec8143e10 0x7efec8145250 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7efec806b410 tx=0x7efec000a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efeaf7fe700 1 -- 172.21.15.111:0/3031573879 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec000ae30 con 0x7efec8143e10 2023-11-28T13:06:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efeaf7fe700 1 -- 172.21.15.111:0/3031573879 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec00133f0 con 0x7efec8143e10 2023-11-28T13:06:22.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efecd056700 1 -- 172.21.15.111:0/3031573879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec806d500 con 0x7efec8143e10 2023-11-28T13:06:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efeaf7fe700 1 -- 172.21.15.111:0/3031573879 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec00137a0 con 0x7efec8143e10 2023-11-28T13:06:22.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.256+0000 7efecd056700 1 -- 172.21.15.111:0/3031573879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec80c9300 con 0x7efec8143e10 2023-11-28T13:06:22.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.257+0000 7efecd056700 1 -- 172.21.15.111:0/3031573879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec8067880 con 0x7efec8143e10 2023-11-28T13:06:22.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.258+0000 7efeaf7fe700 1 -- 172.21.15.111:0/3031573879 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efec0013900 con 0x7efec8143e10 2023-11-28T13:06:22.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.258+0000 7efeaf7fe700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efeb40682f0 0x7efeb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.258+0000 7efec6d9d700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efeb40682f0 0x7efeb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.258+0000 7efeaf7fe700 1 -- 172.21.15.111:0/3031573879 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7efec0086620 con 0x7efec8143e10 2023-11-28T13:06:22.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.259+0000 7efec6d9d700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efeb40682f0 0x7efeb406a7a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efebc007fd0 tx=0x7efebc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:22.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.262+0000 7efeaf7fe700 1 -- 172.21.15.111:0/3031573879 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efec00530f0 con 0x7efec8143e10 2023-11-28T13:06:22.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.440+0000 7efecd056700 1 -- 172.21.15.111:0/3031573879 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efec8158610 con 0x7efeb40682f0 2023-11-28T13:06:22.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.443+0000 7efeaf7fe700 1 -- 172.21.15.111:0/3031573879 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+73327 (secure 0 0 0) 0x7efec8158610 con 0x7efeb40682f0 2023-11-28T13:06:22.445 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:22.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.445+0000 7efecd056700 1 -- 172.21.15.111:0/3031573879 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efeb40682f0 msgr2=0x7efeb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.445+0000 7efecd056700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efeb40682f0 0x7efeb406a7a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efebc007fd0 tx=0x7efebc009040 comp rx=0 tx=0).stop 2023-11-28T13:06:22.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.445+0000 7efecd056700 1 -- 172.21.15.111:0/3031573879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efec8143e10 msgr2=0x7efec8145250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.445+0000 7efecd056700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efec8143e10 0x7efec8145250 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7efec806b410 tx=0x7efec000a600 comp rx=0 tx=0).stop 2023-11-28T13:06:22.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.445+0000 7efecd056700 1 -- 172.21.15.111:0/3031573879 shutdown_connections 2023-11-28T13:06:22.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.445+0000 7efecd056700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efec8143e10 0x7efec8145250 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.445+0000 7efecd056700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7efeb40682f0 0x7efeb406a7a0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.445+0000 7efecd056700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7efeb0005960 0x7efec8146820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.445+0000 7efecd056700 1 --2- 172.21.15.111:0/3031573879 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7efec81471c0 0x7efec81438d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.445+0000 7efecd056700 1 -- 172.21.15.111:0/3031573879 >> 172.21.15.111:0/3031573879 conn(0x7efec80cb490 msgr2=0x7efec806dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:22.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.445+0000 7efecd056700 1 -- 172.21.15.111:0/3031573879 shutdown_connections 2023-11-28T13:06:22.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.445+0000 7efecd056700 1 -- 172.21.15.111:0/3031573879 wait complete. 2023-11-28T13:06:22.449 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:06:22.458 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2023-11-28T13:06:20.442690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":81596,"kb_used_data":7804,"kb_used_omap":0,"kb_used_meta":73728,"kb_avail":754893124,"statfs":{"total":773094113280,"available":773010558976,"internally_reserved":0,"allocated":7991296,"data_stored":4378158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":75497472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":72,"apply_latency_ms":72,"commit_latency_ns":72000000,"apply_latency_ns":72000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12321,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-6,"num_read_kb":-5,"num_write":-72,"num_write_kb":-35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001182"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":58,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-11-28T13:03:01.375035+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T13:03:01.375035+0000","last_peered":"2023-11-28T13:03:01.375035+0000","last_clean":"2023-11-28T13:03:01.375035+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:03:01.375035+0000","last_undegraded":"2023-11-28T13:03:01.375035+0000","last_fullsized":"2023-11-28T13:03:01.375035+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-11-28T13:05:06.401329+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T13:05:06.401329+0000","last_peered":"2023-11-28T13:05:06.401329+0000","last_clean":"2023-11-28T13:05:06.401329+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:05:06.401329+0000","last_undegraded":"2023-11-28T13:05:06.401329+0000","last_fullsized":"2023-11-28T13:05:06.401329+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-11-28T13:05:06.388515+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T13:05:06.388515+0000","last_peered":"2023-11-28T13:05:06.388515+0000","last_clean":"2023-11-28T13:05:06.388515+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:05:06.388515+0000","last_undegraded":"2023-11-28T13:05:06.388515+0000","last_fullsized":"2023-11-28T13:05:06.388515+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-11-28T13:05:06.401348+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T13:05:06.401348+0000","last_peered":"2023-11-28T13:05:06.401348+0000","last_clean":"2023-11-28T13:05:06.401348+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:05:06.401348+0000","last_undegraded":"2023-11-28T13:05:06.401348+0000","last_fullsized":"2023-11-28T13:05:06.401348+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":208,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-11-28T13:02:51.377835+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T13:02:51.377835+0000","last_peered":"2023-11-28T13:02:51.377835+0000","last_clean":"2023-11-28T13:02:51.377835+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:02:51.377835+0000","last_undegraded":"2023-11-28T13:02:51.377835+0000","last_fullsized":"2023-11-28T13:02:51.377835+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":56,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-11-28T13:02:51.377869+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T13:02:51.377869+0000","last_peered":"2023-11-28T13:02:51.377869+0000","last_clean":"2023-11-28T13:02:51.377869+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:02:51.377869+0000","last_undegraded":"2023-11-28T13:02:51.377869+0000","last_fullsized":"2023-11-28T13:02:51.377869+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-11-28T13:03:01.379265+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T13:03:01.379265+0000","last_peered":"2023-11-28T13:03:01.379265+0000","last_clean":"2023-11-28T13:03:01.379265+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:03:01.379265+0000","last_undegraded":"2023-11-28T13:03:01.379265+0000","last_fullsized":"2023-11-28T13:03:01.379265+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":57,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-11-28T13:03:11.375915+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T13:03:11.375915+0000","last_peered":"2023-11-28T13:03:11.375915+0000","last_clean":"2023-11-28T13:03:11.375915+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:03:11.375915+0000","last_undegraded":"2023-11-28T13:03:11.375915+0000","last_fullsized":"2023-11-28T13:03:11.375915+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":65,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-11-28T13:05:21.396757+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T13:05:21.396757+0000","last_peered":"2023-11-28T13:05:21.396757+0000","last_clean":"2023-11-28T13:05:21.396757+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:05:21.396757+0000","last_undegraded":"2023-11-28T13:05:21.396757+0000","last_fullsized":"2023-11-28T13:05:21.396757+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11012,"kb_used_data":1404,"kb_used_omap":0,"kb_used_meta":9600,"kb_avail":94360828,"statfs":{"total":96636764160,"available":96625487872,"internally_reserved":0,"allocated":1437696,"data_stored":984927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9830400},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.64700000000000002},"min":{"1min":0.51200000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.79800000000000004,"5min":1.194,"15min":3.3279999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.58999999999999997,"15min":0.64800000000000002},"min":{"1min":0.46300000000000002,"5min":0.372,"15min":0.27900000000000003},"max":{"1min":0.77000000000000002,"5min":0.81799999999999995,"15min":3.2919999999999998},"last":0.61399999999999999}]},{"osd":1,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.40300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.52400000000000002},"min":{"1min":0.436,"5min":0.34999999999999998,"15min":0.30499999999999999},"max":{"1min":0.76000000000000001,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.54300000000000004}]},{"osd":2,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.56200000000000006},"min":{"1min":0.44600000000000001,"5min":0.35699999999999998,"15min":0.28399999999999997},"max":{"1min":0.751,"5min":1.149,"15min":1.149},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57499999999999996,"15min":0.55200000000000005},"min":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.35699999999999998},"max":{"1min":0.72399999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.57399999999999995}]},{"osd":3,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60299999999999998,"15min":0.56000000000000005},"min":{"1min":0.52000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82899999999999996,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58799999999999997,"15min":0.57299999999999995},"min":{"1min":0.433,"5min":0.40699999999999997,"15min":0.38600000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.51900000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47599999999999998,"15min":0.50800000000000001},"min":{"1min":0.35099999999999998,"5min":0.33300000000000002,"15min":0.316},"max":{"1min":0.78400000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.627},{"interface":"front","average":{"1min":0.495,"5min":0.50600000000000001,"15min":0.52600000000000002},"min":{"1min":0.373,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.73899999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.72199999999999998}]},{"osd":5,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47799999999999998,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71399999999999997,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53500000000000003,"15min":0.56000000000000005},"min":{"1min":0.36899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.74399999999999999}]},{"osd":6,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52400000000000002,"15min":0.56000000000000005},"min":{"1min":0.38200000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77800000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58899999999999997,"15min":0.60799999999999998},"min":{"1min":0.434,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79900000000000004,"5min":1.274,"15min":1.274},"last":0.76600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574943,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9932,"kb_used_data":836,"kb_used_omap":0,"kb_used_meta":9088,"kb_avail":94361908,"statfs":{"total":96636764160,"available":96626593792,"internally_reserved":0,"allocated":856064,"data_stored":400921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9306112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.437,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.89600000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.52400000000000002},"min":{"1min":0.33900000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72799999999999998,"5min":0.755,"15min":0.80400000000000005},"last":0.504}]},{"osd":1,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56699999999999995,"15min":0.53500000000000003},"min":{"1min":0.46100000000000002,"5min":0.32400000000000001,"15min":0.309},"max":{"1min":0.752,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40300000000000002,"5min":0.33000000000000002,"15min":0.29599999999999999},"max":{"1min":0.76700000000000002,"5min":0.86599999999999999,"15min":1.016},"last":0.45800000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.39900000000000002,"5min":0.34300000000000003,"15min":0.34100000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.91700000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.75700000000000001,"5min":0.92400000000000004,"15min":0.96299999999999997},"last":0.48999999999999999}]},{"osd":3,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.371},"max":{"1min":0.71399999999999997,"5min":0.80700000000000005,"15min":1.5920000000000001},"last":0.378},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59399999999999997,"15min":0.58399999999999996},"min":{"1min":0.504,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76400000000000001,"5min":0.84999999999999998,"15min":0.94099999999999995},"last":0.51300000000000001}]},{"osd":4,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46100000000000002,"15min":0.51200000000000001},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74199999999999999,"5min":0.84199999999999997,"15min":1},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48199999999999998,"15min":0.52200000000000002},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72699999999999998,"5min":0.753,"15min":0.89200000000000002},"last":0.439}]},{"osd":5,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.54600000000000004},"min":{"1min":0.36399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.871,"5min":0.871,"15min":1.052},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.59499999999999997},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.81299999999999994,"5min":0.88900000000000001,"15min":1.258},"last":0.52600000000000002}]},{"osd":7,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.52100000000000002,"15min":0.56200000000000006},"min":{"1min":0.32100000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.79200000000000004,"5min":0.80100000000000005,"15min":0.97599999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56100000000000005,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.89500000000000002,"5min":0.90400000000000003,"15min":1.1060000000000001},"last":0.54700000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9940,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":9088,"kb_avail":94361900,"statfs":{"total":96636764160,"available":96626585600,"internally_reserved":0,"allocated":864256,"data_stored":406347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9306112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35099999999999998,"15min":0.33500000000000002},"min":{"1min":0.23799999999999999,"5min":0.224,"15min":0.219},"max":{"1min":0.496,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.36499999999999999},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60299999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.28999999999999998}]},{"osd":2,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.373,"15min":0.34699999999999998},"min":{"1min":0.29199999999999998,"5min":0.22,"15min":0.20599999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40799999999999997,"15min":0.38100000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.25},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.80100000000000005},"last":0.375}]},{"osd":3,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.434,"15min":0.40400000000000003},"min":{"1min":0.32800000000000001,"5min":0.27200000000000002,"15min":0.245},"max":{"1min":0.57399999999999995,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.41299999999999998,"15min":0.40000000000000002},"min":{"1min":0.27800000000000002,"5min":0.26000000000000001,"15min":0.251},"max":{"1min":0.58399999999999996,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.33100000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.60499999999999998,"15min":0.57999999999999996},"min":{"1min":0.40200000000000002,"5min":0.36199999999999999,"15min":0.34000000000000002},"max":{"1min":0.80000000000000004,"5min":0.91700000000000004,"15min":1.028},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.624,"15min":0.59599999999999997},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.34799999999999998},"max":{"1min":0.76100000000000001,"5min":0.98699999999999999,"15min":1.0660000000000001},"last":0.497}]},{"osd":5,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.59999999999999998},"min":{"1min":0.47299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78100000000000003,"5min":0.94999999999999996,"15min":1.042},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63500000000000001,"15min":0.58499999999999996},"min":{"1min":0.44700000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78300000000000003,"5min":1.05,"15min":1.05},"last":0.53400000000000003}]},{"osd":6,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.626,"15min":0.59599999999999997},"min":{"1min":0.438,"5min":0.40600000000000003,"15min":0.38900000000000001},"max":{"1min":0.80100000000000005,"5min":0.96199999999999997,"15min":1.006},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63400000000000001,"15min":0.60999999999999999},"min":{"1min":0.48699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.81899999999999995,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.54400000000000004}]},{"osd":7,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.64200000000000002,"15min":0.61499999999999999},"min":{"1min":0.51600000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64000000000000001,"15min":0.59599999999999997},"min":{"1min":0.47799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80400000000000005,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.47299999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574942,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11068,"kb_used_data":1396,"kb_used_omap":0,"kb_used_meta":9664,"kb_avail":94360772,"statfs":{"total":96636764160,"available":96625430528,"internally_reserved":0,"allocated":1429504,"data_stored":984327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9895936},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.33700000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51300000000000001,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38200000000000001,"15min":0.35499999999999998},"min":{"1min":0.23000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.50700000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.26800000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38200000000000001,"15min":0.35799999999999998},"min":{"1min":0.27300000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.48399999999999999,"5min":0.57099999999999995,"15min":0.59399999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41199999999999998,"15min":0.38100000000000001},"min":{"1min":0.28699999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50600000000000001,"5min":0.623,"15min":0.623},"last":0.313}]},{"osd":3,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38900000000000001,"15min":0.377},"min":{"1min":0.247,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.68000000000000005},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.441,"15min":0.40500000000000003},"min":{"1min":0.28799999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52200000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.34899999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.622,"15min":0.60699999999999998},"min":{"1min":0.51400000000000001,"5min":0.309,"15min":0.28199999999999997},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.496},{"interface":"front","average":{"1min":0.629,"5min":0.626,"15min":0.58799999999999997},"min":{"1min":0.49299999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83299999999999996,"5min":0.88,"15min":0.88},"last":0.41999999999999998}]},{"osd":5,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.58299999999999996},"min":{"1min":0.42999999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79600000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64100000000000001,"15min":0.625},"min":{"1min":0.441,"5min":0.434,"15min":0.39600000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.50800000000000001}]},{"osd":6,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.64300000000000002,"15min":0.61599999999999999},"min":{"1min":0.51700000000000002,"5min":0.46999999999999997,"15min":0.41199999999999998},"max":{"1min":0.84799999999999998,"5min":0.89800000000000002,"15min":0.91900000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65300000000000002,"15min":0.627},"min":{"1min":0.50600000000000001,"5min":0.41499999999999998,"15min":0.33900000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.98599999999999999},"last":0.42999999999999999}]},{"osd":7,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.65900000000000003,"15min":0.63900000000000001},"min":{"1min":0.52900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.0149999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65800000000000003,"15min":0.63700000000000001},"min":{"1min":0.45700000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.47499999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574942,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9816,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":8960,"kb_avail":94362024,"statfs":{"total":96636764160,"available":96626712576,"internally_reserved":0,"allocated":868352,"data_stored":412617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9175040},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.33000000000000002},"min":{"1min":0.255,"5min":0.22500000000000001,"15min":0.20999999999999999},"max":{"1min":0.48799999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"min":{"1min":0.22600000000000001,"5min":0.21199999999999999,"15min":0.184},"max":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.35999999999999999}]},{"osd":1,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.39800000000000002,"15min":0.38700000000000001},"min":{"1min":0.28799999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.51800000000000002,"5min":0.59499999999999997,"15min":0.67000000000000004},"last":0.377},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.374},"min":{"1min":0.26300000000000001,"5min":0.25700000000000001,"15min":0.24199999999999999},"max":{"1min":0.56999999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.42799999999999999}]},{"osd":3,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39200000000000002,"15min":0.35599999999999998},"min":{"1min":0.27900000000000003,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":0.505,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.443,"15min":0.42499999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.55700000000000005,"5min":0.70599999999999996,"15min":0.77500000000000002},"last":0.40000000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64000000000000001,"15min":0.60599999999999998},"min":{"1min":0.46200000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85799999999999998,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63,"15min":0.57899999999999996},"min":{"1min":0.44900000000000001,"5min":0.42199999999999999,"15min":0.34300000000000003},"max":{"1min":0.86399999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.60799999999999998}]},{"osd":5,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.60499999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.91900000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.63400000000000001,"15min":0.60799999999999998},"min":{"1min":0.52500000000000002,"5min":0.41699999999999998,"15min":0.379},"max":{"1min":0.877,"5min":0.877,"15min":1.0589999999999999},"last":0.76600000000000001}]},{"osd":6,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64800000000000002,"15min":0.59699999999999998},"min":{"1min":0.503,"5min":0.41999999999999998,"15min":0.36599999999999999},"max":{"1min":0.83899999999999997,"5min":0.89100000000000001,"15min":1.002},"last":0.496},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65300000000000002,"15min":0.63300000000000001},"min":{"1min":0.48399999999999999,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.82799999999999996,"5min":0.96799999999999997,"15min":0.97799999999999998},"last":0.59599999999999997}]},{"osd":7,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65600000000000003,"15min":0.63600000000000001},"min":{"1min":0.438,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86199999999999999,"5min":0.93500000000000005,"15min":0.95099999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.65800000000000003,"15min":0.61399999999999999},"min":{"1min":0.53000000000000003,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":0.86899999999999999,"5min":0.92400000000000004,"15min":1.0349999999999999},"last":0.64400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574942,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9920,"kb_used_data":824,"kb_used_omap":0,"kb_used_meta":9088,"kb_avail":94361920,"statfs":{"total":96636764160,"available":96626606080,"internally_reserved":0,"allocated":843776,"data_stored":394071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9306112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.33500000000000002},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38200000000000001,"15min":0.34499999999999997},"min":{"1min":0.24099999999999999,"5min":0.23200000000000001,"15min":0.20000000000000001},"max":{"1min":0.55200000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.313}]},{"osd":1,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.38600000000000001,"15min":0.35399999999999998},"min":{"1min":0.25700000000000001,"5min":0.22,"15min":0.21299999999999999},"max":{"1min":0.53000000000000003,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40500000000000003,"15min":0.38300000000000001},"min":{"1min":0.317,"5min":0.254,"15min":0.23799999999999999},"max":{"1min":0.54600000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.33600000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.45000000000000001,"15min":0.47499999999999998},"min":{"1min":0.29299999999999998,"5min":0.252,"15min":0.251},"max":{"1min":0.58299999999999996,"5min":0.65100000000000002,"15min":2.6869999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45700000000000002,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.30499999999999999,"15min":0.28199999999999997},"max":{"1min":0.60099999999999998,"5min":0.66800000000000004,"15min":2.7160000000000002},"last":0.35899999999999999}]},{"osd":4,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63600000000000001,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81999999999999995,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63,"15min":0.84599999999999997},"min":{"1min":0.442,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82899999999999996,"5min":0.93899999999999995,"15min":8.2889999999999997},"last":0.46800000000000003}]},{"osd":5,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.48599999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.63300000000000001,"15min":0.66000000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.72999999999999998,"5min":0.88200000000000001,"15min":2.165},"last":0.52600000000000002}]},{"osd":6,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65200000000000002,"15min":0.63600000000000001},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.88400000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.63500000000000001},"min":{"1min":0.53100000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.81299999999999994,"5min":0.95399999999999996,"15min":1.1419999999999999},"last":0.47599999999999998}]},{"osd":7,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66100000000000003,"15min":0.626},"min":{"1min":0.48399999999999999,"5min":0.46000000000000002,"15min":0.45600000000000002},"max":{"1min":0.82399999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66000000000000003,"15min":0.85199999999999998},"min":{"1min":0.45900000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85999999999999999,"5min":0.89600000000000002,"15min":7.0899999999999999},"last":0.55600000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574940,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10108,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":9280,"kb_avail":94361732,"statfs":{"total":96636764160,"available":96626413568,"internally_reserved":0,"allocated":839680,"data_stored":394029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9502720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.66200000000000003,"5min":0.78500000000000003,"15min":1.0049999999999999},"last":0.371},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56000000000000005,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.66600000000000004,"5min":0.873,"15min":0.873},"last":0.496}]},{"osd":1,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55300000000000005,"15min":0.52100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71399999999999997,"5min":0.76600000000000001,"15min":0.95699999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56999999999999995,"15min":0.56499999999999995},"min":{"1min":0.45500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69999999999999996,"5min":0.77300000000000002,"15min":1.1060000000000001},"last":0.44}]},{"osd":2,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59799999999999998,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68999999999999995,"5min":0.77300000000000002,"15min":0.91800000000000004},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60999999999999999,"15min":0.60199999999999998},"min":{"1min":0.48999999999999999,"5min":0.41499999999999998,"15min":0.36399999999999999},"max":{"1min":0.67800000000000005,"5min":0.81100000000000005,"15min":1.069},"last":0.47599999999999998}]},{"osd":3,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55200000000000005,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.70099999999999996,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.46300000000000002,"5min":0.41099999999999998,"15min":0.38900000000000001},"max":{"1min":0.71099999999999997,"5min":0.81599999999999995,"15min":0.879},"last":0.40699999999999997}]},{"osd":5,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47399999999999998,"15min":0.503},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.315},"max":{"1min":0.752,"5min":0.752,"15min":0.86099999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.51000000000000001},"min":{"1min":0.371,"5min":0.309,"15min":0.309},"max":{"1min":0.68200000000000005,"5min":0.73799999999999999,"15min":0.80400000000000005},"last":0.54800000000000004}]},{"osd":6,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.49099999999999999,"15min":0.51300000000000001},"min":{"1min":0.35399999999999998,"5min":0.32500000000000001,"15min":0.30499999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.84299999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53000000000000003,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.35199999999999998,"15min":0.34499999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.96399999999999997},"last":0.57699999999999996}]},{"osd":7,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73999999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.36599999999999999},"max":{"1min":0.80400000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.56899999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574942,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9800,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":8960,"kb_avail":94362040,"statfs":{"total":96636764160,"available":96626728960,"internally_reserved":0,"allocated":851968,"data_stored":400919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9175040},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.378,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72699999999999998,"5min":0.74399999999999999,"15min":1.819},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.53600000000000003},"min":{"1min":0.40500000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.42099999999999999}]},{"osd":1,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.53300000000000003},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.29799999999999999},"max":{"1min":0.70999999999999996,"5min":0.76500000000000001,"15min":0.90000000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56000000000000005,"15min":0.52700000000000002},"min":{"1min":0.315,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70799999999999996,"5min":0.77400000000000002,"15min":0.78900000000000003},"last":0.54100000000000004}]},{"osd":2,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.59399999999999997},"min":{"1min":0.42499999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.74099999999999999,"5min":0.84899999999999998,"15min":1.0089999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58299999999999996,"15min":0.623},"min":{"1min":0.45600000000000002,"5min":0.38200000000000001,"15min":0.35099999999999998},"max":{"1min":0.68799999999999994,"5min":0.78100000000000003,"15min":2.4020000000000001},"last":0.48099999999999998}]},{"osd":3,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57999999999999996,"15min":0.55800000000000005},"min":{"1min":0.45400000000000001,"5min":0.35199999999999998,"15min":0.33600000000000002},"max":{"1min":0.752,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57299999999999995,"15min":0.64500000000000002},"min":{"1min":0.44600000000000001,"5min":0.36599999999999999,"15min":0.35799999999999998},"max":{"1min":0.69699999999999995,"5min":0.81999999999999995,"15min":2.702},"last":0.46000000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46500000000000002,"15min":0.56499999999999995},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.27300000000000002},"max":{"1min":0.76100000000000001,"5min":0.76800000000000002,"15min":3.0739999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.29899999999999999},"max":{"1min":0.65700000000000003,"5min":0.748,"15min":0.93000000000000005},"last":0.5}]},{"osd":6,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48099999999999998,"15min":0.54000000000000004},"min":{"1min":0.38800000000000001,"5min":0.33500000000000002,"15min":0.28699999999999998},"max":{"1min":0.60799999999999998,"5min":0.71999999999999997,"15min":1.1819999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.51300000000000001,"15min":0.65200000000000002},"min":{"1min":0.35399999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":3.7679999999999998},"last":0.44600000000000001}]},{"osd":7,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.57399999999999995},"min":{"1min":0.377,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.65400000000000003,"5min":0.79300000000000004,"15min":0.83699999999999997},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55400000000000005,"15min":0.627},"min":{"1min":0.40999999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.748,"5min":0.78000000000000003,"15min":1.2230000000000001},"last":0.51300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:06:22.461 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:06:22.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.578+0000 7f8eb2d07700 1 Processor -- start 2023-11-28T13:06:22.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.578+0000 7f8eb2d07700 1 -- start start 2023-11-28T13:06:22.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.579+0000 7f8eb2d07700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f8eac0b2220 0x7f8eac0cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.579+0000 7f8eb2d07700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8eac0cb380 0x7f8eac0ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.579+0000 7f8eb2d07700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8eac0cee40 0x7f8eac0cd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.579+0000 7f8eb2d07700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eac0cf2c0 con 0x7f8eac0cb380 2023-11-28T13:06:22.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.579+0000 7f8eb2d07700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eac08ae90 con 0x7f8eac0b2220 2023-11-28T13:06:22.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.579+0000 7f8eb2d07700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac0cc7b0 con 0x7f8eac0cee40 2023-11-28T13:06:22.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.579+0000 7f8eabfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8eac0cb380 0x7f8eac0ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.579+0000 7f8eabfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8eac0cb380 0x7f8eac0ce900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38742/0 (socket says 172.21.15.111:38742) 2023-11-28T13:06:22.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.579+0000 7f8eabfff700 1 -- 172.21.15.111:0/3362929243 learned_addr learned my addr 172.21.15.111:0/3362929243 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:22.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eb0aa3700 1 --2- 172.21.15.111:0/3362929243 >> v2:172.21.15.149:3300/0 conn(0x7f8eac0b2220 0x7f8eac0cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eb12a4700 1 --2- 172.21.15.111:0/3362929243 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8eac0cee40 0x7f8eac0cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eabfff700 1 -- 172.21.15.111:0/3362929243 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8eac0cee40 msgr2=0x7f8eac0cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:22.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eabfff700 1 --2- 172.21.15.111:0/3362929243 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8eac0cee40 0x7f8eac0cd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eabfff700 1 -- 172.21.15.111:0/3362929243 >> v2:172.21.15.149:3300/0 conn(0x7f8eac0b2220 msgr2=0x7f8eac0cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:22.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eabfff700 1 --2- 172.21.15.111:0/3362929243 >> v2:172.21.15.149:3300/0 conn(0x7f8eac0b2220 0x7f8eac0cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eabfff700 1 -- 172.21.15.111:0/3362929243 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac0b25f0 con 0x7f8eac0cb380 2023-11-28T13:06:22.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eb12a4700 1 --2- 172.21.15.111:0/3362929243 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8eac0cee40 0x7f8eac0cd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:22.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eb0aa3700 1 --2- 172.21.15.111:0/3362929243 >> v2:172.21.15.149:3300/0 conn(0x7f8eac0b2220 0x7f8eac0cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:22.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eabfff700 1 --2- 172.21.15.111:0/3362929243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eac0cb380 0x7f8eac0ce900 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f8e980098f0 tx=0x7f8e98009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd13a59fd5722319 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:22.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eab7fe700 1 -- 172.21.15.111:0/3362929243 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9800b860 con 0x7f8eac0cb380 2023-11-28T13:06:22.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eab7fe700 1 -- 172.21.15.111:0/3362929243 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9800b9c0 con 0x7f8eac0cb380 2023-11-28T13:06:22.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eab7fe700 1 -- 172.21.15.111:0/3362929243 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9800bd50 con 0x7f8eac0cb380 2023-11-28T13:06:22.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eb2d07700 1 -- 172.21.15.111:0/3362929243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eac0cb380 msgr2=0x7f8eac0ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eb2d07700 1 --2- 172.21.15.111:0/3362929243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eac0cb380 0x7f8eac0ce900 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f8e980098f0 tx=0x7f8e98009c00 comp rx=0 tx=0).stop 2023-11-28T13:06:22.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eb2d07700 1 -- 172.21.15.111:0/3362929243 shutdown_connections 2023-11-28T13:06:22.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eb2d07700 1 --2- 172.21.15.111:0/3362929243 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8eac0cee40 0x7f8eac0cd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eb2d07700 1 --2- 172.21.15.111:0/3362929243 >> v2:172.21.15.149:3300/0 conn(0x7f8eac0b2220 0x7f8eac0cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eb2d07700 1 --2- 172.21.15.111:0/3362929243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eac0cb380 0x7f8eac0ce900 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eb2d07700 1 -- 172.21.15.111:0/3362929243 >> 172.21.15.111:0/3362929243 conn(0x7f8eac09a040 msgr2=0x7f8eac0d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:22.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.580+0000 7f8eb2d07700 1 -- 172.21.15.111:0/3362929243 shutdown_connections 2023-11-28T13:06:22.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.581+0000 7f8eb2d07700 1 -- 172.21.15.111:0/3362929243 wait complete. 2023-11-28T13:06:22.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.582+0000 7f8eb2d07700 1 Processor -- start 2023-11-28T13:06:22.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.582+0000 7f8eb2d07700 1 -- start start 2023-11-28T13:06:22.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb2d07700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8eac0b2220 0x7f8eac158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb2d07700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8eac0cb380 0x7f8eac0ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb2d07700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eac0cee40 0x7f8eac0c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb2d07700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac0ca6a0 con 0x7f8eac0cee40 2023-11-28T13:06:22.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb2d07700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac0c9ad0 con 0x7f8eac0b2220 2023-11-28T13:06:22.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb2d07700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac0c9d70 con 0x7f8eac0cb380 2023-11-28T13:06:22.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb12a4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eac0cee40 0x7f8eac0c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb12a4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eac0cee40 0x7f8eac0c7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38752/0 (socket says 172.21.15.111:38752) 2023-11-28T13:06:22.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb12a4700 1 -- 172.21.15.111:0/2333088591 learned_addr learned my addr 172.21.15.111:0/2333088591 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:22.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb0aa3700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8eac0b2220 0x7f8eac158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eabfff700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8eac0cb380 0x7f8eac0ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb12a4700 1 -- 172.21.15.111:0/2333088591 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8eac0cb380 msgr2=0x7f8eac0ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb12a4700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8eac0cb380 0x7f8eac0ae8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb12a4700 1 -- 172.21.15.111:0/2333088591 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8eac0b2220 msgr2=0x7f8eac158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb12a4700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8eac0b2220 0x7f8eac158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb12a4700 1 -- 172.21.15.111:0/2333088591 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e98009550 con 0x7f8eac0cee40 2023-11-28T13:06:22.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb12a4700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eac0cee40 0x7f8eac0c7e00 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400cd50 tx=0x7f8ea400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:22.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8ea97fa700 1 -- 172.21.15.111:0/2333088591 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea400cea0 con 0x7f8eac0cee40 2023-11-28T13:06:22.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8ea97fa700 1 -- 172.21.15.111:0/2333088591 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea400f480 con 0x7f8eac0cee40 2023-11-28T13:06:22.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb2d07700 1 -- 172.21.15.111:0/2333088591 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eac0cccf0 con 0x7f8eac0cee40 2023-11-28T13:06:22.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8ea97fa700 1 -- 172.21.15.111:0/2333088591 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea400f830 con 0x7f8eac0cee40 2023-11-28T13:06:22.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.583+0000 7f8eb2d07700 1 -- 172.21.15.111:0/2333088591 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eac0cca10 con 0x7f8eac0cee40 2023-11-28T13:06:22.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.584+0000 7f8eb2d07700 1 -- 172.21.15.111:0/2333088591 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eac0ccfd0 con 0x7f8eac0cee40 2023-11-28T13:06:22.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.585+0000 7f8ea97fa700 1 -- 172.21.15.111:0/2333088591 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8ea4009030 con 0x7f8eac0cee40 2023-11-28T13:06:22.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.585+0000 7f8ea97fa700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8e900682f0 0x7f8e9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.585+0000 7f8ea97fa700 1 -- 172.21.15.111:0/2333088591 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f8ea4086380 con 0x7f8eac0cee40 2023-11-28T13:06:22.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.585+0000 7f8eb0aa3700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8e900682f0 0x7f8e9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.586+0000 7f8eb0aa3700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8e900682f0 0x7f8e9006a7a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f8eac1f50b0 tx=0x7f8ea0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:22.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.589+0000 7f8ea97fa700 1 -- 172.21.15.111:0/2333088591 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8ea4052d40 con 0x7f8eac0cee40 2023-11-28T13:06:22.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.767+0000 7f8eb2d07700 1 -- 172.21.15.111:0/2333088591 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8eac067170 con 0x7f8e900682f0 2023-11-28T13:06:22.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.770+0000 7f8ea97fa700 1 -- 172.21.15.111:0/2333088591 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+73355 (secure 0 0 0) 0x7f8eac067170 con 0x7f8e900682f0 2023-11-28T13:06:22.772 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:22.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.772+0000 7f8eb2d07700 1 -- 172.21.15.111:0/2333088591 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8e900682f0 msgr2=0x7f8e9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.772+0000 7f8eb2d07700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8e900682f0 0x7f8e9006a7a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f8eac1f50b0 tx=0x7f8ea0008040 comp rx=0 tx=0).stop 2023-11-28T13:06:22.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.772+0000 7f8eb2d07700 1 -- 172.21.15.111:0/2333088591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eac0cee40 msgr2=0x7f8eac0c7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.772+0000 7f8eb2d07700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eac0cee40 0x7f8eac0c7e00 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400cd50 tx=0x7f8ea400a560 comp rx=0 tx=0).stop 2023-11-28T13:06:22.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.772+0000 7f8eb2d07700 1 -- 172.21.15.111:0/2333088591 shutdown_connections 2023-11-28T13:06:22.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.772+0000 7f8eb2d07700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eac0cee40 0x7f8eac0c7e00 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.772+0000 7f8eb2d07700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f8e900682f0 0x7f8e9006a7a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.772+0000 7f8eb2d07700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f8eac0b2220 0x7f8eac158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.772+0000 7f8eb2d07700 1 --2- 172.21.15.111:0/2333088591 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f8eac0cb380 0x7f8eac0ae8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.772+0000 7f8eb2d07700 1 -- 172.21.15.111:0/2333088591 >> 172.21.15.111:0/2333088591 conn(0x7f8eac09a040 msgr2=0x7f8eac0cdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:22.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.772+0000 7f8eb2d07700 1 -- 172.21.15.111:0/2333088591 shutdown_connections 2023-11-28T13:06:22.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.772+0000 7f8eb2d07700 1 -- 172.21.15.111:0/2333088591 wait complete. 2023-11-28T13:06:22.776 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:06:22.786 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2023-11-28T13:06:22.444550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":151,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":82472,"kb_used_data":8040,"kb_used_omap":0,"kb_used_meta":74368,"kb_avail":754892248,"statfs":{"total":773094113280,"available":773009661952,"internally_reserved":0,"allocated":8232960,"data_stored":4491880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":76152832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12321,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-6,"num_read_kb":-5,"num_write":-72,"num_write_kb":-35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.486326"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:18.976456+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T13:06:18.976456+0000","last_peered":"2023-11-28T13:06:18.976456+0000","last_clean":"2023-11-28T13:06:18.976456+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:06:18.976456+0000","last_undegraded":"2023-11-28T13:06:18.976456+0000","last_fullsized":"2023-11-28T13:06:18.976456+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":58,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.485662+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T13:06:19.485662+0000","last_peered":"2023-11-28T13:06:19.485662+0000","last_clean":"2023-11-28T13:06:19.485662+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:06:19.485662+0000","last_undegraded":"2023-11-28T13:06:19.485662+0000","last_fullsized":"2023-11-28T13:06:19.485662+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.485288+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T13:06:19.485288+0000","last_peered":"2023-11-28T13:06:19.485288+0000","last_clean":"2023-11-28T13:06:19.485288+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:06:19.485288+0000","last_undegraded":"2023-11-28T13:06:19.485288+0000","last_fullsized":"2023-11-28T13:06:19.485288+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":58,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.486018+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T13:06:19.486018+0000","last_peered":"2023-11-28T13:06:19.486018+0000","last_clean":"2023-11-28T13:06:19.486018+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:06:19.486018+0000","last_undegraded":"2023-11-28T13:06:19.486018+0000","last_fullsized":"2023-11-28T13:06:19.486018+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":211,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.485304+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T13:06:19.485304+0000","last_peered":"2023-11-28T13:06:19.485304+0000","last_clean":"2023-11-28T13:06:19.485304+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:06:19.485304+0000","last_undegraded":"2023-11-28T13:06:19.485304+0000","last_fullsized":"2023-11-28T13:06:19.485304+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":59,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.485388+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T13:06:19.485388+0000","last_peered":"2023-11-28T13:06:19.485388+0000","last_clean":"2023-11-28T13:06:19.485388+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:06:19.485388+0000","last_undegraded":"2023-11-28T13:06:19.485388+0000","last_fullsized":"2023-11-28T13:06:19.485388+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":59,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.487237+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T13:06:19.487237+0000","last_peered":"2023-11-28T13:06:19.487237+0000","last_clean":"2023-11-28T13:06:19.487237+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:06:19.487237+0000","last_undegraded":"2023-11-28T13:06:19.487237+0000","last_fullsized":"2023-11-28T13:06:19.487237+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:18.976119+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T13:06:18.976119+0000","last_peered":"2023-11-28T13:06:18.976119+0000","last_clean":"2023-11-28T13:06:18.976119+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:06:18.976119+0000","last_undegraded":"2023-11-28T13:06:18.976119+0000","last_fullsized":"2023-11-28T13:06:18.976119+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:18.978940+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T13:06:18.978940+0000","last_peered":"2023-11-28T13:06:18.978940+0000","last_clean":"2023-11-28T13:06:18.978940+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:06:18.978940+0000","last_undegraded":"2023-11-28T13:06:18.978940+0000","last_fullsized":"2023-11-28T13:06:18.978940+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574942,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11176,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":9728,"kb_avail":94360664,"statfs":{"total":96636764160,"available":96625319936,"internally_reserved":0,"allocated":1474560,"data_stored":1004261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9961472},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.64700000000000002},"min":{"1min":0.51200000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.79800000000000004,"5min":1.194,"15min":3.3279999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.58999999999999997,"15min":0.64800000000000002},"min":{"1min":0.46300000000000002,"5min":0.372,"15min":0.27900000000000003},"max":{"1min":0.77000000000000002,"5min":0.81799999999999995,"15min":3.2919999999999998},"last":0.64500000000000002}]},{"osd":1,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.40300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.52400000000000002},"min":{"1min":0.436,"5min":0.34999999999999998,"15min":0.30499999999999999},"max":{"1min":0.76000000000000001,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.51700000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.56200000000000006},"min":{"1min":0.44600000000000001,"5min":0.35699999999999998,"15min":0.28399999999999997},"max":{"1min":0.751,"5min":1.149,"15min":1.149},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57499999999999996,"15min":0.55200000000000005},"min":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.35699999999999998},"max":{"1min":0.72399999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.66400000000000003}]},{"osd":3,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60299999999999998,"15min":0.56000000000000005},"min":{"1min":0.52000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82899999999999996,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.435},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58799999999999997,"15min":0.57299999999999995},"min":{"1min":0.433,"5min":0.40699999999999997,"15min":0.38600000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.48399999999999999}]},{"osd":4,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47599999999999998,"15min":0.50800000000000001},"min":{"1min":0.35099999999999998,"5min":0.33300000000000002,"15min":0.316},"max":{"1min":0.78400000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.50600000000000001,"15min":0.52600000000000002},"min":{"1min":0.373,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.73899999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67700000000000005}]},{"osd":5,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47799999999999998,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71399999999999997,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53500000000000003,"15min":0.56000000000000005},"min":{"1min":0.36899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.75700000000000001}]},{"osd":6,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52400000000000002,"15min":0.56000000000000005},"min":{"1min":0.38200000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77800000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58899999999999997,"15min":0.60799999999999998},"min":{"1min":0.434,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79900000000000004,"5min":1.274,"15min":1.274},"last":0.81000000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574944,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10020,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361820,"statfs":{"total":96636764160,"available":96626503680,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.437,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.89600000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.52400000000000002},"min":{"1min":0.33900000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72799999999999998,"5min":0.755,"15min":0.80400000000000005},"last":0.59099999999999997}]},{"osd":1,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56699999999999995,"15min":0.53500000000000003},"min":{"1min":0.46100000000000002,"5min":0.32400000000000001,"15min":0.309},"max":{"1min":0.752,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40300000000000002,"5min":0.33000000000000002,"15min":0.29599999999999999},"max":{"1min":0.76700000000000002,"5min":0.86599999999999999,"15min":1.016},"last":0.67800000000000005}]},{"osd":2,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.39900000000000002,"5min":0.34300000000000003,"15min":0.34100000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.91700000000000004},"last":0.623},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.75700000000000001,"5min":0.92400000000000004,"15min":0.96299999999999997},"last":0.502}]},{"osd":3,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.371},"max":{"1min":0.71399999999999997,"5min":0.80700000000000005,"15min":1.5920000000000001},"last":0.439},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59399999999999997,"15min":0.58399999999999996},"min":{"1min":0.504,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76400000000000001,"5min":0.84999999999999998,"15min":0.94099999999999995},"last":0.73499999999999999}]},{"osd":4,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46100000000000002,"15min":0.51200000000000001},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74199999999999999,"5min":0.84199999999999997,"15min":1},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48199999999999998,"15min":0.52200000000000002},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72699999999999998,"5min":0.753,"15min":0.89200000000000002},"last":0.54600000000000004}]},{"osd":5,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.54600000000000004},"min":{"1min":0.36399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.871,"5min":0.871,"15min":1.052},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.59499999999999997},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.81299999999999994,"5min":0.88900000000000001,"15min":1.258},"last":0.754}]},{"osd":7,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.52100000000000002,"15min":0.56200000000000006},"min":{"1min":0.32100000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.79200000000000004,"5min":0.80100000000000005,"15min":0.97599999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56100000000000005,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.89500000000000002,"5min":0.90400000000000003,"15min":1.1060000000000001},"last":0.83999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10020,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361820,"statfs":{"total":96636764160,"available":96626503680,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35099999999999998,"15min":0.33500000000000002},"min":{"1min":0.23799999999999999,"5min":0.224,"15min":0.219},"max":{"1min":0.496,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.36499999999999999},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60299999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.373,"15min":0.34699999999999998},"min":{"1min":0.29199999999999998,"5min":0.22,"15min":0.20599999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40799999999999997,"15min":0.38100000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.25},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.80100000000000005},"last":0.33300000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.434,"15min":0.40400000000000003},"min":{"1min":0.32800000000000001,"5min":0.27200000000000002,"15min":0.245},"max":{"1min":0.57399999999999995,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.41299999999999998,"15min":0.40000000000000002},"min":{"1min":0.27800000000000002,"5min":0.26000000000000001,"15min":0.251},"max":{"1min":0.58399999999999996,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.28699999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.60499999999999998,"15min":0.57999999999999996},"min":{"1min":0.40200000000000002,"5min":0.36199999999999999,"15min":0.34000000000000002},"max":{"1min":0.80000000000000004,"5min":0.91700000000000004,"15min":1.028},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.624,"15min":0.59599999999999997},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.34799999999999998},"max":{"1min":0.76100000000000001,"5min":0.98699999999999999,"15min":1.0660000000000001},"last":0.624}]},{"osd":5,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.59999999999999998},"min":{"1min":0.47299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78100000000000003,"5min":0.94999999999999996,"15min":1.042},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63500000000000001,"15min":0.58499999999999996},"min":{"1min":0.44700000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78300000000000003,"5min":1.05,"15min":1.05},"last":0.60499999999999998}]},{"osd":6,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.626,"15min":0.59599999999999997},"min":{"1min":0.438,"5min":0.40600000000000003,"15min":0.38900000000000001},"max":{"1min":0.80100000000000005,"5min":0.96199999999999997,"15min":1.006},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63400000000000001,"15min":0.60999999999999999},"min":{"1min":0.48699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.81899999999999995,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.53400000000000003}]},{"osd":7,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.64200000000000002,"15min":0.61499999999999999},"min":{"1min":0.51600000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64000000000000001,"15min":0.59599999999999997},"min":{"1min":0.47799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80400000000000005,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.63500000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574943,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11176,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":9728,"kb_avail":94360664,"statfs":{"total":96636764160,"available":96625319936,"internally_reserved":0,"allocated":1474560,"data_stored":1004261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9961472},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.33700000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51300000000000001,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.37},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38200000000000001,"15min":0.35499999999999998},"min":{"1min":0.23000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.50700000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.38300000000000001}]},{"osd":2,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38200000000000001,"15min":0.35799999999999998},"min":{"1min":0.27300000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.48399999999999999,"5min":0.57099999999999995,"15min":0.59399999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41199999999999998,"15min":0.38100000000000001},"min":{"1min":0.28699999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50600000000000001,"5min":0.623,"15min":0.623},"last":0.26700000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38900000000000001,"15min":0.377},"min":{"1min":0.247,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.68000000000000005},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.441,"15min":0.40500000000000003},"min":{"1min":0.28799999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52200000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.29599999999999999}]},{"osd":4,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.622,"15min":0.60699999999999998},"min":{"1min":0.51400000000000001,"5min":0.309,"15min":0.28199999999999997},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.626,"15min":0.58799999999999997},"min":{"1min":0.49299999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83299999999999996,"5min":0.88,"15min":0.88},"last":0.70999999999999996}]},{"osd":5,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.58299999999999996},"min":{"1min":0.42999999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79600000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64100000000000001,"15min":0.625},"min":{"1min":0.441,"5min":0.434,"15min":0.39600000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.68500000000000005}]},{"osd":6,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.64300000000000002,"15min":0.61599999999999999},"min":{"1min":0.51700000000000002,"5min":0.46999999999999997,"15min":0.41199999999999998},"max":{"1min":0.84799999999999998,"5min":0.89800000000000002,"15min":0.91900000000000004},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65300000000000002,"15min":0.627},"min":{"1min":0.50600000000000001,"5min":0.41499999999999998,"15min":0.33900000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.98599999999999999},"last":0.76000000000000001}]},{"osd":7,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.65900000000000003,"15min":0.63900000000000001},"min":{"1min":0.52900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.0149999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65800000000000003,"15min":0.63700000000000001},"min":{"1min":0.45700000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.61499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574943,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9892,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94361948,"statfs":{"total":96636764160,"available":96626634752,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.33000000000000002},"min":{"1min":0.255,"5min":0.22500000000000001,"15min":0.20999999999999999},"max":{"1min":0.48799999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"min":{"1min":0.22600000000000001,"5min":0.21199999999999999,"15min":0.184},"max":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.23200000000000001}]},{"osd":1,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.39800000000000002,"15min":0.38700000000000001},"min":{"1min":0.28799999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.51800000000000002,"5min":0.59499999999999997,"15min":0.67000000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.374},"min":{"1min":0.26300000000000001,"5min":0.25700000000000001,"15min":0.24199999999999999},"max":{"1min":0.56999999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.32100000000000001}]},{"osd":3,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39200000000000002,"15min":0.35599999999999998},"min":{"1min":0.27900000000000003,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":0.505,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.31},{"interface":"front","average":{"1min":0.437,"5min":0.443,"15min":0.42499999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.55700000000000005,"5min":0.70599999999999996,"15min":0.77500000000000002},"last":0.35599999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64000000000000001,"15min":0.60599999999999998},"min":{"1min":0.46200000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85799999999999998,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63,"15min":0.57899999999999996},"min":{"1min":0.44900000000000001,"5min":0.42199999999999999,"15min":0.34300000000000003},"max":{"1min":0.86399999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.42099999999999999}]},{"osd":5,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.60499999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.91900000000000004},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.63400000000000001,"15min":0.60799999999999998},"min":{"1min":0.52500000000000002,"5min":0.41699999999999998,"15min":0.379},"max":{"1min":0.877,"5min":0.877,"15min":1.0589999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64800000000000002,"15min":0.59699999999999998},"min":{"1min":0.503,"5min":0.41999999999999998,"15min":0.36599999999999999},"max":{"1min":0.83899999999999997,"5min":0.89100000000000001,"15min":1.002},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65300000000000002,"15min":0.63300000000000001},"min":{"1min":0.48399999999999999,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.82799999999999996,"5min":0.96799999999999997,"15min":0.97799999999999998},"last":0.56799999999999995}]},{"osd":7,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65600000000000003,"15min":0.63600000000000001},"min":{"1min":0.438,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86199999999999999,"5min":0.93500000000000005,"15min":0.95099999999999996},"last":0.437},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.65800000000000003,"15min":0.61399999999999999},"min":{"1min":0.53000000000000003,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":0.86899999999999999,"5min":0.92400000000000004,"15min":1.0349999999999999},"last":0.51600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574943,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10084,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9216,"kb_avail":94361756,"statfs":{"total":96636764160,"available":96626438144,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9437184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.33500000000000002},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38200000000000001,"15min":0.34499999999999997},"min":{"1min":0.24099999999999999,"5min":0.23200000000000001,"15min":0.20000000000000001},"max":{"1min":0.55200000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.33700000000000002}]},{"osd":1,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.38600000000000001,"15min":0.35399999999999998},"min":{"1min":0.25700000000000001,"5min":0.22,"15min":0.21299999999999999},"max":{"1min":0.53000000000000003,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40500000000000003,"15min":0.38300000000000001},"min":{"1min":0.317,"5min":0.254,"15min":0.23799999999999999},"max":{"1min":0.54600000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34999999999999998}]},{"osd":2,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.45000000000000001,"15min":0.47499999999999998},"min":{"1min":0.29299999999999998,"5min":0.252,"15min":0.251},"max":{"1min":0.58299999999999996,"5min":0.65100000000000002,"15min":2.6869999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45700000000000002,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.30499999999999999,"15min":0.28199999999999997},"max":{"1min":0.60099999999999998,"5min":0.66800000000000004,"15min":2.7160000000000002},"last":0.373}]},{"osd":4,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63600000000000001,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81999999999999995,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63,"15min":0.84599999999999997},"min":{"1min":0.442,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82899999999999996,"5min":0.93899999999999995,"15min":8.2889999999999997},"last":0.69199999999999995}]},{"osd":5,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.48599999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.63300000000000001,"15min":0.66000000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.72999999999999998,"5min":0.88200000000000001,"15min":2.165},"last":0.79900000000000004}]},{"osd":6,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65200000000000002,"15min":0.63600000000000001},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.88400000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.63500000000000001},"min":{"1min":0.53100000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.81299999999999994,"5min":0.95399999999999996,"15min":1.1419999999999999},"last":0.82399999999999995}]},{"osd":7,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66100000000000003,"15min":0.626},"min":{"1min":0.48399999999999999,"5min":0.46000000000000002,"15min":0.45600000000000002},"max":{"1min":0.82399999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66000000000000003,"15min":0.85199999999999998},"min":{"1min":0.45900000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85999999999999999,"5min":0.89600000000000002,"15min":7.0899999999999999},"last":0.61699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":10212,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9344,"kb_avail":94361628,"statfs":{"total":96636764160,"available":96626307072,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9568256},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.66200000000000003,"5min":0.78500000000000003,"15min":1.0049999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56000000000000005,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.66600000000000004,"5min":0.873,"15min":0.873},"last":0.65600000000000003}]},{"osd":1,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55300000000000005,"15min":0.52100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71399999999999997,"5min":0.76600000000000001,"15min":0.95699999999999996},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56999999999999995,"15min":0.56499999999999995},"min":{"1min":0.45500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69999999999999996,"5min":0.77300000000000002,"15min":1.1060000000000001},"last":0.627}]},{"osd":2,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59799999999999998,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68999999999999995,"5min":0.77300000000000002,"15min":0.91800000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60999999999999999,"15min":0.60199999999999998},"min":{"1min":0.48999999999999999,"5min":0.41499999999999998,"15min":0.36399999999999999},"max":{"1min":0.67800000000000005,"5min":0.81100000000000005,"15min":1.069},"last":0.53200000000000003}]},{"osd":3,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55200000000000005,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.70099999999999996,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.46300000000000002,"5min":0.41099999999999998,"15min":0.38900000000000001},"max":{"1min":0.71099999999999997,"5min":0.81599999999999995,"15min":0.879},"last":0.56399999999999995}]},{"osd":5,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47399999999999998,"15min":0.503},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.315},"max":{"1min":0.752,"5min":0.752,"15min":0.86099999999999999},"last":0.752},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.51000000000000001},"min":{"1min":0.371,"5min":0.309,"15min":0.309},"max":{"1min":0.68200000000000005,"5min":0.73799999999999999,"15min":0.80400000000000005},"last":0.46300000000000002}]},{"osd":6,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.49099999999999999,"15min":0.51300000000000001},"min":{"1min":0.35399999999999998,"5min":0.32500000000000001,"15min":0.30499999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.84299999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53000000000000003,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.35199999999999998,"15min":0.34499999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.96399999999999997},"last":0.78300000000000003}]},{"osd":7,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73999999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.36599999999999999},"max":{"1min":0.80400000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.79400000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574943,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9892,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94361948,"statfs":{"total":96636764160,"available":96626634752,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.378,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72699999999999998,"5min":0.74399999999999999,"15min":1.819},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.53600000000000003},"min":{"1min":0.40500000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.66900000000000004}]},{"osd":1,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.53300000000000003},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.29799999999999999},"max":{"1min":0.70999999999999996,"5min":0.76500000000000001,"15min":0.90000000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56000000000000005,"15min":0.52700000000000002},"min":{"1min":0.315,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70799999999999996,"5min":0.77400000000000002,"15min":0.78900000000000003},"last":0.51400000000000001}]},{"osd":2,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.59399999999999997},"min":{"1min":0.42499999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.74099999999999999,"5min":0.84899999999999998,"15min":1.0089999999999999},"last":0.63},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58299999999999996,"15min":0.623},"min":{"1min":0.45600000000000002,"5min":0.38200000000000001,"15min":0.35099999999999998},"max":{"1min":0.68799999999999994,"5min":0.78100000000000003,"15min":2.4020000000000001},"last":0.64100000000000001}]},{"osd":3,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57999999999999996,"15min":0.55800000000000005},"min":{"1min":0.45400000000000001,"5min":0.35199999999999998,"15min":0.33600000000000002},"max":{"1min":0.752,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57299999999999995,"15min":0.64500000000000002},"min":{"1min":0.44600000000000001,"5min":0.36599999999999999,"15min":0.35799999999999998},"max":{"1min":0.69699999999999995,"5min":0.81999999999999995,"15min":2.702},"last":0.56799999999999995}]},{"osd":4,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46500000000000002,"15min":0.56499999999999995},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.27300000000000002},"max":{"1min":0.76100000000000001,"5min":0.76800000000000002,"15min":3.0739999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.29899999999999999},"max":{"1min":0.65700000000000003,"5min":0.748,"15min":0.93000000000000005},"last":0.61699999999999999}]},{"osd":6,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48099999999999998,"15min":0.54000000000000004},"min":{"1min":0.38800000000000001,"5min":0.33500000000000002,"15min":0.28699999999999998},"max":{"1min":0.60799999999999998,"5min":0.71999999999999997,"15min":1.1819999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.51300000000000001,"15min":0.65200000000000002},"min":{"1min":0.35399999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":3.7679999999999998},"last":0.60499999999999998}]},{"osd":7,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.57399999999999995},"min":{"1min":0.377,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.65400000000000003,"5min":0.79300000000000004,"15min":0.83699999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55400000000000005,"15min":0.627},"min":{"1min":0.40999999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.748,"5min":0.78000000000000003,"15min":1.2230000000000001},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:06:22.789 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-28T13:06:22.789 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:06:22.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.907+0000 7f565924f700 1 Processor -- start 2023-11-28T13:06:22.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.907+0000 7f565924f700 1 -- start start 2023-11-28T13:06:22.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.908+0000 7f565924f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f56540ca680 0x7f56540cca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.908+0000 7f565924f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565415aab0 con 0x7f565415a6e0 2023-11-28T13:06:22.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.908+0000 7f565924f700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565406a480 con 0x7f56540ca210 2023-11-28T13:06:22.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.908+0000 7f565924f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56540ccf80 con 0x7f56540ca680 2023-11-28T13:06:22.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.908+0000 7f565359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f56540ca680 0x7f56540cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.908+0000 7f5652d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f565415a6e0 0x7f565406b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39534/0 (socket says 172.21.15.111:39534) 2023-11-28T13:06:22.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.908+0000 7f5652d9d700 1 -- 172.21.15.111:0/806147534 learned_addr learned my addr 172.21.15.111:0/806147534 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:22.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.908+0000 7f565359e700 1 -- 172.21.15.111:0/806147534 >> v1:172.21.15.149:6789/0 conn(0x7f56540ca210 legacy=0x7f56540cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:22.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.908+0000 7f565359e700 1 -- 172.21.15.111:0/806147534 >> v1:172.21.15.111:6789/0 conn(0x7f565415a6e0 legacy=0x7f565406b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.908+0000 7f5651d9b700 1 -- 172.21.15.111:0/806147534 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274045055 0 0) 0x7f565415aab0 con 0x7f565415a6e0 2023-11-28T13:06:22.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.908+0000 7f565359e700 1 -- 172.21.15.111:0/806147534 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565406a740 con 0x7f56540ca680 2023-11-28T13:06:22.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.909+0000 7f565359e700 1 --2- 172.21.15.111:0/806147534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56540ca680 0x7f56540cca40 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f564c009d90 tx=0x7f564c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b96f6c4d9f0fb9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:22.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.909+0000 7f5651d9b700 1 -- 172.21.15.111:0/806147534 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c00e070 con 0x7f56540ca680 2023-11-28T13:06:22.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.909+0000 7f5651d9b700 1 -- 172.21.15.111:0/806147534 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564c00bb90 con 0x7f56540ca680 2023-11-28T13:06:22.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.909+0000 7f5651d9b700 1 -- 172.21.15.111:0/806147534 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c003a20 con 0x7f56540ca680 2023-11-28T13:06:22.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.909+0000 7f565924f700 1 -- 172.21.15.111:0/806147534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56540ca680 msgr2=0x7f56540cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.909+0000 7f565924f700 1 --2- 172.21.15.111:0/806147534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56540ca680 0x7f56540cca40 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f564c009d90 tx=0x7f564c00b330 comp rx=0 tx=0).stop 2023-11-28T13:06:22.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.909+0000 7f565924f700 1 -- 172.21.15.111:0/806147534 shutdown_connections 2023-11-28T13:06:22.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.909+0000 7f565924f700 1 --2- 172.21.15.111:0/806147534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56540ca680 0x7f56540cca40 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.909+0000 7f565924f700 1 -- 172.21.15.111:0/806147534 >> 172.21.15.111:0/806147534 conn(0x7f565406db10 msgr2=0x7f565406df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:22.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.909+0000 7f565924f700 1 -- 172.21.15.111:0/806147534 shutdown_connections 2023-11-28T13:06:22.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.909+0000 7f565924f700 1 -- 172.21.15.111:0/806147534 wait complete. 2023-11-28T13:06:22.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.911+0000 7f565924f700 1 Processor -- start 2023-11-28T13:06:22.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.911+0000 7f565924f700 1 -- start start 2023-11-28T13:06:22.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.911+0000 7f565924f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f56540ca210 0x7f56541554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565924f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56540ca680 0x7f56540c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565924f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f565415a6e0 0x7f56540b16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565924f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56540c7ef0 con 0x7f56540ca680 2023-11-28T13:06:22.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565924f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56540c81d0 con 0x7f565415a6e0 2023-11-28T13:06:22.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565924f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56540c84b0 con 0x7f56540ca210 2023-11-28T13:06:22.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56540ca680 0x7f56540c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56540ca680 0x7f56540c7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38764/0 (socket says 172.21.15.111:38764) 2023-11-28T13:06:22.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565259c700 1 -- 172.21.15.111:0/1988318302 learned_addr learned my addr 172.21.15.111:0/1988318302 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:22.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f5652d9d700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f56540ca210 0x7f56541554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565359e700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f565415a6e0 0x7f56540b16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565259c700 1 -- 172.21.15.111:0/1988318302 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f56540ca210 msgr2=0x7f56541554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565259c700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f56540ca210 0x7f56541554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565259c700 1 -- 172.21.15.111:0/1988318302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f565415a6e0 msgr2=0x7f56540b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:22.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565259c700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f565415a6e0 0x7f56540b16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:22.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565259c700 1 -- 172.21.15.111:0/1988318302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c009a70 con 0x7f56540ca680 2023-11-28T13:06:22.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565259c700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56540ca680 0x7f56540c7650 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f564400b4a0 tx=0x7f564400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:22.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f563b7fe700 1 -- 172.21.15.111:0/1988318302 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644011830 con 0x7f56540ca680 2023-11-28T13:06:22.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f563b7fe700 1 -- 172.21.15.111:0/1988318302 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5644011990 con 0x7f56540ca680 2023-11-28T13:06:22.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565924f700 1 -- 172.21.15.111:0/1988318302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565406ad00 con 0x7f56540ca680 2023-11-28T13:06:22.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f563b7fe700 1 -- 172.21.15.111:0/1988318302 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644011d40 con 0x7f56540ca680 2023-11-28T13:06:22.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.912+0000 7f565924f700 1 -- 172.21.15.111:0/1988318302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565406aa20 con 0x7f56540ca680 2023-11-28T13:06:22.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.913+0000 7f565924f700 1 -- 172.21.15.111:0/1988318302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565406afe0 con 0x7f56540ca680 2023-11-28T13:06:22.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.914+0000 7f563b7fe700 1 -- 172.21.15.111:0/1988318302 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f564400f050 con 0x7f56540ca680 2023-11-28T13:06:22.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.914+0000 7f563b7fe700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56400682f0 0x7f564006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:22.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.914+0000 7f563b7fe700 1 -- 172.21.15.111:0/1988318302 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f56440856b0 con 0x7f56540ca680 2023-11-28T13:06:22.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.914+0000 7f5652d9d700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56400682f0 0x7f564006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:22.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.914+0000 7f5652d9d700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56400682f0 0x7f564006a7a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5648000f60 tx=0x7f5648007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:22.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:22.918+0000 7f563b7fe700 1 -- 172.21.15.111:0/1988318302 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5644052070 con 0x7f56540ca680 2023-11-28T13:06:23.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.098+0000 7f565924f700 1 -- 172.21.15.111:0/1988318302 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5654067170 con 0x7f56400682f0 2023-11-28T13:06:23.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.101+0000 7f563b7fe700 1 -- 172.21.15.111:0/1988318302 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+73355 (secure 0 0 0) 0x7f5654067170 con 0x7f56400682f0 2023-11-28T13:06:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:23.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.103+0000 7f565924f700 1 -- 172.21.15.111:0/1988318302 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56400682f0 msgr2=0x7f564006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:23.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.103+0000 7f565924f700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56400682f0 0x7f564006a7a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5648000f60 tx=0x7f5648007c00 comp rx=0 tx=0).stop 2023-11-28T13:06:23.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.103+0000 7f565924f700 1 -- 172.21.15.111:0/1988318302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56540ca680 msgr2=0x7f56540c7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:23.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.103+0000 7f565924f700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56540ca680 0x7f56540c7650 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f564400b4a0 tx=0x7f564400b860 comp rx=0 tx=0).stop 2023-11-28T13:06:23.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.103+0000 7f565924f700 1 -- 172.21.15.111:0/1988318302 shutdown_connections 2023-11-28T13:06:23.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.103+0000 7f565924f700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f56400682f0 0x7f564006a7a0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:23.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.103+0000 7f565924f700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f565415a6e0 0x7f56540b16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:23.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.103+0000 7f565924f700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f56540ca210 0x7f56541554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:23.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.103+0000 7f565924f700 1 --2- 172.21.15.111:0/1988318302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56540ca680 0x7f56540c7650 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:23.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.103+0000 7f565924f700 1 -- 172.21.15.111:0/1988318302 >> 172.21.15.111:0/1988318302 conn(0x7f565406db10 msgr2=0x7f56540ae740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:23.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.103+0000 7f565924f700 1 -- 172.21.15.111:0/1988318302 shutdown_connections 2023-11-28T13:06:23.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.103+0000 7f565924f700 1 -- 172.21.15.111:0/1988318302 wait complete. 2023-11-28T13:06:23.107 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:06:23.117 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2023-11-28T13:06:22.444550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":151,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":82472,"kb_used_data":8040,"kb_used_omap":0,"kb_used_meta":74368,"kb_avail":754892248,"statfs":{"total":773094113280,"available":773009661952,"internally_reserved":0,"allocated":8232960,"data_stored":4491880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":76152832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12321,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-6,"num_read_kb":-5,"num_write":-72,"num_write_kb":-35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.486326"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:18.976456+0000","last_change":"2023-11-28T12:58:50.572462+0000","last_active":"2023-11-28T13:06:18.976456+0000","last_peered":"2023-11-28T13:06:18.976456+0000","last_clean":"2023-11-28T13:06:18.976456+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:06:18.976456+0000","last_undegraded":"2023-11-28T13:06:18.976456+0000","last_fullsized":"2023-11-28T13:06:18.976456+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":58,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.485662+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T13:06:19.485662+0000","last_peered":"2023-11-28T13:06:19.485662+0000","last_clean":"2023-11-28T13:06:19.485662+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:06:19.485662+0000","last_undegraded":"2023-11-28T13:06:19.485662+0000","last_fullsized":"2023-11-28T13:06:19.485662+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.485288+0000","last_change":"2023-11-28T12:58:50.648097+0000","last_active":"2023-11-28T13:06:19.485288+0000","last_peered":"2023-11-28T13:06:19.485288+0000","last_clean":"2023-11-28T13:06:19.485288+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:06:19.485288+0000","last_undegraded":"2023-11-28T13:06:19.485288+0000","last_fullsized":"2023-11-28T13:06:19.485288+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":58,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.486018+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T13:06:19.486018+0000","last_peered":"2023-11-28T13:06:19.486018+0000","last_clean":"2023-11-28T13:06:19.486018+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:06:19.486018+0000","last_undegraded":"2023-11-28T13:06:19.486018+0000","last_fullsized":"2023-11-28T13:06:19.486018+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":211,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.485304+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T13:06:19.485304+0000","last_peered":"2023-11-28T13:06:19.485304+0000","last_clean":"2023-11-28T13:06:19.485304+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:06:19.485304+0000","last_undegraded":"2023-11-28T13:06:19.485304+0000","last_fullsized":"2023-11-28T13:06:19.485304+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":59,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.485388+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T13:06:19.485388+0000","last_peered":"2023-11-28T13:06:19.485388+0000","last_clean":"2023-11-28T13:06:19.485388+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:06:19.485388+0000","last_undegraded":"2023-11-28T13:06:19.485388+0000","last_fullsized":"2023-11-28T13:06:19.485388+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":59,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.487237+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T13:06:19.487237+0000","last_peered":"2023-11-28T13:06:19.487237+0000","last_clean":"2023-11-28T13:06:19.487237+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:06:19.487237+0000","last_undegraded":"2023-11-28T13:06:19.487237+0000","last_fullsized":"2023-11-28T13:06:19.487237+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:18.976119+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T13:06:18.976119+0000","last_peered":"2023-11-28T13:06:18.976119+0000","last_clean":"2023-11-28T13:06:18.976119+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:06:18.976119+0000","last_undegraded":"2023-11-28T13:06:18.976119+0000","last_fullsized":"2023-11-28T13:06:18.976119+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:12:12.298416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:18.978940+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T13:06:18.978940+0000","last_peered":"2023-11-28T13:06:18.978940+0000","last_clean":"2023-11-28T13:06:18.978940+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:06:18.978940+0000","last_undegraded":"2023-11-28T13:06:18.978940+0000","last_fullsized":"2023-11-28T13:06:18.978940+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574942,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11176,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":9728,"kb_avail":94360664,"statfs":{"total":96636764160,"available":96625319936,"internally_reserved":0,"allocated":1474560,"data_stored":1004261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9961472},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.64700000000000002},"min":{"1min":0.51200000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.79800000000000004,"5min":1.194,"15min":3.3279999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.58999999999999997,"15min":0.64800000000000002},"min":{"1min":0.46300000000000002,"5min":0.372,"15min":0.27900000000000003},"max":{"1min":0.77000000000000002,"5min":0.81799999999999995,"15min":3.2919999999999998},"last":0.64500000000000002}]},{"osd":1,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.40300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.52400000000000002},"min":{"1min":0.436,"5min":0.34999999999999998,"15min":0.30499999999999999},"max":{"1min":0.76000000000000001,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.51700000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.56200000000000006},"min":{"1min":0.44600000000000001,"5min":0.35699999999999998,"15min":0.28399999999999997},"max":{"1min":0.751,"5min":1.149,"15min":1.149},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57499999999999996,"15min":0.55200000000000005},"min":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.35699999999999998},"max":{"1min":0.72399999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.66400000000000003}]},{"osd":3,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60299999999999998,"15min":0.56000000000000005},"min":{"1min":0.52000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82899999999999996,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.435},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58799999999999997,"15min":0.57299999999999995},"min":{"1min":0.433,"5min":0.40699999999999997,"15min":0.38600000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.48399999999999999}]},{"osd":4,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47599999999999998,"15min":0.50800000000000001},"min":{"1min":0.35099999999999998,"5min":0.33300000000000002,"15min":0.316},"max":{"1min":0.78400000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.50600000000000001,"15min":0.52600000000000002},"min":{"1min":0.373,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.73899999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67700000000000005}]},{"osd":5,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47799999999999998,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71399999999999997,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53500000000000003,"15min":0.56000000000000005},"min":{"1min":0.36899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.75700000000000001}]},{"osd":6,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52400000000000002,"15min":0.56000000000000005},"min":{"1min":0.38200000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77800000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58899999999999997,"15min":0.60799999999999998},"min":{"1min":0.434,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79900000000000004,"5min":1.274,"15min":1.274},"last":0.81000000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574944,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10020,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361820,"statfs":{"total":96636764160,"available":96626503680,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.437,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.89600000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.52400000000000002},"min":{"1min":0.33900000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72799999999999998,"5min":0.755,"15min":0.80400000000000005},"last":0.59099999999999997}]},{"osd":1,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56699999999999995,"15min":0.53500000000000003},"min":{"1min":0.46100000000000002,"5min":0.32400000000000001,"15min":0.309},"max":{"1min":0.752,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40300000000000002,"5min":0.33000000000000002,"15min":0.29599999999999999},"max":{"1min":0.76700000000000002,"5min":0.86599999999999999,"15min":1.016},"last":0.67800000000000005}]},{"osd":2,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.39900000000000002,"5min":0.34300000000000003,"15min":0.34100000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.91700000000000004},"last":0.623},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.75700000000000001,"5min":0.92400000000000004,"15min":0.96299999999999997},"last":0.502}]},{"osd":3,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.371},"max":{"1min":0.71399999999999997,"5min":0.80700000000000005,"15min":1.5920000000000001},"last":0.439},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59399999999999997,"15min":0.58399999999999996},"min":{"1min":0.504,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76400000000000001,"5min":0.84999999999999998,"15min":0.94099999999999995},"last":0.73499999999999999}]},{"osd":4,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46100000000000002,"15min":0.51200000000000001},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74199999999999999,"5min":0.84199999999999997,"15min":1},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48199999999999998,"15min":0.52200000000000002},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72699999999999998,"5min":0.753,"15min":0.89200000000000002},"last":0.54600000000000004}]},{"osd":5,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.54600000000000004},"min":{"1min":0.36399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.871,"5min":0.871,"15min":1.052},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.59499999999999997},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.81299999999999994,"5min":0.88900000000000001,"15min":1.258},"last":0.754}]},{"osd":7,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.52100000000000002,"15min":0.56200000000000006},"min":{"1min":0.32100000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.79200000000000004,"5min":0.80100000000000005,"15min":0.97599999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56100000000000005,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.89500000000000002,"5min":0.90400000000000003,"15min":1.1060000000000001},"last":0.83999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10020,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361820,"statfs":{"total":96636764160,"available":96626503680,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35099999999999998,"15min":0.33500000000000002},"min":{"1min":0.23799999999999999,"5min":0.224,"15min":0.219},"max":{"1min":0.496,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.36499999999999999},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60299999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.373,"15min":0.34699999999999998},"min":{"1min":0.29199999999999998,"5min":0.22,"15min":0.20599999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40799999999999997,"15min":0.38100000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.25},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.80100000000000005},"last":0.33300000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.434,"15min":0.40400000000000003},"min":{"1min":0.32800000000000001,"5min":0.27200000000000002,"15min":0.245},"max":{"1min":0.57399999999999995,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.41299999999999998,"15min":0.40000000000000002},"min":{"1min":0.27800000000000002,"5min":0.26000000000000001,"15min":0.251},"max":{"1min":0.58399999999999996,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.28699999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.60499999999999998,"15min":0.57999999999999996},"min":{"1min":0.40200000000000002,"5min":0.36199999999999999,"15min":0.34000000000000002},"max":{"1min":0.80000000000000004,"5min":0.91700000000000004,"15min":1.028},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.624,"15min":0.59599999999999997},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.34799999999999998},"max":{"1min":0.76100000000000001,"5min":0.98699999999999999,"15min":1.0660000000000001},"last":0.624}]},{"osd":5,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.59999999999999998},"min":{"1min":0.47299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78100000000000003,"5min":0.94999999999999996,"15min":1.042},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63500000000000001,"15min":0.58499999999999996},"min":{"1min":0.44700000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78300000000000003,"5min":1.05,"15min":1.05},"last":0.60499999999999998}]},{"osd":6,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.626,"15min":0.59599999999999997},"min":{"1min":0.438,"5min":0.40600000000000003,"15min":0.38900000000000001},"max":{"1min":0.80100000000000005,"5min":0.96199999999999997,"15min":1.006},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63400000000000001,"15min":0.60999999999999999},"min":{"1min":0.48699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.81899999999999995,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.53400000000000003}]},{"osd":7,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.64200000000000002,"15min":0.61499999999999999},"min":{"1min":0.51600000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64000000000000001,"15min":0.59599999999999997},"min":{"1min":0.47799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80400000000000005,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.63500000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574943,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11176,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":9728,"kb_avail":94360664,"statfs":{"total":96636764160,"available":96625319936,"internally_reserved":0,"allocated":1474560,"data_stored":1004261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9961472},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.33700000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51300000000000001,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.37},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38200000000000001,"15min":0.35499999999999998},"min":{"1min":0.23000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.50700000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.38300000000000001}]},{"osd":2,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38200000000000001,"15min":0.35799999999999998},"min":{"1min":0.27300000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.48399999999999999,"5min":0.57099999999999995,"15min":0.59399999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41199999999999998,"15min":0.38100000000000001},"min":{"1min":0.28699999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50600000000000001,"5min":0.623,"15min":0.623},"last":0.26700000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38900000000000001,"15min":0.377},"min":{"1min":0.247,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.68000000000000005},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.441,"15min":0.40500000000000003},"min":{"1min":0.28799999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52200000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.29599999999999999}]},{"osd":4,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.622,"15min":0.60699999999999998},"min":{"1min":0.51400000000000001,"5min":0.309,"15min":0.28199999999999997},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.626,"15min":0.58799999999999997},"min":{"1min":0.49299999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83299999999999996,"5min":0.88,"15min":0.88},"last":0.70999999999999996}]},{"osd":5,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.58299999999999996},"min":{"1min":0.42999999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79600000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64100000000000001,"15min":0.625},"min":{"1min":0.441,"5min":0.434,"15min":0.39600000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.68500000000000005}]},{"osd":6,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.64300000000000002,"15min":0.61599999999999999},"min":{"1min":0.51700000000000002,"5min":0.46999999999999997,"15min":0.41199999999999998},"max":{"1min":0.84799999999999998,"5min":0.89800000000000002,"15min":0.91900000000000004},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65300000000000002,"15min":0.627},"min":{"1min":0.50600000000000001,"5min":0.41499999999999998,"15min":0.33900000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.98599999999999999},"last":0.76000000000000001}]},{"osd":7,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.65900000000000003,"15min":0.63900000000000001},"min":{"1min":0.52900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.0149999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65800000000000003,"15min":0.63700000000000001},"min":{"1min":0.45700000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.61499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574943,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9892,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94361948,"statfs":{"total":96636764160,"available":96626634752,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.33000000000000002},"min":{"1min":0.255,"5min":0.22500000000000001,"15min":0.20999999999999999},"max":{"1min":0.48799999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"min":{"1min":0.22600000000000001,"5min":0.21199999999999999,"15min":0.184},"max":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.23200000000000001}]},{"osd":1,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.39800000000000002,"15min":0.38700000000000001},"min":{"1min":0.28799999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.51800000000000002,"5min":0.59499999999999997,"15min":0.67000000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.374},"min":{"1min":0.26300000000000001,"5min":0.25700000000000001,"15min":0.24199999999999999},"max":{"1min":0.56999999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.32100000000000001}]},{"osd":3,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39200000000000002,"15min":0.35599999999999998},"min":{"1min":0.27900000000000003,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":0.505,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.31},{"interface":"front","average":{"1min":0.437,"5min":0.443,"15min":0.42499999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.55700000000000005,"5min":0.70599999999999996,"15min":0.77500000000000002},"last":0.35599999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64000000000000001,"15min":0.60599999999999998},"min":{"1min":0.46200000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85799999999999998,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63,"15min":0.57899999999999996},"min":{"1min":0.44900000000000001,"5min":0.42199999999999999,"15min":0.34300000000000003},"max":{"1min":0.86399999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.42099999999999999}]},{"osd":5,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.60499999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.91900000000000004},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.63400000000000001,"15min":0.60799999999999998},"min":{"1min":0.52500000000000002,"5min":0.41699999999999998,"15min":0.379},"max":{"1min":0.877,"5min":0.877,"15min":1.0589999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64800000000000002,"15min":0.59699999999999998},"min":{"1min":0.503,"5min":0.41999999999999998,"15min":0.36599999999999999},"max":{"1min":0.83899999999999997,"5min":0.89100000000000001,"15min":1.002},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65300000000000002,"15min":0.63300000000000001},"min":{"1min":0.48399999999999999,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.82799999999999996,"5min":0.96799999999999997,"15min":0.97799999999999998},"last":0.56799999999999995}]},{"osd":7,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65600000000000003,"15min":0.63600000000000001},"min":{"1min":0.438,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86199999999999999,"5min":0.93500000000000005,"15min":0.95099999999999996},"last":0.437},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.65800000000000003,"15min":0.61399999999999999},"min":{"1min":0.53000000000000003,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":0.86899999999999999,"5min":0.92400000000000004,"15min":1.0349999999999999},"last":0.51600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574943,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10084,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9216,"kb_avail":94361756,"statfs":{"total":96636764160,"available":96626438144,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9437184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.33500000000000002},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38200000000000001,"15min":0.34499999999999997},"min":{"1min":0.24099999999999999,"5min":0.23200000000000001,"15min":0.20000000000000001},"max":{"1min":0.55200000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.33700000000000002}]},{"osd":1,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.38600000000000001,"15min":0.35399999999999998},"min":{"1min":0.25700000000000001,"5min":0.22,"15min":0.21299999999999999},"max":{"1min":0.53000000000000003,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40500000000000003,"15min":0.38300000000000001},"min":{"1min":0.317,"5min":0.254,"15min":0.23799999999999999},"max":{"1min":0.54600000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34999999999999998}]},{"osd":2,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.45000000000000001,"15min":0.47499999999999998},"min":{"1min":0.29299999999999998,"5min":0.252,"15min":0.251},"max":{"1min":0.58299999999999996,"5min":0.65100000000000002,"15min":2.6869999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45700000000000002,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.30499999999999999,"15min":0.28199999999999997},"max":{"1min":0.60099999999999998,"5min":0.66800000000000004,"15min":2.7160000000000002},"last":0.373}]},{"osd":4,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63600000000000001,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81999999999999995,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63,"15min":0.84599999999999997},"min":{"1min":0.442,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82899999999999996,"5min":0.93899999999999995,"15min":8.2889999999999997},"last":0.69199999999999995}]},{"osd":5,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.48599999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.63300000000000001,"15min":0.66000000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.72999999999999998,"5min":0.88200000000000001,"15min":2.165},"last":0.79900000000000004}]},{"osd":6,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65200000000000002,"15min":0.63600000000000001},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.88400000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.63500000000000001},"min":{"1min":0.53100000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.81299999999999994,"5min":0.95399999999999996,"15min":1.1419999999999999},"last":0.82399999999999995}]},{"osd":7,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66100000000000003,"15min":0.626},"min":{"1min":0.48399999999999999,"5min":0.46000000000000002,"15min":0.45600000000000002},"max":{"1min":0.82399999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66000000000000003,"15min":0.85199999999999998},"min":{"1min":0.45900000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85999999999999999,"5min":0.89600000000000002,"15min":7.0899999999999999},"last":0.61699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574941,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":10212,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9344,"kb_avail":94361628,"statfs":{"total":96636764160,"available":96626307072,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9568256},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.66200000000000003,"5min":0.78500000000000003,"15min":1.0049999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56000000000000005,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.66600000000000004,"5min":0.873,"15min":0.873},"last":0.65600000000000003}]},{"osd":1,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55300000000000005,"15min":0.52100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71399999999999997,"5min":0.76600000000000001,"15min":0.95699999999999996},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56999999999999995,"15min":0.56499999999999995},"min":{"1min":0.45500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69999999999999996,"5min":0.77300000000000002,"15min":1.1060000000000001},"last":0.627}]},{"osd":2,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59799999999999998,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68999999999999995,"5min":0.77300000000000002,"15min":0.91800000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60999999999999999,"15min":0.60199999999999998},"min":{"1min":0.48999999999999999,"5min":0.41499999999999998,"15min":0.36399999999999999},"max":{"1min":0.67800000000000005,"5min":0.81100000000000005,"15min":1.069},"last":0.53200000000000003}]},{"osd":3,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55200000000000005,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.70099999999999996,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.46300000000000002,"5min":0.41099999999999998,"15min":0.38900000000000001},"max":{"1min":0.71099999999999997,"5min":0.81599999999999995,"15min":0.879},"last":0.56399999999999995}]},{"osd":5,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47399999999999998,"15min":0.503},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.315},"max":{"1min":0.752,"5min":0.752,"15min":0.86099999999999999},"last":0.752},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.51000000000000001},"min":{"1min":0.371,"5min":0.309,"15min":0.309},"max":{"1min":0.68200000000000005,"5min":0.73799999999999999,"15min":0.80400000000000005},"last":0.46300000000000002}]},{"osd":6,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.49099999999999999,"15min":0.51300000000000001},"min":{"1min":0.35399999999999998,"5min":0.32500000000000001,"15min":0.30499999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.84299999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53000000000000003,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.35199999999999998,"15min":0.34499999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.96399999999999997},"last":0.78300000000000003}]},{"osd":7,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73999999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.36599999999999999},"max":{"1min":0.80400000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.79400000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574943,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9892,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94361948,"statfs":{"total":96636764160,"available":96626634752,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.378,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72699999999999998,"5min":0.74399999999999999,"15min":1.819},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.53600000000000003},"min":{"1min":0.40500000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.66900000000000004}]},{"osd":1,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.53300000000000003},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.29799999999999999},"max":{"1min":0.70999999999999996,"5min":0.76500000000000001,"15min":0.90000000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56000000000000005,"15min":0.52700000000000002},"min":{"1min":0.315,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70799999999999996,"5min":0.77400000000000002,"15min":0.78900000000000003},"last":0.51400000000000001}]},{"osd":2,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.59399999999999997},"min":{"1min":0.42499999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.74099999999999999,"5min":0.84899999999999998,"15min":1.0089999999999999},"last":0.63},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58299999999999996,"15min":0.623},"min":{"1min":0.45600000000000002,"5min":0.38200000000000001,"15min":0.35099999999999998},"max":{"1min":0.68799999999999994,"5min":0.78100000000000003,"15min":2.4020000000000001},"last":0.64100000000000001}]},{"osd":3,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57999999999999996,"15min":0.55800000000000005},"min":{"1min":0.45400000000000001,"5min":0.35199999999999998,"15min":0.33600000000000002},"max":{"1min":0.752,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57299999999999995,"15min":0.64500000000000002},"min":{"1min":0.44600000000000001,"5min":0.36599999999999999,"15min":0.35799999999999998},"max":{"1min":0.69699999999999995,"5min":0.81999999999999995,"15min":2.702},"last":0.56799999999999995}]},{"osd":4,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46500000000000002,"15min":0.56499999999999995},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.27300000000000002},"max":{"1min":0.76100000000000001,"5min":0.76800000000000002,"15min":3.0739999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.29899999999999999},"max":{"1min":0.65700000000000003,"5min":0.748,"15min":0.93000000000000005},"last":0.61699999999999999}]},{"osd":6,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48099999999999998,"15min":0.54000000000000004},"min":{"1min":0.38800000000000001,"5min":0.33500000000000002,"15min":0.28699999999999998},"max":{"1min":0.60799999999999998,"5min":0.71999999999999997,"15min":1.1819999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.51300000000000001,"15min":0.65200000000000002},"min":{"1min":0.35399999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":3.7679999999999998},"last":0.60499999999999998}]},{"osd":7,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.57399999999999995},"min":{"1min":0.377,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.65400000000000003,"5min":0.79300000000000004,"15min":0.83699999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55400000000000005,"15min":0.627},"min":{"1min":0.40999999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.748,"5min":0.78000000000000003,"15min":1.2230000000000001},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:06:23.119 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T13:06:23.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.237+0000 7f99170fe700 1 Processor -- start 2023-11-28T13:06:23.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.237+0000 7f99170fe700 1 -- start start 2023-11-28T13:06:23.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.237+0000 7f99170fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f9910072300 0x7f99100712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:23.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.237+0000 7f99170fe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f991015cc50 0x7f991006b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:23.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.237+0000 7f99170fe700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99100717f0 con 0x7f99100cceb0 2023-11-28T13:06:23.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.237+0000 7f99170fe700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99100726d0 con 0x7f9910072300 2023-11-28T13:06:23.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.237+0000 7f99170fe700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991006fb90 con 0x7f991015cc50 2023-11-28T13:06:23.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f991569b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f99100cceb0 0x7f991006a610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39552/0 (socket says 172.21.15.111:39552) 2023-11-28T13:06:23.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f991569b700 1 -- 172.21.15.111:0/3362722066 learned_addr learned my addr 172.21.15.111:0/3362722066 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:23.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f9914e9a700 1 --2- 172.21.15.111:0/3362722066 >> v2:172.21.15.149:3300/0 conn(0x7f9910072300 0x7f99100712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:23.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f990ffff700 1 --2- 172.21.15.111:0/3362722066 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f991015cc50 0x7f991006b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:23.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f990f7fe700 1 -- 172.21.15.111:0/3362722066 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226993914 0 0) 0x7f99100717f0 con 0x7f99100cceb0 2023-11-28T13:06:23.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f990f7fe700 1 -- 172.21.15.111:0/3362722066 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99000035b0 con 0x7f99100cceb0 2023-11-28T13:06:23.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f9914e9a700 1 -- 172.21.15.111:0/3362722066 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f991015cc50 msgr2=0x7f991006b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:23.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f9914e9a700 1 --2- 172.21.15.111:0/3362722066 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f991015cc50 0x7f991006b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:23.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f9914e9a700 1 -- 172.21.15.111:0/3362722066 >> v1:172.21.15.111:6789/0 conn(0x7f99100cceb0 legacy=0x7f991006a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:23.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f9914e9a700 1 -- 172.21.15.111:0/3362722066 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991015d0f0 con 0x7f9910072300 2023-11-28T13:06:23.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f9914e9a700 1 --2- 172.21.15.111:0/3362722066 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9910072300 0x7f99100712b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9904009840 tx=0x7f9904009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e496175961590755 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:23.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f990f7fe700 1 -- 172.21.15.111:0/3362722066 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904011830 con 0x7f9910072300 2023-11-28T13:06:23.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f990f7fe700 1 -- 172.21.15.111:0/3362722066 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9904011990 con 0x7f9910072300 2023-11-28T13:06:23.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.238+0000 7f990f7fe700 1 -- 172.21.15.111:0/3362722066 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904011d20 con 0x7f9910072300 2023-11-28T13:06:23.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.239+0000 7f99170fe700 1 -- 172.21.15.111:0/3362722066 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9910072300 msgr2=0x7f99100712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:23.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.239+0000 7f99170fe700 1 --2- 172.21.15.111:0/3362722066 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9910072300 0x7f99100712b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9904009840 tx=0x7f9904009b50 comp rx=0 tx=0).stop 2023-11-28T13:06:23.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.239+0000 7f99170fe700 1 -- 172.21.15.111:0/3362722066 shutdown_connections 2023-11-28T13:06:23.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.239+0000 7f99170fe700 1 --2- 172.21.15.111:0/3362722066 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f9910072300 0x7f99100712b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:23.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.239+0000 7f99170fe700 1 --2- 172.21.15.111:0/3362722066 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f991015cc50 0x7f991006b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:23.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.239+0000 7f99170fe700 1 -- 172.21.15.111:0/3362722066 >> 172.21.15.111:0/3362722066 conn(0x7f99100cf2e0 msgr2=0x7f99100cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:23.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.239+0000 7f99170fe700 1 -- 172.21.15.111:0/3362722066 shutdown_connections 2023-11-28T13:06:23.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.239+0000 7f99170fe700 1 -- 172.21.15.111:0/3362722066 wait complete. 2023-11-28T13:06:23.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.240+0000 7f99170fe700 1 Processor -- start 2023-11-28T13:06:23.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.240+0000 7f99170fe700 1 -- start start 2023-11-28T13:06:23.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.241+0000 7f99170fe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9910072300 0x7f99101e8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:23.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.241+0000 7f99170fe700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f99100cceb0 0x7f991014cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:23.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.241+0000 7f99170fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f991015cc50 0x7f99101e6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:23.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.241+0000 7f99170fe700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99100aca50 con 0x7f991015cc50 2023-11-28T13:06:23.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.241+0000 7f99170fe700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99100acd30 con 0x7f99100cceb0 2023-11-28T13:06:23.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.241+0000 7f99170fe700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99101e7d80 con 0x7f9910072300 2023-11-28T13:06:23.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.241+0000 7f991569b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f991015cc50 0x7f99101e6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:23.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.241+0000 7f991569b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f991015cc50 0x7f99101e6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38780/0 (socket says 172.21.15.111:38780) 2023-11-28T13:06:23.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.241+0000 7f991569b700 1 -- 172.21.15.111:0/235915039 learned_addr learned my addr 172.21.15.111:0/235915039 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:23.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.241+0000 7f9914e9a700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9910072300 0x7f99101e8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:23.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.241+0000 7f990ffff700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f99100cceb0 0x7f991014cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:23.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.242+0000 7f991569b700 1 -- 172.21.15.111:0/235915039 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9910072300 msgr2=0x7f99101e8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:23.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.242+0000 7f991569b700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9910072300 0x7f99101e8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:23.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.242+0000 7f991569b700 1 -- 172.21.15.111:0/235915039 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f99100cceb0 msgr2=0x7f991014cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:23.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.242+0000 7f991569b700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f99100cceb0 0x7f991014cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:23.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.242+0000 7f991569b700 1 -- 172.21.15.111:0/235915039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99040094a0 con 0x7f991015cc50 2023-11-28T13:06:23.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.242+0000 7f990ffff700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f99100cceb0 0x7f991014cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:23.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.242+0000 7f991569b700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f991015cc50 0x7f99101e6cd0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f990800cb70 tx=0x7f990800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:23.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.242+0000 7f990d7fa700 1 -- 172.21.15.111:0/235915039 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990800ab10 con 0x7f991015cc50 2023-11-28T13:06:23.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.242+0000 7f990d7fa700 1 -- 172.21.15.111:0/235915039 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990800ac70 con 0x7f991015cc50 2023-11-28T13:06:23.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.242+0000 7f990d7fa700 1 -- 172.21.15.111:0/235915039 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9908012400 con 0x7f991015cc50 2023-11-28T13:06:23.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.242+0000 7f99170fe700 1 -- 172.21.15.111:0/235915039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991015d6b0 con 0x7f991015cc50 2023-11-28T13:06:23.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.242+0000 7f99170fe700 1 -- 172.21.15.111:0/235915039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991015d3d0 con 0x7f991015cc50 2023-11-28T13:06:23.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.243+0000 7f99170fe700 1 -- 172.21.15.111:0/235915039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991015d990 con 0x7f991015cc50 2023-11-28T13:06:23.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.243+0000 7f990d7fa700 1 -- 172.21.15.111:0/235915039 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9908019030 con 0x7f991015cc50 2023-11-28T13:06:23.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.244+0000 7f990d7fa700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f98f40682f0 0x7f98f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:23.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.244+0000 7f990d7fa700 1 -- 172.21.15.111:0/235915039 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f99080857d0 con 0x7f991015cc50 2023-11-28T13:06:23.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.244+0000 7f9914e9a700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f98f40682f0 0x7f98f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:23.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.244+0000 7f9914e9a700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f98f40682f0 0x7f98f406a7a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9904000c00 tx=0x7f990401c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:23.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.248+0000 7f990d7fa700 1 -- 172.21.15.111:0/235915039 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9908052170 con 0x7f991015cc50 2023-11-28T13:06:23.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.430+0000 7f99170fe700 1 -- 172.21.15.111:0/235915039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9910067170 con 0x7f991015cc50 2023-11-28T13:06:23.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.431+0000 7f990d7fa700 1 -- 172.21.15.111:0/235915039 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+14652 (secure 0 0 0) 0x7f9908055790 con 0x7f991015cc50 2023-11-28T13:06:23.433 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:23.433 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":72,"fsid":"a1b2d00c-aeec-4b5d-93e3-41820c4189ba","created":"2023-11-28T12:58:34.836080+0000","modified":"2023-11-28T13:06:19.971008+0000","last_up_change":"2023-11-28T12:58:46.162493+0000","last_in_change":"2023-11-28T12:58:37.396611+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T12:58:46.376939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T12:58:48.865894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e275bd31-fa21-4fe2-8a50-63c225873180","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":103527},{"type":"v1","addr":"172.21.15.111:6801","nonce":103527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":103527},{"type":"v1","addr":"172.21.15.111:6803","nonce":103527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":103527},{"type":"v1","addr":"172.21.15.111:6807","nonce":103527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":103527},{"type":"v1","addr":"172.21.15.111:6805","nonce":103527}]},"public_addr":"172.21.15.111:6801/103527","cluster_addr":"172.21.15.111:6803/103527","heartbeat_back_addr":"172.21.15.111:6807/103527","heartbeat_front_addr":"172.21.15.111:6805/103527","state":["exists","up"]},{"osd":1,"uuid":"457b5866-4c8b-4f07-a3c7-95914260e791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":103531},{"type":"v1","addr":"172.21.15.111:6825","nonce":103531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":103531},{"type":"v1","addr":"172.21.15.111:6827","nonce":103531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":103531},{"type":"v1","addr":"172.21.15.111:6831","nonce":103531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":103531},{"type":"v1","addr":"172.21.15.111:6829","nonce":103531}]},"public_addr":"172.21.15.111:6825/103531","cluster_addr":"172.21.15.111:6827/103531","heartbeat_back_addr":"172.21.15.111:6831/103531","heartbeat_front_addr":"172.21.15.111:6829/103531","state":["exists","up"]},{"osd":2,"uuid":"4bb372c7-9bd8-4865-b4ce-b2e09840b087","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6811","nonce":103530},{"type":"v1","addr":"172.21.15.111:6813","nonce":103530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":103530},{"type":"v1","addr":"172.21.15.111:6818","nonce":103530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":103530},{"type":"v1","addr":"172.21.15.111:6823","nonce":103530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":103530},{"type":"v1","addr":"172.21.15.111:6821","nonce":103530}]},"public_addr":"172.21.15.111:6813/103530","cluster_addr":"172.21.15.111:6818/103530","heartbeat_back_addr":"172.21.15.111:6823/103530","heartbeat_front_addr":"172.21.15.111:6821/103530","state":["exists","up"]},{"osd":3,"uuid":"bfaea526-60c3-440f-9e13-9befd332e228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":103538},{"type":"v1","addr":"172.21.15.111:6809","nonce":103538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":103538},{"type":"v1","addr":"172.21.15.111:6812","nonce":103538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6817","nonce":103538},{"type":"v1","addr":"172.21.15.111:6819","nonce":103538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":103538},{"type":"v1","addr":"172.21.15.111:6815","nonce":103538}]},"public_addr":"172.21.15.111:6809/103538","cluster_addr":"172.21.15.111:6812/103538","heartbeat_back_addr":"172.21.15.111:6819/103538","heartbeat_front_addr":"172.21.15.111:6815/103538","state":["exists","up"]},{"osd":4,"uuid":"e82f656b-9400-4bca-b566-10c0b17ad4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":103794},{"type":"v1","addr":"172.21.15.149:6801","nonce":103794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":103794},{"type":"v1","addr":"172.21.15.149:6803","nonce":103794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":103794},{"type":"v1","addr":"172.21.15.149:6807","nonce":103794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":103794},{"type":"v1","addr":"172.21.15.149:6805","nonce":103794}]},"public_addr":"172.21.15.149:6801/103794","cluster_addr":"172.21.15.149:6803/103794","heartbeat_back_addr":"172.21.15.149:6807/103794","heartbeat_front_addr":"172.21.15.149:6805/103794","state":["exists","up"]},{"osd":5,"uuid":"0b2c93fb-4843-4b7a-9407-e4f8fac2ff11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":103795},{"type":"v1","addr":"172.21.15.149:6809","nonce":103795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":103795},{"type":"v1","addr":"172.21.15.149:6811","nonce":103795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":103795},{"type":"v1","addr":"172.21.15.149:6815","nonce":103795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":103795},{"type":"v1","addr":"172.21.15.149:6813","nonce":103795}]},"public_addr":"172.21.15.149:6809/103795","cluster_addr":"172.21.15.149:6811/103795","heartbeat_back_addr":"172.21.15.149:6815/103795","heartbeat_front_addr":"172.21.15.149:6813/103795","state":["exists","up"]},{"osd":6,"uuid":"a7aa32aa-5916-4040-9c3a-cadf59085b16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":103796},{"type":"v1","addr":"172.21.15.149:6817","nonce":103796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":103796},{"type":"v1","addr":"172.21.15.149:6819","nonce":103796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":103796},{"type":"v1","addr":"172.21.15.149:6823","nonce":103796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":103796},{"type":"v1","addr":"172.21.15.149:6821","nonce":103796}]},"public_addr":"172.21.15.149:6817/103796","cluster_addr":"172.21.15.149:6819/103796","heartbeat_back_addr":"172.21.15.149:6823/103796","heartbeat_front_addr":"172.21.15.149:6821/103796","state":["exists","up"]},{"osd":7,"uuid":"22bb9d02-3198-40c1-b318-120d1ca3224f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":103827},{"type":"v1","addr":"172.21.15.149:6825","nonce":103827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":103827},{"type":"v1","addr":"172.21.15.149:6827","nonce":103827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":103827},{"type":"v1","addr":"172.21.15.149:6831","nonce":103827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":103827},{"type":"v1","addr":"172.21.15.149:6829","nonce":103827}]},"public_addr":"172.21.15.149:6825/103827","cluster_addr":"172.21.15.149:6827/103827","heartbeat_back_addr":"172.21.15.149:6831/103827","heartbeat_front_addr":"172.21.15.149:6829/103827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.599775+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.636311+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.591608+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.587772+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.614275+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.618438+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.590685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T12:58:42.633802+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6837/1897250522":"2023-11-29T13:06:15.951945+0000","172.21.15.111:6835/1897250522":"2023-11-29T13:06:15.951945+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T13:06:23.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.433+0000 7f99170fe700 1 -- 172.21.15.111:0/235915039 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f98f40682f0 msgr2=0x7f98f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:23.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.433+0000 7f99170fe700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f98f40682f0 0x7f98f406a7a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9904000c00 tx=0x7f990401c040 comp rx=0 tx=0).stop 2023-11-28T13:06:23.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.433+0000 7f99170fe700 1 -- 172.21.15.111:0/235915039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f991015cc50 msgr2=0x7f99101e6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:23.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.433+0000 7f99170fe700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f991015cc50 0x7f99101e6cd0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f990800cb70 tx=0x7f990800cf30 comp rx=0 tx=0).stop 2023-11-28T13:06:23.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.434+0000 7f99170fe700 1 -- 172.21.15.111:0/235915039 shutdown_connections 2023-11-28T13:06:23.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.434+0000 7f99170fe700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f991015cc50 0x7f99101e6cd0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:23.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.434+0000 7f99170fe700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f9910072300 0x7f99101e8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:23.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.434+0000 7f99170fe700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f98f40682f0 0x7f98f406a7a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:23.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.434+0000 7f99170fe700 1 --2- 172.21.15.111:0/235915039 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f99100cceb0 0x7f991014cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:23.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.434+0000 7f99170fe700 1 -- 172.21.15.111:0/235915039 >> 172.21.15.111:0/235915039 conn(0x7f99100cf2e0 msgr2=0x7f99100cb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:23.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.434+0000 7f99170fe700 1 -- 172.21.15.111:0/235915039 shutdown_connections 2023-11-28T13:06:23.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:23.434+0000 7f99170fe700 1 -- 172.21.15.111:0/235915039 wait complete. 2023-11-28T13:06:24.448 INFO:tasks.ceph:Scrubbing osd.0 2023-11-28T13:06:24.448 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-11-28T13:06:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.583+0000 7f74bf091700 1 Processor -- start 2023-11-28T13:06:24.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.583+0000 7f74bf091700 1 -- start start 2023-11-28T13:06:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.584+0000 7f74bf091700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74b81604b0 0x7f74b80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.584+0000 7f74bf091700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f74b80cc550 0x7f74b80cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.584+0000 7f74bf091700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b815cd90 con 0x7f74b80b0d70 2023-11-28T13:06:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.584+0000 7f74bf091700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b80ccf10 con 0x7f74b80cc550 2023-11-28T13:06:24.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.584+0000 7f74bf091700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b8160920 con 0x7f74b81604b0 2023-11-28T13:06:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.584+0000 7f74bd62e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f74b80cc550 0x7f74b80cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.584+0000 7f74bd62e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f74b80cc550 0x7f74b80cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38788/0 (socket says 172.21.15.111:38788) 2023-11-28T13:06:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.584+0000 7f74bd62e700 1 -- 172.21.15.111:0/3817904811 learned_addr learned my addr 172.21.15.111:0/3817904811 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.585+0000 7f74bd62e700 1 -- 172.21.15.111:0/3817904811 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74b81604b0 msgr2=0x7f74b80ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.585+0000 7f74b7fff700 1 --2- 172.21.15.111:0/3817904811 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74b81604b0 0x7f74b80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.585+0000 7f74bd62e700 1 --2- 172.21.15.111:0/3817904811 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74b81604b0 0x7f74b80ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.585+0000 7f74bd62e700 1 -- 172.21.15.111:0/3817904811 >> v1:172.21.15.149:6789/0 conn(0x7f74b80b0d70 legacy=0x7f74b80b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.585+0000 7f74bd62e700 1 -- 172.21.15.111:0/3817904811 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b81613e0 con 0x7f74b80cc550 2023-11-28T13:06:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.585+0000 7f74b7fff700 1 --2- 172.21.15.111:0/3817904811 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74b81604b0 0x7f74b80ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.585+0000 7f74bd62e700 1 --2- 172.21.15.111:0/3817904811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74b80cc550 0x7f74b80cc9d0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009e50 tx=0x7f74b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234d1884a7d58b4f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.585+0000 7f74b77fe700 1 -- 172.21.15.111:0/3817904811 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b000e070 con 0x7f74b80cc550 2023-11-28T13:06:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.585+0000 7f74b77fe700 1 -- 172.21.15.111:0/3817904811 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b0003a20 con 0x7f74b80cc550 2023-11-28T13:06:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.585+0000 7f74b77fe700 1 -- 172.21.15.111:0/3817904811 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b0003dd0 con 0x7f74b80cc550 2023-11-28T13:06:24.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.586+0000 7f74bf091700 1 -- 172.21.15.111:0/3817904811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74b80cc550 msgr2=0x7f74b80cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:24.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.586+0000 7f74bf091700 1 --2- 172.21.15.111:0/3817904811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74b80cc550 0x7f74b80cc9d0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f74b0009e50 tx=0x7f74b000b330 comp rx=0 tx=0).stop 2023-11-28T13:06:24.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.586+0000 7f74bf091700 1 -- 172.21.15.111:0/3817904811 shutdown_connections 2023-11-28T13:06:24.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.586+0000 7f74bf091700 1 --2- 172.21.15.111:0/3817904811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74b80cc550 0x7f74b80cc9d0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.586+0000 7f74bf091700 1 --2- 172.21.15.111:0/3817904811 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74b81604b0 0x7f74b80ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.586+0000 7f74bf091700 1 -- 172.21.15.111:0/3817904811 >> 172.21.15.111:0/3817904811 conn(0x7f74b8098ee0 msgr2=0x7f74b806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.586+0000 7f74bf091700 1 -- 172.21.15.111:0/3817904811 shutdown_connections 2023-11-28T13:06:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.586+0000 7f74bf091700 1 -- 172.21.15.111:0/3817904811 wait complete. 2023-11-28T13:06:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.587+0000 7f74bf091700 1 Processor -- start 2023-11-28T13:06:24.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.587+0000 7f74bf091700 1 -- start start 2023-11-28T13:06:24.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bf091700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74b80b0d70 0x7f74b814a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:24.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bf091700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74b80cc550 0x7f74b80c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:24.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bf091700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f74b81604b0 0x7f74b80cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:24.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bf091700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b80c7fe0 con 0x7f74b80b0d70 2023-11-28T13:06:24.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bf091700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b814a6b0 con 0x7f74b81604b0 2023-11-28T13:06:24.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bf091700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b80c6f40 con 0x7f74b80cc550 2023-11-28T13:06:24.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bce2d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74b80b0d70 0x7f74b814a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:24.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bce2d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74b80b0d70 0x7f74b814a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38790/0 (socket says 172.21.15.111:38790) 2023-11-28T13:06:24.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bce2d700 1 -- 172.21.15.111:0/3007247491 learned_addr learned my addr 172.21.15.111:0/3007247491 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:24.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74b7fff700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74b80cc550 0x7f74b80c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:24.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bd62e700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f74b81604b0 0x7f74b80cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:24.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bce2d700 1 -- 172.21.15.111:0/3007247491 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74b80cc550 msgr2=0x7f74b80c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:24.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bce2d700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f74b80cc550 0x7f74b80c89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bce2d700 1 -- 172.21.15.111:0/3007247491 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f74b81604b0 msgr2=0x7f74b80cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:24.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bce2d700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f74b81604b0 0x7f74b80cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bce2d700 1 -- 172.21.15.111:0/3007247491 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b0009b30 con 0x7f74b80b0d70 2023-11-28T13:06:24.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bce2d700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74b80b0d70 0x7f74b814a170 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00c4b0 tx=0x7f74ac00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:24.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74b57fa700 1 -- 172.21.15.111:0/3007247491 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac011790 con 0x7f74b80b0d70 2023-11-28T13:06:24.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74b57fa700 1 -- 172.21.15.111:0/3007247491 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac0118f0 con 0x7f74b80b0d70 2023-11-28T13:06:24.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74bf091700 1 -- 172.21.15.111:0/3007247491 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b8160e20 con 0x7f74b80b0d70 2023-11-28T13:06:24.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.588+0000 7f74b57fa700 1 -- 172.21.15.111:0/3007247491 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac011ca0 con 0x7f74b80b0d70 2023-11-28T13:06:24.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.589+0000 7f74bf091700 1 -- 172.21.15.111:0/3007247491 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b8160b40 con 0x7f74b80b0d70 2023-11-28T13:06:24.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.589+0000 7f74bf091700 1 -- 172.21.15.111:0/3007247491 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f74b8161100 con 0x7f74b80b0d70 2023-11-28T13:06:24.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.590+0000 7f74b57fa700 1 -- 172.21.15.111:0/3007247491 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f74ac010050 con 0x7f74b80b0d70 2023-11-28T13:06:24.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.590+0000 7f74b57fa700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f749c0682f0 0x7f749c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:24.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.590+0000 7f74b57fa700 1 -- 172.21.15.111:0/3007247491 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f74ac0853c0 con 0x7f74b80b0d70 2023-11-28T13:06:24.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.590+0000 7f74b7fff700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f749c0682f0 0x7f749c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:24.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.590+0000 7f74b57fa700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x7f749c06eb90 0x7f749c070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:24.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.590+0000 7f74b57fa700 1 -- 172.21.15.111:0/3007247491 --> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f749c0716b0 con 0x7f749c06eb90 2023-11-28T13:06:24.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.590+0000 7f74b57fa700 1 -- 172.21.15.111:0/3007247491 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74ac087a70 con 0x7f74b80b0d70 2023-11-28T13:06:24.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.591+0000 7f74bd62e700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x7f749c06eb90 0x7f749c070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:24.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.591+0000 7f74b7fff700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f749c0682f0 0x7f749c06a7a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f74b80cd170 tx=0x7f74a4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:24.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.591+0000 7f74bd62e700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x7f749c06eb90 0x7f749c070fe0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:24.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.591+0000 7f74b57fa700 1 -- 172.21.15.111:0/3007247491 <== osd.0 v2:172.21.15.111:6800/103527 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f749c0716b0 con 0x7f749c06eb90 2023-11-28T13:06:24.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.606+0000 7f74bf091700 1 -- 172.21.15.111:0/3007247491 --> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f74b80ce300 con 0x7f749c06eb90 2023-11-28T13:06:24.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.606+0000 7f74b57fa700 1 -- 172.21.15.111:0/3007247491 <== osd.0 v2:172.21.15.111:6800/103527 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f74b80ce300 con 0x7f749c06eb90 2023-11-28T13:06:24.607 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:06:24.608 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-28T13:06:24.608 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:06:24.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.606+0000 7f74bf091700 1 -- 172.21.15.111:0/3007247491 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x7f749c06eb90 msgr2=0x7f749c070fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:24.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.606+0000 7f74bf091700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.111:6800/103527,v1:172.21.15.111:6801/103527] conn(0x7f749c06eb90 0x7f749c070fe0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.606+0000 7f74bf091700 1 -- 172.21.15.111:0/3007247491 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f749c0682f0 msgr2=0x7f749c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:24.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.606+0000 7f74bf091700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f749c0682f0 0x7f749c06a7a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f74b80cd170 tx=0x7f74a4008040 comp rx=0 tx=0).stop 2023-11-28T13:06:24.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.606+0000 7f74bf091700 1 -- 172.21.15.111:0/3007247491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74b80b0d70 msgr2=0x7f74b814a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:24.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.606+0000 7f74bf091700 1 --2- 172.21.15.111:0/3007247491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74b80b0d70 0x7f74b814a170 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00c4b0 tx=0x7f74ac00c7c0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.606+0000 7f74bd62e700 1 -- 172.21.15.111:0/3007247491 reap_dead start 2023-11-28T13:06:24.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.606+0000 7f74bf091700 1 -- 172.21.15.111:0/3007247491 shutdown_connections 2023-11-28T13:06:24.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.606+0000 7f74bf091700 1 -- 172.21.15.111:0/3007247491 >> 172.21.15.111:0/3007247491 conn(0x7f74b8098ee0 msgr2=0x7f74b8072cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:24.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.606+0000 7f74bf091700 1 -- 172.21.15.111:0/3007247491 shutdown_connections 2023-11-28T13:06:24.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.606+0000 7f74bf091700 1 -- 172.21.15.111:0/3007247491 wait complete. 2023-11-28T13:06:24.619 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-11-28T13:06:24.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.739+0000 7feefd3fc700 1 Processor -- start 2023-11-28T13:06:24.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.739+0000 7feefd3fc700 1 -- start start 2023-11-28T13:06:24.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.740+0000 7feefd3fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7feef80cd460 0x7feef806c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:24.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.740+0000 7feefd3fc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef80b88e0 con 0x7feef80b8510 2023-11-28T13:06:24.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.740+0000 7feefd3fc700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef80cd8f0 con 0x7feef80d0d60 2023-11-28T13:06:24.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.740+0000 7feefd3fc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef80cdbd0 con 0x7feef80cd460 2023-11-28T13:06:24.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.740+0000 7feef67fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7feef80cd460 0x7feef806c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:24.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.740+0000 7feef67fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7feef80cd460 0x7feef806c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38802/0 (socket says 172.21.15.111:38802) 2023-11-28T13:06:24.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.740+0000 7feef67fc700 1 -- 172.21.15.111:0/1188119949 learned_addr learned my addr 172.21.15.111:0/1188119949 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:24.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.740+0000 7feef6ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7feef80b8510 0x7feef806b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39578/0 (socket says 172.21.15.111:39578) 2023-11-28T13:06:24.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.740+0000 7feef67fc700 1 -- 172.21.15.111:0/1188119949 >> v1:172.21.15.149:6789/0 conn(0x7feef80d0d60 legacy=0x7feef80d1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:24.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.740+0000 7feef67fc700 1 -- 172.21.15.111:0/1188119949 >> v1:172.21.15.111:6789/0 conn(0x7feef80b8510 legacy=0x7feef806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:24.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.740+0000 7feef5ffb700 1 -- 172.21.15.111:0/1188119949 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566883516 0 0) 0x7feef80b88e0 con 0x7feef80b8510 2023-11-28T13:06:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.740+0000 7feef67fc700 1 -- 172.21.15.111:0/1188119949 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef80cfd10 con 0x7feef80cd460 2023-11-28T13:06:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.740+0000 7feef67fc700 1 --2- 172.21.15.111:0/1188119949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feef80cd460 0x7feef806c080 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7feeec0098f0 tx=0x7feeec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f3f285219e4a386 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.741+0000 7feef5ffb700 1 -- 172.21.15.111:0/1188119949 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec00b8a0 con 0x7feef80cd460 2023-11-28T13:06:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.741+0000 7feef5ffb700 1 -- 172.21.15.111:0/1188119949 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec00ba00 con 0x7feef80cd460 2023-11-28T13:06:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.741+0000 7feef5ffb700 1 -- 172.21.15.111:0/1188119949 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec00bd90 con 0x7feef80cd460 2023-11-28T13:06:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.741+0000 7feefd3fc700 1 -- 172.21.15.111:0/1188119949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feef80cd460 msgr2=0x7feef806c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.741+0000 7feefd3fc700 1 --2- 172.21.15.111:0/1188119949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feef80cd460 0x7feef806c080 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7feeec0098f0 tx=0x7feeec009c00 comp rx=0 tx=0).stop 2023-11-28T13:06:24.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.741+0000 7feefd3fc700 1 -- 172.21.15.111:0/1188119949 shutdown_connections 2023-11-28T13:06:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.741+0000 7feefd3fc700 1 --2- 172.21.15.111:0/1188119949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feef80cd460 0x7feef806c080 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.741+0000 7feefd3fc700 1 -- 172.21.15.111:0/1188119949 >> 172.21.15.111:0/1188119949 conn(0x7feef80719a0 msgr2=0x7feef8071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.741+0000 7feefd3fc700 1 -- 172.21.15.111:0/1188119949 shutdown_connections 2023-11-28T13:06:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.741+0000 7feefd3fc700 1 -- 172.21.15.111:0/1188119949 wait complete. 2023-11-28T13:06:24.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.742+0000 7feefd3fc700 1 Processor -- start 2023-11-28T13:06:24.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.742+0000 7feefd3fc700 1 -- start start 2023-11-28T13:06:24.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feefd3fc700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7feef80b8510 0x7feef815ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:24.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feefd3fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feef80cd460 0x7feef81e1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:24.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feefd3fc700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feef80d0d60 0x7feef80b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:24.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feefd3fc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef81e3410 con 0x7feef80cd460 2023-11-28T13:06:24.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feefd3fc700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef81e2370 con 0x7feef80d0d60 2023-11-28T13:06:24.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feefd3fc700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef81e2650 con 0x7feef80b8510 2023-11-28T13:06:24.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feef67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feef80cd460 0x7feef81e1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:24.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feef67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feef80cd460 0x7feef81e1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38808/0 (socket says 172.21.15.111:38808) 2023-11-28T13:06:24.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feef67fc700 1 -- 172.21.15.111:0/1248233883 learned_addr learned my addr 172.21.15.111:0/1248233883 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:24.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feef77fe700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feef80d0d60 0x7feef80b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:24.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feef6ffd700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7feef80b8510 0x7feef815ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:24.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feef67fc700 1 -- 172.21.15.111:0/1248233883 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7feef80b8510 msgr2=0x7feef815ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:24.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feef67fc700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7feef80b8510 0x7feef815ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feef67fc700 1 -- 172.21.15.111:0/1248233883 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feef80d0d60 msgr2=0x7feef80b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:24.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.743+0000 7feef67fc700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feef80d0d60 0x7feef80b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.744+0000 7feef67fc700 1 -- 172.21.15.111:0/1248233883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec009550 con 0x7feef80cd460 2023-11-28T13:06:24.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.744+0000 7feef77fe700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feef80d0d60 0x7feef80b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:24.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.744+0000 7feef67fc700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feef80cd460 0x7feef81e1940 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7feeec013000 tx=0x7feeec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.744+0000 7feedf7fe700 1 -- 172.21.15.111:0/1248233883 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec011a70 con 0x7feef80cd460 2023-11-28T13:06:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.744+0000 7feedf7fe700 1 -- 172.21.15.111:0/1248233883 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec01f070 con 0x7feef80cd460 2023-11-28T13:06:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.744+0000 7feefd3fc700 1 -- 172.21.15.111:0/1248233883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef80d29c0 con 0x7feef80cd460 2023-11-28T13:06:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.744+0000 7feedf7fe700 1 -- 172.21.15.111:0/1248233883 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec011dd0 con 0x7feef80cd460 2023-11-28T13:06:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.744+0000 7feefd3fc700 1 -- 172.21.15.111:0/1248233883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef80cfff0 con 0x7feef80cd460 2023-11-28T13:06:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.745+0000 7feefd3fc700 1 -- 172.21.15.111:0/1248233883 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef80d2b50 con 0x7feef80cd460 2023-11-28T13:06:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.745+0000 7feedf7fe700 1 -- 172.21.15.111:0/1248233883 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feeec00f050 con 0x7feef80cd460 2023-11-28T13:06:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.745+0000 7feedf7fe700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7feee40682f0 0x7feee406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.746+0000 7feedf7fe700 1 -- 172.21.15.111:0/1248233883 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7feeec021070 con 0x7feef80cd460 2023-11-28T13:06:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.746+0000 7feef6ffd700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7feee40682f0 0x7feee406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.746+0000 7feef6ffd700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7feee40682f0 0x7feee406a7a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7feee8000f60 tx=0x7feee8007c40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.750+0000 7feedf7fe700 1 -- 172.21.15.111:0/1248233883 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feeec053fb0 con 0x7feef80cd460 2023-11-28T13:06:24.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.934+0000 7feefd3fc700 1 -- 172.21.15.111:0/1248233883 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7feef80b2a30 con 0x7feee40682f0 2023-11-28T13:06:24.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.934+0000 7feedf7fe700 1 -- 172.21.15.111:0/1248233883 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7feef80b2a30 con 0x7feee40682f0 2023-11-28T13:06:24.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.936+0000 7feefd3fc700 1 -- 172.21.15.111:0/1248233883 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7feee40682f0 msgr2=0x7feee406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:24.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.936+0000 7feefd3fc700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7feee40682f0 0x7feee406a7a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7feee8000f60 tx=0x7feee8007c40 comp rx=0 tx=0).stop 2023-11-28T13:06:24.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.936+0000 7feefd3fc700 1 -- 172.21.15.111:0/1248233883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feef80cd460 msgr2=0x7feef81e1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:24.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.936+0000 7feefd3fc700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feef80cd460 0x7feef81e1940 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7feeec013000 tx=0x7feeec004750 comp rx=0 tx=0).stop 2023-11-28T13:06:24.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.936+0000 7feefd3fc700 1 -- 172.21.15.111:0/1248233883 shutdown_connections 2023-11-28T13:06:24.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.936+0000 7feefd3fc700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7feee40682f0 0x7feee406a7a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.936+0000 7feefd3fc700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7feef80d0d60 0x7feef80b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.936+0000 7feefd3fc700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7feef80b8510 0x7feef815ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.936+0000 7feefd3fc700 1 --2- 172.21.15.111:0/1248233883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feef80cd460 0x7feef81e1940 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:24.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.936+0000 7feefd3fc700 1 -- 172.21.15.111:0/1248233883 >> 172.21.15.111:0/1248233883 conn(0x7feef80719a0 msgr2=0x7feef80b6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:24.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.936+0000 7feefd3fc700 1 -- 172.21.15.111:0/1248233883 shutdown_connections 2023-11-28T13:06:24.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:24.936+0000 7feefd3fc700 1 -- 172.21.15.111:0/1248233883 wait complete. 2023-11-28T13:06:24.939 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 0 to deep-scrub 2023-11-28T13:06:24.949 INFO:tasks.ceph:Scrubbing osd.1 2023-11-28T13:06:24.950 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-11-28T13:06:25.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.072+0000 7fe05dc70700 1 Processor -- start 2023-11-28T13:06:25.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.072+0000 7fe05dc70700 1 -- start start 2023-11-28T13:06:25.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.073+0000 7fe05dc70700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe0580b2360 0x7fe0580c6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.073+0000 7fe05dc70700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0580d0b90 con 0x7fe0580ce900 2023-11-28T13:06:25.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.073+0000 7fe05dc70700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0580b3db0 con 0x7fe0580c5890 2023-11-28T13:06:25.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.073+0000 7fe05dc70700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0580c6cc0 con 0x7fe0580b2360 2023-11-28T13:06:25.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.073+0000 7fe0577fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe0580b2360 0x7fe0580c6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.073+0000 7fe056ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe0580ce900 0x7fe0580b4240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39594/0 (socket says 172.21.15.111:39594) 2023-11-28T13:06:25.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.073+0000 7fe056ffd700 1 -- 172.21.15.111:0/4119758972 learned_addr learned my addr 172.21.15.111:0/4119758972 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:25.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.073+0000 7fe0577fe700 1 -- 172.21.15.111:0/4119758972 >> v1:172.21.15.149:6789/0 conn(0x7fe0580c5890 legacy=0x7fe0580b4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:25.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.073+0000 7fe0577fe700 1 -- 172.21.15.111:0/4119758972 >> v1:172.21.15.111:6789/0 conn(0x7fe0580ce900 legacy=0x7fe0580b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.073+0000 7fe0577fe700 1 -- 172.21.15.111:0/4119758972 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0580c6eb0 con 0x7fe0580b2360 2023-11-28T13:06:25.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.073+0000 7fe0567fc700 1 -- 172.21.15.111:0/4119758972 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340540319 0 0) 0x7fe048001110 con 0x7fe0580ce900 2023-11-28T13:06:25.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.073+0000 7fe0577fe700 1 --2- 172.21.15.111:0/4119758972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0580b2360 0x7fe0580c6780 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fe04c009820 tx=0x7fe04c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8168a717f5c3a393 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.074+0000 7fe0567fc700 1 -- 172.21.15.111:0/4119758972 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c005e30 con 0x7fe0580b2360 2023-11-28T13:06:25.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.074+0000 7fe0567fc700 1 -- 172.21.15.111:0/4119758972 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04c004030 con 0x7fe0580b2360 2023-11-28T13:06:25.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.074+0000 7fe0567fc700 1 -- 172.21.15.111:0/4119758972 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c0043c0 con 0x7fe0580b2360 2023-11-28T13:06:25.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.074+0000 7fe05dc70700 1 -- 172.21.15.111:0/4119758972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0580b2360 msgr2=0x7fe0580c6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.074+0000 7fe05dc70700 1 --2- 172.21.15.111:0/4119758972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0580b2360 0x7fe0580c6780 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fe04c009820 tx=0x7fe04c009b30 comp rx=0 tx=0).stop 2023-11-28T13:06:25.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.074+0000 7fe05dc70700 1 -- 172.21.15.111:0/4119758972 shutdown_connections 2023-11-28T13:06:25.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.074+0000 7fe05dc70700 1 --2- 172.21.15.111:0/4119758972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0580b2360 0x7fe0580c6780 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.074+0000 7fe05dc70700 1 -- 172.21.15.111:0/4119758972 >> 172.21.15.111:0/4119758972 conn(0x7fe05809a040 msgr2=0x7fe0580d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:25.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.074+0000 7fe05dc70700 1 -- 172.21.15.111:0/4119758972 shutdown_connections 2023-11-28T13:06:25.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.074+0000 7fe05dc70700 1 -- 172.21.15.111:0/4119758972 wait complete. 2023-11-28T13:06:25.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.075+0000 7fe05dc70700 1 Processor -- start 2023-11-28T13:06:25.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.076+0000 7fe05dc70700 1 -- start start 2023-11-28T13:06:25.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.076+0000 7fe05dc70700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe0580b2360 0x7fe0580b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.076+0000 7fe05dc70700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0580c5890 0x7fe058153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.076+0000 7fe05dc70700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe0580ce900 0x7fe058154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.076+0000 7fe05dc70700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe058156610 con 0x7fe0580c5890 2023-11-28T13:06:25.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.076+0000 7fe05dc70700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0580ae890 con 0x7fe0580b2360 2023-11-28T13:06:25.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.076+0000 7fe05dc70700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0580aeb70 con 0x7fe0580ce900 2023-11-28T13:06:25.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.076+0000 7fe056ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0580c5890 0x7fe058153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.076+0000 7fe056ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0580c5890 0x7fe058153d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38822/0 (socket says 172.21.15.111:38822) 2023-11-28T13:06:25.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.076+0000 7fe056ffd700 1 -- 172.21.15.111:0/398077461 learned_addr learned my addr 172.21.15.111:0/398077461 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:25.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe0577fe700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe0580b2360 0x7fe0580b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe057fff700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe0580ce900 0x7fe058154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe056ffd700 1 -- 172.21.15.111:0/398077461 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe0580ce900 msgr2=0x7fe058154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe056ffd700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe0580ce900 0x7fe058154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe056ffd700 1 -- 172.21.15.111:0/398077461 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe0580b2360 msgr2=0x7fe0580b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe056ffd700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe0580b2360 0x7fe0580b5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe056ffd700 1 -- 172.21.15.111:0/398077461 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c009430 con 0x7fe0580c5890 2023-11-28T13:06:25.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe0577fe700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fe0580b2360 0x7fe0580b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:25.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe057fff700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fe0580ce900 0x7fe058154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:25.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe056ffd700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0580c5890 0x7fe058153d40 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe04800c550 tx=0x7fe04800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe03ffff700 1 -- 172.21.15.111:0/398077461 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe048004c80 con 0x7fe0580c5890 2023-11-28T13:06:25.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe03ffff700 1 -- 172.21.15.111:0/398077461 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe048004e00 con 0x7fe0580c5890 2023-11-28T13:06:25.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe05dc70700 1 -- 172.21.15.111:0/398077461 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0580c7470 con 0x7fe0580c5890 2023-11-28T13:06:25.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe03ffff700 1 -- 172.21.15.111:0/398077461 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe048007310 con 0x7fe0580c5890 2023-11-28T13:06:25.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.077+0000 7fe05dc70700 1 -- 172.21.15.111:0/398077461 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0580c7190 con 0x7fe0580c5890 2023-11-28T13:06:25.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.078+0000 7fe05dc70700 1 -- 172.21.15.111:0/398077461 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe0580b5060 con 0x7fe0580c5890 2023-11-28T13:06:25.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.078+0000 7fe03ffff700 1 -- 172.21.15.111:0/398077461 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe04801a020 con 0x7fe0580c5890 2023-11-28T13:06:25.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.079+0000 7fe03ffff700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe0440682f0 0x7fe04406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.079+0000 7fe03ffff700 1 -- 172.21.15.111:0/398077461 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fe0480854d0 con 0x7fe0580c5890 2023-11-28T13:06:25.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.079+0000 7fe0577fe700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe0440682f0 0x7fe04406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.079+0000 7fe03ffff700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] conn(0x7fe04406eb90 0x7fe044070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.079+0000 7fe03ffff700 1 -- 172.21.15.111:0/398077461 --> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe0440716b0 con 0x7fe04406eb90 2023-11-28T13:06:25.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.079+0000 7fe03ffff700 1 -- 172.21.15.111:0/398077461 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe048087b80 con 0x7fe0580c5890 2023-11-28T13:06:25.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.079+0000 7fe057fff700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] conn(0x7fe04406eb90 0x7fe044070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.079+0000 7fe0577fe700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe0440682f0 0x7fe04406a7a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe04c014040 tx=0x7fe04c008070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.079+0000 7fe057fff700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] conn(0x7fe04406eb90 0x7fe044070fe0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.080+0000 7fe03ffff700 1 -- 172.21.15.111:0/398077461 <== osd.1 v2:172.21.15.111:6824/103531 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe0440716b0 con 0x7fe04406eb90 2023-11-28T13:06:25.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.094+0000 7fe05dc70700 1 -- 172.21.15.111:0/398077461 --> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe0580d0b90 con 0x7fe04406eb90 2023-11-28T13:06:25.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.094+0000 7fe03ffff700 1 -- 172.21.15.111:0/398077461 <== osd.1 v2:172.21.15.111:6824/103531 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe0580d0b90 con 0x7fe04406eb90 2023-11-28T13:06:25.096 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:06:25.096 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-28T13:06:25.096 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:06:25.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.095+0000 7fe05dc70700 1 -- 172.21.15.111:0/398077461 >> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] conn(0x7fe04406eb90 msgr2=0x7fe044070fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.095+0000 7fe05dc70700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.111:6824/103531,v1:172.21.15.111:6825/103531] conn(0x7fe04406eb90 0x7fe044070fe0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.095+0000 7fe05dc70700 1 -- 172.21.15.111:0/398077461 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe0440682f0 msgr2=0x7fe04406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.095+0000 7fe05dc70700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fe0440682f0 0x7fe04406a7a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe04c014040 tx=0x7fe04c008070 comp rx=0 tx=0).stop 2023-11-28T13:06:25.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.095+0000 7fe05dc70700 1 -- 172.21.15.111:0/398077461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0580c5890 msgr2=0x7fe058153d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.095+0000 7fe05dc70700 1 --2- 172.21.15.111:0/398077461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0580c5890 0x7fe058153d40 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe04800c550 tx=0x7fe04800c910 comp rx=0 tx=0).stop 2023-11-28T13:06:25.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.095+0000 7fe057fff700 1 -- 172.21.15.111:0/398077461 reap_dead start 2023-11-28T13:06:25.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.095+0000 7fe05dc70700 1 -- 172.21.15.111:0/398077461 shutdown_connections 2023-11-28T13:06:25.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.095+0000 7fe05dc70700 1 -- 172.21.15.111:0/398077461 >> 172.21.15.111:0/398077461 conn(0x7fe05809a040 msgr2=0x7fe0580cf650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:25.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.095+0000 7fe05dc70700 1 -- 172.21.15.111:0/398077461 shutdown_connections 2023-11-28T13:06:25.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.095+0000 7fe05dc70700 1 -- 172.21.15.111:0/398077461 wait complete. 2023-11-28T13:06:25.108 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-11-28T13:06:25.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.227+0000 7f876a40d700 1 Processor -- start 2023-11-28T13:06:25.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.227+0000 7f876a40d700 1 -- start start 2023-11-28T13:06:25.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.228+0000 7f876a40d700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f876415a3c0 0x7f876406f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.228+0000 7f876a40d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f87640cb790 0x7f876406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.228+0000 7f876a40d700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876406b490 con 0x7f876406a7b0 2023-11-28T13:06:25.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.228+0000 7f876a40d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876406abf0 con 0x7f87640cb790 2023-11-28T13:06:25.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.228+0000 7f876a40d700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876406f680 con 0x7f876415a3c0 2023-11-28T13:06:25.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.228+0000 7f87637fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f87640cb790 0x7f876406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.228+0000 7f87637fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f87640cb790 0x7f876406a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38832/0 (socket says 172.21.15.111:38832) 2023-11-28T13:06:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.228+0000 7f87637fe700 1 -- 172.21.15.111:0/1868886381 learned_addr learned my addr 172.21.15.111:0/1868886381 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.228+0000 7f8763fff700 1 --2- 172.21.15.111:0/1868886381 >> v2:172.21.15.149:3300/0 conn(0x7f876415a3c0 0x7f876406f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f87637fe700 1 -- 172.21.15.111:0/1868886381 >> v2:172.21.15.149:3300/0 conn(0x7f876415a3c0 msgr2=0x7f876406f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f87637fe700 1 --2- 172.21.15.111:0/1868886381 >> v2:172.21.15.149:3300/0 conn(0x7f876415a3c0 0x7f876406f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f87637fe700 1 -- 172.21.15.111:0/1868886381 >> v1:172.21.15.149:6789/0 conn(0x7f876406a7b0 legacy=0x7f876415df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f87637fe700 1 -- 172.21.15.111:0/1868886381 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876415a790 con 0x7f87640cb790 2023-11-28T13:06:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f87637fe700 1 --2- 172.21.15.111:0/1868886381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87640cb790 0x7f876406a270 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f87640710e0 tx=0x7f8758009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6470bc40b8ed1b6a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f8762ffd700 1 -- 172.21.15.111:0/1868886381 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875800b870 con 0x7f87640cb790 2023-11-28T13:06:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f8762ffd700 1 -- 172.21.15.111:0/1868886381 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875800b9d0 con 0x7f87640cb790 2023-11-28T13:06:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f8762ffd700 1 -- 172.21.15.111:0/1868886381 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875800bd60 con 0x7f87640cb790 2023-11-28T13:06:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f876a40d700 1 -- 172.21.15.111:0/1868886381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87640cb790 msgr2=0x7f876406a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f876a40d700 1 --2- 172.21.15.111:0/1868886381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87640cb790 0x7f876406a270 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f87640710e0 tx=0x7f8758009bd0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f876a40d700 1 -- 172.21.15.111:0/1868886381 shutdown_connections 2023-11-28T13:06:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f876a40d700 1 --2- 172.21.15.111:0/1868886381 >> v2:172.21.15.149:3300/0 conn(0x7f876415a3c0 0x7f876406f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f876a40d700 1 --2- 172.21.15.111:0/1868886381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87640cb790 0x7f876406a270 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f876a40d700 1 -- 172.21.15.111:0/1868886381 >> 172.21.15.111:0/1868886381 conn(0x7f87640b0db0 msgr2=0x7f87640b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:25.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f876a40d700 1 -- 172.21.15.111:0/1868886381 shutdown_connections 2023-11-28T13:06:25.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.229+0000 7f876a40d700 1 -- 172.21.15.111:0/1868886381 wait complete. 2023-11-28T13:06:25.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.231+0000 7f876a40d700 1 Processor -- start 2023-11-28T13:06:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.231+0000 7f876a40d700 1 -- start start 2023-11-28T13:06:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.231+0000 7f876a40d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f876406a7b0 0x7f87640b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.231+0000 7f876a40d700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f87640cb790 0x7f876415d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f876a40d700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f876415a3c0 0x7f87640c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f876a40d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87641f0d00 con 0x7f876406a7b0 2023-11-28T13:06:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f876a40d700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87641f0180 con 0x7f87640cb790 2023-11-28T13:06:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f876a40d700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87641f0460 con 0x7f876415a3c0 2023-11-28T13:06:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f8763fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f876406a7b0 0x7f87640b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f8763fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f876406a7b0 0x7f87640b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38836/0 (socket says 172.21.15.111:38836) 2023-11-28T13:06:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f8763fff700 1 -- 172.21.15.111:0/3805721921 learned_addr learned my addr 172.21.15.111:0/3805721921 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f87637fe700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f87640cb790 0x7f876415d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f87689aa700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f876415a3c0 0x7f87640c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f8763fff700 1 -- 172.21.15.111:0/3805721921 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f876415a3c0 msgr2=0x7f87640c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f8763fff700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f876415a3c0 0x7f87640c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f8763fff700 1 -- 172.21.15.111:0/3805721921 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f87640cb790 msgr2=0x7f876415d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f8763fff700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f87640cb790 0x7f876415d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f8763fff700 1 -- 172.21.15.111:0/3805721921 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8758009550 con 0x7f876406a7b0 2023-11-28T13:06:25.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f87637fe700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f87640cb790 0x7f876415d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:25.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f8763fff700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f876406a7b0 0x7f87640b2fb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f875400e810 tx=0x7f875400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f8760ff9700 1 -- 172.21.15.111:0/3805721921 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8754013610 con 0x7f876406a7b0 2023-11-28T13:06:25.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f8760ff9700 1 -- 172.21.15.111:0/3805721921 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8754013770 con 0x7f876406a7b0 2023-11-28T13:06:25.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f876a40d700 1 -- 172.21.15.111:0/3805721921 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f876406b9c0 con 0x7f876406a7b0 2023-11-28T13:06:25.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f8760ff9700 1 -- 172.21.15.111:0/3805721921 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8754013b20 con 0x7f876406a7b0 2023-11-28T13:06:25.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.232+0000 7f876a40d700 1 -- 172.21.15.111:0/3805721921 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f876406b6e0 con 0x7f876406a7b0 2023-11-28T13:06:25.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.233+0000 7f876a40d700 1 -- 172.21.15.111:0/3805721921 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8764067170 con 0x7f876406a7b0 2023-11-28T13:06:25.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.234+0000 7f8760ff9700 1 -- 172.21.15.111:0/3805721921 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8754023070 con 0x7f876406a7b0 2023-11-28T13:06:25.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.234+0000 7f8760ff9700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87500682f0 0x7f875006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.234+0000 7f8760ff9700 1 -- 172.21.15.111:0/3805721921 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f8754085800 con 0x7f876406a7b0 2023-11-28T13:06:25.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.234+0000 7f87637fe700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87500682f0 0x7f875006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.234+0000 7f87637fe700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87500682f0 0x7f875006a7a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8758014040 tx=0x7f8758008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.238+0000 7f8760ff9700 1 -- 172.21.15.111:0/3805721921 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8754052220 con 0x7f876406a7b0 2023-11-28T13:06:25.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.419+0000 7f876a40d700 1 -- 172.21.15.111:0/3805721921 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8764066450 con 0x7f87500682f0 2023-11-28T13:06:25.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.420+0000 7f8760ff9700 1 -- 172.21.15.111:0/3805721921 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8764066450 con 0x7f87500682f0 2023-11-28T13:06:25.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.422+0000 7f876a40d700 1 -- 172.21.15.111:0/3805721921 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87500682f0 msgr2=0x7f875006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.422+0000 7f876a40d700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87500682f0 0x7f875006a7a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8758014040 tx=0x7f8758008030 comp rx=0 tx=0).stop 2023-11-28T13:06:25.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.422+0000 7f876a40d700 1 -- 172.21.15.111:0/3805721921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f876406a7b0 msgr2=0x7f87640b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.422+0000 7f876a40d700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f876406a7b0 0x7f87640b2fb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f875400e810 tx=0x7f875400ebd0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.422+0000 7f876a40d700 1 -- 172.21.15.111:0/3805721921 shutdown_connections 2023-11-28T13:06:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.422+0000 7f876a40d700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f876415a3c0 0x7f87640c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.422+0000 7f876a40d700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f876406a7b0 0x7f87640b2fb0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.422+0000 7f876a40d700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f87500682f0 0x7f875006a7a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.422+0000 7f876a40d700 1 --2- 172.21.15.111:0/3805721921 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f87640cb790 0x7f876415d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.422+0000 7f876a40d700 1 -- 172.21.15.111:0/3805721921 >> 172.21.15.111:0/3805721921 conn(0x7f87640b0db0 msgr2=0x7f87640c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:25.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.422+0000 7f876a40d700 1 -- 172.21.15.111:0/3805721921 shutdown_connections 2023-11-28T13:06:25.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.422+0000 7f876a40d700 1 -- 172.21.15.111:0/3805721921 wait complete. 2023-11-28T13:06:25.426 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 1 to deep-scrub 2023-11-28T13:06:25.435 INFO:tasks.ceph:Scrubbing osd.2 2023-11-28T13:06:25.436 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-11-28T13:06:25.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.558+0000 7fd35b23f700 1 Processor -- start 2023-11-28T13:06:25.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.558+0000 7fd35b23f700 1 -- start start 2023-11-28T13:06:25.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd35b23f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd3540b0d70 0x7fd3541604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd35b23f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd3540b1140 0x7fd3540ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd35b23f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35415cd90 con 0x7fd3540b15b0 2023-11-28T13:06:25.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd35b23f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3541613e0 con 0x7fd3540b1140 2023-11-28T13:06:25.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd35b23f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd354160b40 con 0x7fd3540b0d70 2023-11-28T13:06:25.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd353fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd3540b1140 0x7fd3540ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd3597dc700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd3540b15b0 0x7fd3540cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39626/0 (socket says 172.21.15.111:39626) 2023-11-28T13:06:25.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd3597dc700 1 -- 172.21.15.111:0/664829801 learned_addr learned my addr 172.21.15.111:0/664829801 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:25.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd358fdb700 1 --2- 172.21.15.111:0/664829801 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd3540b0d70 0x7fd3541604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd353fff700 1 -- 172.21.15.111:0/664829801 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd3540b0d70 msgr2=0x7fd3541604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:25.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd3537fe700 1 -- 172.21.15.111:0/664829801 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249170972 0 0) 0x7fd34c001490 con 0x7fd3540b15b0 2023-11-28T13:06:25.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd353fff700 1 --2- 172.21.15.111:0/664829801 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd3540b0d70 0x7fd3541604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd353fff700 1 -- 172.21.15.111:0/664829801 >> v1:172.21.15.111:6789/0 conn(0x7fd3540b15b0 legacy=0x7fd3540cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd353fff700 1 -- 172.21.15.111:0/664829801 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd354160e20 con 0x7fd3540b1140 2023-11-28T13:06:25.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd358fdb700 1 --2- 172.21.15.111:0/664829801 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd3540b0d70 0x7fd3541604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:25.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.559+0000 7fd353fff700 1 --2- 172.21.15.111:0/664829801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3540b1140 0x7fd3540ce6c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fd340009840 tx=0x7fd340009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf8f76bde93ef251 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.560+0000 7fd3537fe700 1 -- 172.21.15.111:0/664829801 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd340011830 con 0x7fd3540b1140 2023-11-28T13:06:25.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.560+0000 7fd3537fe700 1 -- 172.21.15.111:0/664829801 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd340011990 con 0x7fd3540b1140 2023-11-28T13:06:25.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.560+0000 7fd3537fe700 1 -- 172.21.15.111:0/664829801 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd340011d20 con 0x7fd3540b1140 2023-11-28T13:06:25.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.560+0000 7fd35b23f700 1 -- 172.21.15.111:0/664829801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3540b1140 msgr2=0x7fd3540ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.560+0000 7fd35b23f700 1 --2- 172.21.15.111:0/664829801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3540b1140 0x7fd3540ce6c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fd340009840 tx=0x7fd340009b50 comp rx=0 tx=0).stop 2023-11-28T13:06:25.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.560+0000 7fd35b23f700 1 -- 172.21.15.111:0/664829801 shutdown_connections 2023-11-28T13:06:25.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.560+0000 7fd35b23f700 1 --2- 172.21.15.111:0/664829801 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd3540b0d70 0x7fd3541604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.560+0000 7fd35b23f700 1 --2- 172.21.15.111:0/664829801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3540b1140 0x7fd3540ce6c0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.560+0000 7fd35b23f700 1 -- 172.21.15.111:0/664829801 >> 172.21.15.111:0/664829801 conn(0x7fd354098ee0 msgr2=0x7fd35406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:25.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.560+0000 7fd35b23f700 1 -- 172.21.15.111:0/664829801 shutdown_connections 2023-11-28T13:06:25.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.560+0000 7fd35b23f700 1 -- 172.21.15.111:0/664829801 wait complete. 2023-11-28T13:06:25.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.562+0000 7fd35b23f700 1 Processor -- start 2023-11-28T13:06:25.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.562+0000 7fd35b23f700 1 -- start start 2023-11-28T13:06:25.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd35b23f700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd3540b0d70 0x7fd35414ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd35b23f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3540b1140 0x7fd35415f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd35b23f700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd3540b15b0 0x7fd35414d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd35b23f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35415e3d0 con 0x7fd3540b1140 2023-11-28T13:06:25.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd35b23f700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35415e6b0 con 0x7fd3540b0d70 2023-11-28T13:06:25.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd35b23f700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35415e990 con 0x7fd3540b15b0 2023-11-28T13:06:25.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd353fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3540b1140 0x7fd35415f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd353fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3540b1140 0x7fd35415f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38852/0 (socket says 172.21.15.111:38852) 2023-11-28T13:06:25.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd353fff700 1 -- 172.21.15.111:0/2649693108 learned_addr learned my addr 172.21.15.111:0/2649693108 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:25.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd358fdb700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd3540b0d70 0x7fd35414ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd3597dc700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd3540b15b0 0x7fd35414d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd353fff700 1 -- 172.21.15.111:0/2649693108 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd3540b15b0 msgr2=0x7fd35414d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd353fff700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fd3540b15b0 0x7fd35414d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd353fff700 1 -- 172.21.15.111:0/2649693108 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd3540b0d70 msgr2=0x7fd35414ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd353fff700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fd3540b0d70 0x7fd35414ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd353fff700 1 -- 172.21.15.111:0/2649693108 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3400094a0 con 0x7fd3540b1140 2023-11-28T13:06:25.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd353fff700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3540b1140 0x7fd35415f470 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fd340000c00 tx=0x7fd340004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.563+0000 7fd3517fa700 1 -- 172.21.15.111:0/2649693108 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3400036a0 con 0x7fd3540b1140 2023-11-28T13:06:25.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.564+0000 7fd3517fa700 1 -- 172.21.15.111:0/2649693108 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34001f070 con 0x7fd3540b1140 2023-11-28T13:06:25.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.564+0000 7fd3517fa700 1 -- 172.21.15.111:0/2649693108 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd340003a40 con 0x7fd3540b1140 2023-11-28T13:06:25.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.564+0000 7fd35b23f700 1 -- 172.21.15.111:0/2649693108 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3540af340 con 0x7fd3540b1140 2023-11-28T13:06:25.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.564+0000 7fd35b23f700 1 -- 172.21.15.111:0/2649693108 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd354161100 con 0x7fd3540b1140 2023-11-28T13:06:25.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.564+0000 7fd35b23f700 1 -- 172.21.15.111:0/2649693108 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd3540cec00 con 0x7fd3540b1140 2023-11-28T13:06:25.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.565+0000 7fd3517fa700 1 -- 172.21.15.111:0/2649693108 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd34000f050 con 0x7fd3540b1140 2023-11-28T13:06:25.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.565+0000 7fd3517fa700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd3380682f0 0x7fd33806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.565+0000 7fd3517fa700 1 -- 172.21.15.111:0/2649693108 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fd340021070 con 0x7fd3540b1140 2023-11-28T13:06:25.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.565+0000 7fd358fdb700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd3380682f0 0x7fd33806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.565+0000 7fd3517fa700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] conn(0x7fd33806eb90 0x7fd338070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.565+0000 7fd3517fa700 1 -- 172.21.15.111:0/2649693108 --> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3380716b0 con 0x7fd33806eb90 2023-11-28T13:06:25.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.565+0000 7fd3517fa700 1 -- 172.21.15.111:0/2649693108 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3400899d0 con 0x7fd3540b1140 2023-11-28T13:06:25.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.565+0000 7fd3597dc700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] conn(0x7fd33806eb90 0x7fd338070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.566+0000 7fd358fdb700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd3380682f0 0x7fd33806a7a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd34800abe0 tx=0x7fd348008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.566+0000 7fd3597dc700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] conn(0x7fd33806eb90 0x7fd338070fe0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.566+0000 7fd3517fa700 1 -- 172.21.15.111:0/2649693108 <== osd.2 v2:172.21.15.111:6811/103530 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd3380716b0 con 0x7fd33806eb90 2023-11-28T13:06:25.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.581+0000 7fd35b23f700 1 -- 172.21.15.111:0/2649693108 --> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd3540cdfc0 con 0x7fd33806eb90 2023-11-28T13:06:25.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.581+0000 7fd3517fa700 1 -- 172.21.15.111:0/2649693108 <== osd.2 v2:172.21.15.111:6811/103530 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fd3540cdfc0 con 0x7fd33806eb90 2023-11-28T13:06:25.582 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:06:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-28T13:06:25.583 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:06:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.581+0000 7fd35b23f700 1 -- 172.21.15.111:0/2649693108 >> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] conn(0x7fd33806eb90 msgr2=0x7fd338070fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.581+0000 7fd35b23f700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.111:6811/103530,v1:172.21.15.111:6813/103530] conn(0x7fd33806eb90 0x7fd338070fe0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.581+0000 7fd35b23f700 1 -- 172.21.15.111:0/2649693108 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd3380682f0 msgr2=0x7fd33806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.581+0000 7fd35b23f700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fd3380682f0 0x7fd33806a7a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd34800abe0 tx=0x7fd348008040 comp rx=0 tx=0).stop 2023-11-28T13:06:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.581+0000 7fd35b23f700 1 -- 172.21.15.111:0/2649693108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3540b1140 msgr2=0x7fd35415f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.581+0000 7fd35b23f700 1 --2- 172.21.15.111:0/2649693108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3540b1140 0x7fd35415f470 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fd340000c00 tx=0x7fd340004750 comp rx=0 tx=0).stop 2023-11-28T13:06:25.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.581+0000 7fd3597dc700 1 -- 172.21.15.111:0/2649693108 reap_dead start 2023-11-28T13:06:25.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.581+0000 7fd35b23f700 1 -- 172.21.15.111:0/2649693108 shutdown_connections 2023-11-28T13:06:25.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.581+0000 7fd35b23f700 1 -- 172.21.15.111:0/2649693108 >> 172.21.15.111:0/2649693108 conn(0x7fd354098ee0 msgr2=0x7fd35415c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:25.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.581+0000 7fd35b23f700 1 -- 172.21.15.111:0/2649693108 shutdown_connections 2023-11-28T13:06:25.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.581+0000 7fd35b23f700 1 -- 172.21.15.111:0/2649693108 wait complete. 2023-11-28T13:06:25.594 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-11-28T13:06:25.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.715+0000 7f117b31c700 1 Processor -- start 2023-11-28T13:06:25.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.715+0000 7f117b31c700 1 -- start start 2023-11-28T13:06:25.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f117b31c700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f117406ba90 0x7f11740cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f117b31c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11740cb810 con 0x7f117415c7b0 2023-11-28T13:06:25.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f117b31c700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11740cc650 con 0x7f11740d0490 2023-11-28T13:06:25.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f117b31c700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11740cba40 con 0x7f117406ba90 2023-11-28T13:06:25.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f11790b8700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f117415c7b0 0x7f11740ad8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39638/0 (socket says 172.21.15.111:39638) 2023-11-28T13:06:25.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f11790b8700 1 -- 172.21.15.111:0/1653015497 learned_addr learned my addr 172.21.15.111:0/1653015497 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:25.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f11798b9700 1 --2- 172.21.15.111:0/1653015497 >> v2:172.21.15.149:3300/0 conn(0x7f117406ba90 0x7f11740cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f116bfff700 1 -- 172.21.15.111:0/1653015497 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412569456 0 0) 0x7f11740cb810 con 0x7f117415c7b0 2023-11-28T13:06:25.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f116bfff700 1 -- 172.21.15.111:0/1653015497 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11600035b0 con 0x7f117415c7b0 2023-11-28T13:06:25.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f11798b9700 1 -- 172.21.15.111:0/1653015497 >> v1:172.21.15.149:6789/0 conn(0x7f11740d0490 legacy=0x7f117406b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f11798b9700 1 -- 172.21.15.111:0/1653015497 >> v1:172.21.15.111:6789/0 conn(0x7f117415c7b0 legacy=0x7f11740ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f116bfff700 1 -- 172.21.15.111:0/1653015497 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1488266759 0 0) 0x7f11600035b0 con 0x7f117415c7b0 2023-11-28T13:06:25.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f11798b9700 1 -- 172.21.15.111:0/1653015497 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11740cbd20 con 0x7f117406ba90 2023-11-28T13:06:25.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.716+0000 7f11798b9700 1 --2- 172.21.15.111:0/1653015497 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f117406ba90 0x7f11740cb2d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f116c009d90 tx=0x7f116c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8944f2917610eae8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.717+0000 7f116bfff700 1 -- 172.21.15.111:0/1653015497 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c00e070 con 0x7f117406ba90 2023-11-28T13:06:25.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.717+0000 7f116bfff700 1 -- 172.21.15.111:0/1653015497 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116c003a20 con 0x7f117406ba90 2023-11-28T13:06:25.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.717+0000 7f116bfff700 1 -- 172.21.15.111:0/1653015497 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11700017f0 con 0x7f117406ba90 2023-11-28T13:06:25.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.717+0000 7f117b31c700 1 -- 172.21.15.111:0/1653015497 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f117406ba90 msgr2=0x7f11740cb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.717+0000 7f117b31c700 1 --2- 172.21.15.111:0/1653015497 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f117406ba90 0x7f11740cb2d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f116c009d90 tx=0x7f116c00b330 comp rx=0 tx=0).stop 2023-11-28T13:06:25.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.717+0000 7f117b31c700 1 -- 172.21.15.111:0/1653015497 shutdown_connections 2023-11-28T13:06:25.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.717+0000 7f117b31c700 1 --2- 172.21.15.111:0/1653015497 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f117406ba90 0x7f11740cb2d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.717+0000 7f117b31c700 1 -- 172.21.15.111:0/1653015497 >> 172.21.15.111:0/1653015497 conn(0x7f117409a040 msgr2=0x7f11740d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:25.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.717+0000 7f117b31c700 1 -- 172.21.15.111:0/1653015497 shutdown_connections 2023-11-28T13:06:25.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.717+0000 7f117b31c700 1 -- 172.21.15.111:0/1653015497 wait complete. 2023-11-28T13:06:25.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.718+0000 7f117b31c700 1 Processor -- start 2023-11-28T13:06:25.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.718+0000 7f117b31c700 1 -- start start 2023-11-28T13:06:25.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f117b31c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f117406ba90 0x7f11741f1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f117b31c700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11740d0490 0x7f117406fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f117b31c700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f117415c7b0 0x7f11740703e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f117b31c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11740b2bf0 con 0x7f117406ba90 2023-11-28T13:06:25.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f117b31c700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117415e0b0 con 0x7f11740d0490 2023-11-28T13:06:25.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f117b31c700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117415e390 con 0x7f117415c7b0 2023-11-28T13:06:25.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f11790b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f117406ba90 0x7f11741f1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f11790b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f117406ba90 0x7f11741f1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38868/0 (socket says 172.21.15.111:38868) 2023-11-28T13:06:25.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f11790b8700 1 -- 172.21.15.111:0/3635383454 learned_addr learned my addr 172.21.15.111:0/3635383454 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:25.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f11788b7700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11740d0490 0x7f117406fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f11798b9700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f117415c7b0 0x7f11740703e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f11790b8700 1 -- 172.21.15.111:0/3635383454 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f117415c7b0 msgr2=0x7f11740703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f11790b8700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f117415c7b0 0x7f11740703e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f11790b8700 1 -- 172.21.15.111:0/3635383454 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11740d0490 msgr2=0x7f117406fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.719+0000 7f11790b8700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11740d0490 0x7f117406fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.720+0000 7f11790b8700 1 -- 172.21.15.111:0/3635383454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c009a70 con 0x7f117406ba90 2023-11-28T13:06:25.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.720+0000 7f11790b8700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f117406ba90 0x7f11741f1880 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f117000c570 tx=0x7f117000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.720+0000 7f1169ffb700 1 -- 172.21.15.111:0/3635383454 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11700074e0 con 0x7f117406ba90 2023-11-28T13:06:25.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.720+0000 7f1169ffb700 1 -- 172.21.15.111:0/3635383454 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1170007640 con 0x7f117406ba90 2023-11-28T13:06:25.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.720+0000 7f117b31c700 1 -- 172.21.15.111:0/3635383454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11740cc2e0 con 0x7f117406ba90 2023-11-28T13:06:25.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.720+0000 7f1169ffb700 1 -- 172.21.15.111:0/3635383454 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1170005a70 con 0x7f117406ba90 2023-11-28T13:06:25.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.720+0000 7f117b31c700 1 -- 172.21.15.111:0/3635383454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11740cc000 con 0x7f117406ba90 2023-11-28T13:06:25.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.721+0000 7f117b31c700 1 -- 172.21.15.111:0/3635383454 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117415cb80 con 0x7f117406ba90 2023-11-28T13:06:25.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.721+0000 7f1169ffb700 1 -- 172.21.15.111:0/3635383454 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1170019030 con 0x7f117406ba90 2023-11-28T13:06:25.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.721+0000 7f1169ffb700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f11580682f0 0x7f115806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:25.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.722+0000 7f1169ffb700 1 -- 172.21.15.111:0/3635383454 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f1170084e40 con 0x7f117406ba90 2023-11-28T13:06:25.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.722+0000 7f11788b7700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f11580682f0 0x7f115806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:25.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.722+0000 7f11788b7700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f11580682f0 0x7f115806a7a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f11741f5480 tx=0x7f1164009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:25.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.726+0000 7f1169ffb700 1 -- 172.21.15.111:0/3635383454 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1170051860 con 0x7f117406ba90 2023-11-28T13:06:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.909+0000 7f117b31c700 1 -- 172.21.15.111:0/3635383454 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1174067170 con 0x7f11580682f0 2023-11-28T13:06:25.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.909+0000 7f1169ffb700 1 -- 172.21.15.111:0/3635383454 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1174067170 con 0x7f11580682f0 2023-11-28T13:06:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.911+0000 7f117b31c700 1 -- 172.21.15.111:0/3635383454 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f11580682f0 msgr2=0x7f115806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.911+0000 7f117b31c700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f11580682f0 0x7f115806a7a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f11741f5480 tx=0x7f1164009040 comp rx=0 tx=0).stop 2023-11-28T13:06:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.911+0000 7f117b31c700 1 -- 172.21.15.111:0/3635383454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f117406ba90 msgr2=0x7f11741f1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.911+0000 7f117b31c700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f117406ba90 0x7f11741f1880 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f117000c570 tx=0x7f117000c880 comp rx=0 tx=0).stop 2023-11-28T13:06:25.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.911+0000 7f117b31c700 1 -- 172.21.15.111:0/3635383454 shutdown_connections 2023-11-28T13:06:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.911+0000 7f117b31c700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f117415c7b0 0x7f11740703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.911+0000 7f117b31c700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f117406ba90 0x7f11741f1880 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.911+0000 7f117b31c700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f11580682f0 0x7f115806a7a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.911+0000 7f117b31c700 1 --2- 172.21.15.111:0/3635383454 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f11740d0490 0x7f117406fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.911+0000 7f117b31c700 1 -- 172.21.15.111:0/3635383454 >> 172.21.15.111:0/3635383454 conn(0x7f117409a040 msgr2=0x7f117406b380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.911+0000 7f117b31c700 1 -- 172.21.15.111:0/3635383454 shutdown_connections 2023-11-28T13:06:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:25.911+0000 7f117b31c700 1 -- 172.21.15.111:0/3635383454 wait complete. 2023-11-28T13:06:25.914 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 2 to deep-scrub 2023-11-28T13:06:25.924 INFO:tasks.ceph:Scrubbing osd.3 2023-11-28T13:06:25.925 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-11-28T13:06:26.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.048+0000 7fb446d74700 1 Processor -- start 2023-11-28T13:06:26.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.048+0000 7fb446d74700 1 -- start start 2023-11-28T13:06:26.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb446d74700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb4400b0d70 0x7fb4401604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb446d74700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb4400b1140 0x7fb4400ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb446d74700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44015cd90 con 0x7fb4400b15b0 2023-11-28T13:06:26.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb446d74700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4401613e0 con 0x7fb4400b1140 2023-11-28T13:06:26.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb446d74700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb440160b40 con 0x7fb4400b0d70 2023-11-28T13:06:26.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb445311700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb4400b15b0 0x7fb4400cc550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39656/0 (socket says 172.21.15.111:39656) 2023-11-28T13:06:26.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb445311700 1 -- 172.21.15.111:0/3187164124 learned_addr learned my addr 172.21.15.111:0/3187164124 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:26.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb444b10700 1 --2- 172.21.15.111:0/3187164124 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb4400b0d70 0x7fb4401604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43ffff700 1 --2- 172.21.15.111:0/3187164124 >> v2:172.21.15.149:3300/0 conn(0x7fb4400b1140 0x7fb4400ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073860557 0 0) 0x7fb438001490 con 0x7fb4400b15b0 2023-11-28T13:06:26.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4300035b0 con 0x7fb4400b15b0 2023-11-28T13:06:26.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3270452089 0 0) 0x7fb4300035b0 con 0x7fb4400b15b0 2023-11-28T13:06:26.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb4400b0d70 msgr2=0x7fb4401604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:26.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 --2- 172.21.15.111:0/3187164124 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb4400b0d70 0x7fb4401604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 >> v2:172.21.15.149:3300/0 conn(0x7fb4400b1140 msgr2=0x7fb4400ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:26.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 --2- 172.21.15.111:0/3187164124 >> v2:172.21.15.149:3300/0 conn(0x7fb4400b1140 0x7fb4400ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb440160e20 con 0x7fb4400b15b0 2023-11-28T13:06:26.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fb438001b80 con 0x7fb4400b15b0 2023-11-28T13:06:26.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 >> v1:172.21.15.111:6789/0 conn(0x7fb4400b15b0 legacy=0x7fb4400cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 --2- 172.21.15.111:0/3187164124 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb430003a00 0x7fb430003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 --2- 172.21.15.111:0/3187164124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb430004430 0x7fb4300048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 --2- 172.21.15.111:0/3187164124 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb430004f60 0x7fb43000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4401613e0 con 0x7fb430004430 2023-11-28T13:06:26.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb440160b40 con 0x7fb430003a00 2023-11-28T13:06:26.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43000bf50 con 0x7fb430004f60 2023-11-28T13:06:26.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb444b10700 1 --2- 172.21.15.111:0/3187164124 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb4400b0d70 0x7fb4401604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:06:26.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.049+0000 7fb43ffff700 1 --2- 172.21.15.111:0/3187164124 >> v2:172.21.15.149:3300/0 conn(0x7fb4400b1140 0x7fb4400ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:06:26.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.050+0000 7fb43ffff700 1 --2- 172.21.15.111:0/3187164124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb430004430 0x7fb4300048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.050+0000 7fb445311700 1 --2- 172.21.15.111:0/3187164124 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb430004f60 0x7fb43000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.050+0000 7fb444b10700 1 --2- 172.21.15.111:0/3187164124 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb430003a00 0x7fb430003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.050+0000 7fb43ffff700 1 -- 172.21.15.111:0/3187164124 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb430004f60 msgr2=0x7fb43000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:26.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.050+0000 7fb43ffff700 1 --2- 172.21.15.111:0/3187164124 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb430004f60 0x7fb43000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.050+0000 7fb43ffff700 1 -- 172.21.15.111:0/3187164124 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb430003a00 msgr2=0x7fb430003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:26.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.050+0000 7fb43ffff700 1 --2- 172.21.15.111:0/3187164124 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb430003a00 0x7fb430003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.050+0000 7fb43ffff700 1 -- 172.21.15.111:0/3187164124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4300035b0 con 0x7fb430004430 2023-11-28T13:06:26.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.050+0000 7fb43ffff700 1 --2- 172.21.15.111:0/3187164124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb430004430 0x7fb4300048f0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009d10 tx=0x7fb430005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9cb043c78196f6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.050+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c009e60 con 0x7fb430004430 2023-11-28T13:06:26.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.050+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c00e9a0 con 0x7fb430004430 2023-11-28T13:06:26.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.050+0000 7fb43f7fe700 1 -- 172.21.15.111:0/3187164124 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c00ed10 con 0x7fb430004430 2023-11-28T13:06:26.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.051+0000 7fb446d74700 1 -- 172.21.15.111:0/3187164124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb430004430 msgr2=0x7fb4300048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.051+0000 7fb446d74700 1 --2- 172.21.15.111:0/3187164124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb430004430 0x7fb4300048f0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009d10 tx=0x7fb430005880 comp rx=0 tx=0).stop 2023-11-28T13:06:26.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.051+0000 7fb445311700 1 -- 172.21.15.111:0/3187164124 reap_dead start 2023-11-28T13:06:26.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.051+0000 7fb446d74700 1 -- 172.21.15.111:0/3187164124 shutdown_connections 2023-11-28T13:06:26.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.051+0000 7fb446d74700 1 -- 172.21.15.111:0/3187164124 >> 172.21.15.111:0/3187164124 conn(0x7fb440098ee0 msgr2=0x7fb44006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:26.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.051+0000 7fb446d74700 1 -- 172.21.15.111:0/3187164124 shutdown_connections 2023-11-28T13:06:26.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.051+0000 7fb446d74700 1 -- 172.21.15.111:0/3187164124 wait complete. 2023-11-28T13:06:26.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.052+0000 7fb446d74700 1 Processor -- start 2023-11-28T13:06:26.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.052+0000 7fb446d74700 1 -- start start 2023-11-28T13:06:26.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb446d74700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb430004430 0x7fb4401efb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb446d74700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4401f64e0 0x7fb4400ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb446d74700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb4400adee0 0x7fb4400b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb446d74700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4400ae360 con 0x7fb4401f64e0 2023-11-28T13:06:26.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb446d74700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb440156b00 con 0x7fb430004430 2023-11-28T13:06:26.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb446d74700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4401f27f0 con 0x7fb4400adee0 2023-11-28T13:06:26.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb43ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4401f64e0 0x7fb4400ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb43ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4401f64e0 0x7fb4400ad9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38888/0 (socket says 172.21.15.111:38888) 2023-11-28T13:06:26.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb43ffff700 1 -- 172.21.15.111:0/3680326727 learned_addr learned my addr 172.21.15.111:0/3680326727 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:26.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb444b10700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb430004430 0x7fb4401efb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb445311700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb4400adee0 0x7fb4400b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb43ffff700 1 -- 172.21.15.111:0/3680326727 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb4400adee0 msgr2=0x7fb4400b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb43ffff700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb4400adee0 0x7fb4400b2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb43ffff700 1 -- 172.21.15.111:0/3680326727 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb430004430 msgr2=0x7fb4401efb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb43ffff700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb430004430 0x7fb4401efb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb43ffff700 1 -- 172.21.15.111:0/3680326727 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4400af340 con 0x7fb4401f64e0 2023-11-28T13:06:26.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb43ffff700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4401f64e0 0x7fb4400ad9a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fb4300056f0 tx=0x7fb42c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb43d7fa700 1 -- 172.21.15.111:0/3680326727 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c020070 con 0x7fb4401f64e0 2023-11-28T13:06:26.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb43d7fa700 1 -- 172.21.15.111:0/3680326727 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c009e60 con 0x7fb4401f64e0 2023-11-28T13:06:26.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb446d74700 1 -- 172.21.15.111:0/3680326727 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb440161100 con 0x7fb4401f64e0 2023-11-28T13:06:26.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.053+0000 7fb43d7fa700 1 -- 172.21.15.111:0/3680326727 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c003af0 con 0x7fb4401f64e0 2023-11-28T13:06:26.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.054+0000 7fb446d74700 1 -- 172.21.15.111:0/3680326727 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4400cec00 con 0x7fb4401f64e0 2023-11-28T13:06:26.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.054+0000 7fb446d74700 1 -- 172.21.15.111:0/3680326727 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb440067170 con 0x7fb4401f64e0 2023-11-28T13:06:26.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.055+0000 7fb43d7fa700 1 -- 172.21.15.111:0/3680326727 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb42c02c030 con 0x7fb4401f64e0 2023-11-28T13:06:26.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.055+0000 7fb43d7fa700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb4240682f0 0x7fb42406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.055+0000 7fb43d7fa700 1 -- 172.21.15.111:0/3680326727 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fb42c086e10 con 0x7fb4401f64e0 2023-11-28T13:06:26.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.055+0000 7fb444b10700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb4240682f0 0x7fb42406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.055+0000 7fb43d7fa700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] conn(0x7fb42406eb90 0x7fb424070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.055+0000 7fb43d7fa700 1 -- 172.21.15.111:0/3680326727 --> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4240716b0 con 0x7fb42406eb90 2023-11-28T13:06:26.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.055+0000 7fb43d7fa700 1 -- 172.21.15.111:0/3680326727 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb42c0894c0 con 0x7fb4401f64e0 2023-11-28T13:06:26.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.055+0000 7fb445311700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] conn(0x7fb42406eb90 0x7fb424070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.056+0000 7fb444b10700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb4240682f0 0x7fb42406a7a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb4400b5190 tx=0x7fb4340091c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.056+0000 7fb445311700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] conn(0x7fb42406eb90 0x7fb424070fe0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.056+0000 7fb43d7fa700 1 -- 172.21.15.111:0/3680326727 <== osd.3 v2:172.21.15.111:6808/103538 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb4240716b0 con 0x7fb42406eb90 2023-11-28T13:06:26.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.071+0000 7fb446d74700 1 -- 172.21.15.111:0/3680326727 --> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb4400ae2e0 con 0x7fb42406eb90 2023-11-28T13:06:26.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.071+0000 7fb43d7fa700 1 -- 172.21.15.111:0/3680326727 <== osd.3 v2:172.21.15.111:6808/103538 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb4400ae2e0 con 0x7fb42406eb90 2023-11-28T13:06:26.079 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:06:26.079 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-28T13:06:26.079 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:06:26.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.071+0000 7fb446d74700 1 -- 172.21.15.111:0/3680326727 >> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] conn(0x7fb42406eb90 msgr2=0x7fb424070fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.071+0000 7fb446d74700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.111:6808/103538,v1:172.21.15.111:6809/103538] conn(0x7fb42406eb90 0x7fb424070fe0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.071+0000 7fb446d74700 1 -- 172.21.15.111:0/3680326727 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb4240682f0 msgr2=0x7fb42406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.071+0000 7fb446d74700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb4240682f0 0x7fb42406a7a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb4400b5190 tx=0x7fb4340091c0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.071+0000 7fb446d74700 1 -- 172.21.15.111:0/3680326727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4401f64e0 msgr2=0x7fb4400ad9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.071+0000 7fb446d74700 1 --2- 172.21.15.111:0/3680326727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4401f64e0 0x7fb4400ad9a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fb4300056f0 tx=0x7fb42c0047c0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.071+0000 7fb445311700 1 -- 172.21.15.111:0/3680326727 reap_dead start 2023-11-28T13:06:26.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.071+0000 7fb446d74700 1 -- 172.21.15.111:0/3680326727 shutdown_connections 2023-11-28T13:06:26.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.071+0000 7fb446d74700 1 -- 172.21.15.111:0/3680326727 >> 172.21.15.111:0/3680326727 conn(0x7fb440098ee0 msgr2=0x7fb44006dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:26.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.071+0000 7fb446d74700 1 -- 172.21.15.111:0/3680326727 shutdown_connections 2023-11-28T13:06:26.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.071+0000 7fb446d74700 1 -- 172.21.15.111:0/3680326727 wait complete. 2023-11-28T13:06:26.084 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-11-28T13:06:26.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.203+0000 7f70b1662700 1 Processor -- start 2023-11-28T13:06:26.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.203+0000 7f70b1662700 1 -- start start 2023-11-28T13:06:26.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70b1662700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70ac158cb0 0x7f70ac0ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70b1662700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f70ac071020 0x7f70ac06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70b1662700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ac06d870 con 0x7f70ac159080 2023-11-28T13:06:26.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70b1662700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ac06cc60 con 0x7f70ac071020 2023-11-28T13:06:26.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70b1662700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac06cf40 con 0x7f70ac158cb0 2023-11-28T13:06:26.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70ab7fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f70ac071020 0x7f70ac06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70aaffd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70ac158cb0 0x7f70ac0ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70aa7fc700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f70ac159080 0x7f70ac070910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:48216/0 (socket says 172.21.15.111:48216) 2023-11-28T13:06:26.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70aa7fc700 1 -- 172.21.15.111:0/2617453183 learned_addr learned my addr 172.21.15.111:0/2617453183 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:26.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70aaffd700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70ac158cb0 0x7f70ac0ca1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:60644/0 (socket says 172.21.15.111:60644) 2023-11-28T13:06:26.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70ab7fe700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f70ac071020 0x7f70ac06dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:56800/0 (socket says 172.21.15.111:56800) 2023-11-28T13:06:26.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70ab7fe700 1 -- 172.21.15.111:0/2617453183 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70ac158cb0 msgr2=0x7f70ac0ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:26.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70ab7fe700 1 --2- 172.21.15.111:0/2617453183 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70ac158cb0 0x7f70ac0ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70ab7fe700 1 -- 172.21.15.111:0/2617453183 >> v1:172.21.15.149:6789/0 conn(0x7f70ac159080 legacy=0x7f70ac070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70ab7fe700 1 -- 172.21.15.111:0/2617453183 --> v2:172.21.15.149:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac06d220 con 0x7f70ac071020 2023-11-28T13:06:26.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70a9ffb700 1 -- 172.21.15.111:0/2617453183 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780102929 0 0) 0x7f70ac06d870 con 0x7f70ac159080 2023-11-28T13:06:26.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.204+0000 7f70aaffd700 1 --2- 172.21.15.111:0/2617453183 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70ac158cb0 0x7f70ac0ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:06:26.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.205+0000 7f70ab7fe700 1 --2- 172.21.15.111:0/2617453183 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70ac071020 0x7f70ac06dd00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f70a4009d90 tx=0x7f70a400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42524b25c83f33f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.205+0000 7f70a9ffb700 1 -- 172.21.15.111:0/2617453183 <== mon.1 v2:172.21.15.149:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a400e070 con 0x7f70ac071020 2023-11-28T13:06:26.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.205+0000 7f70a9ffb700 1 -- 172.21.15.111:0/2617453183 <== mon.1 v2:172.21.15.149:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a400bb20 con 0x7f70ac071020 2023-11-28T13:06:26.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.205+0000 7f70a9ffb700 1 -- 172.21.15.111:0/2617453183 <== mon.1 v2:172.21.15.149:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a4012430 con 0x7f70ac071020 2023-11-28T13:06:26.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.205+0000 7f70b1662700 1 -- 172.21.15.111:0/2617453183 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70ac071020 msgr2=0x7f70ac06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.205+0000 7f70b1662700 1 --2- 172.21.15.111:0/2617453183 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70ac071020 0x7f70ac06dd00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f70a4009d90 tx=0x7f70a400b330 comp rx=0 tx=0).stop 2023-11-28T13:06:26.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.205+0000 7f70b1662700 1 -- 172.21.15.111:0/2617453183 shutdown_connections 2023-11-28T13:06:26.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.205+0000 7f70b1662700 1 --2- 172.21.15.111:0/2617453183 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70ac071020 0x7f70ac06dd00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.205+0000 7f70b1662700 1 --2- 172.21.15.111:0/2617453183 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70ac158cb0 0x7f70ac0ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.205+0000 7f70b1662700 1 -- 172.21.15.111:0/2617453183 >> 172.21.15.111:0/2617453183 conn(0x7f70ac0cb490 msgr2=0x7f70ac0cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:26.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.205+0000 7f70b1662700 1 -- 172.21.15.111:0/2617453183 shutdown_connections 2023-11-28T13:06:26.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.205+0000 7f70b1662700 1 -- 172.21.15.111:0/2617453183 wait complete. 2023-11-28T13:06:26.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.207+0000 7f70b1662700 1 Processor -- start 2023-11-28T13:06:26.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.207+0000 7f70b1662700 1 -- start start 2023-11-28T13:06:26.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.207+0000 7f70b1662700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70ac071020 0x7f70ac150c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.207+0000 7f70b1662700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70ac158cb0 0x7f70ac15a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.207+0000 7f70b1662700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac0ac270 0x7f70ac15aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.207+0000 7f70b1662700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac0ad2d0 con 0x7f70ac0ac270 2023-11-28T13:06:26.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.207+0000 7f70b1662700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac1511d0 con 0x7f70ac071020 2023-11-28T13:06:26.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.207+0000 7f70b1662700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac0ac750 con 0x7f70ac158cb0 2023-11-28T13:06:26.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70ab7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac0ac270 0x7f70ac15aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70ab7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac0ac270 0x7f70ac15aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38906/0 (socket says 172.21.15.111:38906) 2023-11-28T13:06:26.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70ab7fe700 1 -- 172.21.15.111:0/2944394323 learned_addr learned my addr 172.21.15.111:0/2944394323 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:26.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70aaffd700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70ac071020 0x7f70ac150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70aa7fc700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70ac158cb0 0x7f70ac15a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70ab7fe700 1 -- 172.21.15.111:0/2944394323 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70ac158cb0 msgr2=0x7f70ac15a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70ab7fe700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70ac158cb0 0x7f70ac15a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70ab7fe700 1 -- 172.21.15.111:0/2944394323 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70ac071020 msgr2=0x7f70ac150c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70ab7fe700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70ac071020 0x7f70ac150c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70ab7fe700 1 -- 172.21.15.111:0/2944394323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a000b740 con 0x7f70ac0ac270 2023-11-28T13:06:26.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70ab7fe700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac0ac270 0x7f70ac15aaf0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f70a4014040 tx=0x7f70a4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70937fe700 1 -- 172.21.15.111:0/2944394323 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a4012430 con 0x7f70ac0ac270 2023-11-28T13:06:26.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70937fe700 1 -- 172.21.15.111:0/2944394323 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a4020070 con 0x7f70ac0ac270 2023-11-28T13:06:26.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70b1662700 1 -- 172.21.15.111:0/2944394323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a4009a70 con 0x7f70ac0ac270 2023-11-28T13:06:26.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70937fe700 1 -- 172.21.15.111:0/2944394323 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a4004280 con 0x7f70ac0ac270 2023-11-28T13:06:26.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.208+0000 7f70b1662700 1 -- 172.21.15.111:0/2944394323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ac071b20 con 0x7f70ac0ac270 2023-11-28T13:06:26.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.209+0000 7f70b1662700 1 -- 172.21.15.111:0/2944394323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ac06d500 con 0x7f70ac0ac270 2023-11-28T13:06:26.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.210+0000 7f70937fe700 1 -- 172.21.15.111:0/2944394323 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f70a40043e0 con 0x7f70ac0ac270 2023-11-28T13:06:26.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.210+0000 7f70937fe700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f70980682f0 0x7f709806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.210+0000 7f70937fe700 1 -- 172.21.15.111:0/2944394323 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f70a4022070 con 0x7f70ac0ac270 2023-11-28T13:06:26.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.210+0000 7f70aaffd700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f70980682f0 0x7f709806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.210+0000 7f70aaffd700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f70980682f0 0x7f709806a7a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f70a000bef0 tx=0x7f70a000b420 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.214+0000 7f70937fe700 1 -- 172.21.15.111:0/2944394323 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f70a4054850 con 0x7f70ac0ac270 2023-11-28T13:06:26.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.399+0000 7f70b1662700 1 -- 172.21.15.111:0/2944394323 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f70ac0c9300 con 0x7f70980682f0 2023-11-28T13:06:26.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.399+0000 7f70937fe700 1 -- 172.21.15.111:0/2944394323 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f70ac0c9300 con 0x7f70980682f0 2023-11-28T13:06:26.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.402+0000 7f70b1662700 1 -- 172.21.15.111:0/2944394323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f70980682f0 msgr2=0x7f709806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.402+0000 7f70b1662700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f70980682f0 0x7f709806a7a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f70a000bef0 tx=0x7f70a000b420 comp rx=0 tx=0).stop 2023-11-28T13:06:26.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.402+0000 7f70b1662700 1 -- 172.21.15.111:0/2944394323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac0ac270 msgr2=0x7f70ac15aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.402+0000 7f70b1662700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac0ac270 0x7f70ac15aaf0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f70a4014040 tx=0x7f70a4004ad0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.402+0000 7f70b1662700 1 -- 172.21.15.111:0/2944394323 shutdown_connections 2023-11-28T13:06:26.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.402+0000 7f70b1662700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac0ac270 0x7f70ac15aaf0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.402+0000 7f70b1662700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f70980682f0 0x7f709806a7a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.402+0000 7f70b1662700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f70ac071020 0x7f70ac150c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.402+0000 7f70b1662700 1 --2- 172.21.15.111:0/2944394323 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f70ac158cb0 0x7f70ac15a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.402+0000 7f70b1662700 1 -- 172.21.15.111:0/2944394323 >> 172.21.15.111:0/2944394323 conn(0x7f70ac0cb490 msgr2=0x7f70ac0b4120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:26.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.402+0000 7f70b1662700 1 -- 172.21.15.111:0/2944394323 shutdown_connections 2023-11-28T13:06:26.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.402+0000 7f70b1662700 1 -- 172.21.15.111:0/2944394323 wait complete. 2023-11-28T13:06:26.405 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 3 to deep-scrub 2023-11-28T13:06:26.415 INFO:tasks.ceph:Scrubbing osd.4 2023-11-28T13:06:26.416 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-11-28T13:06:26.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.540+0000 7f316ba28700 1 Processor -- start 2023-11-28T13:06:26.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.540+0000 7f316ba28700 1 -- start start 2023-11-28T13:06:26.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.541+0000 7f316ba28700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f316415a5d0 0x7f31640ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.541+0000 7f316ba28700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f31640b0160 0x7f31640ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.541+0000 7f316ba28700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31640ae8b0 con 0x7f31640cb2d0 2023-11-28T13:06:26.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.541+0000 7f316ba28700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31640cc1b0 con 0x7f31640b0160 2023-11-28T13:06:26.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.541+0000 7f316ba28700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31640cc390 con 0x7f316415a5d0 2023-11-28T13:06:26.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.541+0000 7f3168fc3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f31640b0160 0x7f31640ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.541+0000 7f3168fc3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f31640b0160 0x7f31640ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38912/0 (socket says 172.21.15.111:38912) 2023-11-28T13:06:26.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.541+0000 7f3168fc3700 1 -- 172.21.15.111:0/1986683870 learned_addr learned my addr 172.21.15.111:0/1986683870 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:26.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.541+0000 7f31697c4700 1 --2- 172.21.15.111:0/1986683870 >> v2:172.21.15.149:3300/0 conn(0x7f316415a5d0 0x7f31640ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.542+0000 7f3168fc3700 1 -- 172.21.15.111:0/1986683870 >> v2:172.21.15.149:3300/0 conn(0x7f316415a5d0 msgr2=0x7f31640ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:26.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.542+0000 7f3168fc3700 1 --2- 172.21.15.111:0/1986683870 >> v2:172.21.15.149:3300/0 conn(0x7f316415a5d0 0x7f31640ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.542+0000 7f3168fc3700 1 -- 172.21.15.111:0/1986683870 >> v1:172.21.15.149:6789/0 conn(0x7f31640cb2d0 legacy=0x7f316415e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.542+0000 7f3168fc3700 1 -- 172.21.15.111:0/1986683870 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31640cc670 con 0x7f31640b0160 2023-11-28T13:06:26.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.542+0000 7f3168fc3700 1 --2- 172.21.15.111:0/1986683870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31640b0160 0x7f31640ced90 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f3164067d30 tx=0x7f3160009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c4aa8bf6214954 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.542+0000 7f315bfff700 1 -- 172.21.15.111:0/1986683870 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3160011830 con 0x7f31640b0160 2023-11-28T13:06:26.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.542+0000 7f315bfff700 1 -- 172.21.15.111:0/1986683870 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3160011990 con 0x7f31640b0160 2023-11-28T13:06:26.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.542+0000 7f315bfff700 1 -- 172.21.15.111:0/1986683870 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3160011d20 con 0x7f31640b0160 2023-11-28T13:06:26.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.542+0000 7f316ba28700 1 -- 172.21.15.111:0/1986683870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31640b0160 msgr2=0x7f31640ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.542+0000 7f316ba28700 1 --2- 172.21.15.111:0/1986683870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31640b0160 0x7f31640ced90 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f3164067d30 tx=0x7f3160009bd0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.542+0000 7f316ba28700 1 -- 172.21.15.111:0/1986683870 shutdown_connections 2023-11-28T13:06:26.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.543+0000 7f316ba28700 1 --2- 172.21.15.111:0/1986683870 >> v2:172.21.15.149:3300/0 conn(0x7f316415a5d0 0x7f31640ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.543+0000 7f316ba28700 1 --2- 172.21.15.111:0/1986683870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31640b0160 0x7f31640ced90 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.543+0000 7f316ba28700 1 -- 172.21.15.111:0/1986683870 >> 172.21.15.111:0/1986683870 conn(0x7f316409a040 msgr2=0x7f31640d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:26.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.543+0000 7f316ba28700 1 -- 172.21.15.111:0/1986683870 shutdown_connections 2023-11-28T13:06:26.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.543+0000 7f316ba28700 1 -- 172.21.15.111:0/1986683870 wait complete. 2023-11-28T13:06:26.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.544+0000 7f316ba28700 1 Processor -- start 2023-11-28T13:06:26.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.544+0000 7f316ba28700 1 -- start start 2023-11-28T13:06:26.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.544+0000 7f316ba28700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f31640b0160 0x7f31641ef8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.544+0000 7f316ba28700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31640cb2d0 0x7f31640cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.544+0000 7f316ba28700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f316415a5d0 0x7f31640cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.544+0000 7f316ba28700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31641efe00 con 0x7f31640cb2d0 2023-11-28T13:06:26.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.544+0000 7f316ba28700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31640c4930 con 0x7f316415a5d0 2023-11-28T13:06:26.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.544+0000 7f316ba28700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31640c4c10 con 0x7f31640b0160 2023-11-28T13:06:26.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f3168fc3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31640cb2d0 0x7f31640cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f3168fc3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31640cb2d0 0x7f31640cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38918/0 (socket says 172.21.15.111:38918) 2023-11-28T13:06:26.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f3168fc3700 1 -- 172.21.15.111:0/766534998 learned_addr learned my addr 172.21.15.111:0/766534998 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:26.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f31697c4700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f31640b0160 0x7f31641ef8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f3169fc5700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f316415a5d0 0x7f31640cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f3168fc3700 1 -- 172.21.15.111:0/766534998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f31640b0160 msgr2=0x7f31641ef8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f3168fc3700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f31640b0160 0x7f31641ef8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f3168fc3700 1 -- 172.21.15.111:0/766534998 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f316415a5d0 msgr2=0x7f31640cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f3168fc3700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f316415a5d0 0x7f31640cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f3168fc3700 1 -- 172.21.15.111:0/766534998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3160009550 con 0x7f31640cb2d0 2023-11-28T13:06:26.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f3168fc3700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31640cb2d0 0x7f31640cdaa0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f3160009c10 tx=0x7f3160004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f3159ffb700 1 -- 172.21.15.111:0/766534998 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31600036a0 con 0x7f31640cb2d0 2023-11-28T13:06:26.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f3159ffb700 1 -- 172.21.15.111:0/766534998 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f316001e070 con 0x7f31640cb2d0 2023-11-28T13:06:26.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f316ba28700 1 -- 172.21.15.111:0/766534998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31640ccc30 con 0x7f31640cb2d0 2023-11-28T13:06:26.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f3159ffb700 1 -- 172.21.15.111:0/766534998 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3160003a60 con 0x7f31640cb2d0 2023-11-28T13:06:26.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.545+0000 7f316ba28700 1 -- 172.21.15.111:0/766534998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31640cc950 con 0x7f31640cb2d0 2023-11-28T13:06:26.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.546+0000 7f316ba28700 1 -- 172.21.15.111:0/766534998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f31640ccf10 con 0x7f31640cb2d0 2023-11-28T13:06:26.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.547+0000 7f3159ffb700 1 -- 172.21.15.111:0/766534998 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f316000f050 con 0x7f31640cb2d0 2023-11-28T13:06:26.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.547+0000 7f3159ffb700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f31480682f0 0x7f314806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.547+0000 7f3159ffb700 1 -- 172.21.15.111:0/766534998 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f3160086a20 con 0x7f31640cb2d0 2023-11-28T13:06:26.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.547+0000 7f31697c4700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f31480682f0 0x7f314806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.547+0000 7f3159ffb700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] conn(0x7f314806eb90 0x7f3148070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.547+0000 7f3159ffb700 1 -- 172.21.15.111:0/766534998 --> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f31480716b0 con 0x7f314806eb90 2023-11-28T13:06:26.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.547+0000 7f3159ffb700 1 -- 172.21.15.111:0/766534998 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f316008b2a0 con 0x7f31640cb2d0 2023-11-28T13:06:26.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.547+0000 7f31697c4700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f31480682f0 0x7f314806a7a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f31640c63b0 tx=0x7f3154008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.547+0000 7f3169fc5700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] conn(0x7f314806eb90 0x7f3148070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.548+0000 7f3169fc5700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] conn(0x7f314806eb90 0x7f3148070fe0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.549+0000 7f3159ffb700 1 -- 172.21.15.111:0/766534998 <== osd.4 v2:172.21.15.149:6800/103794 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f31480716b0 con 0x7f314806eb90 2023-11-28T13:06:26.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.562+0000 7f316ba28700 1 -- 172.21.15.111:0/766534998 --> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f31640b46c0 con 0x7f314806eb90 2023-11-28T13:06:26.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.563+0000 7f3159ffb700 1 -- 172.21.15.111:0/766534998 <== osd.4 v2:172.21.15.149:6800/103794 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f31640b46c0 con 0x7f314806eb90 2023-11-28T13:06:26.574 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:06:26.574 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-28T13:06:26.574 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:06:26.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.563+0000 7f316ba28700 1 -- 172.21.15.111:0/766534998 >> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] conn(0x7f314806eb90 msgr2=0x7f3148070fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.563+0000 7f316ba28700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.149:6800/103794,v1:172.21.15.149:6801/103794] conn(0x7f314806eb90 0x7f3148070fe0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.563+0000 7f316ba28700 1 -- 172.21.15.111:0/766534998 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f31480682f0 msgr2=0x7f314806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.564+0000 7f316ba28700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f31480682f0 0x7f314806a7a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f31640c63b0 tx=0x7f3154008040 comp rx=0 tx=0).stop 2023-11-28T13:06:26.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.564+0000 7f316ba28700 1 -- 172.21.15.111:0/766534998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31640cb2d0 msgr2=0x7f31640cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.564+0000 7f316ba28700 1 --2- 172.21.15.111:0/766534998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31640cb2d0 0x7f31640cdaa0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f3160009c10 tx=0x7f3160004750 comp rx=0 tx=0).stop 2023-11-28T13:06:26.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.564+0000 7f3169fc5700 1 -- 172.21.15.111:0/766534998 reap_dead start 2023-11-28T13:06:26.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.564+0000 7f316ba28700 1 -- 172.21.15.111:0/766534998 shutdown_connections 2023-11-28T13:06:26.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.564+0000 7f316ba28700 1 -- 172.21.15.111:0/766534998 >> 172.21.15.111:0/766534998 conn(0x7f316409a040 msgr2=0x7f31640cd330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:26.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.564+0000 7f316ba28700 1 -- 172.21.15.111:0/766534998 shutdown_connections 2023-11-28T13:06:26.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.564+0000 7f316ba28700 1 -- 172.21.15.111:0/766534998 wait complete. 2023-11-28T13:06:26.581 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-11-28T13:06:26.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.701+0000 7f678e3a6700 1 Processor -- start 2023-11-28T13:06:26.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.702+0000 7f678e3a6700 1 -- start start 2023-11-28T13:06:26.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.702+0000 7f678e3a6700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67880cef60 0x7f67880ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.702+0000 7f678e3a6700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f67880b0160 0x7f67880adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.702+0000 7f678e3a6700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67880cc390 con 0x7f678815a5d0 2023-11-28T13:06:26.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.702+0000 7f678e3a6700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67880cc1b0 con 0x7f67880b0160 2023-11-28T13:06:26.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.702+0000 7f678e3a6700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67880cc670 con 0x7f67880cef60 2023-11-28T13:06:26.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f67877fe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67880cef60 0x7f67880ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f678c943700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f67880b0160 0x7f67880adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f67877fe700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67880cef60 0x7f67880ad8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3301/0 says I am v2:172.21.15.111:60674/0 (socket says 172.21.15.111:60674) 2023-11-28T13:06:26.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f6787fff700 1 --1- >> v1:172.21.15.149:6789/0 conn(0x7f678815a5d0 0x7f67880ce850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.149:6789/0 says I am v1:172.21.15.111:48246/0 (socket says 172.21.15.111:48246) 2023-11-28T13:06:26.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f678c943700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f67880b0160 0x7f67880adde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.149:3300/0 says I am v2:172.21.15.111:56828/0 (socket says 172.21.15.111:56828) 2023-11-28T13:06:26.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f67877fe700 1 -- 172.21.15.111:0/3074190773 learned_addr learned my addr 172.21.15.111:0/3074190773 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:26.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f6786ffd700 1 -- 172.21.15.111:0/3074190773 <== mon.1 v1:172.21.15.149:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163057078 0 0) 0x7f67880cc390 con 0x7f678815a5d0 2023-11-28T13:06:26.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f6786ffd700 1 -- 172.21.15.111:0/3074190773 --> v1:172.21.15.149:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67700035b0 con 0x7f678815a5d0 2023-11-28T13:06:26.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f67877fe700 1 -- 172.21.15.111:0/3074190773 >> v2:172.21.15.149:3300/0 conn(0x7f67880b0160 msgr2=0x7f67880adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:26.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f67877fe700 1 --2- 172.21.15.111:0/3074190773 >> v2:172.21.15.149:3300/0 conn(0x7f67880b0160 0x7f67880adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f67877fe700 1 -- 172.21.15.111:0/3074190773 >> v1:172.21.15.149:6789/0 conn(0x7f678815a5d0 legacy=0x7f67880ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f67877fe700 1 -- 172.21.15.111:0/3074190773 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67880cc950 con 0x7f67880cef60 2023-11-28T13:06:26.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f678c943700 1 --2- 172.21.15.111:0/3074190773 >> v2:172.21.15.149:3300/0 conn(0x7f67880b0160 0x7f67880adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T13:06:26.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f67877fe700 1 --2- 172.21.15.111:0/3074190773 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67880cef60 0x7f67880ad8a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f67780098f0 tx=0x7f6778009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c8f9bdf3e48d93e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f6786ffd700 1 -- 172.21.15.111:0/3074190773 <== mon.2 v2:172.21.15.149:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6778011830 con 0x7f67880cef60 2023-11-28T13:06:26.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f6786ffd700 1 -- 172.21.15.111:0/3074190773 <== mon.2 v2:172.21.15.149:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6778011990 con 0x7f67880cef60 2023-11-28T13:06:26.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.703+0000 7f6786ffd700 1 -- 172.21.15.111:0/3074190773 <== mon.2 v2:172.21.15.149:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6778011d20 con 0x7f67880cef60 2023-11-28T13:06:26.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.704+0000 7f678e3a6700 1 -- 172.21.15.111:0/3074190773 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67880cef60 msgr2=0x7f67880ad8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.704+0000 7f678e3a6700 1 --2- 172.21.15.111:0/3074190773 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67880cef60 0x7f67880ad8a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f67780098f0 tx=0x7f6778009c00 comp rx=0 tx=0).stop 2023-11-28T13:06:26.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.704+0000 7f678e3a6700 1 -- 172.21.15.111:0/3074190773 shutdown_connections 2023-11-28T13:06:26.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.704+0000 7f678e3a6700 1 --2- 172.21.15.111:0/3074190773 >> v2:172.21.15.149:3300/0 conn(0x7f67880b0160 0x7f67880adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.704+0000 7f678e3a6700 1 --2- 172.21.15.111:0/3074190773 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67880cef60 0x7f67880ad8a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.704+0000 7f678e3a6700 1 -- 172.21.15.111:0/3074190773 >> 172.21.15.111:0/3074190773 conn(0x7f678809a040 msgr2=0x7f67880d0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:26.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.704+0000 7f678e3a6700 1 -- 172.21.15.111:0/3074190773 shutdown_connections 2023-11-28T13:06:26.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.704+0000 7f678e3a6700 1 -- 172.21.15.111:0/3074190773 wait complete. 2023-11-28T13:06:26.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.705+0000 7f678e3a6700 1 Processor -- start 2023-11-28T13:06:26.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.705+0000 7f678e3a6700 1 -- start start 2023-11-28T13:06:26.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f678e3a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67880b0160 0x7f6788157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f678e3a6700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67880cef60 0x7f67880c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f678e3a6700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f678815a5d0 0x7f67880c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f678e3a6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678806b380 con 0x7f67880b0160 2023-11-28T13:06:26.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f678e3a6700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6788157e70 con 0x7f678815a5d0 2023-11-28T13:06:26.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f678e3a6700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67880c8580 con 0x7f67880cef60 2023-11-28T13:06:26.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f6787fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67880b0160 0x7f6788157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f6787fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67880b0160 0x7f6788157930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38930/0 (socket says 172.21.15.111:38930) 2023-11-28T13:06:26.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f6787fff700 1 -- 172.21.15.111:0/3219597605 learned_addr learned my addr 172.21.15.111:0/3219597605 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:26.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f67877fe700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67880cef60 0x7f67880c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f678c943700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f678815a5d0 0x7f67880c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f6787fff700 1 -- 172.21.15.111:0/3219597605 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67880cef60 msgr2=0x7f67880c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f6787fff700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67880cef60 0x7f67880c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f6787fff700 1 -- 172.21.15.111:0/3219597605 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f678815a5d0 msgr2=0x7f67880c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f6787fff700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f678815a5d0 0x7f67880c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f6787fff700 1 -- 172.21.15.111:0/3219597605 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6780009a70 con 0x7f67880b0160 2023-11-28T13:06:26.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.706+0000 7f6787fff700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67880b0160 0x7f6788157930 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f677c00c490 tx=0x7f677c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.707+0000 7f6784ff9700 1 -- 172.21.15.111:0/3219597605 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c005d00 con 0x7f67880b0160 2023-11-28T13:06:26.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.707+0000 7f6784ff9700 1 -- 172.21.15.111:0/3219597605 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677c005e60 con 0x7f67880b0160 2023-11-28T13:06:26.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.707+0000 7f6784ff9700 1 -- 172.21.15.111:0/3219597605 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c007310 con 0x7f67880b0160 2023-11-28T13:06:26.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.707+0000 7f678e3a6700 1 -- 172.21.15.111:0/3219597605 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6778009550 con 0x7f67880b0160 2023-11-28T13:06:26.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.707+0000 7f678e3a6700 1 -- 172.21.15.111:0/3219597605 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67880ccf10 con 0x7f67880b0160 2023-11-28T13:06:26.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.708+0000 7f678e3a6700 1 -- 172.21.15.111:0/3219597605 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67880ccc30 con 0x7f67880b0160 2023-11-28T13:06:26.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.708+0000 7f6784ff9700 1 -- 172.21.15.111:0/3219597605 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f677c019030 con 0x7f67880b0160 2023-11-28T13:06:26.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.708+0000 7f6784ff9700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f67740682f0 0x7f677406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:26.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.708+0000 7f6784ff9700 1 -- 172.21.15.111:0/3219597605 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f677c084eb0 con 0x7f67880b0160 2023-11-28T13:06:26.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.708+0000 7f67877fe700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f67740682f0 0x7f677406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:26.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.709+0000 7f67877fe700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f67740682f0 0x7f677406a7a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f67880cf350 tx=0x7f6778008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:26.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.713+0000 7f6784ff9700 1 -- 172.21.15.111:0/3219597605 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f677c0518d0 con 0x7f67880b0160 2023-11-28T13:06:26.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.891+0000 7f678e3a6700 1 -- 172.21.15.111:0/3219597605 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f678815a9a0 con 0x7f67740682f0 2023-11-28T13:06:26.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.892+0000 7f6784ff9700 1 -- 172.21.15.111:0/3219597605 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f678815a9a0 con 0x7f67740682f0 2023-11-28T13:06:26.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.894+0000 7f678e3a6700 1 -- 172.21.15.111:0/3219597605 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f67740682f0 msgr2=0x7f677406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.894+0000 7f678e3a6700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f67740682f0 0x7f677406a7a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f67880cf350 tx=0x7f6778008030 comp rx=0 tx=0).stop 2023-11-28T13:06:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.894+0000 7f678e3a6700 1 -- 172.21.15.111:0/3219597605 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67880b0160 msgr2=0x7f6788157930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.894+0000 7f678e3a6700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67880b0160 0x7f6788157930 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f677c00c490 tx=0x7f677c00c850 comp rx=0 tx=0).stop 2023-11-28T13:06:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.894+0000 7f678e3a6700 1 -- 172.21.15.111:0/3219597605 shutdown_connections 2023-11-28T13:06:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.894+0000 7f678e3a6700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f67740682f0 0x7f677406a7a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.894+0000 7f678e3a6700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f678815a5d0 0x7f67880c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.894+0000 7f678e3a6700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67880b0160 0x7f6788157930 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.894+0000 7f678e3a6700 1 --2- 172.21.15.111:0/3219597605 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f67880cef60 0x7f67880c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:26.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.894+0000 7f678e3a6700 1 -- 172.21.15.111:0/3219597605 >> 172.21.15.111:0/3219597605 conn(0x7f678809a040 msgr2=0x7f67880ad8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:26.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.894+0000 7f678e3a6700 1 -- 172.21.15.111:0/3219597605 shutdown_connections 2023-11-28T13:06:26.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:26.894+0000 7f678e3a6700 1 -- 172.21.15.111:0/3219597605 wait complete. 2023-11-28T13:06:26.898 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 4 to deep-scrub 2023-11-28T13:06:26.907 INFO:tasks.ceph:Scrubbing osd.5 2023-11-28T13:06:26.907 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-11-28T13:06:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.030+0000 7faed09e2700 1 Processor -- start 2023-11-28T13:06:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.030+0000 7faed09e2700 1 -- start start 2023-11-28T13:06:27.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faed09e2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faecc15cca0 0x7faecc0ae900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faed09e2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faecc0cf470 0x7faecc06a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faed09e2700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0ac6d0 con 0x7faecc06a7b0 2023-11-28T13:06:27.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faed09e2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc15d070 con 0x7faecc15cca0 2023-11-28T13:06:27.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faed09e2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc06ac20 con 0x7faecc0cf470 2023-11-28T13:06:27.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faeca59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faecc15cca0 0x7faecc0ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faecad9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7faecc06a7b0 0x7faecc1606c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39720/0 (socket says 172.21.15.111:39720) 2023-11-28T13:06:27.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faecad9d700 1 -- 172.21.15.111:0/3544608241 learned_addr learned my addr 172.21.15.111:0/3544608241 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:27.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faec9d9b700 1 --2- 172.21.15.111:0/3544608241 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faecc0cf470 0x7faecc06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faeca59c700 1 -- 172.21.15.111:0/3544608241 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faecc0cf470 msgr2=0x7faecc06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:27.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faeca59c700 1 --2- 172.21.15.111:0/3544608241 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faecc0cf470 0x7faecc06a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faeca59c700 1 -- 172.21.15.111:0/3544608241 >> v1:172.21.15.111:6789/0 conn(0x7faecc06a7b0 legacy=0x7faecc1606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faec959a700 1 -- 172.21.15.111:0/3544608241 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783061290 0 0) 0x7faecc0ac6d0 con 0x7faecc06a7b0 2023-11-28T13:06:27.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faeca59c700 1 -- 172.21.15.111:0/3544608241 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc0acf70 con 0x7faecc15cca0 2023-11-28T13:06:27.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faeca59c700 1 --2- 172.21.15.111:0/3544608241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc15cca0 0x7faecc0ae900 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7faec0009850 tx=0x7faec0009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d8c5ef47f487e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faec959a700 1 -- 172.21.15.111:0/3544608241 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec0005e30 con 0x7faecc15cca0 2023-11-28T13:06:27.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faec959a700 1 -- 172.21.15.111:0/3544608241 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec0006030 con 0x7faecc15cca0 2023-11-28T13:06:27.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.031+0000 7faec959a700 1 -- 172.21.15.111:0/3544608241 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec00063c0 con 0x7faecc15cca0 2023-11-28T13:06:27.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.032+0000 7faed09e2700 1 -- 172.21.15.111:0/3544608241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc15cca0 msgr2=0x7faecc0ae900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.032+0000 7faed09e2700 1 --2- 172.21.15.111:0/3544608241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc15cca0 0x7faecc0ae900 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7faec0009850 tx=0x7faec0009b60 comp rx=0 tx=0).stop 2023-11-28T13:06:27.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.032+0000 7faed09e2700 1 -- 172.21.15.111:0/3544608241 shutdown_connections 2023-11-28T13:06:27.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.032+0000 7faed09e2700 1 --2- 172.21.15.111:0/3544608241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc15cca0 0x7faecc0ae900 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.032+0000 7faed09e2700 1 --2- 172.21.15.111:0/3544608241 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faecc0cf470 0x7faecc06a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.032+0000 7faed09e2700 1 -- 172.21.15.111:0/3544608241 >> 172.21.15.111:0/3544608241 conn(0x7faecc06bca0 msgr2=0x7faecc0cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:27.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.032+0000 7faed09e2700 1 -- 172.21.15.111:0/3544608241 shutdown_connections 2023-11-28T13:06:27.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.032+0000 7faed09e2700 1 -- 172.21.15.111:0/3544608241 wait complete. 2023-11-28T13:06:27.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.033+0000 7faed09e2700 1 Processor -- start 2023-11-28T13:06:27.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.033+0000 7faed09e2700 1 -- start start 2023-11-28T13:06:27.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.034+0000 7faed09e2700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faecc06a7b0 0x7faecc0b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.034+0000 7faed09e2700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faecc0cf470 0x7faecc0b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.034+0000 7faed09e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc15cca0 0x7faecc0b1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.034+0000 7faed09e2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc0c4820 con 0x7faecc15cca0 2023-11-28T13:06:27.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.034+0000 7faed09e2700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc0c4a90 con 0x7faecc0cf470 2023-11-28T13:06:27.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.034+0000 7faed09e2700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc0c4d70 con 0x7faecc06a7b0 2023-11-28T13:06:27.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.034+0000 7faecad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc15cca0 0x7faecc0b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.034+0000 7faecad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc15cca0 0x7faecc0b1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38948/0 (socket says 172.21.15.111:38948) 2023-11-28T13:06:27.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.034+0000 7faecad9d700 1 -- 172.21.15.111:0/3713340724 learned_addr learned my addr 172.21.15.111:0/3713340724 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.035+0000 7faec9d9b700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faecc0cf470 0x7faecc0b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.035+0000 7faecad9d700 1 -- 172.21.15.111:0/3713340724 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faecc06a7b0 msgr2=0x7faecc0b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.035+0000 7faecad9d700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7faecc06a7b0 0x7faecc0b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.035+0000 7faecad9d700 1 -- 172.21.15.111:0/3713340724 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faecc0cf470 msgr2=0x7faecc0b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.035+0000 7faecad9d700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faecc0cf470 0x7faecc0b10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.035+0000 7faecad9d700 1 -- 172.21.15.111:0/3713340724 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec0009430 con 0x7faecc15cca0 2023-11-28T13:06:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.035+0000 7faec9d9b700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7faecc0cf470 0x7faecc0b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.035+0000 7faecad9d700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc15cca0 0x7faecc0b1600 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7faec400cba0 tx=0x7faec400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.035+0000 7faebaffd700 1 -- 172.21.15.111:0/3713340724 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec400ab80 con 0x7faecc15cca0 2023-11-28T13:06:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.035+0000 7faed09e2700 1 -- 172.21.15.111:0/3713340724 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc0aef70 con 0x7faecc15cca0 2023-11-28T13:06:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.035+0000 7faed09e2700 1 -- 172.21.15.111:0/3713340724 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc0af310 con 0x7faecc15cca0 2023-11-28T13:06:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.035+0000 7faebaffd700 1 -- 172.21.15.111:0/3713340724 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec400ace0 con 0x7faecc15cca0 2023-11-28T13:06:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.035+0000 7faebaffd700 1 -- 172.21.15.111:0/3713340724 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec40072d0 con 0x7faecc15cca0 2023-11-28T13:06:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.036+0000 7faed09e2700 1 -- 172.21.15.111:0/3713340724 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faeb0000fc0 con 0x7faecc15cca0 2023-11-28T13:06:27.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.036+0000 7faebaffd700 1 -- 172.21.15.111:0/3713340724 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faec401a030 con 0x7faecc15cca0 2023-11-28T13:06:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.036+0000 7faebaffd700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faeac0682f0 0x7faeac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.037+0000 7faebaffd700 1 -- 172.21.15.111:0/3713340724 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7faec40857a0 con 0x7faecc15cca0 2023-11-28T13:06:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.037+0000 7faeca59c700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faeac0682f0 0x7faeac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.037+0000 7faebaffd700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x7faeac06eb90 0x7faeac070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.037+0000 7faebaffd700 1 -- 172.21.15.111:0/3713340724 --> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faeac0716b0 con 0x7faeac06eb90 2023-11-28T13:06:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.037+0000 7faebaffd700 1 -- 172.21.15.111:0/3713340724 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7faec4087e50 con 0x7faecc15cca0 2023-11-28T13:06:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.037+0000 7faec9d9b700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x7faeac06eb90 0x7faeac070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.037+0000 7faeca59c700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faeac0682f0 0x7faeac06a7a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7faec00097a0 tx=0x7faec0010000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.037+0000 7faec9d9b700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x7faeac06eb90 0x7faeac070fe0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.038+0000 7faebaffd700 1 -- 172.21.15.111:0/3713340724 <== osd.5 v2:172.21.15.149:6808/103795 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7faeac0716b0 con 0x7faeac06eb90 2023-11-28T13:06:27.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.052+0000 7faed09e2700 1 -- 172.21.15.111:0/3713340724 --> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7faeb0002d60 con 0x7faeac06eb90 2023-11-28T13:06:27.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.053+0000 7faebaffd700 1 -- 172.21.15.111:0/3713340724 <== osd.5 v2:172.21.15.149:6808/103795 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7faeb0002d60 con 0x7faeac06eb90 2023-11-28T13:06:27.054 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:06:27.055 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-28T13:06:27.055 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:06:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.053+0000 7faed09e2700 1 -- 172.21.15.111:0/3713340724 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x7faeac06eb90 msgr2=0x7faeac070fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.053+0000 7faed09e2700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.149:6808/103795,v1:172.21.15.149:6809/103795] conn(0x7faeac06eb90 0x7faeac070fe0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.053+0000 7faed09e2700 1 -- 172.21.15.111:0/3713340724 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faeac0682f0 msgr2=0x7faeac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.053+0000 7faed09e2700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7faeac0682f0 0x7faeac06a7a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7faec00097a0 tx=0x7faec0010000 comp rx=0 tx=0).stop 2023-11-28T13:06:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.053+0000 7faed09e2700 1 -- 172.21.15.111:0/3713340724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc15cca0 msgr2=0x7faecc0b1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.053+0000 7faed09e2700 1 --2- 172.21.15.111:0/3713340724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc15cca0 0x7faecc0b1600 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7faec400cba0 tx=0x7faec400cf60 comp rx=0 tx=0).stop 2023-11-28T13:06:27.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.053+0000 7faecad9d700 1 -- 172.21.15.111:0/3713340724 reap_dead start 2023-11-28T13:06:27.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.053+0000 7faed09e2700 1 -- 172.21.15.111:0/3713340724 shutdown_connections 2023-11-28T13:06:27.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.053+0000 7faed09e2700 1 -- 172.21.15.111:0/3713340724 >> 172.21.15.111:0/3713340724 conn(0x7faecc06bca0 msgr2=0x7faecc06edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:27.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.053+0000 7faed09e2700 1 -- 172.21.15.111:0/3713340724 shutdown_connections 2023-11-28T13:06:27.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.053+0000 7faed09e2700 1 -- 172.21.15.111:0/3713340724 wait complete. 2023-11-28T13:06:27.066 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-11-28T13:06:27.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.186+0000 7f578c272700 1 Processor -- start 2023-11-28T13:06:27.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.186+0000 7f578c272700 1 -- start start 2023-11-28T13:06:27.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.187+0000 7f578c272700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f578415c800 0x7f5784160e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.187+0000 7f578c272700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f57840cd790 0x7f57840cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.187+0000 7f578c272700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f578415cbd0 con 0x7f578406bfd0 2023-11-28T13:06:27.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.187+0000 7f578c272700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57840ce150 con 0x7f57840cd790 2023-11-28T13:06:27.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.187+0000 7f578c272700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578406af30 con 0x7f578415c800 2023-11-28T13:06:27.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.187+0000 7f578980d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f578406bfd0 0x7f578406a6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39734/0 (socket says 172.21.15.111:39734) 2023-11-28T13:06:27.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.187+0000 7f578980d700 1 -- 172.21.15.111:0/1488663807 learned_addr learned my addr 172.21.15.111:0/1488663807 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:27.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.187+0000 7f578a00e700 1 --2- 172.21.15.111:0/1488663807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f578415c800 0x7f5784160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.187+0000 7f578a80f700 1 --2- 172.21.15.111:0/1488663807 >> v2:172.21.15.149:3300/0 conn(0x7f57840cd790 0x7f57840cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.187+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798286730 0 0) 0x7f578415cbd0 con 0x7f578406bfd0 2023-11-28T13:06:27.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.187+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57700035b0 con 0x7f578406bfd0 2023-11-28T13:06:27.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1167251286 0 0) 0x7f57700035b0 con 0x7f578406bfd0 2023-11-28T13:06:27.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f578415c800 msgr2=0x7f5784160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:27.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 --2- 172.21.15.111:0/1488663807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f578415c800 0x7f5784160e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 >> v2:172.21.15.149:3300/0 conn(0x7f57840cd790 msgr2=0x7f57840cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:27.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 --2- 172.21.15.111:0/1488663807 >> v2:172.21.15.149:3300/0 conn(0x7f57840cd790 0x7f57840cdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578406b210 con 0x7f578406bfd0 2023-11-28T13:06:27.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f5780001670 con 0x7f578406bfd0 2023-11-28T13:06:27.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 >> v1:172.21.15.111:6789/0 conn(0x7f578406bfd0 legacy=0x7f578406a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 --2- 172.21.15.111:0/1488663807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5770003a00 0x7f5770003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 --2- 172.21.15.111:0/1488663807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5770004430 0x7f57700048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 --2- 172.21.15.111:0/1488663807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f578415c800 0x7f5770004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57840ce150 con 0x7f5770003a00 2023-11-28T13:06:27.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578406af30 con 0x7f5770004430 2023-11-28T13:06:27.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577000ba10 con 0x7f578415c800 2023-11-28T13:06:27.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578a00e700 1 --2- 172.21.15.111:0/1488663807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5770003a00 0x7f5770003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578980d700 1 --2- 172.21.15.111:0/1488663807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5770004430 0x7f57700048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578a80f700 1 --2- 172.21.15.111:0/1488663807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f578415c800 0x7f5770004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578a00e700 1 -- 172.21.15.111:0/1488663807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f578415c800 msgr2=0x7f5770004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:27.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578a00e700 1 --2- 172.21.15.111:0/1488663807 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f578415c800 0x7f5770004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578a00e700 1 -- 172.21.15.111:0/1488663807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5770004430 msgr2=0x7f57700048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:27.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578a00e700 1 --2- 172.21.15.111:0/1488663807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5770004430 0x7f57700048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578a00e700 1 -- 172.21.15.111:0/1488663807 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57700035b0 con 0x7f5770003a00 2023-11-28T13:06:27.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578980d700 1 --2- 172.21.15.111:0/1488663807 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5770004430 0x7f57700048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:27.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578a00e700 1 --2- 172.21.15.111:0/1488663807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5770003a00 0x7f5770003e10 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f5774009b60 tx=0x7f5770005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1705b95b15817e92 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577400e6c0 con 0x7f5770003a00 2023-11-28T13:06:27.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577400e820 con 0x7f5770003a00 2023-11-28T13:06:27.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.188+0000 7f578900c700 1 -- 172.21.15.111:0/1488663807 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577400eb90 con 0x7f5770003a00 2023-11-28T13:06:27.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.189+0000 7f578c272700 1 -- 172.21.15.111:0/1488663807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5770003a00 msgr2=0x7f5770003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.189+0000 7f578c272700 1 --2- 172.21.15.111:0/1488663807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5770003a00 0x7f5770003e10 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f5774009b60 tx=0x7f5770005880 comp rx=0 tx=0).stop 2023-11-28T13:06:27.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.189+0000 7f578a80f700 1 -- 172.21.15.111:0/1488663807 reap_dead start 2023-11-28T13:06:27.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.189+0000 7f578c272700 1 -- 172.21.15.111:0/1488663807 shutdown_connections 2023-11-28T13:06:27.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.189+0000 7f578c272700 1 -- 172.21.15.111:0/1488663807 >> 172.21.15.111:0/1488663807 conn(0x7f578406db10 msgr2=0x7f578406df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:27.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.189+0000 7f578c272700 1 -- 172.21.15.111:0/1488663807 shutdown_connections 2023-11-28T13:06:27.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.189+0000 7f578c272700 1 -- 172.21.15.111:0/1488663807 wait complete. 2023-11-28T13:06:27.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.190+0000 7f578c272700 1 Processor -- start 2023-11-28T13:06:27.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.190+0000 7f578c272700 1 -- start start 2023-11-28T13:06:27.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578c272700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5770003a00 0x7f57840b1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578c272700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57840ca1f0 0x7f57840b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578c272700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f57840b09b0 0x7f57840b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578c272700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57840cb270 con 0x7f57840ca1f0 2023-11-28T13:06:27.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578c272700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57840ca660 con 0x7f57840b09b0 2023-11-28T13:06:27.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578c272700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57840ca940 con 0x7f5770003a00 2023-11-28T13:06:27.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578980d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57840ca1f0 0x7f57840b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578980d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57840ca1f0 0x7f57840b3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38966/0 (socket says 172.21.15.111:38966) 2023-11-28T13:06:27.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578980d700 1 -- 172.21.15.111:0/2039777144 learned_addr learned my addr 172.21.15.111:0/2039777144 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:27.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578a00e700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5770003a00 0x7f57840b1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578a80f700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f57840b09b0 0x7f57840b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578980d700 1 -- 172.21.15.111:0/2039777144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5770003a00 msgr2=0x7f57840b1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578980d700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5770003a00 0x7f57840b1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578980d700 1 -- 172.21.15.111:0/2039777144 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f57840b09b0 msgr2=0x7f57840b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578980d700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f57840b09b0 0x7f57840b0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578980d700 1 -- 172.21.15.111:0/2039777144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578406b7d0 con 0x7f57840ca1f0 2023-11-28T13:06:27.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f578980d700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57840ca1f0 0x7f57840b3ce0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f5780000ff0 tx=0x7f578000edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.191+0000 7f577affd700 1 -- 172.21.15.111:0/2039777144 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578000ee40 con 0x7f57840ca1f0 2023-11-28T13:06:27.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.192+0000 7f577affd700 1 -- 172.21.15.111:0/2039777144 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5780005e20 con 0x7f57840ca1f0 2023-11-28T13:06:27.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.192+0000 7f578c272700 1 -- 172.21.15.111:0/2039777144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f578406b4f0 con 0x7f57840ca1f0 2023-11-28T13:06:27.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.192+0000 7f577affd700 1 -- 172.21.15.111:0/2039777144 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578000c9e0 con 0x7f57840ca1f0 2023-11-28T13:06:27.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.192+0000 7f578c272700 1 -- 172.21.15.111:0/2039777144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578406bab0 con 0x7f57840ca1f0 2023-11-28T13:06:27.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.193+0000 7f578c272700 1 -- 172.21.15.111:0/2039777144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57840cdc10 con 0x7f57840ca1f0 2023-11-28T13:06:27.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.193+0000 7f577affd700 1 -- 172.21.15.111:0/2039777144 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f578000cb40 con 0x7f57840ca1f0 2023-11-28T13:06:27.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.193+0000 7f577affd700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f57680682f0 0x7f576806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.193+0000 7f577affd700 1 -- 172.21.15.111:0/2039777144 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f5780085b00 con 0x7f57840ca1f0 2023-11-28T13:06:27.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.193+0000 7f578a00e700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f57680682f0 0x7f576806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.194+0000 7f578a00e700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f57680682f0 0x7f576806a7a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f57740090b0 tx=0x7f5774008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.198+0000 7f577affd700 1 -- 172.21.15.111:0/2039777144 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5780087e60 con 0x7f57840ca1f0 2023-11-28T13:06:27.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.377+0000 7f578c272700 1 -- 172.21.15.111:0/2039777144 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f5784066d50 con 0x7f57680682f0 2023-11-28T13:06:27.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.377+0000 7f577affd700 1 -- 172.21.15.111:0/2039777144 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5784066d50 con 0x7f57680682f0 2023-11-28T13:06:27.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.379+0000 7f578c272700 1 -- 172.21.15.111:0/2039777144 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f57680682f0 msgr2=0x7f576806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.379+0000 7f578c272700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f57680682f0 0x7f576806a7a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f57740090b0 tx=0x7f5774008030 comp rx=0 tx=0).stop 2023-11-28T13:06:27.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.379+0000 7f578c272700 1 -- 172.21.15.111:0/2039777144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57840ca1f0 msgr2=0x7f57840b3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.379+0000 7f578c272700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57840ca1f0 0x7f57840b3ce0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f5780000ff0 tx=0x7f578000edd0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.379+0000 7f578c272700 1 -- 172.21.15.111:0/2039777144 shutdown_connections 2023-11-28T13:06:27.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.379+0000 7f578c272700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f57680682f0 0x7f576806a7a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.379+0000 7f578c272700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f57840b09b0 0x7f57840b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.379+0000 7f578c272700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f5770003a00 0x7f57840b1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.379+0000 7f578c272700 1 --2- 172.21.15.111:0/2039777144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57840ca1f0 0x7f57840b3ce0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.379+0000 7f578c272700 1 -- 172.21.15.111:0/2039777144 >> 172.21.15.111:0/2039777144 conn(0x7f578406db10 msgr2=0x7f57840c78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:27.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.379+0000 7f578c272700 1 -- 172.21.15.111:0/2039777144 shutdown_connections 2023-11-28T13:06:27.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.379+0000 7f578c272700 1 -- 172.21.15.111:0/2039777144 wait complete. 2023-11-28T13:06:27.383 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 5 to deep-scrub 2023-11-28T13:06:27.392 INFO:tasks.ceph:Scrubbing osd.6 2023-11-28T13:06:27.392 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-11-28T13:06:27.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.515+0000 7f1e9a299700 1 Processor -- start 2023-11-28T13:06:27.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.515+0000 7f1e9a299700 1 -- start start 2023-11-28T13:06:27.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.515+0000 7f1e9a299700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e9415c5e0 0x7f1e9406a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.515+0000 7f1e9a299700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1e9406b720 0x7f1e9406a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e9a299700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e9415c9b0 con 0x7f1e9406acf0 2023-11-28T13:06:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e9a299700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e940cce90 con 0x7f1e9406b720 2023-11-28T13:06:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e9a299700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e94160e60 con 0x7f1e9415c5e0 2023-11-28T13:06:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e937fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1e9406b720 0x7f1e9406a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e98836700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1e9406acf0 0x7f1e940712e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39752/0 (socket says 172.21.15.111:39752) 2023-11-28T13:06:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e98836700 1 -- 172.21.15.111:0/1689835676 learned_addr learned my addr 172.21.15.111:0/1689835676 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e93fff700 1 --2- 172.21.15.111:0/1689835676 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e9415c5e0 0x7f1e9406a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e92ffd700 1 -- 172.21.15.111:0/1689835676 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006906291 0 0) 0x7f1e9415c9b0 con 0x7f1e9406acf0 2023-11-28T13:06:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e92ffd700 1 -- 172.21.15.111:0/1689835676 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e7c0035b0 con 0x7f1e9406acf0 2023-11-28T13:06:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e937fe700 1 -- 172.21.15.111:0/1689835676 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e9415c5e0 msgr2=0x7f1e9406a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e937fe700 1 --2- 172.21.15.111:0/1689835676 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e9415c5e0 0x7f1e9406a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e937fe700 1 -- 172.21.15.111:0/1689835676 >> v1:172.21.15.111:6789/0 conn(0x7f1e9406acf0 legacy=0x7f1e940712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e937fe700 1 -- 172.21.15.111:0/1689835676 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e940cd0a0 con 0x7f1e9406b720 2023-11-28T13:06:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e93fff700 1 --2- 172.21.15.111:0/1689835676 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e9415c5e0 0x7f1e9406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e937fe700 1 --2- 172.21.15.111:0/1689835676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9406b720 0x7f1e9406a7b0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f1e88009840 tx=0x7f1e88009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3ca0512593dfe7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e92ffd700 1 -- 172.21.15.111:0/1689835676 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88011830 con 0x7f1e9406b720 2023-11-28T13:06:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e92ffd700 1 -- 172.21.15.111:0/1689835676 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e88011990 con 0x7f1e9406b720 2023-11-28T13:06:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.516+0000 7f1e92ffd700 1 -- 172.21.15.111:0/1689835676 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88011d20 con 0x7f1e9406b720 2023-11-28T13:06:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.517+0000 7f1e9a299700 1 -- 172.21.15.111:0/1689835676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9406b720 msgr2=0x7f1e9406a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.517+0000 7f1e9a299700 1 --2- 172.21.15.111:0/1689835676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9406b720 0x7f1e9406a7b0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f1e88009840 tx=0x7f1e88009b50 comp rx=0 tx=0).stop 2023-11-28T13:06:27.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.517+0000 7f1e9a299700 1 -- 172.21.15.111:0/1689835676 shutdown_connections 2023-11-28T13:06:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.517+0000 7f1e9a299700 1 --2- 172.21.15.111:0/1689835676 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e9415c5e0 0x7f1e9406a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.517+0000 7f1e9a299700 1 --2- 172.21.15.111:0/1689835676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9406b720 0x7f1e9406a7b0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.517+0000 7f1e9a299700 1 -- 172.21.15.111:0/1689835676 >> 172.21.15.111:0/1689835676 conn(0x7f1e940b0db0 msgr2=0x7f1e940b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.517+0000 7f1e9a299700 1 -- 172.21.15.111:0/1689835676 shutdown_connections 2023-11-28T13:06:27.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.517+0000 7f1e9a299700 1 -- 172.21.15.111:0/1689835676 wait complete. 2023-11-28T13:06:27.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.518+0000 7f1e9a299700 1 Processor -- start 2023-11-28T13:06:27.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.518+0000 7f1e9a299700 1 -- start start 2023-11-28T13:06:27.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e9a299700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e9406acf0 0x7f1e9415f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e9a299700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9406b720 0x7f1e940ad330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e9a299700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e9415c5e0 0x7f1e941530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e9a299700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e940c3b10 con 0x7f1e9406b720 2023-11-28T13:06:27.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e9a299700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e940c3df0 con 0x7f1e9406acf0 2023-11-28T13:06:27.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e9a299700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e940ad870 con 0x7f1e9415c5e0 2023-11-28T13:06:27.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e937fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9406b720 0x7f1e940ad330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e937fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9406b720 0x7f1e940ad330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38982/0 (socket says 172.21.15.111:38982) 2023-11-28T13:06:27.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e937fe700 1 -- 172.21.15.111:0/3479339864 learned_addr learned my addr 172.21.15.111:0/3479339864 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:27.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e93fff700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e9406acf0 0x7f1e9415f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e937fe700 1 -- 172.21.15.111:0/3479339864 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e9415c5e0 msgr2=0x7f1e941530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e98836700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e9415c5e0 0x7f1e941530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e937fe700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e9415c5e0 0x7f1e941530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e937fe700 1 -- 172.21.15.111:0/3479339864 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e9406acf0 msgr2=0x7f1e9415f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e937fe700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e9406acf0 0x7f1e9415f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e937fe700 1 -- 172.21.15.111:0/3479339864 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e880094a0 con 0x7f1e9406b720 2023-11-28T13:06:27.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e93fff700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f1e9406acf0 0x7f1e9415f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:27.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e98836700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f1e9415c5e0 0x7f1e941530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:27.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.519+0000 7f1e937fe700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9406b720 0x7f1e940ad330 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f1e88000c00 tx=0x7f1e88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.520+0000 7f1e90ff9700 1 -- 172.21.15.111:0/3479339864 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e880036a0 con 0x7f1e9406b720 2023-11-28T13:06:27.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.520+0000 7f1e90ff9700 1 -- 172.21.15.111:0/3479339864 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8801f070 con 0x7f1e9406b720 2023-11-28T13:06:27.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.520+0000 7f1e9a299700 1 -- 172.21.15.111:0/3479339864 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e940cd660 con 0x7f1e9406b720 2023-11-28T13:06:27.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.520+0000 7f1e90ff9700 1 -- 172.21.15.111:0/3479339864 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88003a60 con 0x7f1e9406b720 2023-11-28T13:06:27.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.520+0000 7f1e9a299700 1 -- 172.21.15.111:0/3479339864 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e940cd380 con 0x7f1e9406b720 2023-11-28T13:06:27.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.520+0000 7f1e9a299700 1 -- 172.21.15.111:0/3479339864 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1e94160af0 con 0x7f1e9406b720 2023-11-28T13:06:27.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.521+0000 7f1e90ff9700 1 -- 172.21.15.111:0/3479339864 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1e8800f050 con 0x7f1e9406b720 2023-11-28T13:06:27.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.521+0000 7f1e90ff9700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1e800682f0 0x7f1e8006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.521+0000 7f1e90ff9700 1 -- 172.21.15.111:0/3479339864 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f1e88021070 con 0x7f1e9406b720 2023-11-28T13:06:27.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.521+0000 7f1e93fff700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1e800682f0 0x7f1e8006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.522+0000 7f1e90ff9700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x7f1e8006eb90 0x7f1e80070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.522+0000 7f1e90ff9700 1 -- 172.21.15.111:0/3479339864 --> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e800716b0 con 0x7f1e8006eb90 2023-11-28T13:06:27.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.522+0000 7f1e90ff9700 1 -- 172.21.15.111:0/3479339864 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e88087b80 con 0x7f1e9406b720 2023-11-28T13:06:27.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.522+0000 7f1e98836700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x7f1e8006eb90 0x7f1e80070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.522+0000 7f1e93fff700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1e800682f0 0x7f1e8006a7a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009820 tx=0x7f1e84008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.522+0000 7f1e98836700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x7f1e8006eb90 0x7f1e80070fe0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.523+0000 7f1e90ff9700 1 -- 172.21.15.111:0/3479339864 <== osd.6 v2:172.21.15.149:6816/103796 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1e800716b0 con 0x7f1e8006eb90 2023-11-28T13:06:27.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.537+0000 7f1e9a299700 1 -- 172.21.15.111:0/3479339864 --> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1e94067170 con 0x7f1e8006eb90 2023-11-28T13:06:27.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.537+0000 7f1e90ff9700 1 -- 172.21.15.111:0/3479339864 <== osd.6 v2:172.21.15.149:6816/103796 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1e94067170 con 0x7f1e8006eb90 2023-11-28T13:06:27.539 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:06:27.539 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-28T13:06:27.539 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:06:27.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.537+0000 7f1e9a299700 1 -- 172.21.15.111:0/3479339864 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x7f1e8006eb90 msgr2=0x7f1e80070fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.537+0000 7f1e9a299700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.149:6816/103796,v1:172.21.15.149:6817/103796] conn(0x7f1e8006eb90 0x7f1e80070fe0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.538+0000 7f1e9a299700 1 -- 172.21.15.111:0/3479339864 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1e800682f0 msgr2=0x7f1e8006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.538+0000 7f1e9a299700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f1e800682f0 0x7f1e8006a7a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009820 tx=0x7f1e84008040 comp rx=0 tx=0).stop 2023-11-28T13:06:27.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.538+0000 7f1e9a299700 1 -- 172.21.15.111:0/3479339864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9406b720 msgr2=0x7f1e940ad330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.538+0000 7f1e9a299700 1 --2- 172.21.15.111:0/3479339864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9406b720 0x7f1e940ad330 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f1e88000c00 tx=0x7f1e88004750 comp rx=0 tx=0).stop 2023-11-28T13:06:27.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.538+0000 7f1e98836700 1 -- 172.21.15.111:0/3479339864 reap_dead start 2023-11-28T13:06:27.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.538+0000 7f1e9a299700 1 -- 172.21.15.111:0/3479339864 shutdown_connections 2023-11-28T13:06:27.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.538+0000 7f1e9a299700 1 -- 172.21.15.111:0/3479339864 >> 172.21.15.111:0/3479339864 conn(0x7f1e940b0db0 msgr2=0x7f1e9415b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:27.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.538+0000 7f1e9a299700 1 -- 172.21.15.111:0/3479339864 shutdown_connections 2023-11-28T13:06:27.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.538+0000 7f1e9a299700 1 -- 172.21.15.111:0/3479339864 wait complete. 2023-11-28T13:06:27.551 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-11-28T13:06:27.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.669+0000 7f6ae4a66700 1 Processor -- start 2023-11-28T13:06:27.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.669+0000 7f6ae4a66700 1 -- start start 2023-11-28T13:06:27.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.670+0000 7f6ae4a66700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7f6ae00b20f0 0x7f6ae00b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.670+0000 7f6ae4a66700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6ae00b40c0 0x7f6ae006b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.670+0000 7f6ae4a66700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae00b5300 con 0x7f6ae006b8c0 2023-11-28T13:06:27.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.670+0000 7f6ae4a66700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae006bd40 con 0x7f6ae00b40c0 2023-11-28T13:06:27.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.670+0000 7f6ae4a66700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae00cfbb0 con 0x7f6ae00b20f0 2023-11-28T13:06:27.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.670+0000 7f6addd9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6ae00b40c0 0x7f6ae006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.670+0000 7f6addd9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6ae00b40c0 0x7f6ae006b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38990/0 (socket says 172.21.15.111:38990) 2023-11-28T13:06:27.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.671+0000 7f6addd9b700 1 -- 172.21.15.111:0/1043537474 learned_addr learned my addr 172.21.15.111:0/1043537474 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:27.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.671+0000 7f6ade59c700 1 --2- 172.21.15.111:0/1043537474 >> v2:172.21.15.149:3300/0 conn(0x7f6ae00b20f0 0x7f6ae00b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.671+0000 7f6addd9b700 1 -- 172.21.15.111:0/1043537474 >> v2:172.21.15.149:3300/0 conn(0x7f6ae00b20f0 msgr2=0x7f6ae00b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:27.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.671+0000 7f6addd9b700 1 --2- 172.21.15.111:0/1043537474 >> v2:172.21.15.149:3300/0 conn(0x7f6ae00b20f0 0x7f6ae00b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.671+0000 7f6addd9b700 1 -- 172.21.15.111:0/1043537474 >> v1:172.21.15.149:6789/0 conn(0x7f6ae006b8c0 legacy=0x7f6ae00ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.671+0000 7f6addd9b700 1 -- 172.21.15.111:0/1043537474 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae00b46f0 con 0x7f6ae00b40c0 2023-11-28T13:06:27.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.671+0000 7f6addd9b700 1 --2- 172.21.15.111:0/1043537474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae00b40c0 0x7f6ae006b380 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f6ae006e620 tx=0x7f6ad0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a911fbfb15e003 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.671+0000 7f6add59a700 1 -- 172.21.15.111:0/1043537474 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad000b870 con 0x7f6ae00b40c0 2023-11-28T13:06:27.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.671+0000 7f6add59a700 1 -- 172.21.15.111:0/1043537474 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad000b9d0 con 0x7f6ae00b40c0 2023-11-28T13:06:27.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.671+0000 7f6add59a700 1 -- 172.21.15.111:0/1043537474 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad000bd60 con 0x7f6ae00b40c0 2023-11-28T13:06:27.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.672+0000 7f6ae4a66700 1 -- 172.21.15.111:0/1043537474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae00b40c0 msgr2=0x7f6ae006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.672+0000 7f6ae4a66700 1 --2- 172.21.15.111:0/1043537474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae00b40c0 0x7f6ae006b380 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f6ae006e620 tx=0x7f6ad0009bd0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.672+0000 7f6ae4a66700 1 -- 172.21.15.111:0/1043537474 shutdown_connections 2023-11-28T13:06:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.672+0000 7f6ae4a66700 1 --2- 172.21.15.111:0/1043537474 >> v2:172.21.15.149:3300/0 conn(0x7f6ae00b20f0 0x7f6ae00b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.672+0000 7f6ae4a66700 1 --2- 172.21.15.111:0/1043537474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae00b40c0 0x7f6ae006b380 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.672+0000 7f6ae4a66700 1 -- 172.21.15.111:0/1043537474 >> 172.21.15.111:0/1043537474 conn(0x7f6ae009a040 msgr2=0x7f6ae00d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.672+0000 7f6ae4a66700 1 -- 172.21.15.111:0/1043537474 shutdown_connections 2023-11-28T13:06:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.672+0000 7f6ae4a66700 1 -- 172.21.15.111:0/1043537474 wait complete. 2023-11-28T13:06:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.673+0000 7f6ae4a66700 1 Processor -- start 2023-11-28T13:06:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.673+0000 7f6ae4a66700 1 -- start start 2023-11-28T13:06:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ae4a66700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae006b8c0 0x7f6ae015a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ae4a66700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ae00b20f0 0x7f6ae00ca610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ae4a66700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ae00b40c0 0x7f6ae00cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ae4a66700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae01e5c90 con 0x7f6ae006b8c0 2023-11-28T13:06:27.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ae4a66700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae00cb160 con 0x7f6ae00b40c0 2023-11-28T13:06:27.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ae4a66700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae01e5330 con 0x7f6ae00b20f0 2023-11-28T13:06:27.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ade59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae006b8c0 0x7f6ae015a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ade59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae006b8c0 0x7f6ae015a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38996/0 (socket says 172.21.15.111:38996) 2023-11-28T13:06:27.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ade59c700 1 -- 172.21.15.111:0/3095580335 learned_addr learned my addr 172.21.15.111:0/3095580335 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:27.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6addd9b700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ae00b20f0 0x7f6ae00ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6aded9d700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ae00b40c0 0x7f6ae00cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ade59c700 1 -- 172.21.15.111:0/3095580335 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ae00b20f0 msgr2=0x7f6ae00ca610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ade59c700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ae00b20f0 0x7f6ae00ca610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ade59c700 1 -- 172.21.15.111:0/3095580335 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ae00b40c0 msgr2=0x7f6ae00cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ade59c700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ae00b40c0 0x7f6ae00cab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ade59c700 1 -- 172.21.15.111:0/3095580335 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad0009550 con 0x7f6ae006b8c0 2023-11-28T13:06:27.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6aded9d700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ae00b40c0 0x7f6ae00cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:27.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6addd9b700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ae00b20f0 0x7f6ae00ca610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:27.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ade59c700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae006b8c0 0x7f6ae015a4f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400e810 tx=0x7f6ad400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6aceffd700 1 -- 172.21.15.111:0/3095580335 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4013610 con 0x7f6ae006b8c0 2023-11-28T13:06:27.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6aceffd700 1 -- 172.21.15.111:0/3095580335 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad4013770 con 0x7f6ae006b8c0 2023-11-28T13:06:27.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ae4a66700 1 -- 172.21.15.111:0/3095580335 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae00b4cb0 con 0x7f6ae006b8c0 2023-11-28T13:06:27.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6aceffd700 1 -- 172.21.15.111:0/3095580335 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4013b20 con 0x7f6ae006b8c0 2023-11-28T13:06:27.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.674+0000 7f6ae4a66700 1 -- 172.21.15.111:0/3095580335 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae00b49d0 con 0x7f6ae006b8c0 2023-11-28T13:06:27.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.675+0000 7f6ae4a66700 1 -- 172.21.15.111:0/3095580335 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae00b4f90 con 0x7f6ae006b8c0 2023-11-28T13:06:27.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.676+0000 7f6aceffd700 1 -- 172.21.15.111:0/3095580335 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6ad4023070 con 0x7f6ae006b8c0 2023-11-28T13:06:27.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.676+0000 7f6aceffd700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ac00682f0 0x7f6ac006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:27.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.676+0000 7f6aceffd700 1 -- 172.21.15.111:0/3095580335 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f6ad4085850 con 0x7f6ae006b8c0 2023-11-28T13:06:27.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.676+0000 7f6addd9b700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ac00682f0 0x7f6ac006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:27.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.677+0000 7f6addd9b700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ac00682f0 0x7f6ac006a7a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0000c00 tx=0x7f6ad0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:27.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.680+0000 7f6aceffd700 1 -- 172.21.15.111:0/3095580335 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6ad40521f0 con 0x7f6ae006b8c0 2023-11-28T13:06:27.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.862+0000 7f6ae4a66700 1 -- 172.21.15.111:0/3095580335 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ae0067170 con 0x7f6ac00682f0 2023-11-28T13:06:27.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.862+0000 7f6aceffd700 1 -- 172.21.15.111:0/3095580335 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6ae0067170 con 0x7f6ac00682f0 2023-11-28T13:06:27.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.864+0000 7f6ae4a66700 1 -- 172.21.15.111:0/3095580335 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ac00682f0 msgr2=0x7f6ac006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.864+0000 7f6ae4a66700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ac00682f0 0x7f6ac006a7a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0000c00 tx=0x7f6ad0008030 comp rx=0 tx=0).stop 2023-11-28T13:06:27.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.864+0000 7f6ae4a66700 1 -- 172.21.15.111:0/3095580335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae006b8c0 msgr2=0x7f6ae015a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:27.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.864+0000 7f6ae4a66700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae006b8c0 0x7f6ae015a4f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400e810 tx=0x7f6ad400ebd0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.864+0000 7f6ae4a66700 1 -- 172.21.15.111:0/3095580335 shutdown_connections 2023-11-28T13:06:27.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.864+0000 7f6ae4a66700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f6ac00682f0 0x7f6ac006a7a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.864+0000 7f6ae4a66700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f6ae00b40c0 0x7f6ae00cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.864+0000 7f6ae4a66700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae006b8c0 0x7f6ae015a4f0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.864+0000 7f6ae4a66700 1 --2- 172.21.15.111:0/3095580335 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f6ae00b20f0 0x7f6ae00ca610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:27.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.864+0000 7f6ae4a66700 1 -- 172.21.15.111:0/3095580335 >> 172.21.15.111:0/3095580335 conn(0x7f6ae009a040 msgr2=0x7f6ae00d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:27.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.864+0000 7f6ae4a66700 1 -- 172.21.15.111:0/3095580335 shutdown_connections 2023-11-28T13:06:27.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:27.864+0000 7f6ae4a66700 1 -- 172.21.15.111:0/3095580335 wait complete. 2023-11-28T13:06:27.868 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 6 to deep-scrub 2023-11-28T13:06:27.878 INFO:tasks.ceph:Scrubbing osd.7 2023-11-28T13:06:27.878 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-11-28T13:06:28.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.000+0000 7f33a1c06700 1 Processor -- start 2023-11-28T13:06:28.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.000+0000 7f33a1c06700 1 -- start start 2023-11-28T13:06:28.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.001+0000 7f33a1c06700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f339c06b720 0x7f339c06a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.001+0000 7f33a1c06700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f339c06e970 0x7f339c06ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.001+0000 7f33a1c06700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339c15c9b0 con 0x7f339c15c5e0 2023-11-28T13:06:28.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.001+0000 7f33a1c06700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339c0cce90 con 0x7f339c06b720 2023-11-28T13:06:28.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.001+0000 7f33a1c06700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339c160e60 con 0x7f339c06e970 2023-11-28T13:06:28.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.001+0000 7f339affd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f339c06b720 0x7f339c06a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.001+0000 7f339b7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f339c15c5e0 0x7f339c06a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39782/0 (socket says 172.21.15.111:39782) 2023-11-28T13:06:28.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.001+0000 7f339affd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f339c06b720 0x7f339c06a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39006/0 (socket says 172.21.15.111:39006) 2023-11-28T13:06:28.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.001+0000 7f339b7fe700 1 -- 172.21.15.111:0/3107956776 learned_addr learned my addr 172.21.15.111:0/3107956776 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:28.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.001+0000 7f339bfff700 1 --2- 172.21.15.111:0/3107956776 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f339c06e970 0x7f339c06ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.001+0000 7f339affd700 1 -- 172.21.15.111:0/3107956776 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f339c06e970 msgr2=0x7f339c06ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:28.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f339affd700 1 --2- 172.21.15.111:0/3107956776 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f339c06e970 0x7f339c06ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f339affd700 1 -- 172.21.15.111:0/3107956776 >> v1:172.21.15.111:6789/0 conn(0x7f339c15c5e0 legacy=0x7f339c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f339affd700 1 -- 172.21.15.111:0/3107956776 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c0cd0a0 con 0x7f339c06b720 2023-11-28T13:06:28.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f339a7fc700 1 -- 172.21.15.111:0/3107956776 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122770394 0 0) 0x7f339c15c9b0 con 0x7f339c15c5e0 2023-11-28T13:06:28.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f339bfff700 1 --2- 172.21.15.111:0/3107956776 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f339c06e970 0x7f339c06ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:28.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f339affd700 1 --2- 172.21.15.111:0/3107956776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339c06b720 0x7f339c06a980 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f338c009840 tx=0x7f338c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd0996512cdb43d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:28.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f339a7fc700 1 -- 172.21.15.111:0/3107956776 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c011830 con 0x7f339c06b720 2023-11-28T13:06:28.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f339a7fc700 1 -- 172.21.15.111:0/3107956776 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c011990 con 0x7f339c06b720 2023-11-28T13:06:28.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f339a7fc700 1 -- 172.21.15.111:0/3107956776 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c011d20 con 0x7f339c06b720 2023-11-28T13:06:28.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f33a1c06700 1 -- 172.21.15.111:0/3107956776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339c06b720 msgr2=0x7f339c06a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f33a1c06700 1 --2- 172.21.15.111:0/3107956776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339c06b720 0x7f339c06a980 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f338c009840 tx=0x7f338c009b50 comp rx=0 tx=0).stop 2023-11-28T13:06:28.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f33a1c06700 1 -- 172.21.15.111:0/3107956776 shutdown_connections 2023-11-28T13:06:28.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f33a1c06700 1 --2- 172.21.15.111:0/3107956776 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f339c06e970 0x7f339c06ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f33a1c06700 1 --2- 172.21.15.111:0/3107956776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339c06b720 0x7f339c06a980 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f33a1c06700 1 -- 172.21.15.111:0/3107956776 >> 172.21.15.111:0/3107956776 conn(0x7f339c0b0db0 msgr2=0x7f339c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:28.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f33a1c06700 1 -- 172.21.15.111:0/3107956776 shutdown_connections 2023-11-28T13:06:28.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.002+0000 7f33a1c06700 1 -- 172.21.15.111:0/3107956776 wait complete. 2023-11-28T13:06:28.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.004+0000 7f33a1c06700 1 Processor -- start 2023-11-28T13:06:28.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.004+0000 7f33a1c06700 1 -- start start 2023-11-28T13:06:28.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.004+0000 7f33a1c06700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f339c06b720 0x7f339c14e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.004+0000 7f33a1c06700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f339c06e970 0x7f339c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f33a1c06700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339c15c5e0 0x7f339c0b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f33a1c06700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c14ed90 con 0x7f339c15c5e0 2023-11-28T13:06:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f33a1c06700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0acc30 con 0x7f339c06e970 2023-11-28T13:06:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f33a1c06700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0ae440 con 0x7f339c06b720 2023-11-28T13:06:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f339bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339c15c5e0 0x7f339c0b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f339bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339c15c5e0 0x7f339c0b2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39012/0 (socket says 172.21.15.111:39012) 2023-11-28T13:06:28.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f339bfff700 1 -- 172.21.15.111:0/4182547231 learned_addr learned my addr 172.21.15.111:0/4182547231 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:28.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f339b7fe700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f339c06b720 0x7f339c14e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f339affd700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f339c06e970 0x7f339c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f339bfff700 1 -- 172.21.15.111:0/4182547231 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f339c06b720 msgr2=0x7f339c14e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f339bfff700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f339c06b720 0x7f339c14e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f339bfff700 1 -- 172.21.15.111:0/4182547231 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f339c06e970 msgr2=0x7f339c0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f339bfff700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f339c06e970 0x7f339c0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f339bfff700 1 -- 172.21.15.111:0/4182547231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c0094a0 con 0x7f339c15c5e0 2023-11-28T13:06:28.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f339bfff700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339c15c5e0 0x7f339c0b2850 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f339400cf20 tx=0x7f339400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:28.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f3383fff700 1 -- 172.21.15.111:0/4182547231 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339400ad60 con 0x7f339c15c5e0 2023-11-28T13:06:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f3383fff700 1 -- 172.21.15.111:0/4182547231 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3394015440 con 0x7f339c15c5e0 2023-11-28T13:06:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f3383fff700 1 -- 172.21.15.111:0/4182547231 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33940157f0 con 0x7f339c15c5e0 2023-11-28T13:06:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f33a1c06700 1 -- 172.21.15.111:0/4182547231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339c0cd660 con 0x7f339c15c5e0 2023-11-28T13:06:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.005+0000 7f33a1c06700 1 -- 172.21.15.111:0/4182547231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339c0cd380 con 0x7f339c15c5e0 2023-11-28T13:06:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.006+0000 7f33a1c06700 1 -- 172.21.15.111:0/4182547231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f339c160af0 con 0x7f339c15c5e0 2023-11-28T13:06:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.007+0000 7f3383fff700 1 -- 172.21.15.111:0/4182547231 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f339401c030 con 0x7f339c15c5e0 2023-11-28T13:06:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.007+0000 7f3383fff700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f33880682f0 0x7f338806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.007+0000 7f3383fff700 1 -- 172.21.15.111:0/4182547231 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f3394086580 con 0x7f339c15c5e0 2023-11-28T13:06:28.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.007+0000 7f339b7fe700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f33880682f0 0x7f338806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.007+0000 7f3383fff700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] conn(0x7f338806eb90 0x7f3388070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.007+0000 7f3383fff700 1 -- 172.21.15.111:0/4182547231 --> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33880716b0 con 0x7f338806eb90 2023-11-28T13:06:28.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.007+0000 7f3383fff700 1 -- 172.21.15.111:0/4182547231 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33940869c0 con 0x7f339c15c5e0 2023-11-28T13:06:28.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.008+0000 7f339b7fe700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f33880682f0 0x7f338806a7a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3390000f60 tx=0x7f3390007e70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:28.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.008+0000 7f339affd700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] conn(0x7f338806eb90 0x7f3388070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.008+0000 7f339affd700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] conn(0x7f338806eb90 0x7f3388070fe0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:28.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.009+0000 7f3383fff700 1 -- 172.21.15.111:0/4182547231 <== osd.7 v2:172.21.15.149:6824/103827 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f33880716b0 con 0x7f338806eb90 2023-11-28T13:06:28.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.022+0000 7f33a1c06700 1 -- 172.21.15.111:0/4182547231 --> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f339c067170 con 0x7f338806eb90 2023-11-28T13:06:28.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.023+0000 7f3383fff700 1 -- 172.21.15.111:0/4182547231 <== osd.7 v2:172.21.15.149:6824/103827 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f339c067170 con 0x7f338806eb90 2023-11-28T13:06:28.025 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-11-28T13:06:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-28T13:06:28.025 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-11-28T13:06:28.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.023+0000 7f33a1c06700 1 -- 172.21.15.111:0/4182547231 >> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] conn(0x7f338806eb90 msgr2=0x7f3388070fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.023+0000 7f33a1c06700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.149:6824/103827,v1:172.21.15.149:6825/103827] conn(0x7f338806eb90 0x7f3388070fe0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.024+0000 7f33a1c06700 1 -- 172.21.15.111:0/4182547231 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f33880682f0 msgr2=0x7f338806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.024+0000 7f33a1c06700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f33880682f0 0x7f338806a7a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3390000f60 tx=0x7f3390007e70 comp rx=0 tx=0).stop 2023-11-28T13:06:28.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.024+0000 7f33a1c06700 1 -- 172.21.15.111:0/4182547231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339c15c5e0 msgr2=0x7f339c0b2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.024+0000 7f33a1c06700 1 --2- 172.21.15.111:0/4182547231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f339c15c5e0 0x7f339c0b2850 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f339400cf20 tx=0x7f339400cf50 comp rx=0 tx=0).stop 2023-11-28T13:06:28.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.024+0000 7f339bfff700 1 -- 172.21.15.111:0/4182547231 reap_dead start 2023-11-28T13:06:28.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.024+0000 7f33a1c06700 1 -- 172.21.15.111:0/4182547231 shutdown_connections 2023-11-28T13:06:28.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.024+0000 7f33a1c06700 1 -- 172.21.15.111:0/4182547231 >> 172.21.15.111:0/4182547231 conn(0x7f339c0b0db0 msgr2=0x7f339c0aedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:28.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.024+0000 7f33a1c06700 1 -- 172.21.15.111:0/4182547231 shutdown_connections 2023-11-28T13:06:28.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.024+0000 7f33a1c06700 1 -- 172.21.15.111:0/4182547231 wait complete. 2023-11-28T13:06:28.037 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-11-28T13:06:28.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.155+0000 7fb8dc187700 1 Processor -- start 2023-11-28T13:06:28.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.156+0000 7fb8dc187700 1 -- start start 2023-11-28T13:06:28.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.156+0000 7fb8dc187700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fb8d415c5e0 0x7fb8d40ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.156+0000 7fb8dc187700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8d40ceb00 0x7fb8d406d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.156+0000 7fb8dc187700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d406ddb0 con 0x7fb8d406d960 2023-11-28T13:06:28.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.156+0000 7fb8dc187700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d4160d20 con 0x7fb8d415c5e0 2023-11-28T13:06:28.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.156+0000 7fb8dc187700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d415c9b0 con 0x7fb8d40ceb00 2023-11-28T13:06:28.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8da724700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fb8d406d960 0x7fb8d4071380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39798/0 (socket says 172.21.15.111:39798) 2023-11-28T13:06:28.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8da724700 1 -- 172.21.15.111:0/4041055558 learned_addr learned my addr 172.21.15.111:0/4041055558 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:28.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d9f23700 1 --2- 172.21.15.111:0/4041055558 >> v2:172.21.15.149:3300/0 conn(0x7fb8d415c5e0 0x7fb8d40ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d9722700 1 --2- 172.21.15.111:0/4041055558 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8d40ceb00 0x7fb8d406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359357969 0 0) 0x7fb8cc001490 con 0x7fb8d406d960 2023-11-28T13:06:28.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8c00035b0 con 0x7fb8d406d960 2023-11-28T13:06:28.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3944616967 0 0) 0x7fb8c00035b0 con 0x7fb8d406d960 2023-11-28T13:06:28.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8d40ceb00 msgr2=0x7fb8d406d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:28.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 --2- 172.21.15.111:0/4041055558 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8d40ceb00 0x7fb8d406d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 >> v2:172.21.15.149:3300/0 conn(0x7fb8d415c5e0 msgr2=0x7fb8d40ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:28.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 --2- 172.21.15.111:0/4041055558 >> v2:172.21.15.149:3300/0 conn(0x7fb8d415c5e0 0x7fb8d40ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d4160110 con 0x7fb8d406d960 2023-11-28T13:06:28.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fb8cc001b80 con 0x7fb8d406d960 2023-11-28T13:06:28.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 >> v1:172.21.15.111:6789/0 conn(0x7fb8d406d960 legacy=0x7fb8d4071380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 --2- 172.21.15.111:0/4041055558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8c0003a00 0x7fb8c0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 --2- 172.21.15.111:0/4041055558 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8d40ceb00 0x7fb8c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 --2- 172.21.15.111:0/4041055558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c0004d90 0x7fb8c0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d4160d20 con 0x7fb8c0004d90 2023-11-28T13:06:28.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d415c9b0 con 0x7fb8c0003a00 2023-11-28T13:06:28.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c000ba10 con 0x7fb8d40ceb00 2023-11-28T13:06:28.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8da724700 1 --2- 172.21.15.111:0/4041055558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c0004d90 0x7fb8c0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d9f23700 1 --2- 172.21.15.111:0/4041055558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8c0003a00 0x7fb8c0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.157+0000 7fb8d9722700 1 --2- 172.21.15.111:0/4041055558 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8d40ceb00 0x7fb8c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8da724700 1 -- 172.21.15.111:0/4041055558 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8d40ceb00 msgr2=0x7fb8c0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:28.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8da724700 1 --2- 172.21.15.111:0/4041055558 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8d40ceb00 0x7fb8c0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8da724700 1 -- 172.21.15.111:0/4041055558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8c0003a00 msgr2=0x7fb8c0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:28.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8da724700 1 --2- 172.21.15.111:0/4041055558 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8c0003a00 0x7fb8c0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8da724700 1 -- 172.21.15.111:0/4041055558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c00035b0 con 0x7fb8c0004d90 2023-11-28T13:06:28.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8da724700 1 --2- 172.21.15.111:0/4041055558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c0004d90 0x7fb8c0005200 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00a830 tx=0x7fb8c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f35f5002212df2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:28.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc012600 con 0x7fb8c0004d90 2023-11-28T13:06:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc012780 con 0x7fb8c0004d90 2023-11-28T13:06:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8d8f21700 1 -- 172.21.15.111:0/4041055558 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc012b30 con 0x7fb8c0004d90 2023-11-28T13:06:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8dc187700 1 -- 172.21.15.111:0/4041055558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c0004d90 msgr2=0x7fb8c0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8dc187700 1 --2- 172.21.15.111:0/4041055558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8c0004d90 0x7fb8c0005200 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00a830 tx=0x7fb8c0005880 comp rx=0 tx=0).stop 2023-11-28T13:06:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8da724700 1 -- 172.21.15.111:0/4041055558 reap_dead start 2023-11-28T13:06:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8dc187700 1 -- 172.21.15.111:0/4041055558 shutdown_connections 2023-11-28T13:06:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.158+0000 7fb8dc187700 1 -- 172.21.15.111:0/4041055558 >> 172.21.15.111:0/4041055558 conn(0x7fb8d40cf310 msgr2=0x7fb8d40cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:28.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.159+0000 7fb8dc187700 1 -- 172.21.15.111:0/4041055558 shutdown_connections 2023-11-28T13:06:28.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.159+0000 7fb8dc187700 1 -- 172.21.15.111:0/4041055558 wait complete. 2023-11-28T13:06:28.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.160+0000 7fb8dc187700 1 Processor -- start 2023-11-28T13:06:28.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.160+0000 7fb8dc187700 1 -- start start 2023-11-28T13:06:28.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.160+0000 7fb8dc187700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8c0004d90 0x7fb8d40c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.160+0000 7fb8dc187700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8d40cc6e0 0x7fb8d414cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8dc187700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8d40c8000 0x7fb8d40c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8dc187700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d40c7b70 con 0x7fb8d40cc6e0 2023-11-28T13:06:28.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8dc187700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d40c95a0 con 0x7fb8c0004d90 2023-11-28T13:06:28.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8dc187700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d40c7240 con 0x7fb8d40c8000 2023-11-28T13:06:28.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8d9722700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8d40cc6e0 0x7fb8d414cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8d9722700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8d40cc6e0 0x7fb8d414cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39034/0 (socket says 172.21.15.111:39034) 2023-11-28T13:06:28.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8d9722700 1 -- 172.21.15.111:0/1382558182 learned_addr learned my addr 172.21.15.111:0/1382558182 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:28.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8d9722700 1 -- 172.21.15.111:0/1382558182 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8d40c8000 msgr2=0x7fb8d40c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8d9722700 1 --2- 172.21.15.111:0/1382558182 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8d40c8000 0x7fb8d40c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8d9722700 1 -- 172.21.15.111:0/1382558182 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8c0004d90 msgr2=0x7fb8d40c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8d9f23700 1 --2- 172.21.15.111:0/1382558182 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8c0004d90 0x7fb8d40c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8d9722700 1 --2- 172.21.15.111:0/1382558182 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8c0004d90 0x7fb8d40c8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8d9722700 1 -- 172.21.15.111:0/1382558182 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d41606d0 con 0x7fb8d40cc6e0 2023-11-28T13:06:28.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8d9722700 1 --2- 172.21.15.111:0/1382558182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8d40cc6e0 0x7fb8d414cae0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0011a20 tx=0x7fb8d0011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:28.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8c67fc700 1 -- 172.21.15.111:0/1382558182 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d000f940 con 0x7fb8d40cc6e0 2023-11-28T13:06:28.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8c67fc700 1 -- 172.21.15.111:0/1382558182 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8d000faa0 con 0x7fb8d40cc6e0 2023-11-28T13:06:28.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8c67fc700 1 -- 172.21.15.111:0/1382558182 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8d000fe50 con 0x7fb8d40cc6e0 2023-11-28T13:06:28.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8dc187700 1 -- 172.21.15.111:0/1382558182 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d41603f0 con 0x7fb8d40cc6e0 2023-11-28T13:06:28.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.161+0000 7fb8dc187700 1 -- 172.21.15.111:0/1382558182 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d41609b0 con 0x7fb8d40cc6e0 2023-11-28T13:06:28.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.162+0000 7fb8dc187700 1 -- 172.21.15.111:0/1382558182 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d4067170 con 0x7fb8d40cc6e0 2023-11-28T13:06:28.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.163+0000 7fb8c67fc700 1 -- 172.21.15.111:0/1382558182 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb8d0014440 con 0x7fb8d40cc6e0 2023-11-28T13:06:28.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.163+0000 7fb8c67fc700 1 --2- 172.21.15.111:0/1382558182 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8b80682f0 0x7fb8b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.163+0000 7fb8d9f23700 1 --2- 172.21.15.111:0/1382558182 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8b80682f0 0x7fb8b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.163+0000 7fb8c67fc700 1 -- 172.21.15.111:0/1382558182 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fb8d0086390 con 0x7fb8d40cc6e0 2023-11-28T13:06:28.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.163+0000 7fb8d9f23700 1 --2- 172.21.15.111:0/1382558182 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8b80682f0 0x7fb8b806a7a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8009aa0 tx=0x7fb8c8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:28.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.167+0000 7fb8c67fc700 1 -- 172.21.15.111:0/1382558182 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb8d0052ec0 con 0x7fb8d40cc6e0 2023-11-28T13:06:28.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.352+0000 7fb8dc187700 1 -- 172.21.15.111:0/1382558182 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8d406e2d0 con 0x7fb8b80682f0 2023-11-28T13:06:28.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.352+0000 7fb8c67fc700 1 -- 172.21.15.111:0/1382558182 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb8d406e2d0 con 0x7fb8b80682f0 2023-11-28T13:06:28.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.354+0000 7fb8dc187700 1 -- 172.21.15.111:0/1382558182 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8b80682f0 msgr2=0x7fb8b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.354+0000 7fb8dc187700 1 --2- 172.21.15.111:0/1382558182 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8b80682f0 0x7fb8b806a7a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8009aa0 tx=0x7fb8c8008040 comp rx=0 tx=0).stop 2023-11-28T13:06:28.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.354+0000 7fb8dc187700 1 -- 172.21.15.111:0/1382558182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8d40cc6e0 msgr2=0x7fb8d414cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.354+0000 7fb8dc187700 1 --2- 172.21.15.111:0/1382558182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8d40cc6e0 0x7fb8d414cae0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0011a20 tx=0x7fb8d0011de0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.354+0000 7fb8dc187700 1 -- 172.21.15.111:0/1382558182 shutdown_connections 2023-11-28T13:06:28.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.354+0000 7fb8dc187700 1 --2- 172.21.15.111:0/1382558182 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fb8d40c8000 0x7fb8d40c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.354+0000 7fb8dc187700 1 --2- 172.21.15.111:0/1382558182 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fb8b80682f0 0x7fb8b806a7a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.354+0000 7fb8dc187700 1 --2- 172.21.15.111:0/1382558182 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fb8c0004d90 0x7fb8d40c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.354+0000 7fb8dc187700 1 --2- 172.21.15.111:0/1382558182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8d40cc6e0 0x7fb8d414cae0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.354+0000 7fb8dc187700 1 -- 172.21.15.111:0/1382558182 >> 172.21.15.111:0/1382558182 conn(0x7fb8d40cf310 msgr2=0x7fb8d4156c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:28.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.354+0000 7fb8dc187700 1 -- 172.21.15.111:0/1382558182 shutdown_connections 2023-11-28T13:06:28.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.354+0000 7fb8dc187700 1 -- 172.21.15.111:0/1382558182 wait complete. 2023-11-28T13:06:28.357 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 7 to deep-scrub 2023-11-28T13:06:28.367 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:06:28.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.485+0000 7f2fb28b9700 1 Processor -- start 2023-11-28T13:06:28.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.485+0000 7f2fb28b9700 1 -- start start 2023-11-28T13:06:28.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.486+0000 7f2fb28b9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac0cc550 0x7f2fac0cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.486+0000 7f2fb28b9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fac15cd90 con 0x7f2fac1604b0 2023-11-28T13:06:28.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.486+0000 7f2fb28b9700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fac1613e0 con 0x7f2fac0b0d70 2023-11-28T13:06:28.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.486+0000 7f2fb28b9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac160950 con 0x7f2fac0cc550 2023-11-28T13:06:28.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.486+0000 7f2fab7fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2fac1604b0 0x7f2fac0ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39818/0 (socket says 172.21.15.111:39818) 2023-11-28T13:06:28.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.486+0000 7f2fab7fe700 1 -- 172.21.15.111:0/3124838394 learned_addr learned my addr 172.21.15.111:0/3124838394 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:28.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2fb0e56700 1 --2- 172.21.15.111:0/3124838394 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac0cc550 0x7f2fac0cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190101912 0 0) 0x7f2f9c001110 con 0x7f2fac1604b0 2023-11-28T13:06:28.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f980035b0 con 0x7f2fac1604b0 2023-11-28T13:06:28.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2465487765 0 0) 0x7f2f980035b0 con 0x7f2fac1604b0 2023-11-28T13:06:28.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac0cc550 msgr2=0x7f2fac0cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:28.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 --2- 172.21.15.111:0/3124838394 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac0cc550 0x7f2fac0cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 >> v1:172.21.15.149:6789/0 conn(0x7f2fac0b0d70 legacy=0x7f2fac0b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac160b40 con 0x7f2fac1604b0 2023-11-28T13:06:28.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f2f9c001800 con 0x7f2fac1604b0 2023-11-28T13:06:28.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 >> v1:172.21.15.111:6789/0 conn(0x7f2fac1604b0 legacy=0x7f2fac0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 --2- 172.21.15.111:0/3124838394 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac0cc550 0x7f2f98003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 --2- 172.21.15.111:0/3124838394 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f98003fb0 0x7f2f98004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 --2- 172.21.15.111:0/3124838394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f98004d90 0x7f2f98005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9800bac0 con 0x7f2f98004d90 2023-11-28T13:06:28.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9800bc30 con 0x7f2f98003fb0 2023-11-28T13:06:28.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9800bdf0 con 0x7f2fac0cc550 2023-11-28T13:06:28.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2fb0e56700 1 --2- 172.21.15.111:0/3124838394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f98004d90 0x7f2f98005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2fab7fe700 1 --2- 172.21.15.111:0/3124838394 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f98003fb0 0x7f2f98004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2fabfff700 1 --2- 172.21.15.111:0/3124838394 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac0cc550 0x7f2f98003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2fb0e56700 1 -- 172.21.15.111:0/3124838394 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac0cc550 msgr2=0x7f2f98003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:28.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2fb0e56700 1 --2- 172.21.15.111:0/3124838394 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac0cc550 0x7f2f98003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2fb0e56700 1 -- 172.21.15.111:0/3124838394 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f98003fb0 msgr2=0x7f2f98004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:28.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2fb0e56700 1 --2- 172.21.15.111:0/3124838394 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2f98003fb0 0x7f2f98004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2fb0e56700 1 -- 172.21.15.111:0/3124838394 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f980035b0 con 0x7f2f98004d90 2023-11-28T13:06:28.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.487+0000 7f2fb0e56700 1 --2- 172.21.15.111:0/3124838394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f98004d90 0x7f2f98005200 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4009f90 tx=0x7f2f98005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa3041ff03df021 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:28.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.488+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa400ec30 con 0x7f2f98004d90 2023-11-28T13:06:28.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.488+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa400f040 con 0x7f2f98004d90 2023-11-28T13:06:28.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.488+0000 7f2faaffd700 1 -- 172.21.15.111:0/3124838394 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4014480 con 0x7f2f98004d90 2023-11-28T13:06:28.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.488+0000 7f2fb28b9700 1 -- 172.21.15.111:0/3124838394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f98004d90 msgr2=0x7f2f98005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.488+0000 7f2fb28b9700 1 --2- 172.21.15.111:0/3124838394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f98004d90 0x7f2f98005200 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4009f90 tx=0x7f2f98005880 comp rx=0 tx=0).stop 2023-11-28T13:06:28.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.488+0000 7f2fb0e56700 1 -- 172.21.15.111:0/3124838394 reap_dead start 2023-11-28T13:06:28.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.488+0000 7f2fb28b9700 1 -- 172.21.15.111:0/3124838394 shutdown_connections 2023-11-28T13:06:28.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.488+0000 7f2fb28b9700 1 -- 172.21.15.111:0/3124838394 >> 172.21.15.111:0/3124838394 conn(0x7f2fac098ee0 msgr2=0x7f2fac06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:28.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.488+0000 7f2fb28b9700 1 -- 172.21.15.111:0/3124838394 shutdown_connections 2023-11-28T13:06:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.488+0000 7f2fb28b9700 1 -- 172.21.15.111:0/3124838394 wait complete. 2023-11-28T13:06:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.489+0000 7f2fb28b9700 1 Processor -- start 2023-11-28T13:06:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.490+0000 7f2fb28b9700 1 -- start start 2023-11-28T13:06:28.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.490+0000 7f2fb28b9700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fac156330 0x7f2fac0b3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.490+0000 7f2fb28b9700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac156700 0x7f2fac0c5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.490+0000 7f2fb28b9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fac0b2970 0x7f2fac0ad960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.490+0000 7f2fb28b9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0b4480 con 0x7f2fac0b2970 2023-11-28T13:06:28.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.490+0000 7f2fb28b9700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0b2ea0 con 0x7f2fac156330 2023-11-28T13:06:28.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.490+0000 7f2fb28b9700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0b3180 con 0x7f2fac156700 2023-11-28T13:06:28.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.490+0000 7f2fb0e56700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fac0b2970 0x7f2fac0ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fb0e56700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fac0b2970 0x7f2fac0ad960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39056/0 (socket says 172.21.15.111:39056) 2023-11-28T13:06:28.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fb0e56700 1 -- 172.21.15.111:0/4150136429 learned_addr learned my addr 172.21.15.111:0/4150136429 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:28.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fab7fe700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac156700 0x7f2fac0c5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fb0e56700 1 -- 172.21.15.111:0/4150136429 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac156700 msgr2=0x7f2fac0c5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fabfff700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fac156330 0x7f2fac0b3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fb0e56700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac156700 0x7f2fac0c5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fb0e56700 1 -- 172.21.15.111:0/4150136429 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fac156330 msgr2=0x7f2fac0b3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fb0e56700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fac156330 0x7f2fac0b3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fb0e56700 1 -- 172.21.15.111:0/4150136429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac161100 con 0x7f2fac0b2970 2023-11-28T13:06:28.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fb0e56700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fac0b2970 0x7f2fac0ad960 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4016000 tx=0x7f2fa4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:28.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fa8ff9700 1 -- 172.21.15.111:0/4150136429 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4007000 con 0x7f2fac0b2970 2023-11-28T13:06:28.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fa8ff9700 1 -- 172.21.15.111:0/4150136429 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa4009410 con 0x7f2fac0b2970 2023-11-28T13:06:28.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fb28b9700 1 -- 172.21.15.111:0/4150136429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fac160e20 con 0x7f2fac0b2970 2023-11-28T13:06:28.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fa8ff9700 1 -- 172.21.15.111:0/4150136429 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4015d80 con 0x7f2fac0b2970 2023-11-28T13:06:28.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.491+0000 7f2fb28b9700 1 -- 172.21.15.111:0/4150136429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fac0af340 con 0x7f2fac0b2970 2023-11-28T13:06:28.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.492+0000 7f2fb28b9700 1 -- 172.21.15.111:0/4150136429 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fac067170 con 0x7f2fac0b2970 2023-11-28T13:06:28.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.493+0000 7f2fa8ff9700 1 -- 172.21.15.111:0/4150136429 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2fa4030070 con 0x7f2fac0b2970 2023-11-28T13:06:28.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.493+0000 7f2fa8ff9700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f8c0682f0 0x7f2f8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:28.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.493+0000 7f2fa8ff9700 1 -- 172.21.15.111:0/4150136429 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f2fa4024030 con 0x7f2fac0b2970 2023-11-28T13:06:28.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.493+0000 7f2fabfff700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f8c0682f0 0x7f2f8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:28.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.494+0000 7f2fabfff700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f8c0682f0 0x7f2f8c06a7a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f2f940055f0 tx=0x7f2f94009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:28.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.497+0000 7f2fa8ff9700 1 -- 172.21.15.111:0/4150136429 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2fa40544d0 con 0x7f2fac0b2970 2023-11-28T13:06:28.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.674+0000 7f2fb28b9700 1 -- 172.21.15.111:0/4150136429 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2fac0cca60 con 0x7f2f8c0682f0 2023-11-28T13:06:28.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.677+0000 7f2fa8ff9700 1 -- 172.21.15.111:0/4150136429 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+73266 (secure 0 0 0) 0x7f2fac0cca60 con 0x7f2f8c0682f0 2023-11-28T13:06:28.679 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.679+0000 7f2fb28b9700 1 -- 172.21.15.111:0/4150136429 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f8c0682f0 msgr2=0x7f2f8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.679+0000 7f2fb28b9700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f8c0682f0 0x7f2f8c06a7a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f2f940055f0 tx=0x7f2f94009040 comp rx=0 tx=0).stop 2023-11-28T13:06:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.679+0000 7f2fb28b9700 1 -- 172.21.15.111:0/4150136429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fac0b2970 msgr2=0x7f2fac0ad960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.679+0000 7f2fb28b9700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fac0b2970 0x7f2fac0ad960 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4016000 tx=0x7f2fa4009750 comp rx=0 tx=0).stop 2023-11-28T13:06:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.679+0000 7f2fb28b9700 1 -- 172.21.15.111:0/4150136429 shutdown_connections 2023-11-28T13:06:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.679+0000 7f2fb28b9700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fac0b2970 0x7f2fac0ad960 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.679+0000 7f2fb28b9700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2f8c0682f0 0x7f2f8c06a7a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.679+0000 7f2fb28b9700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2fac156330 0x7f2fac0b3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.679+0000 7f2fb28b9700 1 --2- 172.21.15.111:0/4150136429 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2fac156700 0x7f2fac0c5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:28.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.679+0000 7f2fb28b9700 1 -- 172.21.15.111:0/4150136429 >> 172.21.15.111:0/4150136429 conn(0x7f2fac098ee0 msgr2=0x7f2fac0af760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:28.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.679+0000 7f2fb28b9700 1 -- 172.21.15.111:0/4150136429 shutdown_connections 2023-11-28T13:06:28.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:28.679+0000 7f2fb28b9700 1 -- 172.21.15.111:0/4150136429 wait complete. 2023-11-28T13:06:28.683 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:06:28.692 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2023-11-28T13:06:28.445548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":82472,"kb_used_data":8040,"kb_used_omap":0,"kb_used_meta":74368,"kb_avail":754892248,"statfs":{"total":773094113280,"available":773009661952,"internally_reserved":0,"allocated":8232960,"data_stored":4491880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":76152832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":-3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.471814"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-11-28T13:06:25.847158+0000","last_change":"2023-11-28T13:06:25.847137+0000","last_active":"2023-11-28T13:06:25.847158+0000","last_peered":"2023-11-28T13:06:25.847158+0000","last_clean":"2023-11-28T13:06:25.847158+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:06:25.847158+0000","last_undegraded":"2023-11-28T13:06:25.847158+0000","last_fullsized":"2023-11-28T13:06:25.847158+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:06:25.847056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:06:25.847056+0000","last_clean_scrub_stamp":"2023-11-28T13:06:25.847056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:26:35.431945+0000","scrub_duration":0.0034698929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":58,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.485662+0000","last_change":"2023-11-28T12:58:50.206823+0000","last_active":"2023-11-28T13:06:19.485662+0000","last_peered":"2023-11-28T13:06:19.485662+0000","last_clean":"2023-11-28T13:06:19.485662+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:06:19.485662+0000","last_undegraded":"2023-11-28T13:06:19.485662+0000","last_fullsized":"2023-11-28T13:06:19.485662+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:15:56.681008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-11-28T13:06:25.951630+0000","last_change":"2023-11-28T13:06:25.951605+0000","last_active":"2023-11-28T13:06:25.951630+0000","last_peered":"2023-11-28T13:06:25.951630+0000","last_clean":"2023-11-28T13:06:25.951630+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:06:25.951630+0000","last_undegraded":"2023-11-28T13:06:25.951630+0000","last_fullsized":"2023-11-28T13:06:25.951630+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:06:25.951514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:06:25.951514+0000","last_clean_scrub_stamp":"2023-11-28T13:06:25.951514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:53:11.317252+0000","scrub_duration":0.0034822400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":58,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.486018+0000","last_change":"2023-11-28T12:58:50.204377+0000","last_active":"2023-11-28T13:06:19.486018+0000","last_peered":"2023-11-28T13:06:19.486018+0000","last_clean":"2023-11-28T13:06:19.486018+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:06:19.486018+0000","last_undegraded":"2023-11-28T13:06:19.486018+0000","last_fullsized":"2023-11-28T13:06:19.486018+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:11:50.948290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":211,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.485304+0000","last_change":"2023-11-28T12:58:48.490116+0000","last_active":"2023-11-28T13:06:19.485304+0000","last_peered":"2023-11-28T13:06:19.485304+0000","last_clean":"2023-11-28T13:06:19.485304+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:06:19.485304+0000","last_undegraded":"2023-11-28T13:06:19.485304+0000","last_fullsized":"2023-11-28T13:06:19.485304+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:47.164859+0000","last_clean_scrub_stamp":"2023-11-28T12:58:47.164859+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:59:48.809452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":59,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.485388+0000","last_change":"2023-11-28T12:58:50.210847+0000","last_active":"2023-11-28T13:06:19.485388+0000","last_peered":"2023-11-28T13:06:19.485388+0000","last_clean":"2023-11-28T13:06:19.485388+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:06:19.485388+0000","last_undegraded":"2023-11-28T13:06:19.485388+0000","last_fullsized":"2023-11-28T13:06:19.485388+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:20:24.576055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":59,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.487237+0000","last_change":"2023-11-28T12:58:50.572468+0000","last_active":"2023-11-28T13:06:19.487237+0000","last_peered":"2023-11-28T13:06:19.487237+0000","last_clean":"2023-11-28T13:06:19.487237+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:06:19.487237+0000","last_undegraded":"2023-11-28T13:06:19.487237+0000","last_fullsized":"2023-11-28T13:06:19.487237+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:48:29.091528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-11-28T13:06:19.974727+0000","last_change":"2023-11-28T12:58:50.206591+0000","last_active":"2023-11-28T13:06:19.974727+0000","last_peered":"2023-11-28T13:06:19.974727+0000","last_clean":"2023-11-28T13:06:19.974727+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:06:19.974727+0000","last_undegraded":"2023-11-28T13:06:19.974727+0000","last_fullsized":"2023-11-28T13:06:19.974727+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-11-28T13:06:18.978940+0000","last_change":"2023-11-28T12:58:50.201845+0000","last_active":"2023-11-28T13:06:18.978940+0000","last_peered":"2023-11-28T13:06:18.978940+0000","last_clean":"2023-11-28T13:06:18.978940+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:06:18.978940+0000","last_undegraded":"2023-11-28T13:06:18.978940+0000","last_fullsized":"2023-11-28T13:06:18.978940+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T12:58:49.183429+0000","last_clean_scrub_stamp":"2023-11-28T12:58:49.183429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:47:40.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574943,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11176,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":9728,"kb_avail":94360664,"statfs":{"total":96636764160,"available":96625319936,"internally_reserved":0,"allocated":1474560,"data_stored":1004261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9961472},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.64700000000000002},"min":{"1min":0.51200000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.79800000000000004,"5min":1.194,"15min":3.3279999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.58999999999999997,"15min":0.64800000000000002},"min":{"1min":0.46300000000000002,"5min":0.372,"15min":0.27900000000000003},"max":{"1min":0.77000000000000002,"5min":0.81799999999999995,"15min":3.2919999999999998},"last":0.48199999999999998}]},{"osd":1,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.40300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.52400000000000002},"min":{"1min":0.436,"5min":0.34999999999999998,"15min":0.30499999999999999},"max":{"1min":0.76000000000000001,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42399999999999999}]},{"osd":2,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.56200000000000006},"min":{"1min":0.44600000000000001,"5min":0.35699999999999998,"15min":0.28399999999999997},"max":{"1min":0.751,"5min":1.149,"15min":1.149},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57499999999999996,"15min":0.55200000000000005},"min":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.35699999999999998},"max":{"1min":0.72399999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.38800000000000001}]},{"osd":3,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60299999999999998,"15min":0.56000000000000005},"min":{"1min":0.52000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82899999999999996,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58799999999999997,"15min":0.57299999999999995},"min":{"1min":0.433,"5min":0.40699999999999997,"15min":0.38600000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.54100000000000004}]},{"osd":4,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47599999999999998,"15min":0.50800000000000001},"min":{"1min":0.35099999999999998,"5min":0.33300000000000002,"15min":0.316},"max":{"1min":0.78400000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.50600000000000001,"15min":0.52600000000000002},"min":{"1min":0.373,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.73899999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.61699999999999999}]},{"osd":5,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47799999999999998,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71399999999999997,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53500000000000003,"15min":0.56000000000000005},"min":{"1min":0.36899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.51200000000000001}]},{"osd":6,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52400000000000002,"15min":0.56000000000000005},"min":{"1min":0.38200000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77800000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58899999999999997,"15min":0.60799999999999998},"min":{"1min":0.434,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79900000000000004,"5min":1.274,"15min":1.274},"last":0.53700000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574945,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10020,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361820,"statfs":{"total":96636764160,"available":96626503680,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.437,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.89600000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.52400000000000002},"min":{"1min":0.33900000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72799999999999998,"5min":0.755,"15min":0.80400000000000005},"last":0.57599999999999996}]},{"osd":1,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56699999999999995,"15min":0.53500000000000003},"min":{"1min":0.46100000000000002,"5min":0.32400000000000001,"15min":0.309},"max":{"1min":0.752,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40300000000000002,"5min":0.33000000000000002,"15min":0.29599999999999999},"max":{"1min":0.76700000000000002,"5min":0.86599999999999999,"15min":1.016},"last":0.48299999999999998}]},{"osd":2,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.39900000000000002,"5min":0.34300000000000003,"15min":0.34100000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.91700000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.75700000000000001,"5min":0.92400000000000004,"15min":0.96299999999999997},"last":0.60699999999999998}]},{"osd":3,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.371},"max":{"1min":0.71399999999999997,"5min":0.80700000000000005,"15min":1.5920000000000001},"last":0.502},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59399999999999997,"15min":0.58399999999999996},"min":{"1min":0.504,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76400000000000001,"5min":0.84999999999999998,"15min":0.94099999999999995},"last":0.64100000000000001}]},{"osd":4,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46100000000000002,"15min":0.51200000000000001},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74199999999999999,"5min":0.84199999999999997,"15min":1},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48199999999999998,"15min":0.52200000000000002},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72699999999999998,"5min":0.753,"15min":0.89200000000000002},"last":0.627}]},{"osd":5,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.54600000000000004},"min":{"1min":0.36399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.871,"5min":0.871,"15min":1.052},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.59499999999999997},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.81299999999999994,"5min":0.88900000000000001,"15min":1.258},"last":0.70999999999999996}]},{"osd":7,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.52100000000000002,"15min":0.56200000000000006},"min":{"1min":0.32100000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.79200000000000004,"5min":0.80100000000000005,"15min":0.97599999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56100000000000005,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.89500000000000002,"5min":0.90400000000000003,"15min":1.1060000000000001},"last":0.68700000000000006}]}]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10020,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361820,"statfs":{"total":96636764160,"available":96626503680,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35099999999999998,"15min":0.33500000000000002},"min":{"1min":0.23799999999999999,"5min":0.224,"15min":0.219},"max":{"1min":0.496,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.36499999999999999},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60299999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.28299999999999997}]},{"osd":2,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.373,"15min":0.34699999999999998},"min":{"1min":0.29199999999999998,"5min":0.22,"15min":0.20599999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40799999999999997,"15min":0.38100000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.25},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.80100000000000005},"last":0.33100000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.434,"15min":0.40400000000000003},"min":{"1min":0.32800000000000001,"5min":0.27200000000000002,"15min":0.245},"max":{"1min":0.57399999999999995,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.41299999999999998,"15min":0.40000000000000002},"min":{"1min":0.27800000000000002,"5min":0.26000000000000001,"15min":0.251},"max":{"1min":0.58399999999999996,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.25600000000000001}]},{"osd":4,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.60499999999999998,"15min":0.57999999999999996},"min":{"1min":0.40200000000000002,"5min":0.36199999999999999,"15min":0.34000000000000002},"max":{"1min":0.80000000000000004,"5min":0.91700000000000004,"15min":1.028},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.624,"15min":0.59599999999999997},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.34799999999999998},"max":{"1min":0.76100000000000001,"5min":0.98699999999999999,"15min":1.0660000000000001},"last":0.55900000000000005}]},{"osd":5,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.59999999999999998},"min":{"1min":0.47299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78100000000000003,"5min":0.94999999999999996,"15min":1.042},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63500000000000001,"15min":0.58499999999999996},"min":{"1min":0.44700000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78300000000000003,"5min":1.05,"15min":1.05},"last":0.53500000000000003}]},{"osd":6,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.626,"15min":0.59599999999999997},"min":{"1min":0.438,"5min":0.40600000000000003,"15min":0.38900000000000001},"max":{"1min":0.80100000000000005,"5min":0.96199999999999997,"15min":1.006},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63400000000000001,"15min":0.60999999999999999},"min":{"1min":0.48699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.81899999999999995,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.66900000000000004}]},{"osd":7,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.64200000000000002,"15min":0.61499999999999999},"min":{"1min":0.51600000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64000000000000001,"15min":0.59599999999999997},"min":{"1min":0.47799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80400000000000005,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.68500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574944,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11176,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":9728,"kb_avail":94360664,"statfs":{"total":96636764160,"available":96625319936,"internally_reserved":0,"allocated":1474560,"data_stored":1004261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9961472},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.33700000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51300000000000001,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38200000000000001,"15min":0.35499999999999998},"min":{"1min":0.23000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.50700000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.38900000000000001}]},{"osd":2,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38200000000000001,"15min":0.35799999999999998},"min":{"1min":0.27300000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.48399999999999999,"5min":0.57099999999999995,"15min":0.59399999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41199999999999998,"15min":0.38100000000000001},"min":{"1min":0.28699999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50600000000000001,"5min":0.623,"15min":0.623},"last":0.46800000000000003}]},{"osd":3,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38900000000000001,"15min":0.377},"min":{"1min":0.247,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.68000000000000005},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.441,"15min":0.40500000000000003},"min":{"1min":0.28799999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52200000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.59399999999999997}]},{"osd":4,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.622,"15min":0.60699999999999998},"min":{"1min":0.51400000000000001,"5min":0.309,"15min":0.28199999999999997},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.626,"15min":0.58799999999999997},"min":{"1min":0.49299999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83299999999999996,"5min":0.88,"15min":0.88},"last":0.65400000000000003}]},{"osd":5,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.58299999999999996},"min":{"1min":0.42999999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79600000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64100000000000001,"15min":0.625},"min":{"1min":0.441,"5min":0.434,"15min":0.39600000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.68000000000000005}]},{"osd":6,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.64300000000000002,"15min":0.61599999999999999},"min":{"1min":0.51700000000000002,"5min":0.46999999999999997,"15min":0.41199999999999998},"max":{"1min":0.84799999999999998,"5min":0.89800000000000002,"15min":0.91900000000000004},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65300000000000002,"15min":0.627},"min":{"1min":0.50600000000000001,"5min":0.41499999999999998,"15min":0.33900000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.98599999999999999},"last":0.66800000000000004}]},{"osd":7,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.65900000000000003,"15min":0.63900000000000001},"min":{"1min":0.52900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.0149999999999999},"last":0.63},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65800000000000003,"15min":0.63700000000000001},"min":{"1min":0.45700000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.61199999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574944,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9892,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94361948,"statfs":{"total":96636764160,"available":96626634752,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.33000000000000002},"min":{"1min":0.255,"5min":0.22500000000000001,"15min":0.20999999999999999},"max":{"1min":0.48799999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"min":{"1min":0.22600000000000001,"5min":0.21199999999999999,"15min":0.184},"max":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.25900000000000001}]},{"osd":1,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.39800000000000002,"15min":0.38700000000000001},"min":{"1min":0.28799999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.51800000000000002,"5min":0.59499999999999997,"15min":0.67000000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.374},"min":{"1min":0.26300000000000001,"5min":0.25700000000000001,"15min":0.24199999999999999},"max":{"1min":0.56999999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.31}]},{"osd":3,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39200000000000002,"15min":0.35599999999999998},"min":{"1min":0.27900000000000003,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":0.505,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.437,"5min":0.443,"15min":0.42499999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.55700000000000005,"5min":0.70599999999999996,"15min":0.77500000000000002},"last":0.36299999999999999}]},{"osd":4,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64000000000000001,"15min":0.60599999999999998},"min":{"1min":0.46200000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85799999999999998,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63,"15min":0.57899999999999996},"min":{"1min":0.44900000000000001,"5min":0.42199999999999999,"15min":0.34300000000000003},"max":{"1min":0.86399999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.42599999999999999}]},{"osd":5,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.60499999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.91900000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.63400000000000001,"15min":0.60799999999999998},"min":{"1min":0.52500000000000002,"5min":0.41699999999999998,"15min":0.379},"max":{"1min":0.877,"5min":0.877,"15min":1.0589999999999999},"last":0.434}]},{"osd":6,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64800000000000002,"15min":0.59699999999999998},"min":{"1min":0.503,"5min":0.41999999999999998,"15min":0.36599999999999999},"max":{"1min":0.83899999999999997,"5min":0.89100000000000001,"15min":1.002},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65300000000000002,"15min":0.63300000000000001},"min":{"1min":0.48399999999999999,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.82799999999999996,"5min":0.96799999999999997,"15min":0.97799999999999998},"last":0.41999999999999998}]},{"osd":7,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65600000000000003,"15min":0.63600000000000001},"min":{"1min":0.438,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86199999999999999,"5min":0.93500000000000005,"15min":0.95099999999999996},"last":0.504},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.65800000000000003,"15min":0.61399999999999999},"min":{"1min":0.53000000000000003,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":0.86899999999999999,"5min":0.92400000000000004,"15min":1.0349999999999999},"last":0.45000000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574944,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10084,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9216,"kb_avail":94361756,"statfs":{"total":96636764160,"available":96626438144,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9437184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.33500000000000002},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38200000000000001,"15min":0.34499999999999997},"min":{"1min":0.24099999999999999,"5min":0.23200000000000001,"15min":0.20000000000000001},"max":{"1min":0.55200000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.34499999999999997}]},{"osd":1,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.38600000000000001,"15min":0.35399999999999998},"min":{"1min":0.25700000000000001,"5min":0.22,"15min":0.21299999999999999},"max":{"1min":0.53000000000000003,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.307},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40500000000000003,"15min":0.38300000000000001},"min":{"1min":0.317,"5min":0.254,"15min":0.23799999999999999},"max":{"1min":0.54600000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.39300000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.45000000000000001,"15min":0.47499999999999998},"min":{"1min":0.29299999999999998,"5min":0.252,"15min":0.251},"max":{"1min":0.58299999999999996,"5min":0.65100000000000002,"15min":2.6869999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45700000000000002,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.30499999999999999,"15min":0.28199999999999997},"max":{"1min":0.60099999999999998,"5min":0.66800000000000004,"15min":2.7160000000000002},"last":0.41899999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63600000000000001,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81999999999999995,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63,"15min":0.84599999999999997},"min":{"1min":0.442,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82899999999999996,"5min":0.93899999999999995,"15min":8.2889999999999997},"last":0.65400000000000003}]},{"osd":5,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.48599999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.63300000000000001,"15min":0.66000000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.72999999999999998,"5min":0.88200000000000001,"15min":2.165},"last":0.68000000000000005}]},{"osd":6,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65200000000000002,"15min":0.63600000000000001},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.88400000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.63},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.63500000000000001},"min":{"1min":0.53100000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.81299999999999994,"5min":0.95399999999999996,"15min":1.1419999999999999},"last":0.69899999999999995}]},{"osd":7,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66100000000000003,"15min":0.626},"min":{"1min":0.48399999999999999,"5min":0.46000000000000002,"15min":0.45600000000000002},"max":{"1min":0.82399999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66000000000000003,"15min":0.85199999999999998},"min":{"1min":0.45900000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85999999999999999,"5min":0.89600000000000002,"15min":7.0899999999999999},"last":0.67400000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":10212,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9344,"kb_avail":94361628,"statfs":{"total":96636764160,"available":96626307072,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9568256},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.66200000000000003,"5min":0.78500000000000003,"15min":1.0049999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56000000000000005,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.66600000000000004,"5min":0.873,"15min":0.873},"last":0.57399999999999995}]},{"osd":1,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55300000000000005,"15min":0.52100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71399999999999997,"5min":0.76600000000000001,"15min":0.95699999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56999999999999995,"15min":0.56499999999999995},"min":{"1min":0.45500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69999999999999996,"5min":0.77300000000000002,"15min":1.1060000000000001},"last":0.58399999999999996}]},{"osd":2,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59799999999999998,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68999999999999995,"5min":0.77300000000000002,"15min":0.91800000000000004},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60999999999999999,"15min":0.60199999999999998},"min":{"1min":0.48999999999999999,"5min":0.41499999999999998,"15min":0.36399999999999999},"max":{"1min":0.67800000000000005,"5min":0.81100000000000005,"15min":1.069},"last":0.61299999999999999}]},{"osd":3,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55200000000000005,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.70099999999999996,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.46300000000000002,"5min":0.41099999999999998,"15min":0.38900000000000001},"max":{"1min":0.71099999999999997,"5min":0.81599999999999995,"15min":0.879},"last":0.63300000000000001}]},{"osd":5,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47399999999999998,"15min":0.503},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.315},"max":{"1min":0.752,"5min":0.752,"15min":0.86099999999999999},"last":0.38},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.51000000000000001},"min":{"1min":0.371,"5min":0.309,"15min":0.309},"max":{"1min":0.68200000000000005,"5min":0.73799999999999999,"15min":0.80400000000000005},"last":0.48399999999999999}]},{"osd":6,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.49099999999999999,"15min":0.51300000000000001},"min":{"1min":0.35399999999999998,"5min":0.32500000000000001,"15min":0.30499999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.84299999999999997},"last":0.503},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53000000000000003,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.35199999999999998,"15min":0.34499999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.96399999999999997},"last":0.46899999999999997}]},{"osd":7,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73999999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.36599999999999999},"max":{"1min":0.80400000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.59999999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574944,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9892,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94361948,"statfs":{"total":96636764160,"available":96626634752,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.378,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72699999999999998,"5min":0.74399999999999999,"15min":1.819},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.53600000000000003},"min":{"1min":0.40500000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54400000000000004}]},{"osd":1,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.53300000000000003},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.29799999999999999},"max":{"1min":0.70999999999999996,"5min":0.76500000000000001,"15min":0.90000000000000002},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56000000000000005,"15min":0.52700000000000002},"min":{"1min":0.315,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70799999999999996,"5min":0.77400000000000002,"15min":0.78900000000000003},"last":0.47999999999999998}]},{"osd":2,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.59399999999999997},"min":{"1min":0.42499999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.74099999999999999,"5min":0.84899999999999998,"15min":1.0089999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58299999999999996,"15min":0.623},"min":{"1min":0.45600000000000002,"5min":0.38200000000000001,"15min":0.35099999999999998},"max":{"1min":0.68799999999999994,"5min":0.78100000000000003,"15min":2.4020000000000001},"last":0.46600000000000003}]},{"osd":3,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57999999999999996,"15min":0.55800000000000005},"min":{"1min":0.45400000000000001,"5min":0.35199999999999998,"15min":0.33600000000000002},"max":{"1min":0.752,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57299999999999995,"15min":0.64500000000000002},"min":{"1min":0.44600000000000001,"5min":0.36599999999999999,"15min":0.35799999999999998},"max":{"1min":0.69699999999999995,"5min":0.81999999999999995,"15min":2.702},"last":0.58799999999999997}]},{"osd":4,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46500000000000002,"15min":0.56499999999999995},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.27300000000000002},"max":{"1min":0.76100000000000001,"5min":0.76800000000000002,"15min":3.0739999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.29899999999999999},"max":{"1min":0.65700000000000003,"5min":0.748,"15min":0.93000000000000005},"last":0.67800000000000005}]},{"osd":6,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48099999999999998,"15min":0.54000000000000004},"min":{"1min":0.38800000000000001,"5min":0.33500000000000002,"15min":0.28699999999999998},"max":{"1min":0.60799999999999998,"5min":0.71999999999999997,"15min":1.1819999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.51300000000000001,"15min":0.65200000000000002},"min":{"1min":0.35399999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":3.7679999999999998},"last":0.57399999999999995}]},{"osd":7,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.57399999999999995},"min":{"1min":0.377,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.65400000000000003,"5min":0.79300000000000004,"15min":0.83699999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55400000000000005,"15min":0.627},"min":{"1min":0.40999999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.748,"5min":0.78000000000000003,"15min":1.2230000000000001},"last":0.63100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:06:28.696 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-11-28T12:58:49.183429+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=12, tm_min=58, tm_sec=49, tm_wday=1, tm_yday=332, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=13, tm_min=6, tm_sec=23, tm_wday=1, tm_yday=332, tm_isdst=0) 2023-11-28T13:06:28.696 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-11-28T12:58:49.183429+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=12, tm_min=58, tm_sec=49, tm_wday=1, tm_yday=332, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=13, tm_min=6, tm_sec=23, tm_wday=1, tm_yday=332, tm_isdst=0) 2023-11-28T13:06:28.696 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-11-28T12:58:47.164859+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=12, tm_min=58, tm_sec=47, tm_wday=1, tm_yday=332, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=13, tm_min=6, tm_sec=23, tm_wday=1, tm_yday=332, tm_isdst=0) 2023-11-28T13:06:28.696 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-11-28T12:58:49.183429+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=12, tm_min=58, tm_sec=49, tm_wday=1, tm_yday=332, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=13, tm_min=6, tm_sec=23, tm_wday=1, tm_yday=332, tm_isdst=0) 2023-11-28T13:06:28.696 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-11-28T12:58:49.183429+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=12, tm_min=58, tm_sec=49, tm_wday=1, tm_yday=332, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=13, tm_min=6, tm_sec=23, tm_wday=1, tm_yday=332, tm_isdst=0) 2023-11-28T13:06:28.697 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-11-28T12:58:49.183429+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=12, tm_min=58, tm_sec=49, tm_wday=1, tm_yday=332, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=13, tm_min=6, tm_sec=23, tm_wday=1, tm_yday=332, tm_isdst=0) 2023-11-28T13:06:28.697 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-11-28T12:58:49.183429+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=12, tm_min=58, tm_sec=49, tm_wday=1, tm_yday=332, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=28, tm_hour=13, tm_min=6, tm_sec=23, tm_wday=1, tm_yday=332, tm_isdst=0) 2023-11-28T13:06:28.697 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-11-28T13:06:29.674 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:06:29.677 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:06:48.698 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T13:06:48.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.831+0000 7f2c2ea0a700 1 Processor -- start 2023-11-28T13:06:48.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.832+0000 7f2c2ea0a700 1 -- start start 2023-11-28T13:06:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.832+0000 7f2c2ea0a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c280cd460 0x7f2c2806c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.832+0000 7f2c2ea0a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c280b88e0 con 0x7f2c280d0d60 2023-11-28T13:06:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.832+0000 7f2c2ea0a700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c280cd8f0 con 0x7f2c280b8510 2023-11-28T13:06:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.832+0000 7f2c2ea0a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c280cdbd0 con 0x7f2c280cd460 2023-11-28T13:06:48.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.832+0000 7f2c2cfa7700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f2c280d0d60 0x7f2c280d11e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39842/0 (socket says 172.21.15.111:39842) 2023-11-28T13:06:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.832+0000 7f2c2cfa7700 1 -- 172.21.15.111:0/4276116076 learned_addr learned my addr 172.21.15.111:0/4276116076 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146860981 0 0) 0x7f2c280b88e0 con 0x7f2c280d0d60 2023-11-28T13:06:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c140035b0 con 0x7f2c280d0d60 2023-11-28T13:06:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c277fe700 1 --2- 172.21.15.111:0/4276116076 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c280cd460 0x7f2c2806c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2127187015 0 0) 0x7f2c140035b0 con 0x7f2c280d0d60 2023-11-28T13:06:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c280cd460 msgr2=0x7f2c2806c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 --2- 172.21.15.111:0/4276116076 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c280cd460 0x7f2c2806c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:48.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 >> v1:172.21.15.149:6789/0 conn(0x7f2c280b8510 legacy=0x7f2c2806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c280cfd10 con 0x7f2c280d0d60 2023-11-28T13:06:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7f2c200019f0 con 0x7f2c280d0d60 2023-11-28T13:06:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 >> v1:172.21.15.111:6789/0 conn(0x7f2c280d0d60 legacy=0x7f2c280d11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 --2- 172.21.15.111:0/4276116076 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c14003a30 0x7f2c14003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 --2- 172.21.15.111:0/4276116076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c14004850 0x7f2c14004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 --2- 172.21.15.111:0/4276116076 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c280cd460 0x7f2c14005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c280cdbd0 con 0x7f2c14004850 2023-11-28T13:06:48.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1400bb70 con 0x7f2c14003a30 2023-11-28T13:06:48.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1400bd30 con 0x7f2c280cd460 2023-11-28T13:06:48.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c277fe700 1 --2- 172.21.15.111:0/4276116076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c14004850 0x7f2c14004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:48.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c27fff700 1 --2- 172.21.15.111:0/4276116076 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c14003a30 0x7f2c14003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:48.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c2cfa7700 1 --2- 172.21.15.111:0/4276116076 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c280cd460 0x7f2c14005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:48.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c277fe700 1 -- 172.21.15.111:0/4276116076 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c280cd460 msgr2=0x7f2c14005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:48.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c277fe700 1 --2- 172.21.15.111:0/4276116076 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c280cd460 0x7f2c14005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:48.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c277fe700 1 -- 172.21.15.111:0/4276116076 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c14003a30 msgr2=0x7f2c14003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:48.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.833+0000 7f2c277fe700 1 --2- 172.21.15.111:0/4276116076 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c14003a30 0x7f2c14003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:48.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.834+0000 7f2c277fe700 1 -- 172.21.15.111:0/4276116076 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c140035b0 con 0x7f2c14004850 2023-11-28T13:06:48.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.834+0000 7f2c2cfa7700 1 --2- 172.21.15.111:0/4276116076 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c280cd460 0x7f2c14005240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T13:06:48.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.834+0000 7f2c277fe700 1 --2- 172.21.15.111:0/4276116076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c14004850 0x7f2c14004cc0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009890 tx=0x7f2c14005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb9a19b755eacf6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:48.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.834+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1000c070 con 0x7f2c14004850 2023-11-28T13:06:48.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.834+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c10013610 con 0x7f2c14004850 2023-11-28T13:06:48.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.834+0000 7f2c26ffd700 1 -- 172.21.15.111:0/4276116076 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100139a0 con 0x7f2c14004850 2023-11-28T13:06:48.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.834+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/4276116076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c14004850 msgr2=0x7f2c14004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.834+0000 7f2c2ea0a700 1 --2- 172.21.15.111:0/4276116076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c14004850 0x7f2c14004cc0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009890 tx=0x7f2c14005880 comp rx=0 tx=0).stop 2023-11-28T13:06:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.834+0000 7f2c2cfa7700 1 -- 172.21.15.111:0/4276116076 reap_dead start 2023-11-28T13:06:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.834+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/4276116076 shutdown_connections 2023-11-28T13:06:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.834+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/4276116076 >> 172.21.15.111:0/4276116076 conn(0x7f2c280719a0 msgr2=0x7f2c28071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.834+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/4276116076 shutdown_connections 2023-11-28T13:06:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.834+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/4276116076 wait complete. 2023-11-28T13:06:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.836+0000 7f2c2ea0a700 1 Processor -- start 2023-11-28T13:06:48.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.836+0000 7f2c2ea0a700 1 -- start start 2023-11-28T13:06:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2ea0a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c14004850 0x7f2c2814d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2ea0a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c28154580 0x7f2c280ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2ea0a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c280cd050 0x7f2c281f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2ea0a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2815ac80 con 0x7f2c280cd050 2023-11-28T13:06:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2ea0a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2815a070 con 0x7f2c14004850 2023-11-28T13:06:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2ea0a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2815a350 con 0x7f2c28154580 2023-11-28T13:06:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2cfa7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c280cd050 0x7f2c281f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:48.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2cfa7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c280cd050 0x7f2c281f22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39076/0 (socket says 172.21.15.111:39076) 2023-11-28T13:06:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2cfa7700 1 -- 172.21.15.111:0/2729620497 learned_addr learned my addr 172.21.15.111:0/2729620497 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c27fff700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c14004850 0x7f2c2814d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2cfa7700 1 -- 172.21.15.111:0/2729620497 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c28154580 msgr2=0x7f2c280ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c277fe700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c28154580 0x7f2c280ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2cfa7700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c28154580 0x7f2c280ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2cfa7700 1 -- 172.21.15.111:0/2729620497 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c14004850 msgr2=0x7f2c2814d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:48.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2cfa7700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c14004850 0x7f2c2814d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:48.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2cfa7700 1 -- 172.21.15.111:0/2729620497 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c280d29c0 con 0x7f2c280cd050 2023-11-28T13:06:48.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c27fff700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c14004850 0x7f2c2814d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:48.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2cfa7700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c280cd050 0x7f2c281f22e0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f2c20002010 tx=0x7f2c20007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:48.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c24ff9700 1 -- 172.21.15.111:0/2729620497 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2000ed10 con 0x7f2c280cd050 2023-11-28T13:06:48.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c24ff9700 1 -- 172.21.15.111:0/2729620497 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2000ee70 con 0x7f2c280cd050 2023-11-28T13:06:48.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/2729620497 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c280cfff0 con 0x7f2c280cd050 2023-11-28T13:06:48.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c24ff9700 1 -- 172.21.15.111:0/2729620497 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c20013620 con 0x7f2c280cd050 2023-11-28T13:06:48.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.837+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/2729620497 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c280d2b50 con 0x7f2c280cd050 2023-11-28T13:06:48.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.838+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/2729620497 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c28067880 con 0x7f2c280cd050 2023-11-28T13:06:48.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.839+0000 7f2c24ff9700 1 -- 172.21.15.111:0/2729620497 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2c20013780 con 0x7f2c280cd050 2023-11-28T13:06:48.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.839+0000 7f2c24ff9700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c080682f0 0x7f2c0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:48.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.839+0000 7f2c24ff9700 1 -- 172.21.15.111:0/2729620497 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f2c20086270 con 0x7f2c280cd050 2023-11-28T13:06:48.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.839+0000 7f2c27fff700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c080682f0 0x7f2c0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:48.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.840+0000 7f2c27fff700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c080682f0 0x7f2c0806a7a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2c18007fd0 tx=0x7f2c18009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:48.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:48.843+0000 7f2c24ff9700 1 -- 172.21.15.111:0/2729620497 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2c20052c90 con 0x7f2c280cd050 2023-11-28T13:06:49.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.021+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/2729620497 --> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c2815adc0 con 0x7f2c080682f0 2023-11-28T13:06:49.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.024+0000 7f2c24ff9700 1 -- 172.21.15.111:0/2729620497 <== mgr.4110 v2:172.21.15.111:6832/103256 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+73392 (secure 0 0 0) 0x7f2c2815adc0 con 0x7f2c080682f0 2023-11-28T13:06:49.026 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:49.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.026+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/2729620497 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c080682f0 msgr2=0x7f2c0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.026+0000 7f2c2ea0a700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c080682f0 0x7f2c0806a7a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2c18007fd0 tx=0x7f2c18009040 comp rx=0 tx=0).stop 2023-11-28T13:06:49.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.026+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/2729620497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c280cd050 msgr2=0x7f2c281f22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.026+0000 7f2c2ea0a700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c280cd050 0x7f2c281f22e0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f2c20002010 tx=0x7f2c20007c90 comp rx=0 tx=0).stop 2023-11-28T13:06:49.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.026+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/2729620497 shutdown_connections 2023-11-28T13:06:49.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.026+0000 7f2c2ea0a700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c280cd050 0x7f2c281f22e0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.026+0000 7f2c2ea0a700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7f2c080682f0 0x7f2c0806a7a0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.026+0000 7f2c2ea0a700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f2c14004850 0x7f2c2814d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.026+0000 7f2c2ea0a700 1 --2- 172.21.15.111:0/2729620497 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7f2c28154580 0x7f2c280ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.026+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/2729620497 >> 172.21.15.111:0/2729620497 conn(0x7f2c280719a0 msgr2=0x7f2c280b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:49.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.026+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/2729620497 shutdown_connections 2023-11-28T13:06:49.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.026+0000 7f2c2ea0a700 1 -- 172.21.15.111:0/2729620497 wait complete. 2023-11-28T13:06:49.029 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-11-28T13:06:49.039 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2023-11-28T13:06:48.449909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":82472,"kb_used_data":8040,"kb_used_omap":0,"kb_used_meta":74368,"kb_avail":754892248,"statfs":{"total":773094113280,"available":773009661952,"internally_reserved":0,"allocated":8232960,"data_stored":4491880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":76152832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002635"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-11-28T13:06:25.847158+0000","last_change":"2023-11-28T13:06:25.847137+0000","last_active":"2023-11-28T13:06:25.847158+0000","last_peered":"2023-11-28T13:06:25.847158+0000","last_clean":"2023-11-28T13:06:25.847158+0000","last_became_active":"2023-11-28T12:58:50.572102+0000","last_became_peered":"2023-11-28T12:58:50.572102+0000","last_unstale":"2023-11-28T13:06:25.847158+0000","last_undegraded":"2023-11-28T13:06:25.847158+0000","last_fullsized":"2023-11-28T13:06:25.847158+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:06:25.847056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:06:25.847056+0000","last_clean_scrub_stamp":"2023-11-28T13:06:25.847056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:26:35.431945+0000","scrub_duration":0.0034698929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-11-28T13:06:28.867263+0000","last_change":"2023-11-28T13:06:28.867222+0000","last_active":"2023-11-28T13:06:28.867263+0000","last_peered":"2023-11-28T13:06:28.867263+0000","last_clean":"2023-11-28T13:06:28.867263+0000","last_became_active":"2023-11-28T12:58:50.206549+0000","last_became_peered":"2023-11-28T12:58:50.206549+0000","last_unstale":"2023-11-28T13:06:28.867263+0000","last_undegraded":"2023-11-28T13:06:28.867263+0000","last_fullsized":"2023-11-28T13:06:28.867263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:06:28.867121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:06:28.867121+0000","last_clean_scrub_stamp":"2023-11-28T13:06:28.867121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:14:03.767376+0000","scrub_duration":0.0038766109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-11-28T13:06:25.951630+0000","last_change":"2023-11-28T13:06:25.951605+0000","last_active":"2023-11-28T13:06:25.951630+0000","last_peered":"2023-11-28T13:06:25.951630+0000","last_clean":"2023-11-28T13:06:25.951630+0000","last_became_active":"2023-11-28T12:58:50.647801+0000","last_became_peered":"2023-11-28T12:58:50.647801+0000","last_unstale":"2023-11-28T13:06:25.951630+0000","last_undegraded":"2023-11-28T13:06:25.951630+0000","last_fullsized":"2023-11-28T13:06:25.951630+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:06:25.951514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:06:25.951514+0000","last_clean_scrub_stamp":"2023-11-28T13:06:25.951514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:53:11.317252+0000","scrub_duration":0.0034822400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-11-28T13:06:31.886101+0000","last_change":"2023-11-28T13:06:31.886068+0000","last_active":"2023-11-28T13:06:31.886101+0000","last_peered":"2023-11-28T13:06:31.886101+0000","last_clean":"2023-11-28T13:06:31.886101+0000","last_became_active":"2023-11-28T12:58:50.204147+0000","last_became_peered":"2023-11-28T12:58:50.204147+0000","last_unstale":"2023-11-28T13:06:31.886101+0000","last_undegraded":"2023-11-28T13:06:31.886101+0000","last_fullsized":"2023-11-28T13:06:31.886101+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:06:31.885973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:06:31.885973+0000","last_clean_scrub_stamp":"2023-11-28T13:06:31.885973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:43:59.168518+0000","scrub_duration":0.0039370960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"72'119","reported_seq":228,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-11-28T13:06:29.356460+0000","last_change":"2023-11-28T13:06:29.356432+0000","last_active":"2023-11-28T13:06:29.356460+0000","last_peered":"2023-11-28T13:06:29.356460+0000","last_clean":"2023-11-28T13:06:29.356460+0000","last_became_active":"2023-11-28T12:58:48.489211+0000","last_became_peered":"2023-11-28T12:58:48.489211+0000","last_unstale":"2023-11-28T13:06:29.356460+0000","last_undegraded":"2023-11-28T13:06:29.356460+0000","last_fullsized":"2023-11-28T13:06:29.356460+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'119","last_scrub_stamp":"2023-11-28T13:06:29.356333+0000","last_deep_scrub":"72'119","last_deep_scrub_stamp":"2023-11-28T13:06:29.356333+0000","last_clean_scrub_stamp":"2023-11-28T13:06:29.356333+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T16:57:22.080216+0000","scrub_duration":0.0091074950000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-11-28T13:06:31.285691+0000","last_change":"2023-11-28T13:06:31.285658+0000","last_active":"2023-11-28T13:06:31.285691+0000","last_peered":"2023-11-28T13:06:31.285691+0000","last_clean":"2023-11-28T13:06:31.285691+0000","last_became_active":"2023-11-28T12:58:50.210640+0000","last_became_peered":"2023-11-28T12:58:50.210640+0000","last_unstale":"2023-11-28T13:06:31.285691+0000","last_undegraded":"2023-11-28T13:06:31.285691+0000","last_fullsized":"2023-11-28T13:06:31.285691+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:06:31.285556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:06:31.285556+0000","last_clean_scrub_stamp":"2023-11-28T13:06:31.285556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:35:09.927512+0000","scrub_duration":0.0042986480000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-11-28T13:06:30.330270+0000","last_change":"2023-11-28T13:06:30.330244+0000","last_active":"2023-11-28T13:06:30.330270+0000","last_peered":"2023-11-28T13:06:30.330270+0000","last_clean":"2023-11-28T13:06:30.330270+0000","last_became_active":"2023-11-28T12:58:50.572210+0000","last_became_peered":"2023-11-28T12:58:50.572210+0000","last_unstale":"2023-11-28T13:06:30.330270+0000","last_undegraded":"2023-11-28T13:06:30.330270+0000","last_fullsized":"2023-11-28T13:06:30.330270+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:06:30.330136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:06:30.330136+0000","last_clean_scrub_stamp":"2023-11-28T13:06:30.330136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:46:21.406714+0000","scrub_duration":0.0038535330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-11-28T13:06:31.780780+0000","last_change":"2023-11-28T13:06:31.780748+0000","last_active":"2023-11-28T13:06:31.780780+0000","last_peered":"2023-11-28T13:06:31.780780+0000","last_clean":"2023-11-28T13:06:31.780780+0000","last_became_active":"2023-11-28T12:58:50.206292+0000","last_became_peered":"2023-11-28T12:58:50.206292+0000","last_unstale":"2023-11-28T13:06:31.780780+0000","last_undegraded":"2023-11-28T13:06:31.780780+0000","last_fullsized":"2023-11-28T13:06:31.780780+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:06:31.780628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:06:31.780628+0000","last_clean_scrub_stamp":"2023-11-28T13:06:31.780628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T13:12:33.163922+0000","scrub_duration":0.0044410450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":82,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-11-28T13:06:28.273006+0000","last_change":"2023-11-28T13:06:28.272980+0000","last_active":"2023-11-28T13:06:28.273006+0000","last_peered":"2023-11-28T13:06:28.273006+0000","last_clean":"2023-11-28T13:06:28.273006+0000","last_became_active":"2023-11-28T12:58:50.201542+0000","last_became_peered":"2023-11-28T12:58:50.201542+0000","last_unstale":"2023-11-28T13:06:28.273006+0000","last_undegraded":"2023-11-28T13:06:28.273006+0000","last_fullsized":"2023-11-28T13:06:28.273006+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T13:06:28.272843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T13:06:28.272843+0000","last_clean_scrub_stamp":"2023-11-28T13:06:28.272843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:05:53.304710+0000","scrub_duration":0.0036448280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574947,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11176,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":9728,"kb_avail":94360664,"statfs":{"total":96636764160,"available":96625319936,"internally_reserved":0,"allocated":1474560,"data_stored":1004261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9961472},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.61199999999999999,"15min":0.64700000000000002},"min":{"1min":0.51200000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.79800000000000004,"5min":1.194,"15min":3.3279999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.58999999999999997,"15min":0.64800000000000002},"min":{"1min":0.46300000000000002,"5min":0.372,"15min":0.27900000000000003},"max":{"1min":0.77000000000000002,"5min":0.81799999999999995,"15min":3.2919999999999998},"last":0.58799999999999997}]},{"osd":1,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.40300000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.52400000000000002},"min":{"1min":0.436,"5min":0.34999999999999998,"15min":0.30499999999999999},"max":{"1min":0.76000000000000001,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.56499999999999995}]},{"osd":2,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59299999999999997,"15min":0.56200000000000006},"min":{"1min":0.44600000000000001,"5min":0.35699999999999998,"15min":0.28399999999999997},"max":{"1min":0.751,"5min":1.149,"15min":1.149},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57499999999999996,"15min":0.55200000000000005},"min":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.35699999999999998},"max":{"1min":0.72399999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.51300000000000001}]},{"osd":3,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60299999999999998,"15min":0.56000000000000005},"min":{"1min":0.52000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82899999999999996,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58799999999999997,"15min":0.57299999999999995},"min":{"1min":0.433,"5min":0.40699999999999997,"15min":0.38600000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.63100000000000001}]},{"osd":4,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47599999999999998,"15min":0.50800000000000001},"min":{"1min":0.35099999999999998,"5min":0.33300000000000002,"15min":0.316},"max":{"1min":0.78400000000000003,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.50600000000000001,"15min":0.52600000000000002},"min":{"1min":0.373,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.73899999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.55300000000000005}]},{"osd":5,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47799999999999998,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71399999999999997,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.443},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53500000000000003,"15min":0.56000000000000005},"min":{"1min":0.36899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.48599999999999999}]},{"osd":6,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52400000000000002,"15min":0.56000000000000005},"min":{"1min":0.38200000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77800000000000002,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58899999999999997,"15min":0.60799999999999998},"min":{"1min":0.434,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79900000000000004,"5min":1.274,"15min":1.274},"last":0.57699999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574949,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10020,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361820,"statfs":{"total":96636764160,"available":96626503680,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.437,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.89600000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.52400000000000002},"min":{"1min":0.33900000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72799999999999998,"5min":0.755,"15min":0.80400000000000005},"last":0.59099999999999997}]},{"osd":1,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56699999999999995,"15min":0.53500000000000003},"min":{"1min":0.46100000000000002,"5min":0.32400000000000001,"15min":0.309},"max":{"1min":0.752,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56699999999999995,"15min":0.56299999999999994},"min":{"1min":0.40300000000000002,"5min":0.33000000000000002,"15min":0.29599999999999999},"max":{"1min":0.76700000000000002,"5min":0.86599999999999999,"15min":1.016},"last":0.61899999999999999}]},{"osd":2,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.53600000000000003},"min":{"1min":0.39900000000000002,"5min":0.34300000000000003,"15min":0.34100000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.91700000000000004},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57599999999999996,"15min":0.58099999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.75700000000000001,"5min":0.92400000000000004,"15min":0.96299999999999997},"last":0.73599999999999999}]},{"osd":3,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56200000000000006,"15min":0.58699999999999997},"min":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.371},"max":{"1min":0.71399999999999997,"5min":0.80700000000000005,"15min":1.5920000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59399999999999997,"15min":0.58399999999999996},"min":{"1min":0.504,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76400000000000001,"5min":0.84999999999999998,"15min":0.94099999999999995},"last":0.749}]},{"osd":4,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46100000000000002,"15min":0.51200000000000001},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74199999999999999,"5min":0.84199999999999997,"15min":1},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48199999999999998,"15min":0.52200000000000002},"min":{"1min":0.32500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72699999999999998,"5min":0.753,"15min":0.89200000000000002},"last":0.77700000000000002}]},{"osd":5,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.54600000000000004},"min":{"1min":0.36399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.871,"5min":0.871,"15min":1.052},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.59499999999999997},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.81299999999999994,"5min":0.88900000000000001,"15min":1.258},"last":0.76800000000000002}]},{"osd":7,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.52100000000000002,"15min":0.56200000000000006},"min":{"1min":0.32100000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.79200000000000004,"5min":0.80100000000000005,"15min":0.97599999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56100000000000005,"15min":0.60199999999999998},"min":{"1min":0.39400000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.89500000000000002,"5min":0.90400000000000003,"15min":1.1060000000000001},"last":0.83199999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10020,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9152,"kb_avail":94361820,"statfs":{"total":96636764160,"available":96626503680,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9371648},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35099999999999998,"15min":0.33500000000000002},"min":{"1min":0.23799999999999999,"5min":0.224,"15min":0.219},"max":{"1min":0.496,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.36499999999999999},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60299999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.376}]},{"osd":2,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.373,"15min":0.34699999999999998},"min":{"1min":0.29199999999999998,"5min":0.22,"15min":0.20599999999999999},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40799999999999997,"15min":0.38100000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.25},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.80100000000000005},"last":0.39700000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.434,"15min":0.40400000000000003},"min":{"1min":0.32800000000000001,"5min":0.27200000000000002,"15min":0.245},"max":{"1min":0.57399999999999995,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.41299999999999998,"15min":0.40000000000000002},"min":{"1min":0.27800000000000002,"5min":0.26000000000000001,"15min":0.251},"max":{"1min":0.58399999999999996,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.41399999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.60499999999999998,"15min":0.57999999999999996},"min":{"1min":0.40200000000000002,"5min":0.36199999999999999,"15min":0.34000000000000002},"max":{"1min":0.80000000000000004,"5min":0.91700000000000004,"15min":1.028},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.624,"15min":0.59599999999999997},"min":{"1min":0.51100000000000001,"5min":0.45600000000000002,"15min":0.34799999999999998},"max":{"1min":0.76100000000000001,"5min":0.98699999999999999,"15min":1.0660000000000001},"last":0.68100000000000005}]},{"osd":5,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.59999999999999998},"min":{"1min":0.47299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.78100000000000003,"5min":0.94999999999999996,"15min":1.042},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63500000000000001,"15min":0.58499999999999996},"min":{"1min":0.44700000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78300000000000003,"5min":1.05,"15min":1.05},"last":0.58599999999999997}]},{"osd":6,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.626,"15min":0.59599999999999997},"min":{"1min":0.438,"5min":0.40600000000000003,"15min":0.38900000000000001},"max":{"1min":0.80100000000000005,"5min":0.96199999999999997,"15min":1.006},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63400000000000001,"15min":0.60999999999999999},"min":{"1min":0.48699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.81899999999999995,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.751}]},{"osd":7,"last update":"Tue Nov 28 13:05:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.64200000000000002,"15min":0.61499999999999999},"min":{"1min":0.51600000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80200000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64000000000000001,"15min":0.59599999999999997},"min":{"1min":0.47799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80400000000000005,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.60799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574948,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11176,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":9728,"kb_avail":94360664,"statfs":{"total":96636764160,"available":96625319936,"internally_reserved":0,"allocated":1474560,"data_stored":1004261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9961472},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.33700000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51300000000000001,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.38200000000000001,"15min":0.35499999999999998},"min":{"1min":0.23000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.50700000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.44500000000000001}]},{"osd":2,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38200000000000001,"15min":0.35799999999999998},"min":{"1min":0.27300000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.48399999999999999,"5min":0.57099999999999995,"15min":0.59399999999999997},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41199999999999998,"15min":0.38100000000000001},"min":{"1min":0.28699999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50600000000000001,"5min":0.623,"15min":0.623},"last":0.47299999999999998}]},{"osd":3,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38900000000000001,"15min":0.377},"min":{"1min":0.247,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.68000000000000005},"last":0.496},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.441,"15min":0.40500000000000003},"min":{"1min":0.28799999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52200000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.52300000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.622,"15min":0.60699999999999998},"min":{"1min":0.51400000000000001,"5min":0.309,"15min":0.28199999999999997},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.626,"15min":0.58799999999999997},"min":{"1min":0.49299999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83299999999999996,"5min":0.88,"15min":0.88},"last":0.71699999999999997}]},{"osd":5,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.58299999999999996},"min":{"1min":0.42999999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79600000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64100000000000001,"15min":0.625},"min":{"1min":0.441,"5min":0.434,"15min":0.39600000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.66200000000000003}]},{"osd":6,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.64300000000000002,"15min":0.61599999999999999},"min":{"1min":0.51700000000000002,"5min":0.46999999999999997,"15min":0.41199999999999998},"max":{"1min":0.84799999999999998,"5min":0.89800000000000002,"15min":0.91900000000000004},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65300000000000002,"15min":0.627},"min":{"1min":0.50600000000000001,"5min":0.41499999999999998,"15min":0.33900000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.98599999999999999},"last":0.69499999999999995}]},{"osd":7,"last update":"Tue Nov 28 13:06:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.65900000000000003,"15min":0.63900000000000001},"min":{"1min":0.52900000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.0149999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65800000000000003,"15min":0.63700000000000001},"min":{"1min":0.45700000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.755}]}]},{"osd":2,"up_from":13,"seq":55834574948,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9892,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94361948,"statfs":{"total":96636764160,"available":96626634752,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.33000000000000002},"min":{"1min":0.255,"5min":0.22500000000000001,"15min":0.20999999999999999},"max":{"1min":0.48799999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.35199999999999998},"min":{"1min":0.22600000000000001,"5min":0.21199999999999999,"15min":0.184},"max":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.34699999999999998}]},{"osd":1,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.39800000000000002,"15min":0.38700000000000001},"min":{"1min":0.28799999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.51800000000000002,"5min":0.59499999999999997,"15min":0.67000000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.374},"min":{"1min":0.26300000000000001,"5min":0.25700000000000001,"15min":0.24199999999999999},"max":{"1min":0.56999999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.38300000000000001}]},{"osd":3,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39200000000000002,"15min":0.35599999999999998},"min":{"1min":0.27900000000000003,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":0.505,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.313},{"interface":"front","average":{"1min":0.437,"5min":0.443,"15min":0.42499999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.55700000000000005,"5min":0.70599999999999996,"15min":0.77500000000000002},"last":0.60099999999999998}]},{"osd":4,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64000000000000001,"15min":0.60599999999999998},"min":{"1min":0.46200000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85799999999999998,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.498},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63,"15min":0.57899999999999996},"min":{"1min":0.44900000000000001,"5min":0.42199999999999999,"15min":0.34300000000000003},"max":{"1min":0.86399999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.623}]},{"osd":5,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.60499999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.91900000000000004},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.63400000000000001,"15min":0.60799999999999998},"min":{"1min":0.52500000000000002,"5min":0.41699999999999998,"15min":0.379},"max":{"1min":0.877,"5min":0.877,"15min":1.0589999999999999},"last":0.53000000000000003}]},{"osd":6,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64800000000000002,"15min":0.59699999999999998},"min":{"1min":0.503,"5min":0.41999999999999998,"15min":0.36599999999999999},"max":{"1min":0.83899999999999997,"5min":0.89100000000000001,"15min":1.002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65300000000000002,"15min":0.63300000000000001},"min":{"1min":0.48399999999999999,"5min":0.436,"15min":0.40899999999999997},"max":{"1min":0.82799999999999996,"5min":0.96799999999999997,"15min":0.97799999999999998},"last":0.80800000000000005}]},{"osd":7,"last update":"Tue Nov 28 13:06:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65600000000000003,"15min":0.63600000000000001},"min":{"1min":0.438,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86199999999999999,"5min":0.93500000000000005,"15min":0.95099999999999996},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.65800000000000003,"15min":0.61399999999999999},"min":{"1min":0.53000000000000003,"5min":0.441,"15min":0.40100000000000002},"max":{"1min":0.86899999999999999,"5min":0.92400000000000004,"15min":1.0349999999999999},"last":0.57999999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574948,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10084,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9216,"kb_avail":94361756,"statfs":{"total":96636764160,"available":96626438144,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9437184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36499999999999999,"15min":0.33500000000000002},"min":{"1min":0.23999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38200000000000001,"15min":0.34499999999999997},"min":{"1min":0.24099999999999999,"5min":0.23200000000000001,"15min":0.20000000000000001},"max":{"1min":0.55200000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.375}]},{"osd":1,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.38600000000000001,"15min":0.35399999999999998},"min":{"1min":0.25700000000000001,"5min":0.22,"15min":0.21299999999999999},"max":{"1min":0.53000000000000003,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.315},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40500000000000003,"15min":0.38300000000000001},"min":{"1min":0.317,"5min":0.254,"15min":0.23799999999999999},"max":{"1min":0.54600000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.40200000000000002}]},{"osd":2,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.45000000000000001,"15min":0.47499999999999998},"min":{"1min":0.29299999999999998,"5min":0.252,"15min":0.251},"max":{"1min":0.58299999999999996,"5min":0.65100000000000002,"15min":2.6869999999999998},"last":0.436},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45700000000000002,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.30499999999999999,"15min":0.28199999999999997},"max":{"1min":0.60099999999999998,"5min":0.66800000000000004,"15min":2.7160000000000002},"last":0.46100000000000002}]},{"osd":4,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.63600000000000001,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81999999999999995,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.63,"15min":0.84599999999999997},"min":{"1min":0.442,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82899999999999996,"5min":0.93899999999999995,"15min":8.2889999999999997},"last":0.47399999999999998}]},{"osd":5,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.48599999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.63300000000000001,"15min":0.66000000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.72999999999999998,"5min":0.88200000000000001,"15min":2.165},"last":0.48599999999999999}]},{"osd":6,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.65200000000000002,"15min":0.63600000000000001},"min":{"1min":0.41999999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.88400000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65100000000000002,"15min":0.63500000000000001},"min":{"1min":0.53100000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.81299999999999994,"5min":0.95399999999999996,"15min":1.1419999999999999},"last":0.503}]},{"osd":7,"last update":"Tue Nov 28 13:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.66100000000000003,"15min":0.626},"min":{"1min":0.48399999999999999,"5min":0.46000000000000002,"15min":0.45600000000000002},"max":{"1min":0.82399999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66000000000000003,"15min":0.85199999999999998},"min":{"1min":0.45900000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85999999999999999,"5min":0.89600000000000002,"15min":7.0899999999999999},"last":0.51500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574946,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":10212,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9344,"kb_avail":94361628,"statfs":{"total":96636764160,"available":96626307072,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9568256},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.66200000000000003,"5min":0.78500000000000003,"15min":1.0049999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56000000000000005,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.66600000000000004,"5min":0.873,"15min":0.873},"last":0.49299999999999999}]},{"osd":1,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55300000000000005,"15min":0.52100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71399999999999997,"5min":0.76600000000000001,"15min":0.95699999999999996},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56999999999999995,"15min":0.56499999999999995},"min":{"1min":0.45500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69999999999999996,"5min":0.77300000000000002,"15min":1.1060000000000001},"last":0.47699999999999998}]},{"osd":2,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59799999999999998,"15min":0.56499999999999995},"min":{"1min":0.40000000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68999999999999995,"5min":0.77300000000000002,"15min":0.91800000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60999999999999999,"15min":0.60199999999999998},"min":{"1min":0.48999999999999999,"5min":0.41499999999999998,"15min":0.36399999999999999},"max":{"1min":0.67800000000000005,"5min":0.81100000000000005,"15min":1.069},"last":0.58999999999999997}]},{"osd":3,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.55200000000000005,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.70099999999999996,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59099999999999997,"15min":0.58599999999999997},"min":{"1min":0.46300000000000002,"5min":0.41099999999999998,"15min":0.38900000000000001},"max":{"1min":0.71099999999999997,"5min":0.81599999999999995,"15min":0.879},"last":0.44600000000000001}]},{"osd":5,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.47399999999999998,"15min":0.503},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.315},"max":{"1min":0.752,"5min":0.752,"15min":0.86099999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.51000000000000001},"min":{"1min":0.371,"5min":0.309,"15min":0.309},"max":{"1min":0.68200000000000005,"5min":0.73799999999999999,"15min":0.80400000000000005},"last":0.42099999999999999}]},{"osd":6,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.49099999999999999,"15min":0.51300000000000001},"min":{"1min":0.35399999999999998,"5min":0.32500000000000001,"15min":0.30499999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.84299999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53000000000000003,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.35199999999999998,"15min":0.34499999999999997},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.96399999999999997},"last":0.56399999999999995}]},{"osd":7,"last update":"Tue Nov 28 13:05:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73999999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57799999999999996,"15min":0.58799999999999997},"min":{"1min":0.40600000000000003,"5min":0.39400000000000002,"15min":0.36599999999999999},"max":{"1min":0.80400000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65500000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574948,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9892,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":9024,"kb_avail":94361948,"statfs":{"total":96636764160,"available":96626634752,"internally_reserved":0,"allocated":880640,"data_stored":413893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9240576},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.378,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72699999999999998,"5min":0.74399999999999999,"15min":1.819},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.53600000000000003},"min":{"1min":0.40500000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.69699999999999995}]},{"osd":1,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.53300000000000003},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.29799999999999999},"max":{"1min":0.70999999999999996,"5min":0.76500000000000001,"15min":0.90000000000000002},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56000000000000005,"15min":0.52700000000000002},"min":{"1min":0.315,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70799999999999996,"5min":0.77400000000000002,"15min":0.78900000000000003},"last":0.622}]},{"osd":2,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61299999999999999,"15min":0.59399999999999997},"min":{"1min":0.42499999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.74099999999999999,"5min":0.84899999999999998,"15min":1.0089999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58299999999999996,"15min":0.623},"min":{"1min":0.45600000000000002,"5min":0.38200000000000001,"15min":0.35099999999999998},"max":{"1min":0.68799999999999994,"5min":0.78100000000000003,"15min":2.4020000000000001},"last":0.64800000000000002}]},{"osd":3,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57999999999999996,"15min":0.55800000000000005},"min":{"1min":0.45400000000000001,"5min":0.35199999999999998,"15min":0.33600000000000002},"max":{"1min":0.752,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57299999999999995,"15min":0.64500000000000002},"min":{"1min":0.44600000000000001,"5min":0.36599999999999999,"15min":0.35799999999999998},"max":{"1min":0.69699999999999995,"5min":0.81999999999999995,"15min":2.702},"last":0.55700000000000005}]},{"osd":4,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46500000000000002,"15min":0.56499999999999995},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.27300000000000002},"max":{"1min":0.76100000000000001,"5min":0.76800000000000002,"15min":3.0739999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.54200000000000004},"min":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.29899999999999999},"max":{"1min":0.65700000000000003,"5min":0.748,"15min":0.93000000000000005},"last":0.48699999999999999}]},{"osd":6,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48099999999999998,"15min":0.54000000000000004},"min":{"1min":0.38800000000000001,"5min":0.33500000000000002,"15min":0.28699999999999998},"max":{"1min":0.60799999999999998,"5min":0.71999999999999997,"15min":1.1819999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.502,"5min":0.51300000000000001,"15min":0.65200000000000002},"min":{"1min":0.35399999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":3.7679999999999998},"last":0.53300000000000003}]},{"osd":7,"last update":"Tue Nov 28 13:06:06 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.57399999999999995},"min":{"1min":0.377,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.65400000000000003,"5min":0.79300000000000004,"15min":0.83699999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55400000000000005,"15min":0.627},"min":{"1min":0.40999999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.748,"5min":0.78000000000000003,"15min":1.2230000000000001},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T13:06:49.042 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-28T13:06:49.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.160+0000 7fce39d5a700 1 Processor -- start 2023-11-28T13:06:49.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.160+0000 7fce39d5a700 1 -- start start 2023-11-28T13:06:49.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.161+0000 7fce39d5a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce34159310 0x7fce340ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.161+0000 7fce39d5a700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fce341596e0 0x7fce340ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.161+0000 7fce39d5a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fce3415cfd0 0x7fce3415d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.161+0000 7fce39d5a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce3415d940 con 0x7fce3415cfd0 2023-11-28T13:06:49.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.161+0000 7fce39d5a700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce3415de70 con 0x7fce341596e0 2023-11-28T13:06:49.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.161+0000 7fce39d5a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce340ced10 con 0x7fce34159310 2023-11-28T13:06:49.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.161+0000 7fce33fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fce3415cfd0 0x7fce3415d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.161+0000 7fce33fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fce3415cfd0 0x7fce3415d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39084/0 (socket says 172.21.15.111:39084) 2023-11-28T13:06:49.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.161+0000 7fce33fff700 1 -- 172.21.15.111:0/632218507 learned_addr learned my addr 172.21.15.111:0/632218507 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:49.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce32ffd700 1 --2- 172.21.15.111:0/632218507 >> v2:172.21.15.149:3300/0 conn(0x7fce341596e0 0x7fce340ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce337fe700 1 --2- 172.21.15.111:0/632218507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce34159310 0x7fce340ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce33fff700 1 -- 172.21.15.111:0/632218507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce34159310 msgr2=0x7fce340ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:49.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce33fff700 1 --2- 172.21.15.111:0/632218507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce34159310 0x7fce340ce1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce33fff700 1 -- 172.21.15.111:0/632218507 >> v2:172.21.15.149:3300/0 conn(0x7fce341596e0 msgr2=0x7fce340ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:49.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce33fff700 1 --2- 172.21.15.111:0/632218507 >> v2:172.21.15.149:3300/0 conn(0x7fce341596e0 0x7fce340ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce33fff700 1 -- 172.21.15.111:0/632218507 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce340ceff0 con 0x7fce3415cfd0 2023-11-28T13:06:49.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce32ffd700 1 --2- 172.21.15.111:0/632218507 >> v2:172.21.15.149:3300/0 conn(0x7fce341596e0 0x7fce340ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:49.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce33fff700 1 --2- 172.21.15.111:0/632218507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce3415cfd0 0x7fce3415d400 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fce340b2710 tx=0x7fce2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594876aee3af3dc4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:49.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce327fc700 1 -- 172.21.15.111:0/632218507 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2c00e070 con 0x7fce3415cfd0 2023-11-28T13:06:49.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce327fc700 1 -- 172.21.15.111:0/632218507 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2c00b9c0 con 0x7fce3415cfd0 2023-11-28T13:06:49.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce327fc700 1 -- 172.21.15.111:0/632218507 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2c00bd70 con 0x7fce3415cfd0 2023-11-28T13:06:49.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce39d5a700 1 -- 172.21.15.111:0/632218507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce3415cfd0 msgr2=0x7fce3415d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce39d5a700 1 --2- 172.21.15.111:0/632218507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce3415cfd0 0x7fce3415d400 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fce340b2710 tx=0x7fce2c00b330 comp rx=0 tx=0).stop 2023-11-28T13:06:49.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.162+0000 7fce39d5a700 1 -- 172.21.15.111:0/632218507 shutdown_connections 2023-11-28T13:06:49.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.163+0000 7fce39d5a700 1 --2- 172.21.15.111:0/632218507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce3415cfd0 0x7fce3415d400 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.163+0000 7fce39d5a700 1 --2- 172.21.15.111:0/632218507 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce34159310 0x7fce340ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.163+0000 7fce39d5a700 1 --2- 172.21.15.111:0/632218507 >> v2:172.21.15.149:3300/0 conn(0x7fce341596e0 0x7fce340ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.163+0000 7fce39d5a700 1 -- 172.21.15.111:0/632218507 >> 172.21.15.111:0/632218507 conn(0x7fce3406db10 msgr2=0x7fce3406df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:49.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.163+0000 7fce39d5a700 1 -- 172.21.15.111:0/632218507 shutdown_connections 2023-11-28T13:06:49.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.163+0000 7fce39d5a700 1 -- 172.21.15.111:0/632218507 wait complete. 2023-11-28T13:06:49.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.164+0000 7fce39d5a700 1 Processor -- start 2023-11-28T13:06:49.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.164+0000 7fce39d5a700 1 -- start start 2023-11-28T13:06:49.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce39d5a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce34159310 0x7fce3414e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce39d5a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce341596e0 0x7fce341f2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce39d5a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce3415cfd0 0x7fce340b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce39d5a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce341f2fe0 con 0x7fce3415cfd0 2023-11-28T13:06:49.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce39d5a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce341f32c0 con 0x7fce34159310 2023-11-28T13:06:49.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce39d5a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce340c7d10 con 0x7fce341596e0 2023-11-28T13:06:49.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce33fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce3415cfd0 0x7fce340b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce33fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce3415cfd0 0x7fce340b0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39090/0 (socket says 172.21.15.111:39090) 2023-11-28T13:06:49.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce33fff700 1 -- 172.21.15.111:0/313029971 learned_addr learned my addr 172.21.15.111:0/313029971 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:49.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce337fe700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce34159310 0x7fce3414e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce32ffd700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce341596e0 0x7fce341f2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce33fff700 1 -- 172.21.15.111:0/313029971 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce341596e0 msgr2=0x7fce341f2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce33fff700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce341596e0 0x7fce341f2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce33fff700 1 -- 172.21.15.111:0/313029971 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce34159310 msgr2=0x7fce3414e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce33fff700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce34159310 0x7fce3414e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce33fff700 1 -- 172.21.15.111:0/313029971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c009b30 con 0x7fce3415cfd0 2023-11-28T13:06:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce32ffd700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce341596e0 0x7fce341f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T13:06:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce33fff700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce3415cfd0 0x7fce340b0fe0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fce2c00b370 tx=0x7fce2c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce1bfff700 1 -- 172.21.15.111:0/313029971 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2c014ad0 con 0x7fce3415cfd0 2023-11-28T13:06:49.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce1bfff700 1 -- 172.21.15.111:0/313029971 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2c014c30 con 0x7fce3415cfd0 2023-11-28T13:06:49.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce39d5a700 1 -- 172.21.15.111:0/313029971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce340cb0f0 con 0x7fce3415cfd0 2023-11-28T13:06:49.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce1bfff700 1 -- 172.21.15.111:0/313029971 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2c004040 con 0x7fce3415cfd0 2023-11-28T13:06:49.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.165+0000 7fce39d5a700 1 -- 172.21.15.111:0/313029971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce340cf2d0 con 0x7fce3415cfd0 2023-11-28T13:06:49.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.166+0000 7fce39d5a700 1 -- 172.21.15.111:0/313029971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce340cb3d0 con 0x7fce3415cfd0 2023-11-28T13:06:49.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.166+0000 7fce1bfff700 1 -- 172.21.15.111:0/313029971 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fce2c0041a0 con 0x7fce3415cfd0 2023-11-28T13:06:49.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.167+0000 7fce1bfff700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fce200682f0 0x7fce2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.167+0000 7fce1bfff700 1 -- 172.21.15.111:0/313029971 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fce2c089160 con 0x7fce3415cfd0 2023-11-28T13:06:49.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.167+0000 7fce337fe700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fce200682f0 0x7fce2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.167+0000 7fce337fe700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fce200682f0 0x7fce2006a7a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fce2800a950 tx=0x7fce28008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.171+0000 7fce1bfff700 1 -- 172.21.15.111:0/313029971 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fce2c055b80 con 0x7fce3415cfd0 2023-11-28T13:06:49.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.351+0000 7fce39d5a700 1 -- 172.21.15.111:0/313029971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fce34067170 con 0x7fce3415cfd0 2023-11-28T13:06:49.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.355+0000 7fce1bfff700 1 -- 172.21.15.111:0/313029971 <== mon.0 v2:172.21.15.111:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fce2c055b80 con 0x7fce3415cfd0 2023-11-28T13:06:49.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.356+0000 7fce1bfff700 1 -- 172.21.15.111:0/313029971 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v16)=0 v16) v1 ==== 123+0+0 (secure 0 0 0) 0x7fce2c024ed0 con 0x7fce3415cfd0 2023-11-28T13:06:49.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.358+0000 7fce39d5a700 1 -- 172.21.15.111:0/313029971 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fce200682f0 msgr2=0x7fce2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.358+0000 7fce39d5a700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fce200682f0 0x7fce2006a7a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fce2800a950 tx=0x7fce28008040 comp rx=0 tx=0).stop 2023-11-28T13:06:49.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.358+0000 7fce39d5a700 1 -- 172.21.15.111:0/313029971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce3415cfd0 msgr2=0x7fce340b0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.358+0000 7fce39d5a700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce3415cfd0 0x7fce340b0fe0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fce2c00b370 tx=0x7fce2c004ad0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.358+0000 7fce39d5a700 1 -- 172.21.15.111:0/313029971 shutdown_connections 2023-11-28T13:06:49.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.358+0000 7fce39d5a700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce3415cfd0 0x7fce340b0fe0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.358+0000 7fce39d5a700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fce200682f0 0x7fce2006a7a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.358+0000 7fce39d5a700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fce34159310 0x7fce3414e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.358+0000 7fce39d5a700 1 --2- 172.21.15.111:0/313029971 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fce341596e0 0x7fce341f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.358+0000 7fce39d5a700 1 -- 172.21.15.111:0/313029971 >> 172.21.15.111:0/313029971 conn(0x7fce3406db10 msgr2=0x7fce340b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:49.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.358+0000 7fce39d5a700 1 -- 172.21.15.111:0/313029971 shutdown_connections 2023-11-28T13:06:49.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.358+0000 7fce39d5a700 1 -- 172.21.15.111:0/313029971 wait complete. 2023-11-28T13:06:49.371 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T13:06:49.372 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T13:06:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.490+0000 7fc24d16a700 1 Processor -- start 2023-11-28T13:06:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.490+0000 7fc24d16a700 1 -- start start 2023-11-28T13:06:49.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.490+0000 7fc24d16a700 1 --2- >> v2:172.21.15.149:3300/0 conn(0x7fc2480cdef0 0x7fc2480d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.490+0000 7fc24d16a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2480d05a0 con 0x7fc2480cfc90 2023-11-28T13:06:49.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.490+0000 7fc24d16a700 1 -- --> v1:172.21.15.149:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2480ca260 con 0x7fc2480cc5a0 2023-11-28T13:06:49.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.490+0000 7fc24d16a700 1 -- --> v2:172.21.15.149:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2480ccaa0 con 0x7fc2480cdef0 2023-11-28T13:06:49.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.490+0000 7fc246d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc2480cfc90 0x7fc2480cd7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:39872/0 (socket says 172.21.15.111:39872) 2023-11-28T13:06:49.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc246d9d700 1 -- 172.21.15.111:0/776977484 learned_addr learned my addr 172.21.15.111:0/776977484 (peer_addr_for_me v1:172.21.15.111:0/0) 2023-11-28T13:06:49.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 <== mon.0 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387610947 0 0) 0x7fc238001110 con 0x7fc2480cfc90 2023-11-28T13:06:49.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2300035b0 con 0x7fc2480cfc90 2023-11-28T13:06:49.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc24659c700 1 --2- 172.21.15.111:0/776977484 >> v2:172.21.15.149:3300/0 conn(0x7fc2480cdef0 0x7fc2480d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 <== mon.0 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3382752689 0 0) 0x7fc2300035b0 con 0x7fc2480cfc90 2023-11-28T13:06:49.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 >> v2:172.21.15.149:3300/0 conn(0x7fc2480cdef0 msgr2=0x7fc2480d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:49.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 --2- 172.21.15.111:0/776977484 >> v2:172.21.15.149:3300/0 conn(0x7fc2480cdef0 0x7fc2480d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 >> v1:172.21.15.149:6789/0 conn(0x7fc2480cc5a0 legacy=0x7fc2480ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 --> v1:172.21.15.111:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2480ccd80 con 0x7fc2480cfc90 2023-11-28T13:06:49.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 <== mon.0 v1:172.21.15.111:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 460624011 0 0) 0x7fc238001800 con 0x7fc2480cfc90 2023-11-28T13:06:49.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 >> v1:172.21.15.111:6789/0 conn(0x7fc2480cfc90 legacy=0x7fc2480cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 --2- 172.21.15.111:0/776977484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc230003a00 0x7fc230003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 --2- 172.21.15.111:0/776977484 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc230004850 0x7fc230004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 --2- 172.21.15.111:0/776977484 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc230005200 0x7fc23000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2480ccaa0 con 0x7fc230003a00 2023-11-28T13:06:49.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23000c080 con 0x7fc230004850 2023-11-28T13:06:49.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23000c240 con 0x7fc230005200 2023-11-28T13:06:49.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc246d9d700 1 --2- 172.21.15.111:0/776977484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc230003a00 0x7fc230003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc24659c700 1 --2- 172.21.15.111:0/776977484 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc230004850 0x7fc230004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc24759e700 1 --2- 172.21.15.111:0/776977484 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc230005200 0x7fc23000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc246d9d700 1 -- 172.21.15.111:0/776977484 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc230005200 msgr2=0x7fc23000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc246d9d700 1 --2- 172.21.15.111:0/776977484 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc230005200 0x7fc23000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc246d9d700 1 -- 172.21.15.111:0/776977484 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc230004850 msgr2=0x7fc230004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T13:06:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc246d9d700 1 --2- 172.21.15.111:0/776977484 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc230004850 0x7fc230004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc246d9d700 1 -- 172.21.15.111:0/776977484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2300035b0 con 0x7fc230003a00 2023-11-28T13:06:49.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.491+0000 7fc24759e700 1 -- 172.21.15.111:0/776977484 reap_dead start 2023-11-28T13:06:49.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.492+0000 7fc246d9d700 1 --2- 172.21.15.111:0/776977484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc230003a00 0x7fc230003e10 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc23800c240 tx=0x7fc230005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a91c65e26a7cada server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:49.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.492+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc238007db0 con 0x7fc230003a00 2023-11-28T13:06:49.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.492+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc23800cde0 con 0x7fc230003a00 2023-11-28T13:06:49.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.492+0000 7fc245d9b700 1 -- 172.21.15.111:0/776977484 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc238005ba0 con 0x7fc230003a00 2023-11-28T13:06:49.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.492+0000 7fc24d16a700 1 -- 172.21.15.111:0/776977484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc230003a00 msgr2=0x7fc230003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.492+0000 7fc24d16a700 1 --2- 172.21.15.111:0/776977484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc230003a00 0x7fc230003e10 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc23800c240 tx=0x7fc230005880 comp rx=0 tx=0).stop 2023-11-28T13:06:49.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.492+0000 7fc24d16a700 1 -- 172.21.15.111:0/776977484 shutdown_connections 2023-11-28T13:06:49.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.492+0000 7fc24d16a700 1 --2- 172.21.15.111:0/776977484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc230003a00 0x7fc230003e10 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.492+0000 7fc24d16a700 1 -- 172.21.15.111:0/776977484 >> 172.21.15.111:0/776977484 conn(0x7fc248098ee0 msgr2=0x7fc24806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:49.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.492+0000 7fc24d16a700 1 -- 172.21.15.111:0/776977484 shutdown_connections 2023-11-28T13:06:49.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.492+0000 7fc24d16a700 1 -- 172.21.15.111:0/776977484 wait complete. 2023-11-28T13:06:49.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.494+0000 7fc24d16a700 1 Processor -- start 2023-11-28T13:06:49.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.494+0000 7fc24d16a700 1 -- start start 2023-11-28T13:06:49.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24d16a700 1 --2- >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc230003a00 0x7fc2481f1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24d16a700 1 --2- >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2480b11b0 0x7fc2481ed620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24d16a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2480af340 0x7fc2480af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24d16a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2480afcf0 con 0x7fc2480af340 2023-11-28T13:06:49.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24d16a700 1 -- --> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2480b32e0 con 0x7fc230003a00 2023-11-28T13:06:49.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24d16a700 1 -- --> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2480b19e0 con 0x7fc2480b11b0 2023-11-28T13:06:49.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2480af340 0x7fc2480af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2480af340 0x7fc2480af7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39110/0 (socket says 172.21.15.111:39110) 2023-11-28T13:06:49.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24759e700 1 -- 172.21.15.111:0/672671520 learned_addr learned my addr 172.21.15.111:0/672671520 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-11-28T13:06:49.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc246d9d700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc230003a00 0x7fc2481f1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24659c700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2480b11b0 0x7fc2481ed620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24759e700 1 -- 172.21.15.111:0/672671520 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2480b11b0 msgr2=0x7fc2481ed620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24759e700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2480b11b0 0x7fc2481ed620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24759e700 1 -- 172.21.15.111:0/672671520 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc230003a00 msgr2=0x7fc2481f1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24759e700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc230003a00 0x7fc2481f1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24759e700 1 -- 172.21.15.111:0/672671520 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2480cd340 con 0x7fc2480af340 2023-11-28T13:06:49.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24759e700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2480af340 0x7fc2480af7b0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc2480d0c60 tx=0x7fc24000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:49.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc22f7fe700 1 -- 172.21.15.111:0/672671520 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24000eed0 con 0x7fc2480af340 2023-11-28T13:06:49.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc22f7fe700 1 -- 172.21.15.111:0/672671520 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc240003c10 con 0x7fc2480af340 2023-11-28T13:06:49.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc24d16a700 1 -- 172.21.15.111:0/672671520 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2480cd060 con 0x7fc2480af340 2023-11-28T13:06:49.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.495+0000 7fc22f7fe700 1 -- 172.21.15.111:0/672671520 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2400050d0 con 0x7fc2480af340 2023-11-28T13:06:49.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.496+0000 7fc24d16a700 1 -- 172.21.15.111:0/672671520 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc248143280 con 0x7fc2480af340 2023-11-28T13:06:49.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.496+0000 7fc24d16a700 1 -- 172.21.15.111:0/672671520 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc248067170 con 0x7fc2480af340 2023-11-28T13:06:49.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.497+0000 7fc22f7fe700 1 -- 172.21.15.111:0/672671520 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc240005250 con 0x7fc2480af340 2023-11-28T13:06:49.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.497+0000 7fc22f7fe700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc2340682f0 0x7fc23406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T13:06:49.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.497+0000 7fc22f7fe700 1 -- 172.21.15.111:0/672671520 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5616+0+0 (secure 0 0 0) 0x7fc240085b50 con 0x7fc2480af340 2023-11-28T13:06:49.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.497+0000 7fc246d9d700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc2340682f0 0x7fc23406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T13:06:49.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.498+0000 7fc246d9d700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc2340682f0 0x7fc23406a7a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc238000c00 tx=0x7fc238007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T13:06:49.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.501+0000 7fc22f7fe700 1 -- 172.21.15.111:0/672671520 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc2400524e0 con 0x7fc2480af340 2023-11-28T13:06:49.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.727+0000 7fc24d16a700 1 -- 172.21.15.111:0/672671520 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc248099420 con 0x7fc2480af340 2023-11-28T13:06:49.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.727+0000 7fc22f7fe700 1 -- 172.21.15.111:0/672671520 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3538 (secure 0 0 0) 0x7fc240055b00 con 0x7fc2480af340 2023-11-28T13:06:49.729 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-11-28T13:06:49.729 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5752,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6834/1668572397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6832","nonce":1668572397},{"type":"v1","addr":"172.21.15.149:6834","nonce":1668572397}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5755,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.149:6835/1129092621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6833","nonce":1129092621},{"type":"v1","addr":"172.21.15.149:6835","nonce":1129092621}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6249,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6836/904588136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":904588136},{"type":"v1","addr":"172.21.15.111:6836","nonce":904588136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6903,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/1038752795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6835","nonce":1038752795},{"type":"v1","addr":"172.21.15.111:6837","nonce":1038752795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35}],"filesystems":[]} 2023-11-28T13:06:49.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.729+0000 7fc24d16a700 1 -- 172.21.15.111:0/672671520 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc2340682f0 msgr2=0x7fc23406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.729+0000 7fc24d16a700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc2340682f0 0x7fc23406a7a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc238000c00 tx=0x7fc238007000 comp rx=0 tx=0).stop 2023-11-28T13:06:49.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.729+0000 7fc24d16a700 1 -- 172.21.15.111:0/672671520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2480af340 msgr2=0x7fc2480af7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T13:06:49.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.729+0000 7fc24d16a700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2480af340 0x7fc2480af7b0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc2480d0c60 tx=0x7fc24000ed50 comp rx=0 tx=0).stop 2023-11-28T13:06:49.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.729+0000 7fc24d16a700 1 -- 172.21.15.111:0/672671520 shutdown_connections 2023-11-28T13:06:49.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.729+0000 7fc24d16a700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2480af340 0x7fc2480af7b0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.729+0000 7fc24d16a700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.111:6832/103256,v1:172.21.15.111:6833/103256] conn(0x7fc2340682f0 0x7fc23406a7a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.729+0000 7fc24d16a700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7fc230003a00 0x7fc2481f1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.729+0000 7fc24d16a700 1 --2- 172.21.15.111:0/672671520 >> [v2:172.21.15.149:3301/0,v1:172.21.15.149:6790/0] conn(0x7fc2480b11b0 0x7fc2481ed620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T13:06:49.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.729+0000 7fc24d16a700 1 -- 172.21.15.111:0/672671520 >> 172.21.15.111:0/672671520 conn(0x7fc248098ee0 msgr2=0x7fc24815b940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T13:06:49.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.729+0000 7fc24d16a700 1 -- 172.21.15.111:0/672671520 shutdown_connections 2023-11-28T13:06:49.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-11-28T13:06:49.729+0000 7fc24d16a700 1 -- 172.21.15.111:0/672671520 wait complete. 2023-11-28T13:06:49.733 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 35 2023-11-28T13:06:49.743 DEBUG:tasks.cephfs.filesystem:already dead... 2023-11-28T13:06:49.743 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-28T13:06:49.744 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T13:06:49.744 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:49.752 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T13:06:49.752 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T13:06:49.752 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:49.801 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T13:06:49.801 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T13:06:49.801 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:49.812 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T13:06:49.812 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T13:06:49.813 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:49.863 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T13:06:49.863 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-28T13:06:49.863 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-28T13:06:49.863 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:49.900 INFO:tasks.ceph.osd.0:Stopped 2023-11-28T13:06:49.901 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-28T13:06:49.901 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:49.974 INFO:tasks.ceph.osd.1:Stopped 2023-11-28T13:06:49.974 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-28T13:06:49.974 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:50.040 INFO:tasks.ceph.osd.2:Stopped 2023-11-28T13:06:50.041 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-28T13:06:50.041 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:50.111 INFO:tasks.ceph.osd.3:Stopped 2023-11-28T13:06:50.111 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-28T13:06:50.111 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:50.145 INFO:tasks.ceph.osd.4:Stopped 2023-11-28T13:06:50.146 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-28T13:06:50.146 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:50.218 INFO:tasks.ceph.osd.5:Stopped 2023-11-28T13:06:50.219 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-11-28T13:06:50.219 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:50.289 INFO:tasks.ceph.osd.6:Stopped 2023-11-28T13:06:50.289 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-11-28T13:06:50.289 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:50.370 INFO:tasks.ceph.osd.7:Stopped 2023-11-28T13:06:50.370 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-28T13:06:50.370 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-11-28T13:06:50.370 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:50.394 INFO:tasks.ceph.mgr.y:Stopped 2023-11-28T13:06:50.394 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-28T13:06:50.394 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:50.436 INFO:tasks.ceph.mgr.x:Stopped 2023-11-28T13:06:50.436 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-28T13:06:50.436 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-28T13:06:50.436 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:50.447 INFO:tasks.ceph.mon.a:Stopped 2023-11-28T13:06:50.447 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-28T13:06:50.447 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:50.488 INFO:tasks.ceph.mon.b:Stopped 2023-11-28T13:06:50.489 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-28T13:06:50.489 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T13:06:50.539 INFO:tasks.ceph.mon.c:Stopped 2023-11-28T13:06:50.540 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-28T13:06:50.540 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2023-11-28T13:06:50.573 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T13:06:50.573 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-28T13:06:50.889 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T13:06:50.890 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-28T13:06:51.006 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T13:06:51.006 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-28T13:06:51.079 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T13:06:51.079 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-11-28T13:06:51.156 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi149.front.sepia.ceph.com 2023-11-28T13:06:51.157 DEBUG:teuthology.orchestra.run.smithi149:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-11-28T13:06:51.380 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi149.front.sepia.ceph.com 2023-11-28T13:06:51.381 DEBUG:teuthology.orchestra.run.smithi149:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-11-28T13:06:51.479 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi149.front.sepia.ceph.com 2023-11-28T13:06:51.479 DEBUG:teuthology.orchestra.run.smithi149:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-11-28T13:06:51.556 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi149.front.sepia.ceph.com 2023-11-28T13:06:51.557 DEBUG:teuthology.orchestra.run.smithi149:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-11-28T13:06:51.667 INFO:tasks.ceph:Archiving mon data... 2023-11-28T13:06:51.668 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969/data/mon.a.tgz 2023-11-28T13:06:51.668 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2023-11-28T13:06:51.686 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.Prz991KrGj 2023-11-28T13:06:51.687 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Prz991KrGj 2023-11-28T13:06:51.823 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.Prz991KrGj 2023-11-28T13:06:51.909 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.Prz991KrGj is 301KB 2023-11-28T13:06:52.005 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.Prz991KrGj 2023-11-28T13:06:52.021 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969/data/mon.b.tgz 2023-11-28T13:06:52.022 DEBUG:teuthology.orchestra.run.smithi149:> mktemp 2023-11-28T13:06:52.040 INFO:teuthology.orchestra.run.smithi149.stdout:/tmp/tmp.mZifeRMV9z 2023-11-28T13:06:52.040 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.mZifeRMV9z 2023-11-28T13:06:52.196 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0666 /tmp/tmp.mZifeRMV9z 2023-11-28T13:06:52.283 DEBUG:teuthology.orchestra.remote:smithi149:/tmp/tmp.mZifeRMV9z is 304KB 2023-11-28T13:06:52.377 DEBUG:teuthology.orchestra.run.smithi149:> rm -fr /tmp/tmp.mZifeRMV9z 2023-11-28T13:06:52.393 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969/data/mon.c.tgz 2023-11-28T13:06:52.393 DEBUG:teuthology.orchestra.run.smithi149:> mktemp 2023-11-28T13:06:52.449 INFO:teuthology.orchestra.run.smithi149.stdout:/tmp/tmp.n2ByyJchVe 2023-11-28T13:06:52.449 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.n2ByyJchVe 2023-11-28T13:06:52.574 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0666 /tmp/tmp.n2ByyJchVe 2023-11-28T13:06:52.661 DEBUG:teuthology.orchestra.remote:smithi149:/tmp/tmp.n2ByyJchVe is 304KB 2023-11-28T13:06:52.757 DEBUG:teuthology.orchestra.run.smithi149:> rm -fr /tmp/tmp.n2ByyJchVe 2023-11-28T13:06:52.773 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-28T13:06:52.774 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-28T13:06:52.776 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-28T13:06:52.839 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-28T13:06:52.840 DEBUG:teuthology.orchestra.run.smithi111:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-28T13:06:52.844 DEBUG:teuthology.orchestra.run.smithi149:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-28T13:06:52.881 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-28T13:06:52.917 INFO:teuthology.orchestra.run.smithi149.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-28T13:06:52.922 INFO:tasks.ceph:Archiving crash dumps... 2023-11-28T13:06:52.924 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/crash to /home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969/remote/smithi111/crash 2023-11-28T13:06:52.925 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-28T13:06:52.954 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/crash to /home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969/remote/smithi149/crash 2023-11-28T13:06:52.955 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-28T13:06:52.988 INFO:tasks.ceph:Shutting down logrotate 2023-11-28T13:06:52.988 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:06:52.998 DEBUG:teuthology.orchestra.run.smithi149:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T13:06:53.080 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-28T13:06:53.106 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-28T13:06:53.133 INFO:tasks.ceph:Compressing logs... 2023-11-28T13:06:53.133 DEBUG:teuthology.orchestra.run.smithi111:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T13:06:53.148 DEBUG:teuthology.orchestra.run.smithi149:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T13:06:57.879 INFO:tasks.ceph:Archiving logs... 2023-11-28T13:06:57.880 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969/remote/smithi111/log 2023-11-28T13:06:57.881 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-28T13:06:58.771 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/log/ceph to /home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969/remote/smithi149/log 2023-11-28T13:06:58.772 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-28T13:06:59.321 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-28T13:06:59.330 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-28T13:06:59.331 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-28T13:06:59.333 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-28T13:06:59.424 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-28T13:06:59.434 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-28T13:06:59.435 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T13:06:59.437 DEBUG:teuthology.orchestra.run.smithi149:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T13:06:59.453 INFO:teuthology.orchestra.run.smithi111.stderr:bash: ntpq: command not found 2023-11-28T13:06:59.454 INFO:teuthology.orchestra.run.smithi149.stderr:bash: ntpq: command not found 2023-11-28T13:06:59.458 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T13:06:59.458 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2023-11-28T13:06:59.458 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +914us[ +914us] +/- 97ms 2023-11-28T13:06:59.459 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -357us[ -319us] +/- 65ms 2023-11-28T13:06:59.459 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -282us[ -244us] +/- 80ms 2023-11-28T13:06:59.459 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T13:06:59.460 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T13:06:59.460 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2023-11-28T13:06:59.461 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +921us[ +921us] +/- 97ms 2023-11-28T13:06:59.461 INFO:teuthology.orchestra.run.smithi149.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -342us[ -283us] +/- 65ms 2023-11-28T13:06:59.461 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -269us[ -269us] +/- 80ms 2023-11-28T13:06:59.461 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T13:06:59.461 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-28T13:06:59.473 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-28T13:06:59.473 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-28T13:06:59.482 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-28T13:06:59.607 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-28T13:06:59.705 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T13:06:59.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T13:06:59.738 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T13:06:59.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T13:06:59.772 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-28T13:06:59.783 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-28T13:06:59.797 INFO:teuthology.task.internal:Duration was 1250.755327 seconds 2023-11-28T13:06:59.797 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-28T13:06:59.807 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-28T13:06:59.807 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-28T13:06:59.809 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-28T13:06:59.853 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T13:06:59.859 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T13:07:00.219 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-28T13:07:00.219 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2023-11-28T13:07:00.220 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-28T13:07:00.242 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi149.front.sepia.ceph.com 2023-11-28T13:07:00.243 DEBUG:teuthology.orchestra.run.smithi149:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-28T13:07:00.265 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-28T13:07:00.266 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T13:07:00.286 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T13:07:00.348 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-28T13:07:00.349 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T13:07:00.369 DEBUG:teuthology.orchestra.run.smithi149:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T13:07:00.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-28T13:07:00.479 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-28T13:07:00.479 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-28T13:07:00.506 DEBUG:teuthology.orchestra.run.smithi149:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-28T13:07:00.534 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-28T13:07:00.545 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-28T13:07:00.548 DEBUG:teuthology.orchestra.run.smithi149:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-28T13:07:00.572 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2023-11-28T13:07:00.600 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = core 2023-11-28T13:07:00.615 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-28T13:07:00.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T13:07:00.642 DEBUG:teuthology.orchestra.run.smithi149:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-28T13:07:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T13:07:00.671 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-28T13:07:00.681 INFO:teuthology.task.internal:Transferring archived files... 2023-11-28T13:07:00.682 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969/remote/smithi111 2023-11-28T13:07:00.682 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-28T13:07:00.722 DEBUG:teuthology.misc:Transferring archived files from smithi149:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969/remote/smithi149 2023-11-28T13:07:00.723 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-28T13:07:00.760 INFO:teuthology.task.internal:Removing archive directory... 2023-11-28T13:07:00.761 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-28T13:07:00.763 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-28T13:07:00.811 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-28T13:07:00.823 INFO:teuthology.task.internal:Not uploading archives. 2023-11-28T13:07:00.823 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-28T13:07:00.834 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-28T13:07:00.834 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-28T13:07:00.837 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-28T13:07:00.850 INFO:teuthology.orchestra.run.smithi111.stdout: 262320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 13:07 /home/ubuntu/cephtest 2023-11-28T13:07:00.867 INFO:teuthology.orchestra.run.smithi149.stdout: 262320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 13:07 /home/ubuntu/cephtest 2023-11-28T13:07:00.868 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-28T13:07:00.880 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-28T13:07:00.900 INFO:teuthology.nuke:Checking targets against current locks 2023-11-28T13:07:00.929 DEBUG:teuthology.nuke:shortname: smithi111 2023-11-28T13:07:00.930 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T13:07:00.956 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 12:40:25.366247', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-28T13:07:00.960 DEBUG:teuthology.nuke:shortname: smithi149 2023-11-28T13:07:00.960 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T13:07:00.983 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2023-11-28_12:38:30-fs-wip-neesingh-testing-231123-distro-default-smithi/7469969', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 12:40:25.367605', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-28T13:07:01.014 INFO:teuthology.orchestra.console.smithi111:Power off 2023-11-28T13:07:01.014 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-28T13:07:01.040 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2023-11-28T13:07:01.040 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T13:07:01.059 INFO:teuthology.orchestra.console.smithi149:Power off 2023-11-28T13:07:01.060 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-28T13:07:01.070 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-11-28T13:07:01.083 DEBUG:teuthology.orchestra.console.smithi149:power off output: Chassis Power Control: Down/Off 2023-11-28T13:07:01.083 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T13:07:01.105 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2023-11-28T13:07:05.071 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T13:07:05.107 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T13:07:05.181 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-11-28T13:07:05.218 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2023-11-28T13:07:09.186 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T13:07:09.219 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T13:07:09.297 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-11-28T13:07:09.331 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is off 2023-11-28T13:07:09.431 INFO:teuthology.orchestra.console.smithi149:Power off completed 2023-11-28T13:07:13.301 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T13:07:13.411 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2023-11-28T13:07:13.512 INFO:teuthology.orchestra.console.smithi111:Power off completed 2023-11-28T13:07:13.617 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} duration: 1250.7553267478943 failure_reason: 'Test failure: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub)' flavor: default owner: scheduled_neesingh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7bf12345d9af4830bef79119ff249606 status: fail success: false 2023-11-28T13:07:13.617 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T13:07:13.692 INFO:teuthology.run:FAIL